Data Engineering Workshop (DEW)

Data Warehousing with SQL and NoSQL\n\n\n- Data warehouses\n- Relational databases\n\n\n- SQL operations\n- Transactional vs. analytical\n- Design and performance considerations\n\n- NoSQL\n\n\n- SQL vs. NoSQL\n- NoSQL database types and examples\n\n- Redis\n- Apache Cassandra\n- Apache CouchDB\n- Data Lakes\n\nETL Offload with Hadoop and Spark\n\n\n- Hadoop ecosystem\n- Hadoop Distributed File System (HDFS)\n- Data ingestion tools\n\n\n\n- Apache Flume\n- Apache Sqoop\n\n- Apache Spark\n- ETL schedulers\n\n\n\n- Apache Oozie\n- Apache Airflow\n\n- ETL offload implementation considerations\n\nData Governance, Security and Privacy for Big Data\n\n\n- Data Governance Overview\n- Data Governance Roles\n- Data Governance Models\n- Metadata\n- Master Data Management\n- Security Controls in Hadoop Ecosystem\n- Apache Atlas\n- Apache Ranger\n- Apache Knox\n- Security Considerations in the Cloud\n- General Data Protection Regulation (GDPR)\n- Data Ethics: Avoiding Hidden Biases\nProcessing Streaming and IoT Data\n\n\n- Processing Streaming and IoT Data Overview\n- Streaming and IoT Data Processing Tools Framework\n- Apache Storm\n- Apache Kafka\n- Apache Spark Streaming\n- Apache Flink\n- Pravega\n- Project Nautilus\n- EdgeX Foundry\n\nBuilding Data Pipelines with Python\n\n\n- \n- Introduction to data pipelines\n- Introduction to Python\n\n\n\n- Features of Python\n- Basic Syntax of Python\n- Data Types, Operators, and Conditional Statements\n- User-defined Functions and Classes\n\n- Python libraries\n- Data structures in Python\n- Data pipeline best practices

To complete this course successfully and gain the maximum benefits from\nit, a student should have the following knowledge and skill sets:\n\n\n\n- Experience with a programming language such as Java, R, or Python\n- Familiarity with the non-statistical aspects of the Data Science and Big Data Analytics v2 content\n- Understanding of the data engineer role provided in the Introduction to Data Engineering course

This course is intended for data engineers, data scientists, data architects, data analysts or anyone else who wants to learn and apply data engineering principles and tools. Possible workshop participants include:\n\n\n\n- Current business and data analysts looking to add data engineering to their skillset\n- Database professionals looking to expand their Big Data skills\n- Managers of teams of business intelligence, analytics, and big data professionals