Repository Information

If you want to create tables download GenSynthetic to run the script main.py which will create a folder named csv_tables with tables and synthetic data on it.

---

OMDENA: Fundamentals of Data Engineering: Principles and Techniques for Building Scalable Data Pipelines

This course was designed for those who are interested in career related to: Data engineering, Data analyst, Data Scientist, IT professionals, and Business analysts.

Targeted learning outcome

The course was designed to have a strong understanding of the principles and techniques used in data engineering and to acquaint with design, implement, and manage data systems that can handle large volumes of data. Additionally, the course was designed to build a project where participants can apply the knowledge gained through the couse to practically design and implement a scalable and reliable data piplines.

Syllabus

1. Introduction to Data Engineering

What is Data Engineering?, The role of Data Engineering in Data-Driven Decision Making, Key Concepts and Terminology

2. Python Basics

Variables, Data Types, Loops and control structure, Functions and Classes,
Arrays and Dictionaries

3. Data Ingestion

Data Ingestion Process and Techniques, Batch and Stream Processing, Data Ingestion Tools

4. Data Storage

Relational Databases, SQL Database, NoSQL Databases, Big Data Solutions, Data Warehouses

5. Data Processing

Data Cleaning, Data Transformation, Data Aggregation, Data Processing Tools,

6. Big Data Technologies

Introduction to Hadoop, Introduction to Spark, Introduction to Kafka, Big Data Processing and Analysis

7. Data Quality and Governance

Data Quality, Data Governance Policies, Data Security, Privacy, and Compliance,

Course Features

Lectures: Hands on
Duration: 25 hours
Skill level: beginner, intermediate
Category: Data Science & Machine Learning
Instructor: Anju Mercian