Process data at scale.
A complete big data program — Hadoop, MapReduce, Hive, Pig, Spark, Kafka, Airflow and big data on AWS — from basic to advanced techniques, with real-life projects.
Talk to us about the next batch, schedule and fees
Technologies you’ll work with
Build the pipelines data teams depend on.
12 modules. 60+ topics.
- Introduction to big data — volume, variety, velocity
- Data warehouses
- ETL and ELT
- Introduction to the big data tools and ecosystem
- Batch vs real-time processing
- Linux commands for data engineers
- Hadoop architecture — NameNode, DataNodes, YARN
- HDFS concepts — blocks, replication, fault tolerance
- HDFS shell commands
- Setting up a Hadoop environment
- Understanding MapReduce in a simple manner
- Counters in MapReduce, with code
- Finding the top N elements using MapReduce
- Finding a distinct set of elements using MapReduce
- Building an inverted index using MapReduce
- Calculating median and standard deviation using MapReduce
- Reading different types of files in Hadoop MapReduce
- Data compression in MapReduce
- Introduction to Apache Pig
- Apache Pig setup
- Running Pig in different modes
- Apache Pig architecture
- Pig Latin concepts
- Pig Latin scripts in the Hadoop ecosystem
- User-defined functions (UDFs) in Apache Pig
- Introduction to Hive and its architecture
- HiveQL — databases, tables and queries
- Managed vs external tables
- Partitioning and bucketing
- Joins and built-in functions in Hive
- File formats — ORC, Parquet, Avro
- Why Spark — in-memory processing
- Spark architecture — driver, executors, cluster managers
- RDDs — transformations and actions
- DataFrames and Datasets
- Caching, persistence and partitioning
- Optimizing Spark jobs
- PySpark setup and SparkSession
- Reading and writing data at scale
- Spark SQL queries and window functions
- Data cleaning and transformation with PySpark
- Building ETL pipelines with Spark
- Introduction to Apache Kafka
- Topics, partitions, brokers and consumer groups
- Producers and consumers in Python
- Real-time data pipelines with Kafka
- Spark Structured Streaming with Kafka
- Introduction to Apache Airflow
- DAGs, tasks and operators
- Scheduling and dependencies
- Orchestrating Spark and ETL jobs
- Monitoring and retries
- AWS introduction and account setup
- IAM and S3 for data lakes
- EC2 and AWS Lambda
- AWS Glue — creating an ETL pipeline
- MapReduce on the cloud with Amazon EMR
- Analysing data with Amazon Athena
- Visualising data with Amazon QuickSight
- Monitoring logs with Amazon CloudWatch
- Data warehouse vs data lake
- Designing a data lake on S3
- Data modelling — star and snowflake schemas
- Batch and incremental loading
- Data quality and governance
- End-to-end batch pipeline project — ingest, process, store, report
- Real-time streaming pipeline project
- Building a portfolio of big data projects
- Big data engineer interview questions and practice
For engineers who want to work with big data.
Try our free tutorials first.
Abhishek Kumar Singh
Data Scientist · Consultant · Corporate Trainer
Abhishek works across Hive, Spark, MapReduce, Kafka and Airflow, and has trained data engineers — including Big Data and Data Engineering teams at Capgemini — in practical, project-based batches.
Technology stack
Organizations trained
What learners say about our data engineering training.
This course was easy to understand. The topics were explained clearly and it was very well structured.
For Aws, I think it's the best course available. The course curriculum is also designed to prepare you interview ready.
Overall course is very comprehensive and well structured. Ideal for beginners and professionals aiming to deepen their cloud expertise. Highly recommended!
The course is great and all the topics are in practical manner. The teacher is knowledgeable
Awesome course with practical lessons and explanation
Abhishek is a teacher at heart! He brings in strong teaching experience in data science, AWS concepts such as S3, IAM, EC2, Lambda, VPC , ECS, EKS and SAGE MAKER. He is a thorough gentleman with patience, always handling Q&A with grace and treating students with unwavering respect.
Join the next batch
12 modules across Hadoop, Hive, Pig, Spark, Kafka, Airflow and AWS — with real-life projects and interview preparation.
Book a free demo → Ask about fees on WhatsAppWhat’s included
- Hadoop ecosystem, Spark, Kafka and Airflow
- Big data on AWS — S3, Glue, EMR, Athena
- Batch and streaming pipeline projects
- Big data engineer interview questions and practice
Questions before you join?
Ask about the schedule, fees or whether the program is right for you.
Want to take pipelines into production ML? See our MLOps Training.
Basic programming and SQL help, but the program starts with big data foundations — core concepts, Linux and HDFS — before moving on to Spark, Kafka and the cloud.
Yes — HDFS, MapReduce and Hive explain how distributed data processing works, and many companies still run them. The program then moves on to Spark, Kafka, Airflow and cloud services that modern data teams use every day.
Yes. You’ll build end-to-end batch and streaming pipelines that you can add to your portfolio and discuss in interviews.
Book a free demo or message us on WhatsApp at +91 83182 38637 — we’ll share the schedule, fees and what’s included.
Build pipelines that move the business.
Book a free demo to see how the classes work and get the next batch dates.