Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 21 hours
Course Outline
Module 1: Foundations of Big Data & Spark
- An overview of the Big Data ecosystem and Spark’s role in modern data platforms
- Exploring Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
- Distinguishing between RDD and DataFrame APIs and determining the best approach for each scenario
- Creating and configuring SparkSession while understanding the basics of application settings
Module 2: PySpark DataFrames
- Ingesting and exporting data from enterprise sources and formats: CSV, JSON, Parquet, and Delta
- Mastering PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
- Implementing advanced techniques such as window functions, timestamp handling, and managing nested data
- Enforcing data quality checks and writing reusable, maintainable PySpark code
Module 3: Kubernetes & Docker Essentials for Spark Users
- Grasping the relationship between Docker images and Kubernetes pods in relation to Spark driver and executor processes, from a consumer rather than administrator perspective
- Understanding the mechanics of a Spark job on Kubernetes: pod scheduling, resource requests/limits, and how session settings align with them
- Differentiating between what is controlled via Spark session parameters and what is managed by the platform team through cluster provisioning, and why this boundary is critical for tuning
- Practical exercise: Examine a live Spark-on-Kubernetes job in the sandbox cluster to identify driver and executor pods
Module 4: Deep Dive into Spark Session Configuration on Kubernetes
- Analyzing the trade-offs between executor count and executor size: balancing memory versus cores
- Allocating memory for drivers and executors, understanding overhead, and translating these into pod resource requirements
- Exploring dynamic allocation: its behavior on Kubernetes, when it conserves resources, and when it does not
- Practical exercise: Execute the same job with varying executor and core settings to compare runtime and resource utilization
Module 5: Scaling Dynamics & Cost Efficiency
- Evaluating how scaling nodes impacts job behavior, completion time, and resource usage
- Weighing the performance and cost trade-offs of using many small executors versus fewer large ones
- Tuning shuffle behavior and partitions, including estimating the financial impact of configuration decisions
- Practical exercise: Scale the sandbox cluster from five to ten nodes to observe changes in job completion time and resource consumption
Module 6: Optimizing Data Ingestion & Partitioning
- Addressing the small-files problem: why numerous 1–5 MB Parquet files degrade performance and disrupt partitioning
- Employing repartitioning and coalescing techniques
- Managing partition sizes during both read and write operations
- Writing Parquet efficiently to prevent downstream small-file issues
- Practical exercise: Load a dataset with many small Parquet partitions, apply various repartitioning methods, and compare performance improvements
Module 7: Pandas Memory Control & Failure Analysis
- Identifying root causes and symptoms of out-of-memory errors in Pandas
- Implementing memory-efficient conversion patterns between Spark and Pandas
- Preventing memory spikes when exporting large datasets to CSV
- Utilizing chunked processing and dtype optimization for resource-constrained environments
- Practical exercise: Reproduce a common Pandas out-of-memory issue and resolve it using chunking and dtype adjustments
Module 8: Polars as a Complementary Tool
- Positioning Polars against Pandas: examining performance, lazy evaluation, and memory behavior
- Understanding where Polars fits within modern cloud data stacks and migration plans, including AWS environments, alongside PySpark and Pandas
- Practical exercise: Refactor a Pandas-heavy transformation in Polars to compare memory usage and processing speed
Module 9: Optimizing ETL & ML Workloads
- Applying configuration, partitioning, and memory-management principles to a realistic ETL pipeline
- Understanding optimization specifics for machine-learning workloads sharing the same cluster
- Applying a structured tuning workflow to systematically diagnose cost and performance issues
- Practical exercise: Complete an end-to-end mini-project involving data loading, transformation, and simple model training, with participants tuning Spark configurations independently
Requirements
Participants are expected to have:
- Hands-on experience with Python programming, covering functions, modules, and fundamental object-oriented principles.
- Basic to intermediate proficiency in working with Pandas and tabular data workflows.
- Familiarity with PySpark and Spark DataFrames, including data ingestion, transformations, actions, joins, and aggregations.
- A solid grasp of SQL and data-processing fundamentals, such as filtering, grouping, and dataset joins.
- Basic knowledge of Docker and Kubernetes concepts, like containers, images, and pods. No experience in Kubernetes administration is necessary.
- An understanding of common data formats such as CSV, JSON, and Parquet.
There is no requirement for participants to be Kubernetes administrators or infrastructure specialists. The course is tailored for data engineers, developers, and data scientists who need to understand and optimize their Spark workloads running on Kubernetes from an application and configuration perspective.
Target Audience
This training is designed for professionals who develop, maintain, or enhance data-processing and machine-learning workloads using Python and Spark in cloud or containerized settings.
It is particularly well-suited for:
- Data Engineers working with PySpark, distributed processing, and ETL pipelines.
- Data Scientists handling large datasets or executing machine-learning tasks with Spark, Pandas, or Polars.
- Python Developers building data-intensive applications who aim to enhance memory efficiency and processing speed.
- Machine Learning Engineers managing data preparation and model training on shared Kubernetes or cloud platforms.
- Analytics Engineers working with large data volumes who seek to boost data-processing efficiency.
- DevOps, Platform, and Cloud Engineers supporting Spark on Kubernetes who need to understand how application settings affect resource usage and performance.
- Technical Leads and Solution Architects involved in designing or optimizing modern data platforms.
Testimonials (1)
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.
