Apache Spark Structured Streaming

Master Real-time Stream Processing using PySpark and Databricks

What you'll learn

This course will help you understand Real-time Stream processing using Apache Spark and Databricks Cloud and apply that knowledge to build real-time stream processing solutions. This course is example-driven and follows a working session-like approach. We will take a live coding approach and explain all the needed concepts.

Real-time Stream Processing Concepts, Spark Structured Streaming APIs and Architecture, Working with Streaming Sources and Sinks
Kafka for Data Engineers, Working With Kafka Source and Integrating Spark with Kafka
State-less and State-full Streaming Transformations, Windowing Aggregates using Spark Stream, Watermarking and State Cleanup
Working with Azure Databricks and Streaming on Databricks Platform

Course Prerequisite

What do you need to know before you start this course

Spark Fundamentals and exposure to Spark Dataframe APIs
Programming knowledge using Python programming language

Hear out instructor

Course curriculum

76 lectures - 18 hours video

1. About the Course
  FREE PREVIEW
2. Course Prerequisite
  FREE PREVIEW
3. Source Code and Other Resources
1. Spark Development Environments
  FREE PREVIEW
2. Setup your Databricks Community Cloud Environment
  FREE PREVIEW
3. Working in Databricks Workspace
1. Batch processing to stream processing
2. Your Spark application - Applying Best Practice
3. Your first streaming application - Implementing Stream
4. Stream Processing Model in Spark
5. Create Another Streaming Application
6. Stream Triggers
7. Incremental Batch Processing
8. Streaming Sources and Sinks
9. Creating Chain of Streams
10. Review Rating
1. An Introduction to Kafka
2. Creating Kafka Cluster in Cloud
3. Kafka Core Concepts
4. Producing Data to Kafka Topic
5. Consuming Data from Kafka Topic
6. Working with Kafka Topic Data
7. How to Implement Idempotence
8. Working with Kafka Sink
1. Streaming Aggregates and State Store
2. Incremental Aggregates and Update Mode
3. Spark Streaming Output Modes
4. Statefull Vs Stateless Aggregation
5. Implementing Stateless Streaming Aggregation
6. Timebound Stateful Tumbling Window Aggregation
7. Watermarking and State Store Cleanup
8. Sliding Window Aggregates
9. Review Rating
1. Introduction to Databricks
2. Creating Azure Free Account
3. Azure Portal Overview
4. Creating Azure Databricks Service
5. Introduction to Azure Databricks Workspace
6. Azure Databricks Architecture
7. Creating Azure Databricks Cluster
8. Introduction to Databricks Notebooks
9. Notebooks Magic Commands
10. Databricks Notebooks Utilities
11. Introduction to Databricks Unity Catalog
12. Introduction to Databricks Workflow Jobs
13. Introduction to Databricks Rest API
14. Introduction to Databricks CLI
1. Project Scope and Background
2. Taking out the operational requirement
3. Storage Design
4. Implement Data Security
5. Implement Resource Policies
6. Decouple Data Ingestion
7. Design Bronze Layer
8. Design Silver and Gold Layer
9. Setup your source control
10. Setup your environment
11. Create a workspace
12. Create and Storage Layer
13. Setup Unity Catalog
14. Create Metadata Catalog and External Locations
15. Start Coding
16. Test your code
17. Load historical data
18. Ingest into bronze layer
19. Process the silver layer
20. Handling multiple updates
21. Implementing Gold Layer
22. Creating a run script
23. Preparing for Integration testing
24. Creating Test Data Producer
25. Creating Integration Test for Batch mode
26. Creating Integration Test for Stream mode
27. Implementing CI CD Pipeline
28. Develop Build Pipeline
29. Develop Release Pipeline
30. Creating Databricks CLI Script
1. An End is a new Beginning
1. Live Session 1 - 3rd March 2024: Why Streaming is Required?
2. Live Session 2 - 10th March 2024: Streaming Internals
3. Live Session 3 - 17th March 2024: Streaming Internals continue...
4. Live Session 4 - 7th April 2024: Stateless and Stateful Transformations

About this course

76 Video Lectures
Source Code and Data
Total Support

Features & Support

Total Support

We provide support throughout your learning and answer every question during business hours. For blocker issues, you can also request a one-to-one technical support call.
Completion Certificate

Students who complete ScholarNest Academy courses earn free, verifiable course completion certificates to share with their friends, co-workers, and potential employers.
Future Updates

Any future updates, upgrades, revisions, or topics included in the same course during your course access period will be available at no additional cost.

Course FAQ

How long can I access the course material?

We provide standard 3-year access to the course material from the date of purchase. However, our promotional offers may reduce the access duration for a discounted price. Please check access validity terms and conditions for your promotional offers.
Do you have a refund policy?

Yes. You can ask for a refund within 7 days of your purchase or before completing 15% of the course material, whichever is earlier. We provide a refund after deducting 6% of payment processing charges.
How do you provide support?

We have a Q&A forum where you can ask questions, and our team will answer your queries.

Enroll into self-paced course

Discover your potential, start today and learn at your own schedule

Buy Now

Schedule a free call

Get in touch with your course coordinator to learn more about the course, our instructor-led programs, discount offers, group discounts, corporate training and additional payment methods.

WhatsApp
WhatsApp: +91-93534 65988
Want to speak to your course coordinator? We are just a WhatsApp message or a phone call away.
Email
Email: [email protected]
Drop us an email with all your queries and questions and we will get back to you over the email.

Related Courses:

Schedule a call with course coordinator for bundles, discounts and live sessions

View more courses