Mastering Azure Databricks and Lakehouse

Master Lakehouse Implementation | Azure Databricks | Data Engineer

What you'll learn

Mastering Databricks Cloud is a comprehensive course that will help you become proficient in Databricks Cloud Platform, Lakehouse Architecture, Incremental Data Processing, Unity Catalog, Delta Lake, Autoloader, Delta Live Table, Databricks CLI and Rest API for Automation.

Databricks Cloud - Working with Databricks Platform on Azure Cloud, Designing Lakehouse Architecture, Storage and Security
Delta Lake - Working with Delta Lake using PySpark and Spark SQL, Delta Lake Operations, Optimizations, and Tools
Advanced Databricks Tools - Autoloader, Unity Catalog, Delta Live Tables, Databricks Workflows, Repos, CLI and Rest API

Course Prerequisite

What do you need to know before you start this course

Programming Knowledge Using Python Programming Language and SQL Fundamentals
PySpark Programming and Spark SQL knowledge

Course Content

105 lectures - 12 hours video

1. Ch1 Downloadable Documents
2. Ch 1 Section Overview
  FREE PREVIEW
3. What is Big Data and How it Started
  FREE PREVIEW
4. What is Big Data and How it Started - Notes
5. What is Data Lake and How it works
  FREE PREVIEW
6. What is Data Lake and How it works - Notes
7. Introducing Apache Spark and Databricks Cloud
  FREE PREVIEW
8. Introducing Apache Spark and Databricks Cloud - Notes
9. Introducing Azure Databricks
10. Introducing Azure Databricks - Notes
11. Challanges with the Data Lake Architecture
12. Challanges with the Data Lake Architecture - Notes
13. The Idea of Lakehouse Architecture
14. The Idea of Lakehouse Architecture - Notes
1. Ch2 Downloadable Documents
2. Section Overview
3. Creating Azure Free Account
4. Creating Azure Free Account - Notes
5. Azure Portal Overview
6. Azure Portal Overview - Notes
7. Creating Azure Databricks Service
8. Creating Azure Databricks Service - Notes
9. Introduction to Azure Databricks Workspace
10. Introduction to Azure Databricks Workspace - Notes
1. Ch3 Downloadable Documents
2. Ch3 Downloadable Materials
3. Azure Databricks Architecture
4. Azure Databricks Architecture - Notes
5. Creating Azure Databricks Cluster
6. Creating Azure Databricks Cluster - Notes
7. Azure Databricks Cluster Pool
8. Azure Databricks Cluster Pool - Notes
9. Introduction to Databricks Notebooks
10. Introduction to Databricks Notebooks - Notes
11. Notebook Magic Commands
12. Notebook Magic Commands - Notes
13. Databricks Notebook Utilities
14. Databricks Notebook Utilities - Notes
15. Review Rating
1. Ch4 Downloadable Documents
2. Ch4 Downloadable Materials
3. Ch4 Section Overview
4. Databricks File System and Storage Layer Architecture
5. Databricks File System and Storage Layer Architecture - Notes
6. Exploring DBFS Root
7. Exploring DBFS Root - Notes
8. Overview of Mounting Storage Layer
9. Overview of Mounting Storage Layer - Notes
10. Introduction to Azure Storage Account
11. Introduction to Azure Storage Account - Notes
12. Creating Azure Storage Account
13. Creating Azure Storage Account - Notes
14. Introduction to Azure Storage Explorer
15. Introduction to Azure Storage Explorer - Notes
16. Creating Azure Service Principal and Secret
17. Creating Azure Service Principal and Secret - Notes
18. Granting Access to Storage Account
19. Granting Access to Storage Account - Notes
20. Mounting Your Storage Containers
21. Mounting Your Storage Containers - Notes
22. Overview of Securing your Service Secret
23. Overview of Securing your Service Secret - Notes
24. Creating and Storing Secrets in Azure Key Vault
25. Creating and Storing Secrets in Azure Key Vault - Notes
26. Link Azure Key Vault with Databricks Service
27. Link Azure Key Vault with Databricks Service - Notes
28. Mounting Your Storage Containers
29. Mounting Your Storage Containers - Notes
30. Using Databricks CLI
31. Using Databricks CLI - Notes
32. Review Rating
1. Ch5 Downloadable Documents
2. Ch5 Downloadable Materials
3. Ch5 Section Overview
4. Micro Project - Problem statement
5. Micro Project - Problem statement - Notes
6. Data Ingestion into Bronze Layer
7. Data Ingestion into Bronze Layer - Notes
8. Working with Spark Data Frames
9. Working with Spark Data Frames - Notes
10. Creating Modular Application Structure
11. Creating Modular Application Structure - Notes
12. Using Spark SQL
13. Using Spark SQL - Notes
14. Micro Project Solution Design
15. Micro Project Solution Design - Notes
16. Implementing Silver Layer Transformations
17. Implementing Silver Layer Transformations - Notes
18. Implementing Gold Layer Transformations
19. Implementing Gold Layer Transformations - Notes
20. Creating Databricks Workflow
21. Creating Databricks Workflow - Notes
22. Scheduling and Running Spark Jobs
23. Scheduling and Running Spark Jobs - Notes
1. Ch6 Downloadable Documents
2. Ch6 Downloadable Materials
3. Micro Project - Problem statement
4. Micro Project - Problem statement - Notes
5. Understanding Delta Lake
6. Understanding Delta Lake - Notes
7. Incremental Ingestion into Bronze Layer
8. Incremental Ingestion into Bronze Layer - Notes
9. Silver Layer - Historial and Incremental Data
10. Silver Layer - Historical and Incremental Data - Notes
11. Implementing Silver Layer
12. Implementing Silver Layer - Notes
13. Understanding Silver Layer Implementation
14. Understanding Silver Layer Implementation - Notes
15. Gold Layer - Historial and Incremental Data
16. Gold Layer - Historial and Incremental Data - Notes
17. Implementing Gold Layer
18. Implementing Gold Layer - Notes
19. Understanding Gold Layer Implementation
20. Understanding Gold Layer Implementation - Notes
21. Summarizing Incremental Processing
22. Summarizing Incremental Processing - Notes
23. Data Engineering Roles and Responsibilities
24. Data Engineering Roles and Responsibilities - Notes
1. Downloadable Materials
2. What will you learn in this section
3. Introduction to Unity Catalog
4. Setup Unity Catalog
5. Unity Catalog User Provisioning
6. Working with Securable Objects
1. Downloadable Materials
2. What will you learn in this section
3. Introduction to Delta Lake
4. Creating Delta Table
5. Sharing data for External Delta Table
6. Reading Delta Table
7. Delta Table Operations
8. Delta Table Time Travel
9. Convert Parquet to Delta
10. Delta Table Schema Validation
11. Delta Table Schema Evolution
12. Look Inside Delta Table
13. Delta Table Utilities and Optimization
1. Downloadable Materials
2. What will you learn in this section
3. Architecture and Need for Incremental Ingestion
4. Using Copy Into with Manual Schema Evolution
5. Using Copy Into with Automatic Schema Evolution
6. Streaming Ingestion with Manual Schema Evolution
7. Streaming Ingestion with Automatic Schema Evolution
8. Introduction to Databricks Autoloader
9. Autoloader with Automatic Schema Evolution
1. Downloadable Materials
2. What will you learn in this section
3. Introduction to Databricks DLT
4. Understand DLT Use Case Scenario
5. Setup DLT Scenario Dataset
6. Creating DLT Workload in SQL
7. Creating DLT Pipeline for your Workload
8. Creating DLT Workload in Python
1. Downloadable Materials
2. What will you learn in this section
3. Working with Databricks Repos
4. Working with Databricks Workflows
5. Working with Databricks Rest API
6. Working with Databricks CLI
1. Project Source Code and Resources
2. Project Scope and Background
3. Taking out the operational requirement
4. Storage Design
5. Implement Data Security
6. Implement Resource Policies
7. Decouple Data Ingestion
8. Design Bronze Layer
9. Design Silver and Gold Layer
10. Setup your source control
11. Setup your environment
12. Create a development workspace
13. Create and Configure Storage Layer
14. Create Unity Catalog Metastore
15. Create Catalog and External Locations
16. Start Coding
17. Test your code
18. Load historical data
19. Ingest into bronze layer
20. Process the silver layer
21. Handling multiple updates
22. Implementing Gold Layer
23. Creating a run script
24. Preparing for Integration testing
25. Creating Test Data Producer
26. Creating Integration Test for Batch mode
27. Creating Integration Test for Stream mode
28. Implementing CI CD Pipeline
29. Develop Build Pipeline
30. Develop Release Pipeline
31. Creating Databricks CLI Script

Course Features

PDF & Source Code
Total Support

Features & Support

Total Support

We provide support throughout your learning and answer every question. You may also avail one-to-one and online technical support calls for blocker issues.
Certification Ready

The course covers the curriculum for the Databricks Certified Data Engineer Associate certification exam. By the end of the course, you will be ready to take the certification exam.
Completion Certificate

Students who complete ScholarNest Academy courses earn free, verifiable course completion certificates to share with their friends, co-workers, and potential employers.

Course FAQ

How long can I access the course material?

We provide standard 3-year access to the course material from the date of purchase. However, our promotional offers may reduce the access duration for a discounted price. Please check access validity terms and conditions for the promotional offers.
Do you have a refund policy?

Yes. You can ask for a refund within 7 days of your purchase or before completing 15% of the course material, whichever is earlier. We provide a refund after deducting 6% of payment processing charges.
How do you provide support?

We have a Q&A forum where you can ask questions, and our team will answer your queries.

Enroll into self-paced course

Discover your potential, start today and learn at your own schedule

Buy Now

Schedule a free call

Get in touch with your course coordinator to learn more about the course, our instructor-led programs, discount offers, group discounts, corporate training and additional payment methods.

WhatsApp
WhatsApp: +91-93534 65988
Want to speak to your course coordinator? We are just a WhatsApp message or a phone call away.
Email
Email: [email protected]
Contact us for current promotional offers, course bundles, and international payment currency such as USD, GBP, etc.

Related Courses:

Schedule a call with course coordinator for bundles, discounts and live sessions

View more courses