“Data Engineering with Databricks Cookbook” is a technical working guide that shares knowledge to work successfully with Apache Spark, Delta Lake, and Databricks. It begins by familiarising the learner with how to load and process data using Apache Spark, wherein structured and unstructured data is easily manageable. You will then cover the data transformation strategies, Delta table on management, and streaming data processing to optimize your data operations.
By getting started, you will understand how to enhance the performance of your applications built in Spark and your Delta Lake queries. It also introduces all the key DataOps and DevOps processes to ensure seamless and efficient data operations on Databricks. You will learn about handling workflows related to data pipelines, such as Delta Live Tables and Databricks Workflows, which are crucial for having tidily set processes for data.
Last of all, the book, “Data Engineering with Databricks Cookbook,” presents data governance using Unity Catalog and guides on implementing data policies and measures. By the time you finish this book, you will have learned about constructing a production-grade, fault-tolerant data pipeline system that can be adapted using today’s state-of-the-art technologies. Whether it is for real-time processing of data or for having perfect control over data processing operations, the following recipes are the same that give the best results.
Data Engineering with Databricks Cookbook Table of Contents:
- Data Ingestion and Data Extraction with Apache Spark
- Data Transformation and Data Manipulation with Apache Spark
- Data Management with Delta Lake
- Ingesting Streaming Data
- Processing Streaming Data
- Performance Tuning with Apache Spark
- Performance Tuning in Delta Lake
- Orchestration and Scheduling Data Pipeline with Databricks Workflows
- Building Data Pipelines with Delta Live Tables
- Data Governance with Unity Catalog
- Implementing DataOps and DevOps on Databricks
Who is this course for?
- Data engineers
- Data scientists
- Data practitioners
