diff --git a/Content/01. A Day of Fundamentals - 1 Day/01. Agenda Header.png b/Content/01. A Day of Fundamentals - 1 Day/01. Agenda Header.png new file mode 100644 index 0000000..af51e80 Binary files /dev/null and b/Content/01. A Day of Fundamentals - 1 Day/01. Agenda Header.png differ diff --git a/Content/01. A Day of Fundamentals - 1 Day/01. Agenda.md b/Content/01. A Day of Fundamentals - 1 Day/01. Agenda.md new file mode 100644 index 0000000..1cdc830 --- /dev/null +++ b/Content/01. A Day of Fundamentals - 1 Day/01. Agenda.md @@ -0,0 +1,20 @@ +# Azure Data Integration Pipeline Training +## A Day of Fundamentals +### 1 Day + +![Slide Header](./01.%20Agenda%20Header.png) + +___ + +## Overview + + +___ + +## Agenda + +The following offers an insight into the complete agenda and module breakdown for this workshop. + + + +___ diff --git a/Content/02. Fundamentals to Level 300 - 1 Day/02. Agenda Header.png b/Content/02. Fundamentals to Level 300 - 1 Day/02. Agenda Header.png new file mode 100644 index 0000000..d0d03d7 Binary files /dev/null and b/Content/02. Fundamentals to Level 300 - 1 Day/02. Agenda Header.png differ diff --git a/Content/02. Fundamentals to Level 300 - 1 Day/02. Agenda.md b/Content/02. Fundamentals to Level 300 - 1 Day/02. Agenda.md new file mode 100644 index 0000000..f4874c3 --- /dev/null +++ b/Content/02. Fundamentals to Level 300 - 1 Day/02. Agenda.md @@ -0,0 +1,20 @@ +# Azure Data Integration Pipeline Training +## Fundamentals to Level 300 +### 1 Day + +![Slide Header](./02.%20Agenda.md) + +___ + +## Overview + + +___ + +## Agenda + +The following offers an insight into the complete agenda and module breakdown for this workshop. + + + +___ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/03. Agenda Header.png b/Content/03. Advanced Design & Delivery - 1 Day/03. Agenda Header.png new file mode 100644 index 0000000..68b4c61 Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/03. Agenda Header.png differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/03. Agenda.md b/Content/03. Advanced Design & Delivery - 1 Day/03. Agenda.md new file mode 100644 index 0000000..ed72e46 --- /dev/null +++ b/Content/03. Advanced Design & Delivery - 1 Day/03. Agenda.md @@ -0,0 +1,120 @@ +# Azure Data Integration Pipeline Training +## Advanced Design & Delivery +### 1 Day + +![Slide Header](./03.%20Agenda%20Header.png) + +___ + +## Overview + +In this course we'll quickly cover the fundamentals of data integration pipelines before going much deeper into our Azure resources. + +Within a typical Azure data platform solution for any enterprise grade data analytics or data science workload an umbrella resource is needed to trigger, monitor, and handle the control flow for transforming datasets, with the goal being actionable data insight. Those requirements are met by deploying Azure Data Integration pipelines, delivered using Azure Synapse Analytics or Azure Data Factory. In this session, I'll show you how to create rich dynamic data pipelines and apply these orchestration resources in production. Using scaled architecture design patterns, best practice, data mesh principals, and the latest metadata driven frameworks. We will take a deep dive into the services, considering how to build custom activities, complex pipelines and think about hierarchical design patterns for enterprise grade deployments. All this and more in a complete set of 12 modules (based on real world experience) we will take you through how to implement data integration pipelines in production and delivered advanced orchestration patterns. + +If that's not enough learning for you, a set of hands-on labs will also be made available that you can work through at your own pace. Whether you are new to Azure Data Integration Pipelines or have some experience, you will leave this course with new skills, ideas, and a much deeper understanding of the resources for your future data platform projects. + +___ + +## Agenda + +The following offers an insight into the complete agenda and module breakdown for this course. + +* __Module 1:__ [Pipeline Fundamentals](./Module%201.pdf) + * The History of Azure Orchestration + * Synapse Analytics vs Data Factory + * Integration Components + * Common Activities + * Execution Dependencies + +___ + +* __Module 2:__ [Integration Runtime Design Patterns](./Module%202.pdf) + * Compute Types + * Azure + * Hosted + * SSIS + * Patterns & Configuration + +___ + +* __Module 3:__ [Data Transformation](./Module%203.pdf) + * Data Flows + * Power Query Injection + * Spark Configuration + * Use Cases + +___ + +* __Module 4:__ [Dynamic Pipelines](./Module%204.pdf) + * Expressions & Interpolation + * Simple Metadata Driven Execution + * Dynamic Content Chains + * Reference Names + +___ + +* __Module 5:__ [Pipeline Extensibility](./Module%205.pdf) + * Azure Batch Service + * Tasks + * Compute Pools + * Scaling + * Pipeline Custom Activities + * Azure Management API + * Azure Functions + +___ + +* __Module 6:__ [Execution Parallelism](./Module%206.pdf) + * Control Flow Scale Out + * Concurrency Limitations + * Internal vs External Activities + * Orchestration Framework - [procfwk.com](http://procfwk.com/) + +___ + +* __Module 7:__ [VNet Integration](./Module%207.pdf) + * Private Endpoints + * Managed VNet's + * Firewall Bypass + +___ + +* __Module 8:__ [Security](./Module%208.pdf) + * Service Principals + * Managed Identities + * Azure Key Vault Integration + * Customer Managed Keys + * Pipeline Access & Permissions + +___ + +* __Module 9:__ [Monitoring & Alerting](./Module%209.pdf) + * Studio Monitoring + * Log Analytics & Kusto Queries + * Operational Dashboards + * Advanced Alerting + +___ + +* __Module 10:__ [Solution Testing](./Module%2010.pdf) + * Development Time Validation + * Test Coverage + * NUnit Tests + +___ + +* __Module 11:__ [CI/CD](./Module%2011.pdf) + * Source Control vs Developer UI + * Basic ARM Template Deployments + * Advanced Deployment Patterns + +___ + +* __Module 12:__ [Final Thoughts](./Module%2012.pdf) + * Running Costs + * Conclusions + * Best Practices + + +___ \ No newline at end of file diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 1.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 1.pdf new file mode 100644 index 0000000..ce72441 Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 1.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 10.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 10.pdf new file mode 100644 index 0000000..ca6149d Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 10.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 11.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 11.pdf new file mode 100644 index 0000000..0a541ca Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 11.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 12.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 12.pdf new file mode 100644 index 0000000..3f07b7a Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 12.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 2.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 2.pdf new file mode 100644 index 0000000..366444f Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 2.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 3.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 3.pdf new file mode 100644 index 0000000..25083c5 Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 3.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 4.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 4.pdf new file mode 100644 index 0000000..6fbf2b3 Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 4.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 5.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 5.pdf new file mode 100644 index 0000000..a8988b5 Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 5.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 6.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 6.pdf new file mode 100644 index 0000000..b6f6926 Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 6.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 7.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 7.pdf new file mode 100644 index 0000000..abc7766 Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 7.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 8.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 8.pdf new file mode 100644 index 0000000..da520fc Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 8.pdf differ diff --git a/Content/03. Advanced Design & Delivery - 1 Day/Module 9.pdf b/Content/03. Advanced Design & Delivery - 1 Day/Module 9.pdf new file mode 100644 index 0000000..bb1eaa2 Binary files /dev/null and b/Content/03. Advanced Design & Delivery - 1 Day/Module 9.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/04. Agenda Header.png b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/04. Agenda Header.png new file mode 100644 index 0000000..f5cf011 Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/04. Agenda Header.png differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/04. Agenda.md b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/04. Agenda.md new file mode 100644 index 0000000..1bf6000 --- /dev/null +++ b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/04. Agenda.md @@ -0,0 +1,131 @@ +# Azure Data Integration Pipeline Training +## Advanced Design & Delivery (A Deep Dive) +### 2 Day + +![Slide Header](./04.%20Agenda%20Header.png) + +___ + +## Overview + +In this course we'll quickly cover the fundamentals of data integration pipelines before going much deeper into our Azure resources. + +Within a typical Azure data platform solution for any enterprise grade data analytics or data science workload an umbrella resource is needed to trigger, monitor, and handle the control flow for transforming datasets, with the goal being actionable data insight. Those requirements are met by deploying Azure Data Integration pipelines, delivered using Azure Synapse Analytics or Azure Data Factory. In this session, I'll show you how to create rich dynamic data pipelines and apply these orchestration resources in production. Using scaled architecture design patterns, best practice, data mesh principals, and the latest metadata driven frameworks. We will take a deep dive into the services, considering how to build custom activities, complex pipelines and think about hierarchical design patterns for enterprise grade deployments. All this and more in a complete set of 12 modules (based on real world experience) we will take you through how to implement data integration pipelines in production and delivered advanced orchestation patterns. + +If that's not enough learning for you, a set of hands-on labs will also be made available that you can work through at your own pace. Whether you are new to Azure Data Integration Pipelines or have some experience, you will leave this course with new skills, ideas, and a much deeper understanding of the resources for your future data platform projects. + +___ + +## Agenda + +The following offers an insight into the complete agenda and module breakdown for this course. + +### Day 1 + +* __Module 1:__ [Pipeline Fundamentals](./Module%201.pdf) + * The History of Azure Orchestration + * Synapse Analytics vs Data Factory + * Integration Components + * Common Activities + * Execution Dependencies + +___ + +* __Module 2:__ [Integration Runtime Design Patterns](./Module%202.pdf) + * Compute Types + * Azure + * Hosted + * SSIS + * Patterns & Configuration + +___ + +* __Module 3:__ [Data Transformation](./Module%203.pdf) + * Data Flows + * Power Query Injection + * Spark Configuration + * Use Cases + +___ + +* __Module 4:__ [Dynamic Pipelines](./Module%204.pdf) + * Expressions & Interpolation + * Simple Metadata Driven Execution + * Dynamic Content Chains + * Reference Names + +___ + +* __Module 5:__ [Pipeline Extensibility](./Module%205.pdf) + * Azure Batch Service + * Tasks + * Compute Pools + * Scaling + * Pipeline Custom Activities + * Azure Management API + * Azure Functions + +___ + +* __Module 6:__ [Execution Parallelism](./Module%206.pdf) + * Control Flow Scale Out + * Concurrency Limitations + * Internal vs External Activities + * Orchestration Framework - [procfwk.com](http://procfwk.com/) + +___ + +### Day 2 + +* __An Architects Recap__: [Modules 1 to 6](./Module%201%20-%206%20Recap.pdf) + * Design + * Extract + * Transform + * Load + +___ + +* __Module 7:__ [VNet Integration](./Module%207.pdf) + * Private Endpoints + * Managed VNet's + * Firewall Bypass + +___ + +* __Module 8:__ [Security](./Module%208.pdf) + * Service Principals + * Managed Identities + * Azure Key Vault Integration + * Customer Managed Keys + * Pipeline Access & Permissions + +___ + +* __Module 9:__ [Monitoring & Alerting](./Module%209.pdf) + * Studio Monitoring + * Log Analytics & Kusto Queries + * Operational Dashboards + * Advanced Alerting + +___ + +* __Module 10:__ [Solution Testing](./Module%2010.pdf) + * Development Time Validation + * Test Coverage + * NUnit Tests + +___ + +* __Module 11:__ [CI/CD](./Module%2011.pdf) + * Source Control vs Developer UI + * Basic ARM Template Deployments + * Advanced Deployment Patterns + +___ + +* __Module 12:__ [Final Thoughts](./Module%2012.pdf) + * Running Costs + * Conclusions + * Best Practices + +___ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 1 - 6 Recap.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 1 - 6 Recap.pdf new file mode 100644 index 0000000..b767c65 Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 1 - 6 Recap.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 1.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 1.pdf new file mode 100644 index 0000000..27d576e Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 1.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 10.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 10.pdf new file mode 100644 index 0000000..669c3e5 Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 10.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 11.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 11.pdf new file mode 100644 index 0000000..e42a947 Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 11.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 12.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 12.pdf new file mode 100644 index 0000000..732ea92 Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 12.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 2.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 2.pdf new file mode 100644 index 0000000..5f5d74f Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 2.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 3.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 3.pdf new file mode 100644 index 0000000..22cbf8f Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 3.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 4.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 4.pdf new file mode 100644 index 0000000..d111486 Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 4.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 5.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 5.pdf new file mode 100644 index 0000000..ff3e55a Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 5.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 6.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 6.pdf new file mode 100644 index 0000000..051a9a9 Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 6.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 7.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 7.pdf new file mode 100644 index 0000000..ee412e9 Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 7.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 8.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 8.pdf new file mode 100644 index 0000000..eaa786e Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 8.pdf differ diff --git a/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 9.pdf b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 9.pdf new file mode 100644 index 0000000..4cd580e Binary files /dev/null and b/Content/04. Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 9.pdf differ diff --git a/Content/05. Zero to Level 401 - 3 Day/05. Agenda Header.png b/Content/05. Zero to Level 401 - 3 Day/05. Agenda Header.png new file mode 100644 index 0000000..720f089 Binary files /dev/null and b/Content/05. Zero to Level 401 - 3 Day/05. Agenda Header.png differ diff --git a/Content/05. Zero to Level 401 - 3 Day/05. Agenda.md b/Content/05. Zero to Level 401 - 3 Day/05. Agenda.md new file mode 100644 index 0000000..2f758eb --- /dev/null +++ b/Content/05. Zero to Level 401 - 3 Day/05. Agenda.md @@ -0,0 +1,115 @@ +# Azure Data Integration Pipeline Training +## Zero to Level 401 +### 3 Day + +![Slide Header](./05.%20Agenda%20Header.png) + +___ + +## Overview + + +___ + +## Agenda + +The following offers an insight into the complete agenda and module breakdown for this course. + +* __Module 1:__ [Pipeline Fundamentals]() + * The History of Azure Orchestration + * Synapse Analytics vs Data Factory + * Integration Components + * Common Activities + * Execution Dependencies + +___ + +* __Module 2:__ [Integration Runtime Design Patterns]() + * Compute Types + * Azure + * Hosted + * SSIS + * Patterns & Configuration + +___ + +* __Module 3:__ [Data Transformation]() + * Data Flows + * Power Query Injection + * Spark Configuration + * Use Cases + +___ + +* __Module 4:__ [Dynamic Pipelines]() + * Expressions & Interpolation + * Simple Metadata Driven Execution + * Dynamic Content Chains + * Reference Names + +___ + +* __Module 5:__ [Pipeline Extensibility]() + * Azure Batch Service + * Tasks + * Compute Pools + * Scaling + * Pipeline Custom Activities + * Azure Management API + * Azure Functions + +___ + +* __Module 6:__ [Execution Parallelism]() + * Control Flow Scale Out + * Concurrency Limitations + * Internal vs External Activities + * Orchestration Framework - [procfwk.com](http://procfwk.com/) + +___ + +* __Module 7:__ [VNet Integration]() + * Private Endpoints + * Managed VNet's + * Firewall Bypass + +___ + +* __Module 8:__ [Security]() + * Service Principals + * Managed Identities + * Azure Key Vault Integration + * Customer Managed Keys + * Pipeline Access & Permissions + +___ + +* __Module 9:__ [Monitoring & Alerting]() + * Studio Monitoring + * Log Analytics & Kusto Queries + * Operational Dashboards + * Advanced Alerting + +___ + +* __Module 10:__ [Solution Testing]() + * Development Time Validation + * Test Coverage + * NUnit Tests + +___ + +* __Module 11:__ [CI/CD]() + * Source Control vs Developer UI + * Basic ARM Template Deployments + * Advanced Deployment Patterns + +___ + +* __Module 12:__ [Final Thoughts]() + * Running Costs + * Conclusions + * Best Practices + + +___ diff --git a/training.code-workspace b/training.code-workspace new file mode 100644 index 0000000..6dec11e --- /dev/null +++ b/training.code-workspace @@ -0,0 +1,15 @@ +{ + "folders": [ + { + "path": "." + } + ], + "settings": { + "cSpell.words": [ + "Avanade", + "Kusto", + "mrpaulandrew", + "procfwk" + ] + } +} \ No newline at end of file