diff --git a/Content/A Day of Fundementals - 1 Day/Agenda Header.png b/Content/A Day of Fundementals - 1 Day/Agenda Header.png new file mode 100644 index 0000000..af51e80 Binary files /dev/null and b/Content/A Day of Fundementals - 1 Day/Agenda Header.png differ diff --git a/Content/A Day of Fundementals - 1 Day/Agenda.md b/Content/A Day of Fundementals - 1 Day/Agenda.md new file mode 100644 index 0000000..2900eba --- /dev/null +++ b/Content/A Day of Fundementals - 1 Day/Agenda.md @@ -0,0 +1,20 @@ +# Azure Data Integration Pipeline Training +## A Day of Fundementals + +![Slide Header](.\Agenda%20Header.png) + + +___ + +## Overview + + +___ + +## Agenda + +The following offers an insight into the complete agenda and module breakdown for this workshop. + + + +___ diff --git a/Images/Slide Header.png b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Agenda Header.png similarity index 100% rename from Images/Slide Header.png rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Agenda Header.png diff --git a/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Agenda.md b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Agenda.md new file mode 100644 index 0000000..fc6a765 --- /dev/null +++ b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Agenda.md @@ -0,0 +1,125 @@ +# Azure Data Integration Pipeline Training +## Advanced Design & Delivery (A Deep Dive) + +![Slide Header](.\Agenda%20Header.png) + +___ + +## Overview + +In this course we'll quickly cover the fundamentals of data integration pipelines before going much deeper into our Azure resources. + +Within a typical Azure data platform solution for any enterprise grade data analytics or data science workload an umbrella resource is needed to trigger, monitor, and handle the control flow for transforming datasets, with the goal being actionable data insight. Those requirements are met by deploying Azure Data Integration pipelines, delivered using Azure Synapse Analytics or Azure Data Factory. In this session, I'll show you how to create rich dynamic data pipelines and apply these orchestration resources in production. Using scaled architecture design patterns, best practice, data mesh principals, and the latest metadata driven frameworks. We will take a deep dive into the services, considering how to build custom activities, complex pipelines and think about hierarchical design patterns for enterprise grade deployments. All this and more in a complete set of 12 modules (based on real world experience) we will take you through how to implement data integration pipelines in production and delivered advanced orchestation patterns. + +If that's not enough learning for you, a set of hands-on labs will also be made available that you can work through at your own pace. Whether you are new to Azure Data Integration Pipelines or have some experience, you will leave this course with new skills, ideas, and a much deeper understanding of the resources for your future data platform projects. + +___ + +## Agenda + +The following offers an insight into the complete agenda and module breakdown for this course. + +* __Module 1:__ [Pipeline Fundamentals]() + * The History of Azure Orchestration + * Synapse Analytics vs Data Factory + * Integration Components + * Common Activities + * Execution Dependencies + +___ + +* __Module 2:__ [Integration Runtime Design Patterns]() + * Compute Types + * Azure + * Hosted + * SSIS + * Patterns & Configuration + +___ + +* __Module 3:__ [Data Transformation]() + * Data Flows + * Power Query Injection + * Spark Configuration + * Use Cases + +___ + +* __Module 4:__ [Dynamic Pipelines]() + * Expressions & Interpolation + * Simple Metadata Driven Execution + * Dynamic Content Chains + * Reference Names + +___ + +* __Module 5:__ [Pipeline Extensibility]() + * Azure Batch Service + * Tasks + * Compute Pools + * Scaling + * Pipeline Custom Activities + * Azure Management API + * Azure Functions + +___ + +* __Module 6:__ [Execution Parallelism]() + * Control Flow Scale Out + * Concurrency Limitations + * Internal vs External Activities + * Orchestration Framework - [procfwk.com](http://procfwk.com/) + +___ + +* __An Architects Recap__: [Modules 1 to 6]() + * Design + * Extract + * Transform + * Load + +___ + +* __Module 7:__ [VNet Integration]() + * Private Endpoints + * Managed VNet's + * Firewall Bypass + +___ + +* __Module 8:__ [Security]() + * Service Principals + * Managed Identities + * Azure Key Vault Integration + * Customer Managed Keys + * Pipeline Access & Permissions + +___ + +* __Module 9:__ [Monitoring & Alerting]() + * Studio Monitoring + * Log Analytics & Kusto Queries + * Operational Dashboards + * Advanced Alerting + +* __Module 10:__ [Solution Testing]() + * Development Time Validation + * Test Coverage + * NUnit Tests + +___ + +* __Module 11:__ [CI/CD]() + * Source Control vs Developer UI + * Basic ARM Template Deployments + * Advanced Deployment Patterns + +___ + +* __Module 12:__ [Final Thoughts]() + * Running Costs + * Conclusions + * Best Practices + + +___ diff --git a/Slides/Two Day/Module 1 - 6 Recap.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 1 - 6 Recap.pdf similarity index 100% rename from Slides/Two Day/Module 1 - 6 Recap.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 1 - 6 Recap.pdf diff --git a/Slides/Two Day/Module 1.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 1.pdf similarity index 100% rename from Slides/Two Day/Module 1.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 1.pdf diff --git a/Slides/Two Day/Module 10.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 10.pdf similarity index 100% rename from Slides/Two Day/Module 10.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 10.pdf diff --git a/Slides/Two Day/Module 11.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 11.pdf similarity index 100% rename from Slides/Two Day/Module 11.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 11.pdf diff --git a/Slides/Two Day/Module 12.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 12.pdf similarity index 100% rename from Slides/Two Day/Module 12.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 12.pdf diff --git a/Slides/Two Day/Module 2.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 2.pdf similarity index 100% rename from Slides/Two Day/Module 2.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 2.pdf diff --git a/Slides/Two Day/Module 3.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 3.pdf similarity index 100% rename from Slides/Two Day/Module 3.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 3.pdf diff --git a/Slides/Two Day/Module 4.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 4.pdf similarity index 100% rename from Slides/Two Day/Module 4.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 4.pdf diff --git a/Slides/Two Day/Module 5.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 5.pdf similarity index 100% rename from Slides/Two Day/Module 5.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 5.pdf diff --git a/Slides/Two Day/Module 6.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 6.pdf similarity index 100% rename from Slides/Two Day/Module 6.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 6.pdf diff --git a/Slides/Two Day/Module 7.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 7.pdf similarity index 100% rename from Slides/Two Day/Module 7.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 7.pdf diff --git a/Slides/Two Day/Module 8.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 8.pdf similarity index 100% rename from Slides/Two Day/Module 8.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 8.pdf diff --git a/Slides/Two Day/Module 9.pdf b/Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 9.pdf similarity index 100% rename from Slides/Two Day/Module 9.pdf rename to Content/Advanced Design & Delivery (A Deep Dive) - 2 Day/Module 9.pdf diff --git a/Content/Advanced Design & Delivery - 1 Day/Agenda Header.png b/Content/Advanced Design & Delivery - 1 Day/Agenda Header.png new file mode 100644 index 0000000..68b4c61 Binary files /dev/null and b/Content/Advanced Design & Delivery - 1 Day/Agenda Header.png differ diff --git a/Content/Advanced Design & Delivery - 1 Day/Agenda.md b/Content/Advanced Design & Delivery - 1 Day/Agenda.md new file mode 100644 index 0000000..3e1ce5e --- /dev/null +++ b/Content/Advanced Design & Delivery - 1 Day/Agenda.md @@ -0,0 +1,117 @@ +# Azure Data Integration Pipeline Training +## Advanced Design & Delivery + +![Slide Header](.\Agenda%20Header.png) + +___ + +## Overview + +In this course we'll quickly cover the fundamentals of data integration pipelines before going much deeper into our Azure resources. + +Within a typical Azure data platform solution for any enterprise grade data analytics or data science workload an umbrella resource is needed to trigger, monitor, and handle the control flow for transforming datasets, with the goal being actionable data insight. Those requirements are met by deploying Azure Data Integration pipelines, delivered using Azure Synapse Analytics or Azure Data Factory. In this session, I'll show you how to create rich dynamic data pipelines and apply these orchestration resources in production. Using scaled architecture design patterns, best practice, data mesh principals, and the latest metadata driven frameworks. We will take a deep dive into the services, considering how to build custom activities, complex pipelines and think about hierarchical design patterns for enterprise grade deployments. All this and more in a complete set of 12 modules (based on real world experience) we will take you through how to implement data integration pipelines in production and delivered advanced orchestation patterns. + +If that's not enough learning for you, a set of hands-on labs will also be made available that you can work through at your own pace. Whether you are new to Azure Data Integration Pipelines or have some experience, you will leave this course with new skills, ideas, and a much deeper understanding of the resources for your future data platform projects. + +___ + +## Agenda + +The following offers an insight into the complete agenda and module breakdown for this course. + +* __Module 1:__ [Pipeline Fundamentals]() + * The History of Azure Orchestration + * Synapse Analytics vs Data Factory + * Integration Components + * Common Activities + * Execution Dependencies + +___ + +* __Module 2:__ [Integration Runtime Design Patterns]() + * Compute Types + * Azure + * Hosted + * SSIS + * Patterns & Configuration + +___ + +* __Module 3:__ [Data Transformation]() + * Data Flows + * Power Query Injection + * Spark Configuration + * Use Cases + +___ + +* __Module 4:__ [Dynamic Pipelines]() + * Expressions & Interpolation + * Simple Metadata Driven Execution + * Dynamic Content Chains + * Reference Names + +___ + +* __Module 5:__ [Pipeline Extensibility]() + * Azure Batch Service + * Tasks + * Compute Pools + * Scaling + * Pipeline Custom Activities + * Azure Management API + * Azure Functions + +___ + +* __Module 6:__ [Execution Parallelism]() + * Control Flow Scale Out + * Concurrency Limitations + * Internal vs External Activities + * Orchestration Framework - [procfwk.com](http://procfwk.com/) + +___ + +* __Module 7:__ [VNet Integration]() + * Private Endpoints + * Managed VNet's + * Firewall Bypass + +___ + +* __Module 8:__ [Security]() + * Service Principals + * Managed Identities + * Azure Key Vault Integration + * Customer Managed Keys + * Pipeline Access & Permissions + +___ + +* __Module 9:__ [Monitoring & Alerting]() + * Studio Monitoring + * Log Analytics & Kusto Queries + * Operational Dashboards + * Advanced Alerting + +* __Module 10:__ [Solution Testing]() + * Development Time Validation + * Test Coverage + * NUnit Tests + +___ + +* __Module 11:__ [CI/CD]() + * Source Control vs Developer UI + * Basic ARM Template Deployments + * Advanced Deployment Patterns + +___ + +* __Module 12:__ [Final Thoughts]() + * Running Costs + * Conclusions + * Best Practices + + +___ \ No newline at end of file diff --git a/Slides/One Day/Module 1.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 1.pdf similarity index 100% rename from Slides/One Day/Module 1.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 1.pdf diff --git a/Slides/One Day/Module 10.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 10.pdf similarity index 100% rename from Slides/One Day/Module 10.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 10.pdf diff --git a/Slides/One Day/Module 11.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 11.pdf similarity index 100% rename from Slides/One Day/Module 11.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 11.pdf diff --git a/Slides/One Day/Module 12.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 12.pdf similarity index 100% rename from Slides/One Day/Module 12.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 12.pdf diff --git a/Slides/One Day/Module 2.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 2.pdf similarity index 100% rename from Slides/One Day/Module 2.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 2.pdf diff --git a/Slides/One Day/Module 3.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 3.pdf similarity index 100% rename from Slides/One Day/Module 3.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 3.pdf diff --git a/Slides/One Day/Module 4.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 4.pdf similarity index 100% rename from Slides/One Day/Module 4.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 4.pdf diff --git a/Slides/One Day/Module 5.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 5.pdf similarity index 100% rename from Slides/One Day/Module 5.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 5.pdf diff --git a/Slides/One Day/Module 6.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 6.pdf similarity index 100% rename from Slides/One Day/Module 6.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 6.pdf diff --git a/Slides/One Day/Module 7.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 7.pdf similarity index 100% rename from Slides/One Day/Module 7.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 7.pdf diff --git a/Slides/One Day/Module 8.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 8.pdf similarity index 100% rename from Slides/One Day/Module 8.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 8.pdf diff --git a/Slides/One Day/Module 9.pdf b/Content/Advanced Design & Delivery - 1 Day/Module 9.pdf similarity index 100% rename from Slides/One Day/Module 9.pdf rename to Content/Advanced Design & Delivery - 1 Day/Module 9.pdf diff --git a/Content/Fundamentals to Level 300 - 1 Day/Agenda Header.png b/Content/Fundamentals to Level 300 - 1 Day/Agenda Header.png new file mode 100644 index 0000000..d0d03d7 Binary files /dev/null and b/Content/Fundamentals to Level 300 - 1 Day/Agenda Header.png differ diff --git a/Content/Fundamentals to Level 300 - 1 Day/Agenda.md b/Content/Fundamentals to Level 300 - 1 Day/Agenda.md new file mode 100644 index 0000000..c7a1b3f --- /dev/null +++ b/Content/Fundamentals to Level 300 - 1 Day/Agenda.md @@ -0,0 +1,19 @@ +# Azure Data Integration Pipeline Training +## Fundamentals to Level 300 + +![Slide Header](.\Agenda%20Header.png) + +___ + +## Overview + + +___ + +## Agenda + +The following offers an insight into the complete agenda and module breakdown for this workshop. + + + +___ diff --git a/Slides/The Azure Icon Game.pdf b/Content/The Azure Icon Game.pdf similarity index 100% rename from Slides/The Azure Icon Game.pdf rename to Content/The Azure Icon Game.pdf diff --git a/Content/Zero to Level 401 - 3 Day/Agenda Header.png b/Content/Zero to Level 401 - 3 Day/Agenda Header.png new file mode 100644 index 0000000..720f089 Binary files /dev/null and b/Content/Zero to Level 401 - 3 Day/Agenda Header.png differ diff --git a/Content/Zero to Level 401 - 3 Day/Agenda.md b/Content/Zero to Level 401 - 3 Day/Agenda.md new file mode 100644 index 0000000..3e02b50 --- /dev/null +++ b/Content/Zero to Level 401 - 3 Day/Agenda.md @@ -0,0 +1,112 @@ +# Azure Data Integration Pipeline Training +## Zero to Level 401 + +![Slide Header](.\Agenda%20Header.png) + +___ + +## Overview + + +___ + +## Agenda + +The following offers an insight into the complete agenda and module breakdown for this course. + +* __Module 1:__ [Pipeline Fundamentals]() + * The History of Azure Orchestration + * Synapse Analytics vs Data Factory + * Integration Components + * Common Activities + * Execution Dependencies + +___ + +* __Module 2:__ [Integration Runtime Design Patterns]() + * Compute Types + * Azure + * Hosted + * SSIS + * Patterns & Configuration + +___ + +* __Module 3:__ [Data Transformation]() + * Data Flows + * Power Query Injection + * Spark Configuration + * Use Cases + +___ + +* __Module 4:__ [Dynamic Pipelines]() + * Expressions & Interpolation + * Simple Metadata Driven Execution + * Dynamic Content Chains + * Reference Names + +___ + +* __Module 5:__ [Pipeline Extensibility]() + * Azure Batch Service + * Tasks + * Compute Pools + * Scaling + * Pipeline Custom Activities + * Azure Management API + * Azure Functions + +___ + +* __Module 6:__ [Execution Parallelism]() + * Control Flow Scale Out + * Concurrency Limitations + * Internal vs External Activities + * Orchestration Framework - [procfwk.com](http://procfwk.com/) + +___ + +* __Module 7:__ [VNet Integration]() + * Private Endpoints + * Managed VNet's + * Firewall Bypass + +___ + +* __Module 8:__ [Security]() + * Service Principals + * Managed Identities + * Azure Key Vault Integration + * Customer Managed Keys + * Pipeline Access & Permissions + +___ + +* __Module 9:__ [Monitoring & Alerting]() + * Studio Monitoring + * Log Analytics & Kusto Queries + * Operational Dashboards + * Advanced Alerting + +* __Module 10:__ [Solution Testing]() + * Development Time Validation + * Test Coverage + * NUnit Tests + +___ + +* __Module 11:__ [CI/CD]() + * Source Control vs Developer UI + * Basic ARM Template Deployments + * Advanced Deployment Patterns + +___ + +* __Module 12:__ [Final Thoughts]() + * Running Costs + * Conclusions + * Best Practices + + +___ diff --git a/Images/ReadMe Header.png b/Images/ReadMe Header.png new file mode 100644 index 0000000..35eda9a Binary files /dev/null and b/Images/ReadMe Header.png differ diff --git a/Images/Repo Social Media Header.png b/Images/Repo Social Media Header.png index 3ec3466..57ed657 100644 Binary files a/Images/Repo Social Media Header.png and b/Images/Repo Social Media Header.png differ diff --git a/README.md b/README.md index 7383927..809ef11 100644 --- a/README.md +++ b/README.md @@ -1,138 +1,29 @@ # Azure Data Integration Pipeline Training ## _with Paul Andrew_ -![Slide Header](https://raw.githubusercontent.com/mrpaulandrew/Azure-Data-Integration-Pipelines-Advanced-Design-and-Delivery/main/Images/Slide%20Header.png) +![Slide Header](https://raw.githubusercontent.com/mrpaulandrew/Azure-Data-Integration-Pipeline-Training/main/Images/ReadMe%20Header.png) -Hey friends and welcome to this training course on __Azure Data Integration Pipelines - Advanced Design & Delivery (A Deep Dive)__. Over the next two days we will be all becoming advanced pipeline workers!... And we completely recommend this description when describing your job to non-technical family members. But be warned, if you go on to tell them that the factory of pipes is in the cloud for orchestration and integration you are likely to be branded as crazy. However, here and now that is ok. You are amongst like-minded geeky friends that all want to become cloud pipeline workers that doing data plumbing with data pipes as well :-) +Hey friends and welcome to my training workshops on __Azure Data Integration Pipelines__. + +Over the coming hours or days (depending on your choosen workshop) we will be all becoming pipeline workers!... And I completely recommend this description when describing your job to non-technical family members. But be warned, if you go on to tell them that the factory of pipes is in the cloud for orchestration and integration you are likely to be branded as crazy. However, here and now that is ok. You are amongst like-minded geeky friends that all want to become cloud pipeline workers that doing data plumbing with data pipes as well :-) On a more serious note, throughout our day of training you will quickly notice, like with most technologies, there are an awful lot of different ways you can implement Azure orchestration services and understanding the best way to do something is often the biggest challenge. That said, if you only take away one thing from this training, I will ask that you have an appreciation of this fact; in depends! Then when delviering solutions you take a step back from the requirements and think about the overall technical design and how Azure Integration Pipelines should fit into your platform as a core component. All too often with new and shiny services we start playing around then try to make the technology fit our solution. Rather than thinking about the solution requirements and which technology meets our needs. This is true of all developers, I don't want to preach, so am simply asking that we take a growth mindset. Think about the outputs and the requirements as a goal. - ___ -## Course Overview - -In this course we'll quickly cover the fundamentals of data integration pipelines before going much deeper into our Azure resources. - -Within a typical Azure data platform solution for any enterprise grade data analytics or data science workload an umbrella resource is needed to trigger, monitor, and handle the control flow for transforming datasets, with the goal being actionable data insight. Those requirements are met by deploying Azure Data Integration pipelines, delivered using Azure Synapse Analytics or Azure Data Factory. In this session, I'll show you how to create rich dynamic data pipelines and apply these orchestration resources in production. Using scaled architecture design patterns, best practice, data mesh principals, and the latest metadata driven frameworks. We will take a deep dive into the services, considering how to build custom activities, complex pipelines and think about hierarchical design patterns for enterprise grade deployments. All this and more in a complete set of 12 modules (based on real world experience) we will take you through how to implement data integration pipelines in production and delivered advanced orchestation patterns. - -If that's not enough learning for you, a set of hands-on labs will also be made available that you can work through at your own pace. Whether you are new to Azure Data Integration Pipelines or have some experience, you will leave this course with new skills, ideas, and a much deeper understanding of the resources for your future data platform projects. - -___ - -## Course Prerequisites - -If you've __never__ used Azure Data Integration Pipelines before in either Azure Data Factory or Azure Synapse Analytics, that's ok! However, please watch my 1 hour complete introduction session available via my blog and on YouTube, recorded as part of a recent community MeetUp. Link below: - -https://mrpaulandrew.com/2021/08/23/an-introduction-to-azure-data-integration-pipelines/ - -___ - -## Course Agenda - -The following offers an insight into the complete agenda and module breakdown for this course. - -* __Module 1:__ [Pipeline Fundamentals]() - * The History of Azure Orchestration - * Synapse Analytics vs Data Factory - * Integration Components - * Common Activities - * Execution Dependencies - -___ - -* __Module 2:__ [Integration Runtime Design Patterns]() - * Compute Types - * Azure - * Hosted - * SSIS - * Patterns & Configuration - -___ - -* __Module 3:__ [Data Transformation]() - * Data Flows - * Power Query Injection - * Spark Configuration - * Use Cases - -___ - -* __Module 4:__ [Dynamic Pipelines]() - * Expressions & Interpolation - * Simple Metadata Driven Execution - * Dynamic Content Chains - * Reference Names - -___ - -* __Module 5:__ [Pipeline Extensibility]() - * Azure Batch Service - * Tasks - * Compute Pools - * Scaling - * Pipeline Custom Activities - * Azure Management API - * Azure Functions - -___ - -* __Module 6:__ [Execution Parallelism]() - * Control Flow Scale Out - * Concurrency Limitations - * Internal vs External Activities - * Orchestration Framework - [procfwk.com](http://procfwk.com/) - -___ - -* __Module 7:__ [VNet Integration]() - * Private Endpoints - * Managed VNet's - * Firewall Bypass - -___ - -* __Module 8:__ [Security]() - * Service Principals - * Managed Identities - * Azure Key Vault Integration - * Customer Managed Keys - * Pipeline Access & Permissions - -___ - -* __Module 9:__ [Monitoring & Alerting]() - * Studio Monitoring - * Log Analytics & Kusto Queries - * Operational Dashboards - * Advanced Alerting - -* __Module 10:__ [Solution Testing]() - * Development Time Validation - * Test Coverage - * NUnit Tests - -___ - -* __Module 11:__ [CI/CD]() - * Source Control vs Developer UI - * Basic ARM Template Deployments - * Advanced Deployment Patterns - -___ - -* __Module 12:__ [Final Thoughts]() - * Running Costs - * Conclusions - * Best Practices - +## Workshops + - A Day of Fundementals - 1 Day + - Fundamentals to Level 300 - 1 Day + - Advanced Design & Delivery - 1 Day + - Advanced Design & Delivery (A Deep Dive) - 2 Day + - Zero to Level 401 - 3 Day ___ ## Speaker Biography -Paul Andrew is a Microsoft Data Platform MVP and Technical Architect within the Avanade Centre of Excellence team, with over 15 years’ experience in the industry, working as an engineer and solution architect. Day-to-day Paul is accountable for delivering enterprise grade data insights to international organisations where he wields the complete stack of Azure Data Platform resources. Paul leads delivery teams around the globe implementing the latest design patterns, creating architectural innovations, and defining best practice to ensure technical excellence for customers across a wide variety of industry verticals. Paul is passionate about technology, which is demonstrated in the community, he speaks at events and shares his knowledge gained from real world experience through his blog. Paul maintains the view that his job is also his hobby and doesn’t ever want to take his fingers off the developer’s keyboard while maintaining a growth mindset. +Paul Andrew is a Microsoft Data Platform MVP and Technical Architect within the Avanade Centre of Excellence team, with over 15 years' experience in the industry, working as an engineer and solution architect. Day-to-day Paul is accountable for delivering enterprise grade data insights to international organisations where he wields the complete stack of Azure Data Platform resources. Paul leads delivery teams around the globe implementing the latest design patterns, creating architectural innovations, and defining best practice to ensure technical excellence for customers across a wide variety of industry verticals. Paul is passionate about technology, which is demonstrated in the community, he speaks at events and shares his knowledge gained from real world experience through his blog. Paul maintains the view that his job is also his hobby and doesn’t ever want to take his fingers off the developer’s keyboard while maintaining a growth mindset. ## Speaker Contact Details