Repo reorg

This commit is contained in:
Paul Andrew
2022-08-16 08:42:57 +01:00
parent d969bf2580
commit dca62f67e7
34 changed files with 119 additions and 377 deletions

View File

@@ -2,12 +2,21 @@
## Fundamentals to Level 300
### 1 Day
![Slide Header](./02.%20Agenda.md)
![Slide Header](./02.%20Agenda%20Header.png)
___
## Overview
In this full day of training, well start with the very basics and learn how to orchestrate your Azure data platform from start to finish. You will learn how to build out Azure control flow and data flow components as processing pipelines using Azure Data Factory and Azure Synapse Analytics. Well start by covering the fundamentals within the resources and together build out pipelines that ingest data from local source systems, transform and serve it to consumers. Well then continue taking an end-to-end look at our Azure integration pipeline tools within highly scalable cloud native architectures, dealing with triggering, monitoring, dynamic pipeline content as well as CI/CD practices. Start the day knowing nothing about Azure Data Integration pipelines and leave with the knowledge, slides, demos, and code to apply these resources in your role as a data engineering professional.
___
## Milestones
* Pipelines - the fundamentals
* Getting hands with the basics
* Pipelines - advanced features & extensibility
* Pipelines - real world implementation
___
@@ -15,6 +24,101 @@ ___
The following offers an insight into the complete agenda and module breakdown for this workshop.
* __Module 1:__ [Pipeline Fundamentals](./Module%201.pdf)
* The History of Azure Orchestration
* Synapse Analytics vs Data Factory
* Integration Components
* Common Activities
* Execution Dependencies
___
* __Module 2:__ [Integration Runtime Design Patterns](./Module%202.pdf)
* Compute Types
* Azure
* Hosted
* SSIS
* Patterns & Configuration
___
* __Module 3:__ [Data Transformation](./Module%203.pdf)
* Data Flows
* Power Query Injection
* Spark Configuration
* Use Cases
___
* __Module 4:__ [Dynamic Pipelines](./Module%204.pdf)
* Expressions & Interpolation
* Simple Metadata Driven Execution
* Dynamic Content Chains
* Reference Names
___
* __Module 5:__ [Pipeline Extensibility](./Module%205.pdf)
* Azure Batch Service
* Tasks
* Compute Pools
* Scaling
* Pipeline Custom Activities
* Azure Management API
* Azure Functions
___
* __Module 6:__ [Execution Parallelism](./Module%206.pdf)
* Control Flow Scale Out
* Concurrency Limitations
* Internal vs External Activities
* Orchestration Framework - [procfwk.com](http://procfwk.com/)
___
* __Module 7:__ [VNet Integration](./Module%207.pdf)
* Private Endpoints
* Managed VNet's
* Firewall Bypass
___
* __Module 8:__ [Security](./Module%208.pdf)
* Service Principals
* Managed Identities
* Azure Key Vault Integration
* Customer Managed Keys
* Pipeline Access & Permissions
___
* __Module 9:__ [Monitoring & Alerting](./Module%209.pdf)
* Studio Monitoring
* Log Analytics & Kusto Queries
* Operational Dashboards
* Advanced Alerting
___
* __Module 10:__ [Solution Testing](./Module%2010.pdf)
* Development Time Validation
* Test Coverage
* NUnit Tests
___
* __Module 11:__ [CI/CD](./Module%2011.pdf)
* Source Control vs Developer UI
* Basic ARM Template Deployments
* Advanced Deployment Patterns
___
* __Module 12:__ [Final Thoughts](./Module%2012.pdf)
* Running Costs
* Conclusions
* Best Practices
___