# Azure Data Integration Pipeline Training
## Fundamentals to Level 300
### 1 Day

![Slide Header](./02.%20Agenda%20Header.png)

___

## Overview

In this full day of training, we’ll start with the very basics and learn how to orchestrate your Azure data platform from start to finish. You will learn how to build out Azure control flow and data flow components as processing pipelines using Azure Data Factory and Azure Synapse Analytics. We’ll start by covering the fundamentals within the resources and together build out pipelines that ingest data from local source systems, transform and serve it to consumers. We’ll then continue taking an end-to-end look at our Azure integration pipeline tools within highly scalable cloud native architectures, dealing with triggering, monitoring, dynamic pipeline content as well as CI/CD practices. Start the day knowing nothing about Azure Data Integration pipelines and leave with the knowledge, slides, demos, and code to apply these resources in your role as a data engineering professional.
___

## Milestones

* Pipelines - the fundamentals
* Getting hands with the basics
* Pipelines - advanced features & extensibility
* Pipelines - real world implementation

___

## Agenda

The following offers an insight into the complete agenda and module breakdown for this workshop.

* __Module 1:__ [Pipeline Fundamentals](./Module%201.pdf)
  * The History of Azure Orchestration
  * Synapse Analytics vs Data Factory
  * Integration Components
  * Common Activities
  * Execution Dependencies

___

* __Module 2:__ [Integration Runtime Design Patterns](./Module%202.pdf)
  * Compute Types
    * Azure
    * Hosted
    * SSIS
  * Patterns & Configuration

___

* __Module 3:__ [Data Transformation](./Module%203.pdf)
  * Data Flows
  * Power Query Injection
  * Spark Configuration
  * Use Cases

___

* __Labs:__ [Getting Hands On](https://github.com/mrpaulandrewltd/Azure-Data-Integration-Pipeline-Training/tree/main/Labs)
  * Create Azure Resources
  * Build a Copy Data Pipeline
  * Run SSIS Packages
  * Building a Data Flow
  * Create a Reusable Pipeline
  * Basic Monitoring

___

* __Module 4:__ [Dynamic Pipelines](./Module%204.pdf)
  * Expressions & Interpolation
  * Simple Metadata Driven Execution
  * Dynamic Content Chains
  * Reference Names

___

* __Module 5:__ [Pipeline Extensibility](./Module%205.pdf)
  * Azure Batch Service
    * Tasks
    * Compute Pools
    * Scaling
  * Pipeline Custom Activities
  * Azure Management API
  * Azure Functions 

___

* __Module 6:__ [Execution Parallelism](./Module%206.pdf)
  * Control Flow Scale Out
  * Concurrency Limitations
  * Internal vs External Activities
  * Orchestration Framework - [procfwk.com](http://procfwk.com/)

___

* __Module 7:__ [VNet Integration](./Module%207.pdf)
  * Private Endpoints
  * Managed VNet's
  * Firewall Bypass

___

* __Module 8:__ [Security](./Module%208.pdf)
  * Service Principals
  * Managed Identities
  * Azure Key Vault Integration
  * Customer Managed Keys
  * Pipeline Access & Permissions

___

* __Module 9:__ [Monitoring & Alerting](./Module%209.pdf)
    * Studio Monitoring
    * Log Analytics & Kusto Queries
    * Operational Dashboards
    * Advanced Alerting

___

* __Module 10:__ [Solution Testing](./Module%2010.pdf)
    * Development Time Validation
    * Test Coverage
    * NUnit Tests

___

* __Module 11:__ [CI/CD](./Module%2011.pdf)
    * Source Control vs Developer UI
    * Basic ARM Template Deployments
    * Advanced Deployment Patterns

___

* __Module 12:__ [Final Thoughts](./Module%2012.pdf)
  * Running Costs
  * Conclusions
  * Best Practices


___

[<< Back to root](https://github.com/mrpaulandrewltd/Azure-Data-Integration-Pipeline-Training)