3.6 KiB
Azure Data Integration Pipeline Training
Fundamentals to Level 300
1 Day
Overview
In this full day of training, we’ll start with the very basics and learn how to orchestrate your Azure data platform from start to finish. You will learn how to build out Azure control flow and data flow components as processing pipelines using Azure Data Factory and Azure Synapse Analytics. We’ll start by covering the fundamentals within the resources and together build out pipelines that ingest data from local source systems, transform and serve it to consumers. We’ll then continue taking an end-to-end look at our Azure integration pipeline tools within highly scalable cloud native architectures, dealing with triggering, monitoring, dynamic pipeline content as well as CI/CD practices. Start the day knowing nothing about Azure Data Integration pipelines and leave with the knowledge, slides, demos, and code to apply these resources in your role as a data engineering professional.
Milestones
- Pipelines - the fundamentals
- Getting hands with the basics
- Pipelines - advanced features & extensibility
- Pipelines - real world implementation
Agenda
The following offers an insight into the complete agenda and module breakdown for this workshop.
- Module 1: Pipeline Fundamentals
- The History of Azure Orchestration
- Synapse Analytics vs Data Factory
- Integration Components
- Common Activities
- Execution Dependencies
- Module 2: Integration Runtime Design Patterns
- Compute Types
- Azure
- Hosted
- SSIS
- Patterns & Configuration
- Compute Types
- Module 3: Data Transformation
- Data Flows
- Power Query Injection
- Spark Configuration
- Use Cases
- Labs: Getting Hands On
- Create Azure Resources
- Build a Copy Data Pipeline
- Run SSIS Packages
- Building a Data Flow
- Create a Reusable Pipeline
- Basic Monitoring
- Module 4: Dynamic Pipelines
- Expressions & Interpolation
- Simple Metadata Driven Execution
- Dynamic Content Chains
- Reference Names
- Module 5: Pipeline Extensibility
- Azure Batch Service
- Tasks
- Compute Pools
- Scaling
- Pipeline Custom Activities
- Azure Management API
- Azure Functions
- Azure Batch Service
- Module 6: Execution Parallelism
- Control Flow Scale Out
- Concurrency Limitations
- Internal vs External Activities
- Orchestration Framework - procfwk.com
- Module 7: VNet Integration
- Private Endpoints
- Managed VNet's
- Firewall Bypass
- Module 8: Security
- Service Principals
- Managed Identities
- Azure Key Vault Integration
- Customer Managed Keys
- Pipeline Access & Permissions
- Module 9: Monitoring & Alerting
- Studio Monitoring
- Log Analytics & Kusto Queries
- Operational Dashboards
- Advanced Alerting
- Module 10: Solution Testing
- Development Time Validation
- Test Coverage
- NUnit Tests
- Module 11: CI/CD
- Source Control vs Developer UI
- Basic ARM Template Deployments
- Advanced Deployment Patterns
- Module 12: Final Thoughts
- Running Costs
- Conclusions
- Best Practices