Enterprise-Grade Terraform CI/CD in Azure DevOps - Complete Series

This is the complete guide to building enterprise-grade Terraform CI/CD pipelines in Azure DevOps. Not theory - this is the exact setup I run in production.

This guide is part of our Azure Governance hub covering policy enforcement, compliance frameworks, and enterprise controls.

Why This Series Exists

Most Terraform tutorials show you terraform apply from your laptop. Great for learning. Useless for production.

This series covers:
- ✅ GitOps workflow - All changes through Git and pull requests
- ✅ Automated validation - Status checks on every PR
- ✅ Human approval gates - Two-stage approval before deployment
- ✅ Secret management - Zero secrets in code, everything in Key Vault
- ✅ Multi-environment setup - Dev/Test/Prod with separate state files
- ✅ Production hardening - Drift detection, deployment windows, rollback strategies
- ✅ Troubleshooting - Real production issues and proven solutions

The Complete Series

Part 1: Prerequisites & Architecture

Read Part 1

What you'll build:
- Azure Storage Account for remote state backend
- Azure Key Vault for secret management
- Service Principal for Azure DevOps authentication
- Service connections and variable groups
- Terraform configuration files

Key takeaway: One-time manual setup that enables secure, auditable deployments.

Part 2: Build Pipelines (Status Check & Plan)

Read Part 2

What you'll build:
- Terraform Status Check pipeline (runs on PRs)
- Terraform Plan pipeline (creates deployment artifacts)
- GUI-based classic pipelines (no YAML required)
- Automated validation workflow

Key takeaway: Two pipelines serve different purposes - status checks provide fast feedback, plan pipelines create deployment artifacts.

Part 3: Release Pipeline & Approval Gates

Read Part 3

What you'll build:
- Release pipeline with pre-deployment approval gates
- Terraform apply using approved plan artifacts
- Email notifications for approvals and deployments
- Complete audit trail

Key takeaway: Approval gates prevent unauthorized infrastructure changes. What gets approved is exactly what deploys.

Part 4: Branch Policies & Pull Request Automation

Read Part 4

What you'll build:
- Branch policies that enforce pull request workflow
- Required reviewers and status checks
- Automated validation on every PR
- Clean Git history with squash merges

Key takeaway: Branch policies make it impossible to bypass reviews and automated validation.

Part 5: Production Best Practices & Multi-Environment Setup

Read Part 5

What you'll build:
- Dev/Test/Prod environment separation
- State file isolation (one per environment)
- Environment-specific approval workflows
- Drift detection and deployment windows
- Rollback strategies

Key takeaway: Scale from single environment to enterprise-grade multi-environment infrastructure.

Part 6: Troubleshooting & Common Production Issues

Read Part 6

What you'll learn:
- Authentication and permission issues
- State file corruption and locking
- Resource drift detection and remediation
- Pipeline performance optimization
- Real production war stories

Key takeaway: These are real issues I've encountered in production with proven solutions.

The Complete Workflow

Here's what happens when you deploy infrastructure with this setup:

1. Developer creates feature branch
   ↓
2. Write Terraform code in VSCode
   ↓
3. Commit → Push to Azure DevOps
   ↓
4. Create Pull Request
   ↓
5. [AUTOMATED] Status check pipeline runs
   ├─ terraform init
   ├─ terraform validate
   └─ terraform plan (display output)
   ↓
6. [HUMAN] Code review & approval
   ↓
7. Merge to main (squash commit)
   ↓
8. [AUTOMATED] Build pipeline runs
   ├─ terraform plan
   └─ Create artifact (.tfplan file)
   ↓
9. [HUMAN] Release approval gate
   ├─ Review plan artifact
   └─ Approve deployment
   ↓
10. [AUTOMATED] Release pipeline runs
    └─ terraform apply (using artifact)
    ↓
11. Azure Resources Deployed ✓

Key principle: Every infrastructure change goes through:
- Automated validation (syntax, plan output)
- Human review (code review)
- Human approval (deployment approval)

Why GUI Pipelines (Not YAML)?

I use classic GUI pipelines instead of YAML pipelines for these reasons:

1. Visual debugging - See task order and configuration at a glance
2. Easier troubleshooting - Click into tasks, see logs, no YAML syntax errors
3. Better for teams - Non-developers can understand and modify pipelines
4. Full feature parity - GUI pipelines support all the same features
5. Approval gates - Native support in release pipelines

YAML is great for:
- Version controlling pipeline definitions
- GitOps for pipeline-as-code
- Complex matrix builds

GUI is great for:
- Deployment pipelines with approval workflows
- Visual pipeline management
- Teams that prefer clicking over YAML

Your choice. Both work. I prefer GUI for deployment workflows.

What Makes This Production-Ready?

Security:
- ✅ No secrets in Git (all in Key Vault)
- ✅ Service principal with least-privilege permissions
- ✅ Approval gates prevent unauthorized deployments
- ✅ Complete audit trail (who, what, when, why)

Reliability:
- ✅ State file backups with versioning
- ✅ State locking prevents concurrent modifications
- ✅ Plan artifacts ensure consistency (what's approved is what deploys)
- ✅ Drift detection catches manual portal changes

Scalability:
- ✅ Separate state files per environment
- ✅ Modular Terraform code
- ✅ Performance optimized for large deployments
- ✅ Multi-environment promotion workflow (dev → test → prod)

Compliance:
- ✅ SOC 2 compliant audit trails
- ✅ Change advisory board (CAB) approval workflows
- ✅ Deployment windows restrict when changes can occur
- ✅ Rollback procedures documented and tested

Code Repository

All Terraform code, pipeline configurations, and scripts from this series are available in my GitHub repository:

Repository: github.com/dswann101164

What's included:
- Complete Terraform configuration files
- PowerShell setup scripts
- Pipeline task configurations (exported)
- Branch policy templates
- Troubleshooting runbooks

My Background

I'm an Azure Architect who builds and maintains production infrastructure for a financial services company. I can't use LinkedIn because of my employer (Synovus), so this blog is my portfolio.

Why trust this guide?
- This is my actual production setup (not theoretical)
- I've run this workflow for 2+ years
- I've troubleshot every issue in Part 6 (multiple times)
- I've trained teams on this exact process

What I don't cover:
- Terraform basics (plenty of good tutorials exist)
- Azure fundamentals (assume you know Azure)
- YAML pipelines (I use GUI pipelines by choice)

Who This Series Is For

You should read this if:
- ✅ You're managing Azure infrastructure
- ✅ You want to move from manual deployments to automation
- ✅ You need audit trails for compliance
- ✅ You're building an enterprise deployment pipeline
- ✅ You want approval workflows and governance

This might not be for you if:
- ❌ You're just learning Terraform (start with official tutorials)
- ❌ You're deploying to AWS or GCP (this is Azure-specific)
- ❌ You prefer YAML pipelines (this uses GUI pipelines)
- ❌ You're running hobby projects (this is enterprise-focused)

Time Commitment

Full implementation: 2-3 weeks

Breakdown:
- Part 1 (Prerequisites): 4-6 hours
- Part 2 (Build pipelines): 3-4 hours
- Part 3 (Release pipeline): 2-3 hours
- Part 4 (Branch policies): 1-2 hours
- Part 5 (Multi-environment): 5-7 hours
- Part 6 (Troubleshooting): Ongoing

Maintenance: ~2 hours/month
- Secret rotation
- Provider upgrades (test in dev first)
- Pipeline optimization
- Drift investigation

Common Questions

Q: Why not use GitHub Actions instead of Azure DevOps?

A: GitHub Actions is great. I use Azure DevOps because:
- Native integration with Azure (service connections)
- Better approval workflows (release pipelines)
- Artifact management built-in
- My company already uses Azure DevOps

Q: Can I use YAML pipelines instead of GUI pipelines?

A: Yes. The concepts are identical, just different configuration format. YAML gives you version control for pipeline definitions. GUI gives you visual debugging. Pick what works for your team.

Q: How much does this cost to run?

A: Almost free.
- Azure DevOps: Free for up to 5 users
- Azure Pipelines: Free tier includes 1,800 minutes/month
- Storage account: <$5/month
- Key Vault: <$1/month
- Service principal: Free

Total: ~$6/month for the infrastructure. Your Azure resources cost extra.

Q: What if I'm the only person on my team?

A: This workflow still works. You'll approve your own PRs and releases. The value is in the audit trail, automated validation, and preventing accidental terraform destroy commands.

Q: Can I use this with Terraform Cloud instead of remote state in Azure?

A: Yes, but you'll need to adjust the backend configuration. The rest of the workflow remains the same.

Q: How do I migrate existing infrastructure to this workflow?

A: See Part 6 (Troubleshooting) for the import process. High-level:
1. Import existing resources into state
2. Write Terraform code to match current state
3. Test in dev environment first
4. Gradually migrate resources

What's Next?

After completing this series, consider:

Advanced topics to explore:
- Terraform modules for reusable infrastructure
- Sentinel policies for governance
- Cost estimation in pipelines (using Infracost)
- Terraform testing (using Terratest)
- Multi-subscription deployments
- Hub-and-spoke networking with Terraform

Recommended tools:
- Checkov (static code analysis for Terraform)
- tfsec (security scanner)
- Terraform-docs (auto-generate documentation)
- Pre-commit hooks (validate before commit)

Support This Content

I can't use LinkedIn because of my employer, so:
- ⭐ Star my GitHub repos if you find them useful
- 📝 Link to these posts from your own blog
- 🐛 Report issues or suggest improvements via GitHub
- 🤝 Share this series with your team

Quick Links

• Part 1: Prerequisites & Architecture
• Part 2: Build Pipelines
• Part 3: Release Pipeline & Approval Gates
• Part 4: Branch Policies
• Part 5: Production Best Practices
• Part 6: Troubleshooting

View code on GitHub

This series represents 2+ years of production experience running Terraform in Azure DevOps. All issues documented are real problems I've encountered and solved.