Authenticating Terraform from Azure DevOps via Service Connection Using Workload Identity Federation

☁️ Introduction

Before Terraform can deploy or manage Azure resources, it needs to authenticate — to prove its identity to Azure.
Azure then checks authorization to decide what Terraform is allowed to do.

When Terraform runs from Azure DevOps, one of the ways to authenticate is through a Service Connection (which internally uses a Service Principal).

In this post, we’ll:
✅ Authenticate Terraform using an Azure DevOps Service Connection
✅ Deploy a Resource Group, Virtual Network, and Subnet using Terraform

---

🧩 Prerequisites

✅ Azure Subscription
You’ll need an active subscription and permission to create and manage resources.

✅ Azure DevOps Project
Ensure you have an Azure DevOps Organization and project. For this, we can follow the post Getting Started with Azure DevOps Organization.

✅ Azure DevOps Repository
Terraform code must live in a Git repository.
Create a new repository in Azure Repos (for example, cocan-terraform-infra) to store your Terraform files.
We can follow the post Getting Started with Azure Repos to create and initialize our repo.

✅ Service Connection in Azure DevOps
We’ll use this connection to allow Terraform tasks in the pipeline to authenticate with Azure. For this, we can follow the post Getting Started with Azure DevOps Service Connections Using Workload Identity Federation.

✅ Terraform Installed (Agent)
Your build agent (Microsoft-hosted or self-hosted) must have Terraform installed.
If not, you can use the official Terraform Installer task in your pipeline.

---

🧩 Step 1: Prepare Your Terraform Configuration

Now that you have your Azure DevOps repository (cocan-infra-tf), you can organize your Terraform files inside it using a clear, scalable structure.
Azure DevOps pipelines will automatically pull this code from the repo when they run — there’s no need to execute Terraform locally.

💡 Recommended Structure (Best Practice)

cocan-infra-tf/
 └── environments/
      └── dev/
           └── main.tf

This folder layout allows you to manage multiple environments (such as dev, test, and prod) separately, each with its own configuration and state file.
It also simplifies CI/CD automation later — each environment can have its own pipeline or stage.

---

📄 Create the main.tf File

In your Azure DevOps repository cocan-infra-tf, create the following folder path:

/environments/dev/main.tf

Paste the following Terraform configuration:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
  backend "azurerm" {}
}

provider "azurerm" {
  features {}
}

# Create Resource Group
resource "azurerm_resource_group" "rg" {
  name     = "cocan-dev-rg"
  location = "eastus"
}

💡 Explanation:

• The terraform block defines the required Azure provider and backend (used later for storing state).
• The provider block enables Terraform to interact with Azure resources.
• The azurerm_resource_group resource creates a single Resource Group named nomotic-dev-rg in the East US region.

When your Azure DevOps pipeline runs, it will use your Service Connection (Cocan-infra-serv-connection) with Workload Identity Federation (WIF) to authenticate securely and create this Resource Group in Azure.

---

🧩 Step 2: Install the Terraform Extension in Azure DevOps

Before creating your pipeline YAML file, make sure your Azure DevOps organization has the Terraform extension installed.
This extension provides the Terraform tasks used in your pipeline, such as TerraformInstaller and TerraformTaskV4.

---

⚙️ Steps to Install

1️⃣ Go to the Azure DevOps Marketplace: Terraform Extension by Microsoft DevLabs
2️⃣ Click Get it free
3️⃣ Choose your Azure DevOps organization (for example, communitiescanada.visualstudio.com)
4️⃣ Click Install
5️⃣ Once installed, you’ll see new Terraform tasks available when editing pipelines:

• TerraformInstaller – installs a specific Terraform version on the agent
• TerraformTaskV4 – runs Terraform commands like init, plan, apply, and destroy

---

💡 Note:
If you skip this step, you’ll see an error like:

“A task is missing. The pipeline references a task called TerraformInstaller…”

Installing the extension once fixes it for all pipelines in your organization.

---

🧩 Step 3: Create the Terraform Backend

Terraform needs a backend to store its state file — the record of what resources it manages.
You can choose either a remote backend (recommended) or a local backend (simpler for testing).

---

✅ Option 1: Remote Backend (Recommended)

Store your Terraform state securely in an Azure Storage Account.
You can reuse your existing resource group cocan-infra-ado-rg created for your self-hosted agent.

Run the following commands in Azure Cloud Shell or Azure CLI:

# Create a storage account for Terraform state with a unique name instead of cocantfstatestrgacct
az storage account create \
  -n cocantfstatestrgacct\
  -g cocan-infra-ado-rg \
  -l canadacentral \
  --sku Standard_LRS

# Create a blob container to store the state file
az storage container create \
  --name tfstate \
  --account-name cocantfstatestrgacct

🧠 What this does

Resource	Name	Description
Resource Group	cocan-infra-ado-rg	Holds your backend and VM resources
Storage Account	cocantfstatestrgacct	Stores Terraform state securely
Container	tfstate	Holds your terraform.tfstate file

Once created, your backend values in the pipeline YAML should match these names:

backendAzureRmResourceGroupName: 'cocan-infra-ado-rg'
backendAzureRmStorageAccountName: 'cocantfstatestrgacct'
backendAzureRmContainerName: 'tfstate'
backendAzureRmKey: 'terraform.tfstate'

---

⚙️ Option 2: Local Backend (Simpler for First Test)

If you only want to verify your self-hosted agent and pipeline before setting up Azure storage,
you can use a local backend instead.

1️⃣ Update your main.tf to use a local backend:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }

  backend "local" {
    path = "terraform.tfstate"
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "rg" {
  name     = "cocan-dev-rg"
  location = "canadacentral"
}

2️⃣ Update your YAML pipeline by removing the backend inputs:

# Initialize Terraform (local backend)
- task: TerraformTaskV4@4
  displayName: 'Terraform Init'
  inputs:
    provider: 'azurerm'
    command: 'init'
    workingDirectory: '$(System.DefaultWorkingDirectory)/environments/dev'
    environmentServiceNameAzureRM: 'Cocan-infra-serv-connection'

💡 Terraform will now save the state file (terraform.tfstate) locally inside your agent’s working directory.
This is fine for demos or single-user setups, but not ideal for collaboration.

---

✅ Summary

Backend Type	Use When	Stores State In	Recommended For
Remote (azurerm)	Production or shared projects	Azure Blob Storage	Secure, persistent state
Local	Testing or learning	Local agent disk	Quick, easy setup

---

Next, proceed to Step 3: Create the Pipeline YAML File, where you’ll define the actual pipeline steps using these Terraform tasks.

---

🧩 Step 4: Create the Pipeline YAML File

Now that your Terraform configuration file (main.tf) is ready inside the repository, the next step is to define the pipeline that will run Terraform automatically from Azure DevOps.

💡 Why this step is important:
This YAML file defines the steps Azure DevOps should perform — installing Terraform, initializing the provider, planning changes, and applying them securely through the Service Connection.

---

📁 Folder and File Setup

In your repository cocan-infra-tf, create the following folder and file structure:

cocan-infra-tf/
 ├── .azure-pipelines/
 │    └── pipeline-dev.yml
 └── environments/
      └── dev/
           └── main.tf

This keeps your pipeline files separate and organized while allowing multiple pipelines later (e.g., pipeline-prod.yml, pipeline-test.yml).

---

🧾 Create the File .azure-pipelines/pipeline-dev.yml

Paste the following YAML code:

trigger:
- main

pool:
  vmImage: 'ubuntu-latest'

steps:
# Install Terraform
- task: TerraformInstaller@1
  displayName: 'Install Terraform'
  inputs:
    terraformVersion: 'latest'

# Initialize Terraform
- task: TerraformTaskV4@4
  displayName: 'Terraform Init'
  inputs:
    provider: 'azurerm'
    command: 'init'
    workingDirectory: '$(System.DefaultWorkingDirectory)/environments/dev'
    backendServiceArm: 'Cocan-infra-serv-connection'
    backendAzureRmResourceGroupName: 'cocan-infra-ado-rg'
    backendAzureRmStorageAccountName: 'cocantfstatestorageacct'
    backendAzureRmContainerName: 'tfstate'
    backendAzureRmKey: 'terraform.tfstate'

# Plan Terraform changes
- task: TerraformTaskV4@4
  displayName: 'Terraform Plan'
  inputs:
    provider: 'azurerm'
    command: 'plan'
    workingDirectory: '$(System.DefaultWorkingDirectory)/environments/dev'
    environmentServiceNameAzureRM: 'Cocan-infra-serv-connection'

# Apply Terraform configuration
- task: TerraformTaskV4@4
  displayName: 'Terraform Apply'
  inputs:
    provider: 'azurerm'
    command: 'apply'
    workingDirectory: '$(System.DefaultWorkingDirectory)/environments/dev'
    environmentServiceNameAzureRM: 'Cocan-infra-serv-connection'
    args: '-auto-approve'

---

💡 Explanation:

• The pipeline runs every time code is pushed to the main branch.
• It installs Terraform, initializes the Azure provider, plans, and applies changes.
• The workingDirectory points to your environment folder containing main.tf.
• The pipeline uses your Service Connection (Cocan-infra-serv-connection) to authenticate securely using Workload Identity Federation (WIF) — no credentials or secrets required.

💡 If you encounter a pipeline error mentioning environmentServiceNameAzureRM, try renaming it to environmentServiceName. Azure DevOps occasionally updates this input name.

---

🧩 Step 5: Create and Run the Pipeline in Azure DevOps

Now that the YAML file is saved in your repository, you can create a pipeline in Azure DevOps to use it.

---

🧭 Create the Pipeline

1️⃣ In Azure DevOps, go to Pipelines → New Pipeline (or Create Pipeline)
2️⃣ Choose Azure Repos Git
3️⃣ Select your repository cocan-infra-tf
4️⃣ Choose Existing Azure Pipelines YAML file
5️⃣ Browse to .azure-pipelines/pipeline-dev.yml
6️⃣ Click Continue
7️⃣ Click Run

💡 Note 1:
When you run the pipeline for the first time, you may see a message saying: “This pipeline needs permission to access a resource before this run can continue.”
Simply Click View → Permit to authorize your Service Connection (for example, Cocan-infra-serv-connection).
This one-time approval allows the pipeline to access Azure resources securely in future runs.

💡 Note 2:
If you see an error saying: “No hosted parallelism has been purchased or granted…”
It means your organization doesn’t yet have a free build agent.
Go to https://aka.ms/azpipelines-parallelism-request, sign in, and request a free parallelism grant for your Azure DevOps organization.
Approval usually takes 1–2 business days, after which your pipeline will run normally.

💡 Note 3:
If you don’t want to wait for the free parallelism approval, you can set up a self-hosted agent on your own machine or an Azure VM.
Once registered, update your YAML to use:

pool:
  name: Default

This lets your pipeline run immediately without waiting for Microsoft’s hosted agent approval.

---

⚙️ What Happens When You Run It

Azure DevOps will now:
✅ Clone your repo and detect the YAML pipeline file
✅ Install Terraform on the build agent
✅ Authenticate to Azure using your Service Connection (via WIF)
✅ Initialize Terraform (download providers and configure backend)
✅ Plan the changes (preview resource creation)
✅ Apply the configuration to create the Resource Group

When completed successfully, the log will show:

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

---

🧩 Step 6: Verify in Azure Portal

Go to the Azure Portal → Resource Groups, and look for:

Resource Group Name: cocan-dev-rg
Region: East US

If you see it listed, your pipeline worked perfectly 🎉

---

🧹 Step 7: Clean Up

To avoid charges, destroy the resources when done:

terraform destroy -auto-approve

Or, add a Terraform Destroy task in your pipeline if required for cleanup automation.

---

🔍 Bonus: Why Use a Service Connection?

✅ Centralized and secure authentication management
✅ No secrets or credentials in code
✅ Works seamlessly across multiple pipelines and environments
✅ Easy to rotate credentials via Azure DevOps

---

✅ Summary

✔️ Use Service Connection when running Terraform from Azure DevOps
✔️ No need to manage credentials manually
✔️ Store state in Azure Storage for collaboration
✔️ Keep Terraform runs automated, secure, and repeatable