Automating the rotation of Azure DevOps service connection passwords with Key Vault, Event Grid, Function App and Managed Identity
To deploy on Azure, the common practice is to create an Azure Active Directory (AAD) application, assign it the role of “Contributor” on the resource group, for example, and then configure a service connection with the AAD application on the Azure DevOps account to enable deployment on Azure.
However, this approach has a major drawback: when the Azure AD application password expires, developers must manually generate a new one and update the corresponding service connection in Azure DevOps. This manual process can be tedious and increase the risks associated with password security.
In this article, we will explore a solution to automate the rotation of Azure DevOps service connection passwords using Azure Key Vault, Event Grid, Storage Account, Azure Function, and Managed Identity. This approach will not only save time but also improve the overall security of the deployment process.
It is important to note that this solution is particularly suitable for deployments using Microsoft Hosted Agents. However, if you are using Self Hosted Agents for your deployments, a different approach is recommended. In this case, it is preferable to use a Managed Identity directly on the agent. This will free you from managing and renewing passwords, greatly simplifying the process and enhancing the security of your environment.
Proposed solution for secret rotation automation
To solve the problems related to the manual management of Azure DevOps service connection passwords, we propose an automation solution for secret rotation. Here are the detailed steps to implement this solution:
- Detection of secret expiration: The Key Vault constantly monitors the status of its stored secrets. When a secret is about to expire (SecretNearExpiry event) or has expired (SecretExpired event), the Key Vault triggers the corresponding event.
- Sending events to the Azure storage queue: The events triggered by the Key Vault are sent to the Event Grid topic associated with the Key Vault. This topic is configured to send events to a specific queue in an Azure storage account. The topic uses a System Managed Identity to send the message to the queue. Thus, the traffic stays within the Azure backbone and is secured thanks to the System Managed Identity with the necessary role (least privilege): Storage Queue Data Message Sender (Reference).
- Processing of queue events by the Function App: The Function App constantly monitors the Azure storage queue. When it detects a new message in the queue, it triggers a function to process this event. This function is responsible for rotating the secrets. The function connects to the Storage Account using a User Assigned Managed Identity (UAMI). To do this, the latter has the following roles on the Storage Account: Storage Account Contributor, Storage Blob Data Owner, Storage Queue Data Contributor (Reference).
- Rotation of Azure AD application secrets: If the processed event is a SecretNearExpiry or SecretExpired event, the Function App uses the User Assigned Managed Identity to update the secret of the corresponding Azure AD application. To do this, the User Assign Managed Identity is Owner of the Azure AD application and it has the Graph API Application.ReadWrite.OwnedBy permission; which gives it the necessary rights to update the secret. These two actions cannot be performed via the portal. It is mandatory to go through a script (see below) or API.
- Updating the secret in the Key Vault: The secret in the Key Vault is updated with the new secret generated with the new expiration date. The User Assigned Managed Identity thus has the role of Key Vault Secrets Officer on the Key Vault.
- Updating the password of the Azure DevOps service connection: After updating the secret of the Azure AD application, the Function App also uses the Managed Identity to update the password of the corresponding ARM Service Connection in Azure DevOps.
In summary, this workflow works as follows:
- When a secret in the Key Vault is nearing its expiration or has expired, an event is triggered.
- This event is captured by the Event Grid topic, which then sends it to the Storage Account queue through the created subscription.
- The Function App can then be triggered by these messages in the queue to perform the rotation of the Azure AD application secret, update the secret in the Key Vault, and update the service connection password in Azure DevOps.
Next, we will see all the necessary code to set up this infrastructure and the function code in C#.
Setting up the workflow
All the code is available on the following repo: https://github.com/Thialala/automatic-secrets-rotation.git
Deployment of resources and assignment of the User Assigned Managed Identity Owner of the AAD application
The deployment of the entire infrastructure presented below is done via Infra As Code with Bicep. We have created a Powershell script deploy.ps1 that deploys the main.bicep file and retrieves the principalId of the Managed Identity from the deployment output. This is then used to be added as Owner of the Azure AD application whose client id is provided as input.
param (
[Parameter(Mandatory = $true)]
[string]$ResourceGroupName,
[Parameter(Mandatory = $true)]
[string]$ApplicationClientId
)
# Get the principalId of the Managed Identity from Bicep deployment outputs
$deploymentOutputs = az deployment group create --resource-group $ResourceGroupName --template-file .\main.bicep --query properties.outputs --output json | ConvertFrom-Json
$managedIdentityPrincipalId = $deploymentOutputs.managedIdentityPrincipalId.value
# Add the Managed Identity as an owner of the Azure AD application
az ad app owner add --id $ApplicationClientId --owner-object-id $managedIdentityPrincipalIdThe main.bicep file creates and configures several Azure resources necessary for setting up the workflow. Here is a description of each deployed resource:
- User Assigned Managed Identity: creates the Managed Identity for the application that will allow secure access to various Azure resources.
- Storage Account and Queue: creates a storage account and the queue that will be used to trigger the Azure function. The Shared Access Keys are disabled for the Storage Account.
- Role Assignments: these resources assign the Managed Identity different roles on the storage account to allow appropriate access.
- Key Vault: creates an instance of Azure Key Vault to store secrets. It also assigns the Managed Identity the role of Key Vault Secrets Officer. The Access Policies are disabled and the Key Vault is only accessible via RBAC.
- Application Insights: creates an instance of Application Insights for tracking and monitoring the Function App.
- Azure Function Plan: creates a hosting plan for the Function App.
- Function App: creates the Function App that will host the function that will execute the secret rotation workflow.
- EventGrid Topic: creates an EventGrid topic that will be used to trigger the secret rotation events in Key Vault. The System Assigned Identity is enabled and this latter has the role Storage Queue Data Message Sender.
- EventGrid Subscription: creates an event subscription that publishes in the Storage Account queue which will thus trigger the function when the SecretNearExpiry and SecretExpired events occur in Azure Key Vault.
Assignment of Graph API permission
The managed identity requires the Application.ReadWrite.OwnedBy permission, which gives it the rights to manage the applications it owns. To assign this permission, we have a dedicated PowerShell script named assignGraphPermission.ps1. It is important to note that this action cannot be performed through the Azure portal.
# Input Parameters:
# $appRoleName: The name of the app role to be assigned (e.g., "Application.ReadWrite.OwnedBy")
# $spnObjectId: The object ID of the service principal to which the app role should be assigned
param (
[Parameter(Mandatory=$true)]
[string]$appRoleName,
[Parameter(Mandatory=$true)]
[string]$spnObjectId
)
# Define the Microsoft Graph Application ID
$graphAppId = "00000003-0000-0000-c000-000000000000"
# Retrieve the resource ID of the Microsoft Graph service principal using the Microsoft Graph App ID
$graphResourceId=$(az ad sp show --id $graphAppId --query 'id' --output tsv)
# Retrieve the app role ID for the given appRoleName from the Microsoft Graph service principal
$appRoleId=$(az ad sp show --id $graphAppId --query "appRoles[?value=='$appRoleName' && contains(allowedMemberTypes, 'Application')].id" --output tsv)
# Define the URI for assigning the app role to the service principal
$uri="https://graph.microsoft.com/v1.0/servicePrincipals/$spnObjectId/appRoleAssignments"
Write-Output $uri
# Create the JSON request body containing the required information for the app role assignment
$body="{'principalId':'$spnObjectId','resourceId':'$graphResourceId','appRoleId':'$appRoleId'}"
# Send a POST request to the Microsoft Graph API to create the app role assignment
az rest --method post --uri $uri --body $body --headers "Content-Type=application/json"Even though we could have integrated this script into the previous one, it is generally preferable to separate these responsibilities. In practice, the management of Azure Active Directory is often handled by a specific team, which is separate from the application development teams.
Adding the Managed Identity in Azure DevOps
Recently, Azure DevOps introduced the ability to integrate a Service Principal or a Managed Identity into Azure DevOps, which allows us to avoid the use of a Personal Access Token (PAT). In the current context, we can add the Managed Identity by assigning it a Stakeholder license in Azure DevOps. Subsequently, this identity must be added to the Endpoint Administrators group of the project in question. This approach strengthens security and simplifies identity and access management in the Azure DevOps environment.
C# code of the function
The Azure Function is called KeyVaultSecretNearExpiry and it is triggered by any message in the kv-secrets-near-expiry queue.
The queue message is deserialized into EventGridData which contains information about the secret that is about to expire. The secret name (secretName) and the Key Vault name (keyVaultName) are extracted from the Event Grid data.
Next, the code retrieves the secret from the Key Vault. The retrieved secret contains metadata in the form of tags with the necessary information:
- azureADAppId
- azureADAppName
- azureDevOpsAccountUrl
- azureDevOpsProjectName
- azureDevOpsConnectionName
- SecretDurationInMonths
The corresponding Azure Active Directory application is retrieved using the application ID stored in the secret tags (azureADAppId). If the application is found, a new secret is generated for the application with the appropriate expiration date.
After this, the secret is updated in the Key Vault, and then in the corresponding service connection in Azure DevOps.
In this Azure function, ChainedTokenCredential is used as an authentication strategy to access multiple Azure services: Azure Resource Manager, Graph API, and Azure DevOps. The ChainedTokenCredential attempts to authenticate using a list of TokenCredential in the order they are passed in the constructor. If a non-fatal error (for example, the managed identity service is not available) is encountered during authentication with a TokenCredential, ChainedTokenCredential will move to the next TokenCredential in the list.
In the function code, ChainedTokenCredential is initialized with two types of TokenCredential: DefaultAzureCredential and ManagedIdentityCredential.
- DefaultAzureCredential is a type of TokenCredential that attempts to authenticate using several different methods. It attempts authentication using managed service credentials, credentials based on the local development context (such as Azure CLI, Visual Studio, etc.), end-user credentials, etc.
- ManagedIdentityCredential is a type of TokenCredential that attempts to authenticate using the User Assigned Managed Identity.
The use of ChainedTokenCredential provides flexibility by allowing your application to work both locally during development and in Azure without having to change the authentication code. In addition, it provides redundancy in case of failure of an authentication method.
In particular, in this scenario:
- For Azure Resource Manager, the SecretClient uses the ChainedTokenCredential to authenticate and retrieve secrets from the Key Vault.
- For the Graph API, the GraphServiceClient also uses the ChainedTokenCredential to authenticate and manage Azure AD applications, including the generation and rotation of secrets.
- For Azure DevOps, the ChainedTokenCredential is used to create a VSS (Visual Studio Services) connection. This connection is then used to retrieve and update the Azure DevOps service connection with the new secret.
Thus, the use of ChainedTokenCredential allows the Azure function to securely and flexibly authenticate across multiple Azure services, which is crucial for performing secret rotation.
Testing the workflow
To test the workflow, you first need to create a secret in the Key Vault with an expiration date set to tomorrow, for example. Then, make sure to fill in the six necessary tags: azureADAppId, azureADAppName, azureDevOpsAccountUrl, azureDevOpsProjectName, azureDevOpsConnectionName, and SecretDurationInMonths. Patience is then required — allow it about 10 minutes to trigger the event and execute the function.
I can assure you, without a shadow of a doubt, that this workflow has undergone rigorous testing 😅.
Conclusion
In summary, automating the rotation of Azure DevOps service connection passwords using Azure Functions, Azure Key Vault, and Managed Identity provides a robust and secure solution for managing Azure AD application secrets. By following the workflow described in this article, developers can now avoid the tedious tasks and risks associated with manual password management. This will make CISOs happy 😄.
This workflow is easily adaptable for other use cases such as automatic rotation of Shared Access Keys if we have no choice but to use them.
In the context of this article, it is important to note that the provided Bicep file does not quite meet the preparation criteria for a production environment, nor the optimal security standards. In a production configuration, it would be essential to prohibit public access to all PaaS services, while assigning them a private endpoint. For the Function App, the ideal approach would probably be to opt for the Premium plan (or ASP in an ASE), which offers a private access point and VNET integration, thus providing an additional layer of security.
