Published on

Securely Accessing Azure Key Vault Secrets in AKS with Service Connector and CSI Driver

Authors
  • avatar
    Name
    Jac Timms
    Twitter

Introduction

When working with Azure Kubernetes Service (AKS) and Azure Key Vault, getting the secret access configuration right wasn't immediately obvious from the main MS learn docs. This guide is based on practical experience getting the Service Connector e and CSI Secret Store Driver working together. I'll share the exact steps that worked, the errors we encountered, and how we resolved them.

Note: Service Connector is currently in preview. While this guide reflects what works today, specific details might change as the feature evolves.

Prerequisites

  • An AKS cluster
  • An Azure Key Vault
  • Azure CLI installed
  • kubectl configured to access your cluster

Understanding the Components

1. Service Connector

Service Connector is designed to simplify connecting AKS workloads to Azure services. According to the Apps On Azure Blog, it automatically handles several steps that would otherwise be manual:

  1. Retrieve the OIDC issuer URL
  2. Create Kubernetes service account
  3. Establish federated identity credential trust
  4. Grant permissions to access Azure Services

This automation significantly simplifies the setup process, handling much of the identity and permissions configuration behind the scenes.

2. CSI Secret Store Driver

The Container Storage Interface (CSI) driver enables your pods to access Key Vault secrets by:

  • Mounting secrets as volumes in pods
  • Handling authentication
  • Optionally syncing secrets to Kubernetes secrets

Getting the Required Identity ID

The crucial step we discovered is getting the correct CSI driver's managed identity client ID:

# Get CSI Driver's Managed Identity Client ID
az aks show -g <resource-group> -n <cluster-name> \
  --query 'addonProfiles.azureKeyvaultSecretsProvider.identity.clientId' -o tsv

This command returns the managed identity client ID used by the CSI driver. You'll need this ID for the SecretProviderClass configuration.

Implementation Steps

1. Enable the Key Vault Provider on AKS

az aks enable-addons --addons azure-keyvault-secrets-provider \
    --name myAKSCluster \
    --resource-group myResourceGroup

2. Create Service Connector

Using Terraform:

resource "azapi_resource" "service_connector" {
  type      = "Microsoft.ServiceLinker/linkers@2022-11-01-preview"
  name      = "devsecops_platformt_dev_keyvault_connector"
  parent_id = var.aks_cluster_id
  response_export_values = ["*"]

  body = jsonencode({
    properties = {
      clientType = "none"
      targetService = {
        type = "AzureResource"
        id = var.key_vault_id
        resourceProperties = {
          type = "KeyVault"
          connectAsKubernetesCsiDriver = true
        }
      }
      authInfo = {
        authType = "userAssignedIdentity"
      }
      scope = "default"
    }
  })
}

After creating the Service Connector, it's recommended to validate it in the Azure Portal under the AKS cluster's Service Connector (Preview) section.

3. Create SecretProviderClass

Here's the working configuration with all required elements:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure
  parameters:
    keyvaultName: "your-keyvault-name"
    objects: |
      array:
        - |
          objectName: secret1
          objectType: secret
          objectVersion: ""
    tenantID: "<your-tenant-id>"
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "<csi-driver-managed-identity-client-id>"

The critical settings that made this work:

  • usePodIdentity: "false" - We're not using pod identity
  • useVMManagedIdentity: "true" - Using managed identity authentication
  • userAssignedIdentityID - The CSI driver's managed identity client ID from the earlier command

4. Deploy a Pod Using the Secrets

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keyvault-test-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keyvault-test
  template:
    metadata:
      labels:
        app: keyvault-test
    spec:
      containers:
      - name: test-container
        image: myapp:latest
        volumeMounts:
        - name: secrets-store
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
      - name: secrets-store
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "azure-kvname"

Errors We Encountered and Solutions

1. RBAC Permission Error

RESPONSE 403: 403 Forbidden
ERROR CODE: Forbidden
Caller: appid=<managed-identity-client-id>;
Action: 'Microsoft.KeyVault/vaults/secrets/getSecret/action'

This error indicated we needed to add the correct managed identity ID to the SecretProviderClass configuration.

2. Authentication Configuration Error

failed to mount secrets store objects for pod, err: failed to create auth config, 
error: failed to get credentials, nodePublishSecretRef secret is not set

This was resolved by properly configuring the managed identity settings in the SecretProviderClass.

3. Multiple Identities Error

Multiple user assigned identities exist, please specify the clientId / resourceId 
of the identity in the token request

This error was resolved by explicitly specifying the CSI driver's managed identity client ID in the SecretProviderClass.

Verifying It Works

These commands helped us verify the setup:

# Check pod status and events
kubectl describe pod -n <namespace> <pod-name>

# Verify the secrets are mounted
kubectl exec -n <namespace> <pod-name> -- ls /mnt/secrets-store
kubectl exec -n <namespace> <pod-name> -- cat /mnt/secrets-store/secret1

# Check CSI driver logs if needed
kubectl logs -n kube-system -l app=secrets-store-csi-driver

Why This Works

This configuration works because:

  1. The CSI driver uses its dedicated managed identity to authenticate with Key Vault
  2. The SecretProviderClass correctly specifies which identity to use
  3. The pod mounts the secrets through the CSI driver

The key to success was identifying and using the correct managed identity client ID from the AKS cluster's KeyVault add-on configuration.

Conclusion

The essential steps for success are:

  1. Get the correct CSI driver managed identity client ID
  2. Configure the SecretProviderClass with this ID
  3. Ensure your pod deployment references the SecretProviderClass

This approach provides a reliable way to access Key Vault secrets in AKS using the CSI driver with the correct identity configuration.