GitOps-managed Azure infrastructure using Flux and Crossplane on Minikube.

• Flux watches this Git repo and syncs Kubernetes manifests to the cluster
• Crossplane extends Kubernetes with CRDs for Azure resources
• Together: you push YAML to Git → Azure resources get created/updated/deleted

┌─────────────┐      ┌─────────────┐      ┌─────────────┐      ┌─────────┐
│   Git Repo  │ ───► │    Flux     │ ───► │  Crossplane │ ───► │  Azure  │
│  (GitHub)   │      │ (K8s sync)  │      │ (providers) │      │   API   │
└─────────────┘      └─────────────┘      └─────────────┘      └─────────┘

• Minikube with Docker driver
• kubectl, flux CLI
• Azure CLI (az) logged in
• Docker permissions for your user (sudo usermod -aG docker $USER)

clusters/minikube/
├── flux-system/                    # Flux GitOps toolkit
│   ├── gotk-components.yaml        # Flux controllers
│   ├── gotk-sync.yaml              # Git source + main kustomization
│   ├── crossplane-kustomization.yaml
│   ├── crossplane-config-kustomization.yaml
│   ├── crossplane-provider-config-kustomization.yaml
│   └── azure-resources-kustomization.yaml  # ConfigMap + Kustomization with vars
├── crossplane/                     # Crossplane Helm installation
├── crossplane-config/              # Azure providers
├── crossplane-provider-config/     # Provider authentication
└── azure-resources/                # Actual Azure resources
    ├── base/                       # Templates with ${VAR} placeholders
    │   ├── resource-group.yaml
    │   ├── vnet.yaml
    │   ├── subnets.yaml
    │   ├── storage-account.yaml
    │   └── keyvault.yaml
    └── environments/
        ├── dev/                    # Dev environment
        │   └── kustomization.yaml
        └── prod/                   # Prod environment (ready to deploy)
            └── kustomization.yaml

minikube start

flux bootstrap github \
  --owner=<your-github-username> \
  --repository=flux-demo \
  --path=clusters/minikube \
  --personal

az login
az ad sp create-for-rbac --sdk-auth --role Contributor \
  --scopes /subscriptions/<SUBSCRIPTION_ID> \
  -n "crossplane-sp" > /tmp/azure-credentials.json

kubectl create secret generic azure-credentials \
  -n crossplane-system \
  --from-file=credentials=/tmp/azure-credentials.json

git add . && git commit -m "Add resources" && git push
flux reconcile source git flux-system

Flux applies resources in order via dependsOn:

flux-system
    └── crossplane (Helm install)
            └── crossplane-config (Azure providers)
                    └── crossplane-provider-config (ProviderConfig with credentials)
                            └── azure-resources (actual Azure resources)

flux get kustomizations
flux get sources git

kubectl get providers.pkg.crossplane.io
kubectl get pods -n crossplane-system

kubectl get resourcegroup
kubectl get account.storage.azure.upbound.io
kubectl get virtualnetwork.network.azure.upbound.io

kubectl logs -n flux-system deploy/kustomize-controller --tail=50
kubectl logs -n flux-system deploy/kustomize-controller -f  # follow

flux reconcile source git flux-system
flux reconcile kustomization <name>

az group list -o table
az storage account list -g rg-crossplane-demo -o table
az network vnet list -g rg-crossplane-demo -o table

⚠️ Reality Check: Getting this stack working was frustrating. Crossplane providers are fragile, error messages are cryptic, and debugging requires deep Kubernetes knowledge. This is not a "it just works" solution. Expect to spend significant time troubleshooting.

The provider-family-azure is just the base package. You need separate providers for each resource type:

• provider-azure-storage - for Storage Accounts
• provider-azure-network - for VNets, Subnets, etc.
• provider-azure-compute - for VMs (not installed)

Without the specific provider, you get: no matches for kind "Account".

When a provider is deleted and recreated, CRDs can retain ownerReferences pointing to the OLD ProviderRevision UID. The new provider then fails with:

cannot establish control of object: accounts.storage.azure.upbound.io 
is already controlled by ProviderRevision provider-azure-storage-xxx (UID old-uid-here)

The provider will stay HEALTHY: False forever until you fix this.

Fix: Remove the stale ownerReferences from the CRD:

kubectl patch crd accounts.storage.azure.upbound.io \
  --type=json -p='[{"op": "remove", "path": "/metadata/ownerReferences"}]'

This allows the new ProviderRevision to take ownership.

Even after fixing ownerReferences, providers may need a kick:

kubectl delete provider.pkg.crossplane.io provider-azure-storage
kubectl delete providerrevision --all
# Then let Flux recreate, or apply manually

When a provider pod isn't running but its CRDs still exist, you'll see:

conversion webhook...connection refused

This blocks ALL resources in the same Flux kustomization. One broken provider = everything stuck.

Default interval is 10m. When something breaks, you wait a long time. Consider:

• Shorter intervals for dev: interval: 2m
• Or manual: flux reconcile kustomization <name>

Warning: flux reconcile often hangs/times out when there are errors. Use kubectl directly:

flux get kustomizations  # quick status
kubectl get providers.pkg.crossplane.io  # provider health

ProviderConfig CRD only exists after the Azure provider is installed and healthy. If you try to apply it too early:

no matches for kind "ProviderConfig" in version "azure.upbound.io/v1beta1"

Solution: Separate ProviderConfig into its own kustomization with proper dependsOn.

• Private repo needs SSH key or token in a secret
• Public repo: remove secretRef from gotk-sync.yaml and use HTTPS URL

Azure storage account names are global. If stcrossplanedemo exists anywhere, it fails. Use a unique suffix.

Flux marks a kustomization as "Applied" based on successful API calls, not actual resource existence. If CRDs were deleted, the resources disappear but Flux doesn't know.

Fix: Make a trivial change (add a label) to force reapply:

# Edit a file, commit, push, then:
flux reconcile source git flux-system

1. Providers install CRDs (Custom Resource Definitions) for Azure resource types
2. One provider pod handles ALL resources of its type (not a container per resource)
3. Provider pod watches CRs → calls Azure API → updates status
4. SYNCED=True means Crossplane successfully communicated with Azure
5. READY=True means the resource is provisioned and ready

✅ Do: Make all changes via Git commits
✅ Do: Let Flux handle reconciliation
✅ Do: Use dependsOn for ordering
❌ Don't: kubectl apply directly (breaks GitOps)
❌ Don't: Manually edit resources in cluster

If you must intervene (debugging), understand Flux will revert your changes on next reconcile.

Remove files from azure-resources/, commit, push. Crossplane will delete from Azure.

minikube delete

Or to remove just Crossplane resources:

kubectl delete resourcegroup --all
kubectl delete account.storage.azure.upbound.io --all

• Add health checks to kustomizations
• Add alerting/notifications for failures
• Consider using Crossplane Compositions for reusable patterns
• Add network policies
• Set up proper secret management (External Secrets, Sealed Secrets)
• Deploy prod environment

This repo uses Flux variable substitution for multi-environment deployments.

1. Base templates in azure-resources/base/ use ${VAR} placeholders:

   metadata:
     name: rg-${ENV}-crossplane
   spec:
     forProvider:
       location: ${LOCATION}

2. ConfigMap in flux-system/azure-resources-kustomization.yaml defines variables:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: azure-dev-vars
   data:
     ENV: dev
     LOCATION: westeurope
     VNET_CIDR: "10.10.0.0/16"

3. Flux Kustomization substitutes variables via postBuild:

   spec:
     postBuild:
       substituteFrom:
         - kind: ConfigMap
           name: azure-dev-vars

1. Copy azure-resources-kustomization.yaml
2. Change ConfigMap name and values
3. Update Kustomization path to ./clusters/minikube/azure-resources/environments/prod
4. Add to flux-system/kustomization.yaml

Currently installed providers and their resource counts:

Total: ~147 resource types

Add to crossplane-config/:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-azure-compute
spec:
  package: xpkg.upbound.io/upbound/provider-azure-compute:v1.3.0

Note: Each provider = 1 additional pod (~100-200MB memory).

We chose single repo because:

Multi-repo makes sense when: Different teams deploy their own apps (containers), strict org boundaries, or compliance requirements.

Crossplane automatically detects and corrects drift:

1. Poll interval: Crossplane checks Azure state every ~10 minutes
2. Drift detected: If someone deletes/modifies resources in Azure portal
3. Auto-remediation: Crossplane recreates/updates to match desired state

Test it:

# Delete a subnet in Azure
az network vnet subnet delete -g rg-dev-crossplane --vnet-name vnet-dev-crossplane -n snet-dev-frontend
# Wait ~10 minutes, Crossplane will recreate it
kubectl get subnets -w

GitOps way (recommended):

1. Remove YAML file from repo
2. Commit and push
3. Flux reconciles, removes from K8s
4. Crossplane deletes from Azure

Manual (breaks GitOps, use only for debugging):

kubectl delete resourcegroup rg-dev-crossplane

The Good:

• True GitOps for infrastructure - push YAML, get Azure resources
• Declarative, version-controlled infrastructure
• Automatic drift detection and correction

The Bad:

• Crossplane providers are fragile and error-prone
• Debugging requires deep K8s knowledge (CRDs, ownerReferences, webhooks)
• Error messages are often cryptic
• Provider startup/health issues are common
• Flux + Crossplane adds significant operational complexity

The Ugly:

• Simple tasks (create a storage account) required hours of debugging
• The "stale ownerReferences" bug cost us significant time
• No clear documentation for common failure scenarios
• You need to understand internals to fix basic issues

Would we recommend this for production? Only if you have dedicated platform engineers who understand Kubernetes internals. For simpler use cases, Terraform or Bicep with Azure DevOps/GitHub Actions may be more pragmatic.