GitOps for ML Infrastructure: ArgoCD and Flux Patterns

What's the Problem We're Solving?

Traditional infrastructure management is imperative: you SSH into servers, run commands, and hope things stick. Or you use dashboards to click buttons and pray nobody remembers what they did last week. For ML workloads - where models, data pipelines-real-time-ml-features)-apache-spark))-training-smaller-models), and serving infrastructure all evolve - this becomes a nightmare.

The risks?

• Configuration drift: your cluster state doesn't match your intentions
• No audit trail: who changed what, and when?
• Rollback hell: good luck reverting a manual change at 3 AM
• Inconsistent environments: dev works fine, but prod mysteriously breaks

GitOps flips the script. Instead of "change systems manually," you declare "here's what the system should look like" in Git. A controller watches Git and keeps reality in sync.

The Four Tenets of GitOps

Before we dive into tools, let's ground ourselves in what GitOps actually means:

1. Declarative Infrastructure

You describe the desired state of your infrastructure in code - typically YAML manifests. No imperative commands like "kubectl apply -f" run randomly. Everything flows through Git.

Example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-inference
spec:
  replicas: 3
  selector:
    matchLabels:
      app: model-inference
  template:
    metadata:
      labels:
        app: model-inference
    spec:
      containers:
        - name: inference-server
          image: myregistry.azurecr.io/model-inference:v2.1.0
          resources:
            requests:
              nvidia.com/gpu: "1"
            limits:
              nvidia.com/gpu: "1"

You're not saying "run this deployment now." You're saying "this is the state we want." The GitOps controller makes it happen.

2. Versioned and Immutable

Every change lives in Git. Every commit is a snapshot. Need to rollback? git revert. Need to audit? Check the commit history. Nobody can claim ignorance - it's all there.

3. Automatically Reconciled

A controller (like ArgoCD or Flux) watches Git constantly. When a commit lands, it compares the declared state to actual cluster state. If they differ, the controller syncs automatically.

4. Continuously Monitored

Your controller doesn't just set-and-forget. It continuously checks: "Is the cluster still in the state Git says it should be?" If something drifted (a bad manual change, a pod crash), it re-syncs.

GitOps Repository Structure for ML

Let's design a repo structure that works for ML teams. You'll want clear separation between environments, applications, and Kustomize layers.

gitops-ml-infra/
├── README.md
├── environments/
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   ├── values/
│   │   │   ├── inference.yaml
│   │   │   ├── training.yaml
│   │   │   └── monitoring.yaml
│   │   └── patches/
│   │       ├── replicas.yaml
│   │       └── resources.yaml
│   ├── staging/
│   │   └── ... (similar structure)
│   └── prod/
│       ├── kustomization.yaml
│       ├── values/
│       └── patches/
│       └── sync-policy.yaml  # stricter gates for prod
├── applications/
│   ├── inference-server/
│   │   ├── base/
│   │   │   ├── kustomization.yaml
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   ├── hpa.yaml
│   │   │   └── configmap.yaml
│   │   └── overlays/
│   │       ├── dev/
│   │       ├── staging/
│   │       └── prod/
│   ├── training-scheduler/
│   │   └── (similar structure)
│   ├── model-registry-sync/
│   │   └── (similar structure)
│   └── monitoring/
│       └── (Prometheus, Grafana, Loki)
├── charts/
│   └── ml-infrastructure/
│       ├── values.yaml
│       ├── values-dev.yaml
│       ├── values-prod.yaml
│       └── templates/
└── .github/
    └── workflows/
        ├── model-promotion.yaml
        ├── deploy.yaml
        └── drift-alert.yaml

This structure gives you:

• Environments as first-class citizens: easy to see what's different between dev and prod
• Reusable base manifests: no copy-paste, just Kustomize overlays
• Helm values separated: declarative config per environment
• GitHub Actions for automation: model promotion and deployment triggers

ArgoCD: The GitOps Operator for ML

ArgoCD is a Kubernetes-native GitOps operator. It watches your repo and syncs your cluster. Let's see how it fits into ML workflows.

Installing ArgoCD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
 
# Port-forward to access the UI
kubectl port-forward -n argocd svc/argocd-server 8080:443
# Default password: get it with:
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Declaring Applications

ArgoCD uses an Application custom resource. Here's one for your inference server:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-server
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/gitops-ml-infra
    targetRevision: main
    path: applications/inference-server/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: ml-serving
  syncPolicy:
    automated:
      prune: true # Remove resources deleted from Git
      selfHeal: true # Sync if cluster drifts
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

The syncPolicy is crucial: prune removes resources you deleted from Git, and selfHeal auto-syncs if someone makes manual changes (which they shouldn't, but you know they will).

Sync Waves for Dependencies

ML workloads often have ordering requirements: you can't start training schedulers until the model registry is ready. ArgoCD's sync waves handle this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: model-registry-config
  annotations:
    argocd.argoproj.io/sync-wave: "0" # Deploy first
data:
  registry-url: https://model-registry.internal
 
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: training-scheduler
  annotations:
    argocd.argoproj.io/sync-wave: "1" # Deploy after ConfigMap
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: trainer
              image: myregistry.azurecr.io/trainer:v1.2.0
              env:
                - name: REGISTRY_URL
                  valueFrom:
                    configMapKeyRef:
                      name: model-registry-config
                      key: registry-url

Wave 0 deploys first, wave 1 waits for it to be healthy, and so on.

Health Checks for GPU Deployments

GPU deployments have unique challenges: a pod might be scheduled but waiting for GPU availability. ArgoCD's health checks need tweaking:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: model-inference
  template:
    metadata:
      labels:
        app: model-inference
    spec:
      containers:
        - name: inference
          image: myregistry.azurecr.io/inference:v3.0.0
          resources:
            requests:
              nvidia.com/gpu: "1"
            limits:
              nvidia.com/gpu: "1"
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60 # GPU startup is slow
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /ready
              port: 8000
            initialDelaySeconds: 30

The key: higher initialDelaySeconds for GPU workloads. They need time to initialize.

Drift Detection and Alerting

Here's where GitOps gets powerful: you can automatically detect when reality diverges from Git.

Detecting OutOfSync

ArgoCD constantly compares Git to the cluster. When they diverge, the Application goes OutOfSync. You can trigger alerts:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-failed]
 
  trigger.on-deployed: |
    - when: app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy'
      send: [app-deployed]
 
  service.slack: |
    token: $slack-token
 
  template.app-failed: |
    message: |
      ⚠️ ArgoCD Sync Failed
      App: {{.app.metadata.name}}
      Namespace: {{.app.spec.destination.namespace}}
      Reason: {{.app.status.operationState.message}}
    slack:
      attachments: |
        [{
          "color": "#ff0000",
          "fields": [
            {"title": "Sync Result", "value": "{{.app.status.operationState.syncResult.resources | length}} resources"}
          ]
        }]

This posts to Slack whenever a sync fails, keeping your team in the loop.

Manual Sync Gates for Production

Production deployments should require human approval. Configure this in your Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-server-prod
spec:
  syncPolicy:
    automated: null # Disable auto-sync
    syncOptions:
      - CreateNamespace=true
    # Manual sync required

Now, when a commit lands on main, ArgoCD detects it but doesn't sync automatically. Someone clicks "Sync" in the UI (or you use the CLI). This gives you a human checkpoint.

ML Model Updates with GitOps

The real power of GitOps emerges when you integrate model promotion. Here's the workflow:

Flow: Model is trained → pushed to registry → GitHub Action creates a PR → reviewer approves → merged to main → ArgoCD auto-deploys

Step 1: Automate Model Promotion with GitHub Actions

When your CI pushes a new model image, a workflow creates a PR updating the inference deployment:

# .github/workflows/model-promotion.yaml
name: Promote Model to Prod
 
on:
  push:
    branches: [main]
    paths:
      - "models/**"
 
jobs:
  promote:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
 
      - name: Get latest model image tag
        id: model-info
        run: |
          # Your model registry logic (e.g., query MLflow, check S3)
          MODEL_TAG=$(aws s3api head-object --bucket model-registry --key latest.txt | jq -r '.Metadata.version')
          echo "tag=${MODEL_TAG}" >> $GITHUB_OUTPUT
 
      - name: Update inference deployment
        run: |
          # Update the image tag in the Kustomize overlay
          cd applications/inference-server/overlays/prod
          kustomize edit set image inference=myregistry.azurecr.io/model-inference:${{ steps.model-info.outputs.tag }}
 
      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v4
        with:
          commit-message: "chore: promote model to ${{ steps.model-info.outputs.tag }}"
          title: "Promote Model: ${{ steps.model-info.outputs.tag }}"
          body: |
            ## Model Promotion
            - **Tag**: ${{ steps.model-info.outputs.tag }}
            - **Metrics**: Check model registry for validation results
            - **Rollback**: Revert this PR to rollback
          branch: auto/model-promotion-${{ steps.model-info.outputs.tag }}
          delete-branch: true

Now every model lands as a PR, reviewed before production.

Step 2: Review and Merge

Your ML engineers review the PR (checking metrics, validation results), then merge. This is your governance point.

Step 3: ArgoCD Auto-Deploy

Once merged to main, ArgoCD detects the change and syncs (if auto-sync is enabled):

# In your prod Application spec:
syncPolicy:
  automated:
    prune: true
    selfHeal: true

Boom. New model is live.

Step 4: Rollback via Git Revert

Something's wrong? Inference latency spiked? Just revert the commit:

git revert <commit-hash>
git push

ArgoCD sees the revert and rolls back the image. No manual kubectl commands needed.

Comparing ArgoCD and Flux

Both are excellent GitOps operators. Quick comparison:

For ML teams, ArgoCD's UI is often the sweet spot - non-engineers can watch deployments without learning Flux's architecture.

Drift Detection in Action: A Real Example

Let's trace through what happens when drift occurs:

graph TD
    A["Git Commit<br/>model-inference:v3.1.0"] -->|Pushed| B["GitHub Main"]
    B -->|ArgoCD polls| C["Compare Desired vs Actual"]
    C -->|Mismatch| D["Application = OutOfSync"]
    D -->|If AutoSync Enabled| E["kubectl apply -f"]
    E -->|Pod gets new image| F["Pod ready at v3.1.0"]
    F -->|Health check passes| G["Application = Synced & Healthy"]
 
    H["Someone runs:<br/>kubectl set image..."] -->|Manual change| I["Cluster state changed"]
    I -->|Next reconciliation| J["ArgoCD detects drift"]
    J -->|selfHeal=true| K["Revert to Git state"]
    K -->|Back in sync| L["Git is source of truth"]

This is the beauty of GitOps: humans can try to deviate, but the system constantly pulls them back.

The GitOps ML Infrastructure in Mermaid

Here's how all the pieces fit together:

graph LR
    A["Developer"] -->|"git push model"| B["GitHub Repo"]
    B -->|"Trigger Action"| C["GitHub Actions"]
    C -->|"Update kustomization"| D["Create PR with image tag"]
    D -->|"Review & Merge"| E["main branch"]
    E -->|"Webhook/polling"| F["ArgoCD"]
    F -->|"Compare state"| G{Synced?}
    G -->|"No"| H["kubectl apply"]
    H -->|"Deploy"| I["Kubernetes Cluster"]
    I -->|"Health check"| J["Application CRD"]
    J -->|"Healthy"| K["Inference Server Running"]
    K -->|"Monitor"| L["Prometheus/Grafana"]
    L -->|"Alert on drift"| M["Slack Notification"]

Best Practices for ML + GitOps

1. Never skip code review for model promotion PRs. That YAML change is deploying a model - treat it seriously.

2. Version everything: model images, Helm charts, Kustomize bases. Immutability is your friend.

3. Use sync waves to manage dependencies. Your training scheduler can't run before the model registry is ready.

4. Health checks matter: GPU pods need longer initialDelaySeconds. Don't let ArgoCD mark them unhealthy prematurely.

5. Audit everything: Git is your audit trail. Who promoted which model? Check the commits.

6. Automate where it makes sense, but keep humans in the loop for production. Let GitHub Actions handle image updates, but require approval on merges.

7. Monitor drift: Set up alerts for OutOfSync status. If ArgoCD and the cluster disagree, someone should know.

8. Test in dev/staging first: Your overlays/prod should only see well-tested configurations.

Common Pitfalls and How to Avoid Them

GitOps sounds simple in theory. In practice, teams hit the same walls repeatedly. Let's talk through the gotchas.

Pitfall 1: The "Sync Loop of Death"

Your Application keeps flipping between Synced and OutOfSync. ArgoCD syncs, something changes (maybe a StatefulSet's status), ArgoCD detects drift, syncs again. Forever. You're watching the status page thrash.

What causes this: Mismatch between what Git declares and what Kubernetes actually does. Common culprits:

• Admission controllers that mutate your resources
• HPA (Horizontal Pod Autoscaler) changing replica counts after ArgoCD set them
• Generated fields in status that differ from your manifest

The fix:

# Exclude generated fields from comparison
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-server
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas # HPA will change this, ignore it
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP # Kubernetes assigns this, don't compare
 
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      # Add a sync window to prevent thrashing
      syncOptions:
        - RespectIgnoreDifferences=true

For HPA specifically, tell ArgoCD not to fight the scaler:

# When using HPA, don't let ArgoCD override replicas
apiVersion: v1
kind: Service
metadata:
  name: inference-server
  annotations:
    argocd.argoproj.io/compare-result: IgnoreDifference # Don't fight the autoscaler

Pitfall 2: Breaking Changes During Rollout

You update your inference server image. But the new version has a breaking schema change in its API. Some requests to the old pods fail, others to the new pods fail. Your clients are confused. Rollback is a mess because you're mid-deployment.

The GitOps mistake: You didn't use a rolling update strategy with proper health checks.

The fix:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-inference
spec:
  replicas: 5 # Must have multiple replicas
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1 # 1 extra pod during update
      maxUnavailable: 0 # Never take all pods down
  selector:
    matchLabels:
      app: model-inference
  template:
    metadata:
      labels:
        app: model-inference
    spec:
      terminationGracePeriodSeconds: 30 # Time for graceful shutdown
      containers:
        - name: inference
          image: myregistry.azurecr.io/inference:v4.0.0
 
          # Readiness: is this pod ready to receive traffic?
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 5
            failureThreshold: 2 # 2 failures = mark not ready
 
          # Liveness: is this pod dead?
          livenessProbe:
            httpGet:
              path: /alive
              port: 8000
            initialDelaySeconds: 30
            periodSeconds: 10
            failureThreshold: 3

With this setup, Kubernetes brings up a new pod, waits for it to pass readiness checks, then drains traffic from old pods. If the new pod fails health checks, Kubernetes stops the rollout and keeps running the old version.

Pitfall 3: Model Version Mismatch

Your inference server image expects a specific model version. But your GitOps workflow promotes a new model image without updating the inference server to expect it. Now they're out of sync.

The fix: Couple model and server updates in a single commit:

# applications/inference-server/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
bases:
  - ../../base
 
images:
  - name: model-inference-server
    newTag: v4.1.0 # Updated together
 
patchesJson6902:
  - target:
      group: v1
      version: ""
      kind: ConfigMap
      name: model-config
    patch: |-
      - op: replace
        path: /data/model_version
        value: "llama2-7b-v12"  # Model version must match

Or use Kustomize to generate this atomically:

# Your CI/CD does this in one commit
kustomize edit set image model-inference-server=myregistry.azurecr.io/inference:v4.1.0
kustomize edit set config model-version=llama2-7b-v12
 
git add -A
git commit -m "chore: update inference server and model together"

Pitfall 4: Secrets in Git

You accidentally commit your database credentials to the GitOps repo. Now anyone with read access to your repo has your secrets. And reverting the commit doesn't help - it's in git history forever.

The right approach: Use external secret management. ArgoCD integrates with Vault, AWS Secrets Manager, etc.

# Use ArgoCD's secret management, not kubectl secrets
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-server
spec:
  source:
    repoURL: https://github.com/your-org/gitops-ml-infra
    targetRevision: main
    path: applications/inference-server/overlays/prod
 
    # Reference secrets from Vault, not Git
    plugin:
      name: argocd-vault-plugin
      env:
        - name: AVP_TYPE
          value: vault
        - name: AVP_AUTH_TYPE
          value: k8s

Or use sealed-secrets to encrypt secrets in Git (only the cluster can decrypt):

# Sealed secrets: encrypted at rest in Git
apiVersion: bitnami.com/v1
kind: SealedSecret
metadata:
  name: inference-db-secrets
  namespace: ml-serving
spec:
  encryptedData:
    db_password: AgBvV3rZF8X...= # Encrypted, safe in Git
  template:
    metadata:
      name: inference-db-secrets
      namespace: ml-serving
    type: Opaque

Production Considerations

Disaster Recovery: When Git Becomes Truth

Your cluster breaks. Nodes fail. Disaster scenario: would you rather:

• Manually recreate everything? (Hours, error-prone)
• Apply git apply -k applications/inference-server/overlays/prod/ and have it all come back? (Minutes, reproducible)

GitOps wins because Git is your disaster recovery plan.

But there's a catch: what if someone deletes the cluster entirely? Your ArgoCD controller is gone. Git doesn't help.

The solution: GitOps is half of disaster recovery. The other half is backups.

# Use Velero for cluster backup/restore
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: ml-cluster-daily
  namespace: velero
spec:
  schedule: "0 2 * * *" # 2 AM daily
  template:
    ttl: "720h" # Keep 30 days
    includedNamespaces:
      - ml-serving
      - argocd
      - monitoring
    storageLocation: s3-backup
    # Don't back up ArgoCD Applications—they're in Git
    excludedResources:
      - applications.argoproj.io

Authorization: Who Can Deploy What?

Git permissions and Kubernetes permissions need alignment. If someone can merge to main, they can deploy to prod (via ArgoCD). That's powerful and dangerous.

Best practice:

# RBAC: separate roles for environments
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argocd-deployer-prod
  namespace: argocd
rules:
  - apiGroups: ["argoproj.io"]
    resources: ["applications"]
    verbs: ["get", "list", "patch"]
    resourceNames: ["inference-server-prod"] # Only this app
  - apiGroups: [""]
    resources: ["applications/sync"]
    verbs: ["create"]
    # Allows syncing, but not changing the app itself

Plus, require code review on Git:

# GitHub branch protection
# Settings > Branches > Require pull request reviews before merging
# - Require 1 approval (or 2 for prod)
# - Require status checks to pass (your CI)

Observability: Tracking What GitOps Actually Did

When a deployment fails, your team needs to know: was it a bad code change, a K8s issue, or something else?

# ArgoCD ships metrics you can scrape
apiVersion: v1
kind: Service
metadata:
  name: argocd-metrics
  namespace: argocd
spec:
  ports:
    - name: metrics
      port: 8082
      targetPort: 8082
 
# Prometheus scrape config
prometheus:
  scrape_configs:
    - job_name: "argocd"
      static_configs:
        - targets: ["argocd-metrics:8082"]

Key metrics to alert on:

# OutOfSync applications
argocd_app_info{dest_server!="in-cluster"} and on(name) argocd_app_sync_total{phase="OutOfSync"}
 
# Sync failures
rate(argocd_app_reconcile_total{phase="Failed"}[5m]) > 0
 
# Stale repository information
time() - argocd_git_request_total > 300

Handling Emergencies Without Breaking GitOps

Sometimes you need to hot-patch production and can't wait for a commit + review cycle. GitOps should support emergency bypass, but safely.

# Emergency deployment: bypass GitOps temporarily
kubectl patch deployment model-inference \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"inference","image":"myregistry.azurecr.io/inference:emergency-patch"}]}}}}'
 
# But immediately document this in Git
# so you don't lose track
git checkout -b hotfix/emergency-inference-patch
# Update the kustomization.yaml
git add -A
git commit -m "hotfix: emergency patch for inference server - follow-up PR required"
git push
 
# Then ArgoCD will re-sync and bring you back to declared state
# (Or selfHeal=false to prevent auto-revert)

The key: manual changes should be temporary. Document them and get them into Git promptly.

Advanced Patterns: Multi-Cluster ML Deployments

As your ML platform grows, you might deploy models across multiple clusters - one in us-east-1 for low-latency serving, another in us-west-2 for disaster recovery, and a third in a different region for compliance.

GitOps Across Multiple Clusters

ArgoCD can manage multiple destination clusters from a single repository:

# applications/inference-server/base/application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-server-multi-cluster
spec:
  source:
    repoURL: https://github.com/your-org/gitops-ml-infra
    targetRevision: main
    path: applications/inference-server/overlays/prod
 
  # Deploy to multiple clusters
  destinations:
    - server: https://cluster-us-east-1.k8s.aws/
      namespace: ml-serving
      name: us-east-1
    - server: https://cluster-us-west-2.k8s.aws/
      namespace: ml-serving
      name: us-west-2
    - server: https://cluster-eu-west-1.k8s.aws/
      namespace: ml-serving
      name: eu-west-1
 
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

ArgoCD synchronizes all three clusters. A single commit to the Git repository automatically rolls out your model across all regions, with the same configuration, same versioning, same auditability.

Handling Regional Configuration Differences

But clusters often need different configurations. The us-west-2 cluster might have fewer GPUs and need smaller replicas. The eu-west-1 cluster must comply with GDPR, so it needs different secret storage.

Use Kustomize overlays per cluster:

applications/inference-server/
├── base/
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── us-east-1/
    │   ├── kustomization.yaml
    │   └── replicas.yaml
    ├── us-west-2/
    │   ├── kustomization.yaml
    │   └── replicas.yaml
    └── eu-west-1/
        ├── kustomization.yaml
        ├── replicas.yaml
        └── secrets.yaml

Each overlay specifies regional differences. When ArgoCD syncs, it applies the base, then layers on region-specific patches.

Why GitOps Matters for ML Infrastructure: The Business Case

Beyond)) the technical elegance, GitOps makes business sense for ML teams. Let me quantify it:

Time Savings

Without GitOps:

• Deploy a model: SSH into 3 clusters, run kubectl manually, hope you remember the right commands. 30 minutes per deployment.
• Rollback a bad model: manually revert image tags in three places, verify each cluster. 45 minutes.
• Audit who deployed what: check person's email, Slack messages, maybe kubectl history. 20 minutes.

With GitOps:

• Deploy a model: git push to main. 2 minutes (mostly waiting for CI).
• Rollback: git revert, push. 5 minutes.
• Audit: check Git history, see commit message, author, timestamp. 30 seconds.

Annual savings (assuming 100 model deployments/year): 30×100 + 40×10 (rollbacks) - 2×100 - 5×10 = 2,500 hours. That's 1.25 FTE engineers freed up.

Reliability

Without GitOps:

• 30% of deployments have drift (someone applied something manually). You don't know which clusters are actually running.
• Rollback success rate: 85%. Sometimes it works, sometimes you missed a namespace.

With GitOps:

• 0% drift. Git is the source of truth. Always.
• Rollback success rate: 99.9%. It's just git revert.

Fewer incidents. Less oncall stress. Better sleep.

Team Autonomy

Without GitOps:

• Junior engineers can't deploy. They don't know kubectl well enough. They might break something.
• All deploys go through a senior engineer or ops person. Bottleneck.

With GitOps:

• Junior engineers create a PR. It goes through code review. Senior engineer approves. Deploying is merging.
• Anyone who can write YAML and Git can deploy.

Your team scales faster because knowledge isn't hoarded.

Common Pitfalls and How to Avoid Them: Deep Dive

We covered pitfalls earlier, but let me expand on the most critical ones:

The Sync Loop of Death: Root Causes and Solutions

The sync loop happens because ArgoCD compares desired state (Git) to actual state (cluster) and sometimes they never match. Let me walk through the actual root causes we see:

Cause 1: HPA and Manual Replica Scaling

You've deployed an HPA that scales replicas based on CPU. Git declares 5 replicas. HPA scales to 10 when traffic spikes. ArgoCD sees mismatch: Git says 5, cluster has 10. ArgoCD "fixes" it by scaling back to 5, killing 5 pods mid-request. Traffic spikes again. HPA scales to 10. Loop.

# WRONG: HPA and fixed replicas fight
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-server
spec:
  replicas: 5 # ArgoCD will enforce this
  # ...
 
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-server
  minReplicas: 5
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

CORRECT: Let HPA control replicas, don't enforce them in Git

# RIGHT: Remove replicas from Deployment, let HPA control it
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inference-server
spec:
  # DON'T specify replicas, HPA will manage it
  selector:
    matchLabels:
      app: inference-server
  template:
    # ...
 
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-server-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-server
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

And tell ArgoCD to ignore replica differences:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-server
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas # HPA will change this, don't compare

Cause 2: Mutating Admission Controllers

Your cluster runs an admission controller that injects sidecars (Istio, network policies, etc.). It adds a sidecar container to every pod. Git doesn't declare sidecars. ArgoCD sees them in the cluster, thinks it's drift, tries to remove them. The admission controller adds them back. Loop.

Solution:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: inference-server
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/template/spec/containers
          # ^ Ignore sidecar injection differences added by admission controllers

Or better, declare the sidecars explicitly in your manifests so you know what you're getting.

Handling Secrets in Git: Safe Patterns

Never commit credentials to Git. Ever. But you still need to manage secrets through GitOps.

Pattern 1: Sealed Secrets

Encrypt secrets at rest in Git. Only the cluster can decrypt them.

# Install sealed-secrets
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/controller.yaml
 
# Create a secret locally
kubectl create secret generic my-db-secret \
  --from-literal=password=supersecret \
  --dry-run=client -o yaml > secret.yaml
 
# Seal it (requires cluster's public key)
kubeseal -f secret.yaml -w sealed-secret.yaml
 
# Now it's safe to commit
git add sealed-secret.yaml
git commit -m "Add encrypted DB secret"

The sealed secret looks like:

apiVersion: bitnami.com/v1
kind: SealedSecret
metadata:
  name: my-db-secret
spec:
  encryptedData:
    password: AgBvV3rZF8X...=
  template:
    metadata:
      name: my-db-secret
    type: Opaque

Only your cluster can decrypt AgBvV3rZF8X...=. If Git is compromised, an attacker gets encrypted blobs.

Pattern 2: External Secrets Operator (ESO)

Don't store secrets in Git at all. Reference external secret management:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
 
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: db-secret
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/db/password
    - secretKey: username
      remoteRef:
        key: prod/db/username

Git tracks the ExternalSecret (which references a secret by name), but the actual credentials live in AWS Secrets Manager. Secure, auditable, GitOps-friendly.

Model Promotion and Inference Server Versioning

In ML systems specifically, GitOps becomes the mechanism for promoting models between environments. A common pattern is to declare the model version you want to run in your Git manifests. Development environment might reference model v1.0. Staging references v1.1. Production references v1.0 (still testing v1.1). To promote a model to production, you update the manifest and create a pull request. Code review, testing in staging, then merge. The production inference server automatically deploys the new model.

This pattern eliminates manual coordination and creates an immutable record of which models ran when. You can look at Git history and see exactly when model-v1.1 was promoted to production, who approved it, and what the diff was. If model v1.1 performs poorly in production, you can simply revert the commit, rolling back the model within minutes. This would be impossible with manual deployment.

The pattern extends to inference server configuration. Maybe you want to adjust the serving framework version, or the batch size, or the quantization-pipeline-pipelines-training-orchestration)-fundamentals))-automated-model-compression)-production-inference-deployment)-llms) settings. All of this lives in Git. Every configuration change is a pull request with full history. Your team understands exactly how your production inference is configured at any given moment.

Handling Stateful Workloads: Training Jobs and Data Pipelines

Pure GitOps works great for stateless services - inference servers, API gateways, dashboards. But ML workloads often involve state: training jobs that generate models, data pipelines that process datasets. How do you apply GitOps to these?

The pattern is to declare the training job template in Git, but let Kubernetes manage the actual job lifecycle. Your Git repo contains the job definition - which training script to run, what hyperparameters, where to save the model. You don't declare which jobs have finished or what their status is. Git declares intent; Kubernetes and your jobs declare outcomes.

For data pipelines, the pattern is similar. Your pipeline-parallelism) definition lives in Git - which steps to run, in what order, with what dependencies. The actual execution is managed by a workflow orchestrator like Airflow or Argo Workflows. Git declares the pipeline structure. The orchestrator executes it. If you need to change the pipeline, you update Git, merge through code review, and the orchestrator picks up the new definition.

This hybrid approach gets you the best of both worlds: declarative infrastructure-as-code for your platform, with state management delegated to systems that are designed to handle it.

Detecting and Preventing Configuration Drift

Despite GitOps's promise of deterministic state, drift still happens. An engineer might manually patch a Kubernetes resource to debug an issue and forget to commit the change. A controller might add annotations or labels that aren't in your manifests. Network policies might be modified to debug connectivity. Before you realize it, your actual state has diverged from your declared state.

Mature GitOps systems implement drift detection that runs continuously. ArgoCD has built-in refresh intervals that check for drift. Flux has similar capabilities. Going further, some teams implement policy-as-code tools like Kyverno or OPA to enforce that all resources conform to expected standards. If drift is detected, automated remediation can restore declared state, or it can alert for manual review if the drift is significant.

The philosophy matters here: should GitOps aggressively correct drift, or should it alert and let humans decide? Aggressive correction is simpler but risky - what if the drift is intentional? Alert-based approaches are safer but require operational discipline. Your team needs to respond to drift alerts promptly, or they'll accumulate.

Debugging and Troubleshooting GitOps Issues

When something goes wrong with GitOps, debugging can be tricky because the problem might be in your manifests, your Git history, your controller, or the interaction between them. A deployment might be stuck in OutOfSync state with no obvious reason. An application might show Healthy in ArgoCD but Degraded in your monitoring.

The key to effective debugging is layered investigation. First, check if the Git repository matches the cluster: is ArgoCD showing the resources as in sync? If not, what's the difference? ArgoCD's UI shows diffs visually, making this easy. If the cluster differs from Git, the question becomes: did someone manually change the cluster, or did Git get out of sync? Check your Git history and branch status.

Second, check the controller's logs. ArgoCD has detailed logging of what it's doing. Why did a resource fail to apply? Was there a validation error? A missing namespace? A circular dependency? The logs tell the story. Flux similarly logs reconciliation attempts with detailed error messages.

Third, understand the resource status. A Pod might be stuck in ImagePullBackOff because the container image doesn't exist. A Service might have no endpoints because no Pods match its selector. These aren't GitOps problems; they're Kubernetes problems that GitOps inherited. Debugging requires understanding both layers.

Fourth, use the kubectl tools to understand what's actually running. kubectl get all shows you everything. kubectl describe shows you details and events. kubectl logs shows you application output. These tools help you understand what Kubernetes sees, which might differ from what ArgoCD claims.

Testing GitOps Changes Safely

GitOps makes deployments safe through auditability and rollback capability, but deploying bad code to production is still bad. Teams practicing safe GitOps implement pull request reviews and automated testing before merging to main. Some teams run integration tests that apply the manifests to a test cluster and validate the results. Others use preview environments that spin up with each pull request, letting reviewers see the actual deployment before merging.

A common pattern is to require manual approval for production changes. Development and staging merge automatically on every commit. Production requires a team member to review and click "sync" in ArgoCD. This gives you the benefits of GitOps - immutability, auditability, reproducibility - while keeping humans in the loop for the most critical decisions.

Another pattern is to use different branches for different environments. The develop branch syncs to staging. The main branch syncs to production. Merging from develop to main is a deliberate promotion. You can test thoroughly in staging, then promote. This creates a clear progression from development to production.

Some teams go further with progressive deployment strategies. ArgoCD supports canary and blue-green deployments through custom Kubernetes resources. You deploy the new version to a small percentage of traffic first, verify it works, then roll out to everyone. If something goes wrong, you're only affecting a small subset of users.

Conclusion: Building Confidence in Your ML Platform

GitOps transforms ML infrastructure from a source of anxiety into a source of confidence. You know exactly what's running, why it's running, when it changed, and who changed it. You can deploy with a Pull Request. You can rollback with a git revert.

For ML teams specifically:

• Model promotion is reproducible: A commit that says "deploy resnet50-v3.2" is self-documenting.
• Training pipelines are version-controlled: Your hyperparameters, training job definitions, and infrastructure all live together.
• Disasters are recoverable: Your Git repo is a complete backup of your infrastructure.
• Teams move faster: Code review replaces manual verification.

Wrapping up: GitOps transforms ML infrastructure from a manual, fragile process into a declarative, auditable, reproducible system. By treating your Kubernetes manifests as code, living in Git, you get:

• Reproducibility: exact cluster state at any point in history
• Auditability: every change is a commit with metadata
• Rollback safety: revert a commit to roll back a deployment
• Team autonomy: non-ops engineers can understand and contribute
• Disaster recovery: your Git repo is your infrastructure backup

Tools like ArgoCD make this practical. Start small - sync one application - then scale to your full ML platform. Watch out for the pitfalls we covered, instrument your system, and build in safety gates. Your future self (and your oncall team) will thank you.

The Human Dimension of GitOps

GitOps sounds great in theory, but the real value emerges from how it changes your team's behavior and culture. When your infrastructure lives in Git with code review, something shifts. Suddenly, deploying a model isn't a casual action - it's a deliberate change you're committing to history. Your team gets better at documenting why they made changes. "Why did we upgrade from inference server v2.1.0 to v3.0.0?" You can check the commit message. "Did we test this new configuration before deploying to prod?" You can see the PR review and discussion.

This creates organizational alignment that pure GitOps tooling alone can't achieve. Your data scientists get visibility into what's running where. Your infrastructure engineers can see exactly what models are deployed. When a model underperforms, you can trace back to when it was deployed and what changed that day. The entire organization benefits from transparency that emerges naturally from treating infrastructure as code.

But there's a darker side to consider. With GitOps, Git becomes the control plane. If someone gets access to your repository, they can deploy anything to production. Security becomes paramount. You need branch protection rules to prevent direct commits to main. You need multiple approvers for production changes. You need audit logging to see who approved what. You need signed commits so you can verify that changes came from trusted developers. Getting this right is harder than it looks, but the security payoff is enormous.

Operational Maturity with GitOps

Teams that run production GitOps for years develop patterns that newcomers miss. One critical pattern is the difference between your desired state and your actual state. Git is your desired state. The Kubernetes cluster is your actual state. These should always match, but they often don't due to operator failures, network issues, or controller timeouts. Mature teams implement monitoring that constantly checks this divergence. An ArgoCD application that shows "Synced" but has stale status information in the CRD is actually out of sync. You need detailed metrics to surface this.

Another mature pattern is automatic reconciliation with human gates for production. Your development environment syncs automatically whenever code is merged. Your production environment requires a human click before syncing. This gives you the benefits of GitOps (immutability, auditability, reproducibility) while keeping humans in the loop for the most critical changes. Some teams go further and require multiple approvals for production, or implement sync windows that only allow deployments during business hours.

Disaster recovery becomes simpler with GitOps. Your entire infrastructure is declared in Git. If your Kubernetes cluster dies catastrophically, you can restore from your Git repository. However, this requires discipline. Every resource your applications depend on must be in Git. That includes ConfigMaps, Secrets, PersistentVolumeClaims, and custom resources. Some teams maintain a "golden repo" that serves as the source of truth. Others decentralize Git repos by team. The structure matters less than consistency and discipline.

Evolution and Lessons Learned

GitOps has been around for a decade, and the field has matured significantly. Early adopters learned hard lessons. One was discovering that not everything should live in the control plane through Git. Some state is runtime state - the exact replicas that HPA scaled to, the IP addresses assigned to load balancers, the actual pod IPs. Including these in your desired state causes infinite reconciliation loops. Mature GitOps systems explicitly exclude these from comparison.

Another lesson was discovering that Git history itself becomes a valuable artifact. Teams that maintain clean commit messages and organized branches can trace the evolution of their infrastructure. Teams that throw random commits with cryptic messages find their history useless. Investing in commit hygiene pays dividends when you're debugging production issues months later and need to understand why a configuration changed.

The final lesson was understanding that GitOps is a tool, not a solution. It makes deployments reproducible and auditable, but it doesn't solve every problem. You still need excellent monitoring. You still need runbooks for when things break. You still need team discipline. GitOps enables these things, but it doesn't replace them. The teams that get the most value are the ones that treat GitOps as the foundation for a larger culture of infrastructure automation and observability.

Scaling GitOps Across Enterprises

As organizations grow from a handful of teams to dozens, GitOps governance becomes increasingly important. A single ArgoCD instance handling all deployments becomes a bottleneck. Your repository structure needs to scale. Your rollout process needs guardrails-infrastructure-content-safety-llm-applications) to prevent mistakes at scale. This is where the operational complexity of GitOps accelerates.

Many enterprises deploy multiple ArgoCD instances in a hub-and-spoke model. Central hub handles shared infrastructure. Spoke instances in each region or team handle their workloads. Shared governance policies are enforced through namespace RBAC and policy-as-code tools like Open Policy Agent. This gives you scale with reasonable oversight.

Repository structure also matters at scale. Some organizations use a monorepo with all applications. Others use a folder-per-team structure. Still others use separate repositories per application. There's no universally right answer, but consistency is critical. Every team should follow the same repository patterns, the same deployment directory structures, the same secrets management approach. This reduces cognitive load and prevents mistakes from inconsistent configuration.

The Cost-Benefit Analysis of GitOps

Implementing GitOps requires investment. You need to learn new tools. You need to refactor existing infrastructure into code. You need to build CI/CD pipelines. You need to train your team. For a five-person startup with manual deployments, this might be overkill. The cost of tooling and training exceeds the benefit.

But as you grow, the calculus shifts. At fifty people with dozens of services, manual deployments become impossible. Drift creeps in. People make mistakes. Troubleshooting production issues takes hours. GitOps investment pays for itself through reduced incident response time, faster deployments, better auditability, and team velocity.

The key is graduating to GitOps gradually. Start with one application. Get comfortable with the workflow. Expand to critical services first. Build reusable patterns. Train your team progressively. Don't try to convert your entire infrastructure overnight. That's how GitOps projects fail - too much change, too fast, insufficient training.