GitOpsGitOps

Part of the DevOps Roadmap knowledge base.

Overview

GitOps is an operating model for delivering infrastructure and applications in which Git is the single source of truth for the desired state of the system. Everything that defines your runtime environment — Kubernetes manifests, Helm values, Kustomize overlays, infrastructure definitions — lives in Git. An automated agent running in the cluster continuously compares what is actually running against what Git says should be running, and reconciles the difference. Deploying becomes git merge; rolling back becomes git revert.

This model brings the guarantees of software engineering to operations: every change is a reviewed pull request, the full history is auditable, and the environment can be rebuilt from the repository at any time. Because a controller pulls changes rather than a pipeline pushing them, credentials for the cluster never need to leave the cluster — a major security improvement over classic CI-driven deployment.

GitOps builds directly on the previous roadmap topics: containers give you immutable artifacts, Kubernetes provides declarative APIs and reconciliation, and GitOps closes the loop by putting the desired state under version control. Argo CD and Flux are the two dominant CNCF-graduated implementations.

A mental model

GitOps is the reconciliation loop of Kubernetes (spec vs status), extended so that the spec itself lives in Git instead of only in etcd. The agent turns your Git repository into a second desired-state store and continuously drives the cluster toward it. Two consequences follow directly:

• The cluster is no longer a place where state is authored — only where it converges. Any change made outside Git is, by definition, drift the agent will report or undo.
• Recovery is trivial by construction. To rebuild an environment, point a fresh agent at the repo; the desired state re-materializes. Your cluster becomes cattle, and Git becomes the herd’s pedigree.

GitOps is not just “Kubernetes + Git.” The same pattern applies to anything with a declarative API and a reconciler — Terraform state, Crossplane-managed cloud resources, cluster add-ons — but Kubernetes is where it is most mature.

Fundamentals

The four GitOps principles (OpenGitOps)

1. Declarative — the entire desired state is expressed declaratively (YAML manifests, not scripts of imperative steps).
2. Versioned and immutable — the desired state is stored in a way that enforces immutability and full version history (Git, with signed, append-only commits).
3. Pulled automatically — software agents automatically pull the desired state from the source, rather than a human or pipeline pushing it in.
4. Continuously reconciled — agents continuously observe actual state and attempt to converge it toward the desired state, correcting drift without human intervention.

Push vs pull deployment models

In practice CI still exists in GitOps — it builds, tests, and pushes images, then updates an image tag in the config repo (directly or via an image automation controller). CD is fully owned by the GitOps agent. The clean split is: CI produces artifacts and proposes desired state; CD (the agent) realizes it.

Repository structure

A common pattern is separating application source from deployment configuration:

app-repo/                    # source code + Dockerfile; CI builds and pushes the image
config-repo/
├── apps/
│   └── web/
│       ├── base/                    # shared manifests (Kustomize base)
│       └── overlays/
│           ├── dev/kustomization.yaml
│           ├── staging/
│           └── prod/
└── clusters/
    ├── dev/
    │   └── apps.yaml                # Argo CD Applications / Flux Kustomizations
    └── prod/
        └── apps.yaml

Separate repos prevent CI loops (image-bump commits re-triggering builds), allow different access controls (developers on app-repo, platform/SRE on config-repo), and keep audit history clean. The clusters/ tree holds the root definitions each cluster’s agent points at; apps/ holds the actual workload manifests. Some teams instead use a monorepo with strict path-based ownership — either works, but keep app source and rendered config logically separate.

What GitOps does not solve on its own

• It is not a full CD story by itself. You still need CI to build/test/sign images and something (image automation or a promotion tool) to move versions between environments.
• Ordering and dependencies across many apps need sync waves / dependency ordering, not just “apply everything.”
• Database migrations and one-shot tasks don’t fit the pure reconcile model cleanly; use Jobs with sync hooks and careful sequencing.
• Secrets cannot go in Git in plaintext — a dedicated strategy is mandatory (covered below).

Key Concepts

Argo CD vs Flux

Both watch Git, render manifests (Helm/Kustomize/plain YAML), diff against the cluster, and reconcile on an interval or webhook. Choosing between them is mostly about whether you want an opinionated app-centric UI (Argo CD) or a composable controller toolkit (Flux); many organizations succeed with either.

The reconciliation loop in detail

1. Fetch — the agent clones/pulls the target revision (a branch, tag, or commit SHA) from Git (Argo CD polls ~3 min by default, plus webhooks; Flux uses a per-source interval).
2. Render — it runs kustomize build / helm template to produce the concrete desired manifests.
3. Diff — it compares rendered desired state against live cluster state (server-side diff, ignoring fields it shouldn’t manage).
4. Sync — it applies the difference: create, update, and (if pruning is on) delete resources removed from Git.
5. Report — it surfaces status: Synced/OutOfSync, Healthy/Degraded, and events for humans and alerting.

Example Argo CD Application manifest

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io   # cascade-delete managed resources with the App
spec:
  project: default
  source:
    repoURL: https://github.com/example/config-repo.git
    targetRevision: main
    path: apps/web/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: web
  syncPolicy:
    automated:
      prune: true        # delete resources removed from Git
      selfHeal: true     # revert manual changes in the cluster
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true
    retry:
      limit: 3
      backoff: { duration: 5s, factor: 2, maxDuration: 3m }

With selfHeal: true, a manual kubectl edit is reverted within seconds — the cluster always matches Git. With prune: true, deleting a manifest from Git deletes the resource from the cluster. Both are powerful and dangerous; enable them for production only once your review and rollback discipline is solid.

App-of-apps and ApplicationSet

To manage many applications, define a root Application whose source path contains more Application manifests (the app-of-apps pattern) — bootstrapping an entire cluster becomes one kubectl apply of the root App. ApplicationSet generalizes this with generators that stamp out Applications from a single template:

• Git generator — one Application per directory or file in a repo (add a folder, get an app).
• Cluster generator — one Application per registered cluster (fan the same app across a fleet).
• List / matrix / pull-request generators — explicit lists, cross-products, and ephemeral preview environments per open PR.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: web-all-envs, namespace: argocd }
spec:
  generators:
    - list:
        elements:
          - { env: dev,     cluster: https://kubernetes.default.svc }
          - { env: staging, cluster: https://kubernetes.default.svc }
          - { env: prod,    cluster: https://kubernetes.default.svc }
  template:
    metadata: { name: 'web-{{env}}' }
    spec:
      project: default
      source:
        repoURL: https://github.com/example/config-repo.git
        targetRevision: main
        path: 'apps/web/overlays/{{env}}'
      destination: { server: '{{cluster}}', namespace: web }
      syncPolicy: { automated: { prune: true, selfHeal: true } }

Environment promotion

Promotion is a Git operation, not a pipeline stage:

1. CI builds web:1.5.0, pushes it, and updates the tag in overlays/dev.
2. After validation in dev, a PR copies the version bump to overlays/staging, then after staging soak, another PR to overlays/prod.
3. Each promotion is a reviewed, revertible commit, so “what is running where” is answerable by reading Git.

Prefer directory-per-environment over branch-per-environment: branches drift, and merges between them create noise and accidental promotion of unrelated changes. Directory overlays make each environment’s full state explicit and diffable. Tools like Kargo automate multi-stage promotion (freight, verification, and gated progression) on top of GitOps, and Argo CD’s ApplicationSet PR generator gives per-PR preview environments.

Secrets in GitOps

Plaintext secrets can never be committed — Git history is forever and repos get cloned widely. Three standard approaches, each keeping Git safe in a different way:

# ExternalSecret: Git holds a pointer, not the secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: db-credentials }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws-sm, kind: ClusterSecretStore }
  target: { name: db-credentials }        # the native K8s Secret that gets created
  data:
    - secretKey: password
      remoteRef: { key: prod/db, property: password }

Rule of thumb: Sealed Secrets for small setups with no secret store; SOPS when you want everything (including secrets) in Git but encrypted; External Secrets Operator when you already run Vault or a cloud secret manager and want Git to hold zero secret material.

Drift detection and reconciliation

The agent’s reconciliation loop computes a diff between rendered desired state and live state. Drift can be handled two ways:

• Detect and alert — surface OutOfSync status and notify a human, who reviews and either reconciles or codifies the change into Git. Good early on, when you don’t yet trust auto-correction.
• Auto-correct (self-heal) — the agent immediately reverts the out-of-band change so the cluster snaps back to Git. Good once mature, but it means a hotfix applied by hand will vanish — deliberate, but surprising the first time it happens.

Either way, out-of-band changes lose: the cluster is no longer a place where state is authored, only where it converges. Some fields must be excluded from drift detection (e.g., HPA-managed replica counts, webhook-injected fields) via ignoreDifferences so the agent and other controllers don’t fight over them.

Progressive delivery on top of GitOps

GitOps handles what is deployed; progressive delivery handles how safely it rolls out. Argo Rollouts (with Argo CD) and Flagger (with Flux) add canary and blue-green strategies driven by real metrics: shift 10% of traffic, watch error rate and latency (Prometheus), and automatically promote or roll back. The rollout definition still lives in Git, so the safety policy is versioned alongside the app.

Best Practices

1. Separate application code and deployment config repos — independent release cadence, cleaner permissions, and no CI trigger loops from image-bump commits.
2. Make Git the only write path to production — lock down direct kubectl write access; if it’s not in Git, it doesn’t exist and shouldn’t survive the next reconcile.
3. Graduate to automated sync (prune + self-heal) only after maturity — start with manual sync and diff review, then enable auto-sync per environment once your discipline and alerting are solid; keep prod stricter than dev.
4. Use directory-per-environment, not branch-per-environment — avoids long-lived branch drift and painful, error-prone cross-environment merges.
5. Pin versions everywhere — image digests/tags, Helm chart versions, and Git commit SHAs (not floating branches) for bases; reproducibility is the entire point.
6. Never commit plaintext secrets — pick one secrets strategy and standardize — Sealed Secrets, SOPS, or External Secrets Operator, applied consistently across all teams so there’s one audited path.
7. Protect the config repo like production — branch protection, required reviews, CODEOWNERS, and signed commits; a merge here is a deployment, so treat the review as a change-approval gate.
8. Validate manifests in CI before merge — kustomize build, helm template, kubeconform, and policy checks (OPA/Conftest/Kyverno) catch schema and policy errors before the agent ever tries to apply them.
9. Show the rendered diff in PRs — run argocd app diff / flux diff kustomization as a PR check so reviewers see the actual cluster-facing change, not just the template edit.
10. Alert on sync failures and drift — wire Argo CD notifications / Flux alerts into Slack or your incident tooling; a silent OutOfSync or Degraded is a silent outage risk.
11. Bootstrap clusters with app-of-apps or ApplicationSets — cluster recovery becomes: create cluster, install the agent, point it at the repo, and let it re-materialize everything.
12. Roll back with git revert, not manual edits — this preserves the audit trail and keeps Git truthful; a manual fix will be undone by self-heal anyway.
13. Use sync waves / dependency ordering — deploy CRDs and namespaces before the workloads that need them, and databases before the apps that connect to them.
14. Exclude controller-managed fields from drift — use ignoreDifferences for HPA replica counts and admission-injected fields so GitOps and other controllers don’t thrash.
15. Layer progressive delivery for risky services — Argo Rollouts / Flagger for metric-gated canaries so a bad version is caught and rolled back automatically, with the policy itself in Git.
16. Keep the agent’s own config in Git too — manage Argo CD/Flux via GitOps (self-management) so the platform is as reproducible and auditable as the apps it deploys.

References

• OpenGitOps principles
• Argo CD documentation · ApplicationSet
• Flux documentation · GitOps Toolkit
• Sealed Secrets
• SOPS
• External Secrets Operator
• Argo Rollouts · Flagger
• Kargo (promotion)
• kubeconform · Conftest / OPA
• CNCF GitOps Working Group
• roadmap.sh — DevOps: GitOps