GitOps Breaks in Production, Not in the Diagram

The diagram always works

On the slide, GitOps is four clean words: declarative, versioned, automated, reconciled. Git holds the desired state, an agent makes the cluster match, and a rollback is just a revert. Every part of that is true, and none of it is where teams get stuck.

They get stuck on contact with production. The model does not break in the diagram; it breaks on the three things the diagram leaves out: state that drifts, secrets that cannot live in Git, and changes that have to move between environments without a human hand-editing production. Get those three right and GitOps holds. Get them wrong and you have a very tidy repository that does not match reality.

Where it actually breaks

Drift: someone runs a quick manual edit during an incident, and now the cluster and the repository disagree. Without reconciliation, that difference is invisible until it bites.

Secrets: the desired state includes credentials, and credentials cannot sit in Git in plain text. This is the single most common place GitOps adoption stalls.

Promotion: a change that works in staging has to reach production as the same change, reviewed, not retyped by hand under pressure.

Drift is the feature, if you let it reconcile

Drift is not a failure of GitOps; catching it is the entire point. A reconciler continuously compares the running cluster to what the repository declares and pulls it back into line.

The discipline is to let it. Once reconciliation is on, that emergency manual edit gets reverted automatically, which feels wrong the first time and right forever after, because it means the repository is the truth and not a suggestion. When you genuinely need an emergency change, the rule is simple: make it in Git, or make it and immediately get it back into Git.

Keep secrets out of Git

You do not put secrets in the repository; you put a reference in the repository and keep the secret in a real store. An external secrets operator syncs from a managed vault or your cloud provider's secret service into the cluster, so the desired state stays declarative and the credential never lands in version control. Pick one approach and standardize on it, because half a team encrypting files and half a team syncing from a store is its own kind of drift.

Promote, do not hand-edit

Moving a change between environments is where discipline pays off. Keep environment differences small and explicit, an overlay or a values file, not a fork, so promotion is a reviewed change from one path to the next rather than a person editing production live. The change that reached staging is the change that reaches production, and Git records who moved it and when.

What Git as the source of truth buys you

Once drift reconciles, secrets stay out of the repository, and promotion is a reviewed path, GitOps delivers what the diagram promised: every change is a commit, every rollback is a revert, and the audit trail writes itself. It is not a complete platform on its own, you still need CI, observability, and an incident process, but as the operating model for how state changes, it is the one that survives production.