The Day Our Green CI Pipeline Deployed an 11-Day-Old Bug to Production

 

The Day Our Deployment Pipeline Deployed the Wrong Version to Production

We had 47 green checkmarks. Every test passed. CI was happy. The deploy button was practically begging to be pressed.

I pressed it.

Twenty minutes later, a customer support ticket came in: "The export feature is completely broken. It's showing data from three weeks ago."

Then another. Then five more.

We hadn't broken the export feature. We had deployed a version of the code from eleven days earlier — quietly overwriting a critical bug fix and reintroducing a data staleness issue we'd already resolved.

The scariest part? Our pipeline said everything was successful. No errors. No warnings. Green across the board. We had just shipped the wrong version of our own application with total confidence.

Here's how that happened, and the changes we made so it can't happen again.

---

The Setup: A Pipeline That Looked Solid

Our deployment flow looked reasonable on paper:

Push to main → Run tests → Build Docker image → Push to registry → Deploy to production

Jenkins pipeline, roughly:

groovy

pipeline {
    agent any
    
    stages {
        stage('Checkout') {
            steps {
                git branch: 'main', url: 'https://github.com/company/order-service.git'
            }
        }
        
        stage('Test') {
            steps {
                sh 'mvn test'
            }
        }
        
        stage('Build') {
            steps {
                sh 'docker build -t order-service:latest .'
            }
        }
        
        stage('Push') {
            steps {
                sh 'docker push registry.company.com/order-service:latest'
            }
        }
        
        stage('Deploy') {
            steps {
                sh 'kubectl set image deployment/order-service order-service=registry.company.com/order-service:latest'
            }
        }
    }
}

Notice anything? I didn't, for months. Everything is tagged :latest.

That single word — latest — was about to cause a very bad Tuesday.

---

The Failure: How "Latest" Betrayed Us

Here's the actual sequence of events, once we reconstructed it from logs:

Step 1: A teammate had an old feature branch, checked out locally from 11 days earlier, that he was using to test something unrelated on his machine.

Step 2: He accidentally ran a local script that pushed a Docker image tagged order-service:latest directly to our registry — bypassing CI entirely — while testing a local Docker build command he'd copy-pasted from an old note.

Step 3: Fifteen minutes later, an unrelated, completely correct PR was merged to main. CI ran, all tests passed, and it triggered a deploy.

Step 4: Here's the killer: our Kubernetes deployment was already configured to pull :latest, and due to imagePullPolicy: IfNotPresent on some nodes, several pods didn't even re-pull the image — they just kept running whatever :latest happened to resolve to on that node, which was now his 11-day-old accidental push.

Step 5: New pods scheduled onto different nodes pulled the "real" latest deploy. Old pods on other nodes kept running the stale one.

We ended up with a production cluster running two different versions of the same service simultaneously — split roughly across nodes — for 20+ minutes, with zero visibility into it.

That's not a deployment bug. That's an entire deployment strategy bug.

---

Why This Happens (The Concept)

This incident had three separate root causes stacked on top of each other:

1. Mutable Tags (:latest is a Trap)

:latest isn't a version — it's a moving pointer. Anyone, anywhere, with push access can silently redefine what "latest" means. There's no audit trail tying a specific deploy to a specific, immutable artifact.

2. No Deployment Provenance

We had no way to answer the question: "What commit SHA is actually running in production right now?" Not without SSH-ing into a pod and checking manually.

3. Inconsistent Image Pull Behavior

imagePullPolicy: IfNotPresent means Kubernetes won't re-pull an image if it already has something cached under that tag name — even if the registry's :latest has moved on. Different nodes, different cache states, different code running.

Put together: a bypass of CI + a mutable tag + inconsistent pull behavior = two versions of production running side-by-side with a green pipeline the whole time.

---

The Fix: Building a Pipeline You Can Actually Trust

Fix #1: Never Deploy :latest. Tag by Commit SHA.

This is the single highest-leverage change we made.

groovy

pipeline {
    agent any
    
    environment {
        IMAGE_TAG = "${env.GIT_COMMIT.take(8)}"
    }
    
    stages {
        stage('Checkout') {
            steps {
                checkout scm
                script {
                    env.IMAGE_TAG = sh(script: "git rev-parse --short=8 HEAD", returnStdout: true).trim()
                }
            }
        }
        
        stage('Test') {
            steps {
                sh 'mvn test'
            }
        }
        
        stage('Build') {
            steps {
                sh "docker build -t order-service:${IMAGE_TAG} ."
            }
        }
        
        stage('Push') {
            steps {
                sh "docker push registry.company.com/order-service:${IMAGE_TAG}"
            }
        }
        
        stage('Deploy') {
            steps {
                sh "kubectl set image deployment/order-service order-service=registry.company.com/order-service:${IMAGE_TAG} --record"
            }
        }
    }
}

Now every image is immutable and traceable to an exact commit. order-service:a3f92c1e will always be exactly that code, forever. No ambiguity, no accidental overwrites, no "which version is this actually running."

Rule I now enforce on every project: if your deployment artifact's tag can mean something different tomorrow than it does today, you don't have reproducible deployments — you have a guessing game.

---

Fix #2: Block Manual Pushes to the Registry

The teammate's accidental push should never have been possible in the first place.

yaml

# Registry access policy (example: AWS ECR repository policy)
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OnlyCIPipelineCanPush",
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
        "ecr:PutImage",
        "ecr:InitiateLayerUpload",
        "ecr:UploadLayerPart",
        "ecr:CompleteLayerUpload"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalArn": "arn:aws:iam::123456789:role/jenkins-ci-role"
        }
      }
    }
  ]
}

Only the CI service role can push images now. Individual developer credentials — however well-intentioned — cannot write directly to the production registry. If it doesn't go through the pipeline, it doesn't exist as a deployable artifact.

---

Fix #3: Explicit imagePullPolicy: Always

yaml

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: order-service
          image: registry.company.com/order-service:a3f92c1e
          imagePullPolicy: Always   # No ambiguity, no stale caches

Combined with commit-SHA tagging, this is almost redundant now (since each deploy uses a unique tag), but it's cheap insurance against any node serving a cached image under a tag it shouldn't.

---

Fix #4: A Deployment Manifest That Answers "What's Running Right Now?"

We built a tiny endpoint that every service exposes, populated at build time:

java

@RestController
public class VersionController {
    
    @Value("${app.build.commit:unknown}")
    private String commitSha;
    
    @Value("${app.build.timestamp:unknown}")
    private String buildTimestamp;
    
    @GetMapping("/actuator/version")
    public Map<String, String> version() {
        Map<String, String> info = new HashMap<>();
        info.put("commit", commitSha);
        info.put("builtAt", buildTimestamp);
        return info;
    }
}

Injected at build time:

groovy

stage('Build') {
    steps {
        sh """
            docker build \
              --build-arg GIT_COMMIT=${IMAGE_TAG} \
              --build-arg BUILD_TIME=\$(date -u +%Y-%m-%dT%H:%M:%SZ) \
              -t order-service:${IMAGE_TAG} .
        """
    }
}

Now, any time, from any environment: curl production-url/actuator/version tells you exactly what's running — no SSH, no guessing, no "let me check with the team who deployed last."

During an incident, this alone can save 20 minutes of "wait, are we even sure what version is live?"

---

Fix #5: A Rollback That Takes 30 Seconds, Not 30 Minutes

Because every deploy is tagged with an immutable commit SHA, rollback became trivial:

bash

# Before: no idea what "the previous version" even means with :latest
# After: exact, unambiguous rollback

kubectl set image deployment/order-service \
  order-service=registry.company.com/order-service:PREVIOUS_KNOWN_GOOD_SHA \
  --record

kubectl rollout status deployment/order-service

We also added a one-command rollback script that pulls the last known-good SHA from our deployment history automatically:

bash

#!/bin/bash
# rollback.sh
LAST_GOOD_SHA=$(kubectl rollout history deployment/order-service | tail -2 | head -1 | awk '{print $1}')
kubectl rollout undo deployment/order-service --to-revision=$LAST_GOOD_SHA
echo "Rolled back to revision: $LAST_GOOD_SHA"

What used to be a frantic 30-minute scramble (figure out what broke → find the last good version → manually redeploy it → hope it works) became a single command with a predictable outcome.

---

The Results

Metric	Before	After
Time to identify "what's currently deployed"	15-20 mins (manual investigation)	<10 seconds (/actuator/version)
Time to roll back a bad deploy	25-30 mins	Under 2 mins
Unauthorized/manual registry pushes possible	Yes	No (IAM-blocked)
Ambiguous "latest" deployments	Constant risk	Eliminated
Incidents caused by tag confusion	1 major (this one)	0 since

The bigger shift wasn't any single fix — it was realizing that a deployment pipeline's job isn't just "make the code run in production." It's "make it provable which code is running in production, and make undoing a mistake boring."

---

The Lessons

1. :latest is not a deployment strategy. It's a liability wearing a deployment strategy's clothes.
If your tags aren't immutable, your deployments aren't reproducible — and if they're not reproducible, you can't debug them with confidence.

2. CI passing green doesn't mean production is running what you think it's running.
Those are two separate claims. Bridge that gap explicitly (version endpoints, deployment records) instead of assuming they're the same thing.

3. Anything that CAN bypass your pipeline, eventually WILL bypass your pipeline.
Not out of malice — out of a well-meaning developer testing something locally at the wrong moment. Lock down write access to production artifacts at the infrastructure level, not just as a team policy.

4. Rollback speed matters more than deploy speed.
Everyone optimizes for fast deploys. Fewer teams optimize for "how fast can we undo a mistake at 2 AM." That second number is the one that determines how bad your worst incident gets.

5. "It passed CI" and "it's safe" are different sentences.
Tests validate code behavior. They say nothing about whether the artifact that reaches production is the artifact you think it is. You need both.

---

What I'd Tell a Team Setting Up CI/CD Today

Don't just ask "how do we deploy fast?" Ask "if this deploy goes wrong at 2 AM, how do we prove what's running, and how fast can we undo it?"

If you can't answer that second question in under a minute, you don't have a deployment pipeline — you have a deployment hope.

Tag by commit SHA. Lock down your registry. Expose a version endpoint. Build a rollback script before you need it, not while you need it.

Because the day you need it will not send you a warning.

---

Questions? Comments? Drop them below — I read and reply to every one.

---

Related Reading:

Why Your Spring Boot API is Slow: The N+1 Query Problem (And How I Fixed It in Production)

How My Microservices Architecture Collapsed (And How I Fixed It)   

Build an AI Lead Qualification Agent with Spring Boot