Supply chain security, secrets, and the security pipeline

What it is

Three overlapping concerns that together answer "can I trust the code that runs in production, and the credentials it uses":

Supply chain security is about trusting code you did not write: your dependencies, your build system, and the artifacts you deploy. The threat is that a compromise upstream of you (a malicious package, a tampered build, a poisoned base image) executes in your environment with your privileges.

Secrets management is about credentials: keeping them out of source, rotating them, and detecting when they leak. The threat is a leaked credential used from anywhere.

The security pipeline is where both are enforced: the stages in CI/CD that scan code, dependencies, images and infrastructure, and the admission controls that decide what may run.

What this is confused with: treating these as scanning problems. Scanning finds known bad things. The harder half is provenance (proving an artifact is what you think it is) and elimination (removing static credentials so there is nothing to leak), and scanning is the part that catches what provenance and elimination missed.

The problem it solves

Most of the code in production is not yours. A typical application is a few thousand lines of your code and tens of thousands of transitive dependencies, and each dependency runs with your application's privileges.

The attacks this space addresses are not hypothetical:

SolarWinds (2020):     the BUILD SYSTEM was compromised, injecting a backdoor
                       into signed, legitimately-distributed updates.
                       Scanning the source would not have found it.
Codecov (2021):        a leaked credential in a CI script exfiltrated
                       secrets from thousands of downstream CI pipelines.
event-stream (2018):   a maintainer handed a popular package to an attacker
                       who added a wallet-stealing payload in a dependency.
dependency confusion:  publishing a public package with the name of an
                       internal one, so the build pulls the attacker's.
xz/liblzma (2024):     a multi-year social-engineering campaign inserting a
                       backdoor into a compression library used by sshd.

Every one of these bypassed source-code review, because the malicious code arrived through the build, the dependency tree, or a leaked credential rather than through a pull request. That is the argument for the whole discipline: the pull request is not where the supply-chain threat enters.

Mechanics

Provenance: SBOM, SLSA, Sigstore

An SBOM (Software Bill of Materials) is an inventory of everything in an artifact: every dependency, version and license.

// CycloneDX / SPDX: generated at build time.
{ "components": [
    { "name": "log4j-core", "version": "2.14.1",
      "purl": "pkg:maven/org.apache.logging.log4j/log4j-core@2.14.1" }
]}

The SBOM's value is answering "am I affected" in minutes rather than days. When Log4Shell broke, teams with SBOMs queried them; teams without spent days grepping build files across hundreds of services. The SBOM is the artifact that makes a zero-day response tractable.

SLSA (Supply-chain Levels for Software Artifacts) is a framework for build integrity, with levels of increasing assurance:

SLSA L1:  provenance exists (the build records how it built the artifact)
SLSA L2:  provenance is signed, hosted build service
SLSA L3:  the build is isolated and non-falsifiable; source and build
          are verified

SLSA addresses the SolarWinds threat specifically: L3 requires that the build cannot be tampered with even by someone with access to the build system, which is exactly what SolarWinds subverted. The provenance answers "this artifact was built from this source by this builder," verifiably.

Sigstore is the signing infrastructure that makes provenance practical:

# Sign an artifact with a short-lived, identity-based certificate.
cosign sign --yes registry.example/app@sha256:abc...

# Verify it was signed by the expected identity.
cosign verify --certificate-identity build@example.com \
              --certificate-oidc-issuer https://accounts.google.com \
              registry.example/app@sha256:abc...

Sigstore's innovation is keyless signing: instead of a long-lived signing key that can be stolen, it issues a short-lived certificate bound to an OIDC identity (the CI job's identity), records it in a public transparency log (Rekor), and the certificate expires in minutes. There is no signing key to steal, which is the same "eliminate the static credential" reasoning as workload identity on the SSRF page.

Dependency confusion, and the fix

Internal package:   @company/auth-utils, on your private registry.
Attacker publishes: @company/auth-utils on the PUBLIC npm registry,
                    version 99.0.0.
Your build:         npm sees a higher version on the public registry
                    and pulls the ATTACKER'S package.

The attack exploits the resolver preferring the highest version across all configured registries. The fixes:

1. Scope internal packages to a namespace and configure the registry
   to serve ONLY that namespace from the private registry.
2. Use a lockfile with integrity hashes, so a swapped package fails
   verification.
3. Reserve your internal package names on the public registry
   (publish empty placeholders), so an attacker cannot claim them.
4. A pull-through proxy (Artifactory, Nexus) as the single source,
   configured to never fall back to public for internal namespaces.

The lockfile with integrity hashes is the most broadly effective, because a swapped package has a different hash and the install fails. A build without a committed lockfile is resolving dependencies fresh each time, which is the condition dependency confusion needs.

Secrets: dynamic credentials over rotation

The hierarchy of secret handling, worst to best:

1. Hardcoded in source            -> in git history forever
2. In environment variables       -> in the process, in crash dumps, in logs
3. In a secrets manager, static   -> better, but still a long-lived secret
4. DYNAMIC, short-lived           -> generated on demand, expires in minutes
5. NO SECRET (workload identity)  -> nothing to leak

Dynamic secrets are the change that matters. Instead of a static database password, the application asks a secrets manager for a credential that is created on demand and revoked minutes later:

# Vault generates a database credential per request, valid for 1 hour.
vault read database/creds/app-role
# -> a UNIQUE username and password, auto-revoked after the lease.

A leaked dynamic credential is worthless within the hour, and because each request gets a unique credential, a leak is traceable to the request that leaked it. That is the difference between a static secret (one leak, permanent, untraceable) and a dynamic one (bounded, attributable).

Rotation of static secrets is the fallback, and it is harder than it sounds because rotation must be zero-downtime: the old credential must remain valid while the new one propagates, which means a window where both work.

The rotation problem: you cannot atomically swap a credential across
N running instances. Either:
  - dual-validity: both old and new work during a window (most common)
  - or a coordinated restart, which is downtime

Secret scanning of history is the detective control:

# Scan the entire git history, not just the working tree.
gitleaks detect --source . --log-opts="--all"
trufflehog git file://. --since-commit HEAD~1000

A secret committed and then deleted is still in the history, so scanning must cover history, and a leaked secret must be rotated, not just removed from the repo, because it was public the moment it was pushed. "We deleted the commit" is not remediation; the credential was exposed and must be revoked.

The security pipeline

The stages, each catching a different class:

COMMIT/PR:
  SAST (static analysis)     -> code-level bugs: injection, hardcoded secrets
  secret scanning            -> credentials in the diff
  SCA (software composition) -> known-vulnerable dependencies (CVEs)
  IaC scanning               -> misconfigured Terraform/CloudFormation
                                (public S3 bucket, open security group)

BUILD:
  SBOM generation            -> the inventory
  artifact signing           -> Sigstore/cosign
  container scanning         -> vulnerable OS packages in the image

DEPLOY (admission):
  signature verification     -> reject unsigned images
  policy (OPA/Kyverno)       -> reject privileged pods, images from
                                untrusted registries, missing labels

RUNTIME:
  vulnerability re-scanning  -> a CVE disclosed AFTER deploy
  runtime detection (Falco)  -> anomalous syscalls, unexpected network

The distinction that matters: shift-left catches known issues cheaply, and admission control is the enforcement point. A SAST finding in a PR is a comment; an unsigned image at admission is a hard block. Admission control is where "we scanned it" becomes "it cannot run," and it is the stage teams most often lack.

# Kyverno: reject any image not signed by the expected identity.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
spec:
  rules:
  - name: verify-signature
    match: {any: [{resources: {kinds: [Pod]}}]}
    verifyImages:
    - imageReferences: ["registry.example/*"]
      attestors:
      - entries:
        - keyless:
            subject: "build@example.com"
            issuer: "https://accounts.google.com"

The false-positive problem is what determines whether the pipeline is used or bypassed. A scanner that flags 400 dependency CVEs, most unreachable, trains developers to ignore it. The pipeline's usability is a security property: a noisy gate gets disabled, and a disabled gate is worse than none because it creates the belief that scanning is happening.

A worked example: a pipeline that everyone bypassed

A fintech platform. A security pipeline existed, and developers routinely used the skip-security-checks label to merge, because the pipeline blocked on 400-plus dependency CVEs per PR, most of them in transitive dependencies of test tooling.

Baseline:

PRs merged with security checks skipped:  68%
mean CVEs flagged per PR:                  412
of which reachable/exploitable:            ~3
mean time developers spent triaging:       0 (they skipped)
secrets in git history:                    unknown (never scanned)
images signed:                             no
admission control:                         none
dependency confusion protection:           none

Sixty-eight percent bypass is a pipeline that does not exist. The gate was so noisy that skipping it was the norm, which meant the 3 real CVEs per PR were being ignored alongside the 409 false positives.

Fix 1: reachability, not just presence. The SCA tool flagged every CVE in the dependency tree; most were in code paths never executed.

# Before: flag every CVE in a dependency.
# After: flag CVEs in code paths the application actually reaches
#        (reachability analysis), and use EPSS to prioritise the rest.

CVEs flagged per PR:              412 -> 3-8
of which reachable:               all of them (by construction)
developer triage per PR:          skipped -> ~4 minutes, actually done
skip-label usage:                 68% -> 4%

Reachability analysis took the flag count from 412 to a single digit, and the moment the signal was real, developers stopped skipping. The pipeline became usable and therefore used, which is the whole point: a gate is only a control if it is not bypassed.

Fix 2: secret scanning, which found the expected.

$ gitleaks detect --source . --log-opts="--all"

secrets found in history:           41
  active at the time of scan:        12   (never rotated after commit)
  of which high-value (DB, cloud):    4

Twelve credentials committed to history were still valid. All were rotated, and the four high-value ones were investigated for use (none confirmed, but the logs did not go back far enough to be certain, which is its own finding).

# Added: pre-commit hook + CI gate, blocking on any new secret.
- id: gitleaks
  entry: gitleaks protect --staged

Fix 3: dynamic database credentials, replacing static.

before:  a static DB password in a Kubernetes Secret, shared by all
         replicas of a service, never rotated.
after:   Vault dynamic credentials, unique per pod, 1-hour lease.

static DB credentials:            14 -> 0
mean credential lifetime:         permanent -> 1 hour
credential leak traceability:     none -> per-pod

Fix 4: signing and admission control.

build:   cosign keyless signing of every image, provenance to Rekor.
deploy:  Kyverno policy rejecting any unsigned image or any image not
         from the internal registry.

unsigned images that could run:   any -> none
images from arbitrary registries: allowed -> blocked

The admission control caught a real issue within a week: a developer had referenced a public redis:latest image directly in a manifest, bypassing the internal registry and its scanning. The policy blocked it, which is the enforcement the shift-left scanning could not provide.

Fix 5: dependency confusion protection.

- committed lockfiles with integrity hashes, enforced in CI
- internal packages scoped to @company/, served only from the private
  registry via a pull-through proxy
- placeholder packages reserving the @company names on public npm

Fix 6: SBOM generation, which paid off during the next zero-day.

A CVE was disclosed in a widely-used library.
  before this work: a day of grepping build files across 60 services.
  after: an SBOM query returned the 4 affected services in ~2 minutes.

Final:

                              before      after
skip-label usage              68%         4%
CVEs flagged per PR           412         3-8 (all reachable)
active secrets in history     12          0 (rotated) + gate
static DB credentials         14          0 (dynamic, 1-hour)
images signed                 0%          100%
admission control             none        Kyverno (signature + registry)
dependency confusion          possible    blocked (lockfile + scoping)
zero-day "am I affected"      ~1 day      ~2 min (SBOM)

The single most important change was reachability analysis, because it converted a bypassed gate into a used one, and every other control depended on the pipeline actually running. A noisy pipeline is a disabled pipeline, and the 68 percent bypass rate meant the platform had the appearance of supply-chain security and none of the substance.

Production evidence

The SolarWinds compromise is the canonical build-system attack and the direct motivation for SLSA, which was created by Google and is now a project of the Open Source Security Foundation (OpenSSF). SLSA L3's non-falsifiable-provenance requirement targets exactly the SolarWinds threat.

Sigstore (cosign, Fulcio, Rekor) is an OpenSSF project used by Kubernetes, the npm registry (which now supports Sigstore provenance) and many others. Keyless signing with a transparency log is its defining contribution, and its adoption by package registries is the strongest signal that provenance is becoming table stakes.

Dependency confusion was demonstrated by Alex Birsan (2021), who got code execution inside Apple, Microsoft, PayPal and dozens of others by publishing public packages with internal names. The fix (namespace scoping and lockfiles) is documented by every major package ecosystem in response.

HashiCorp Vault's dynamic secrets are the reference implementation of on-demand, short-lived credentials, and the cloud providers' equivalents (AWS Secrets Manager rotation, short-lived STS credentials) reflect the same direction: away from static long-lived secrets.

Executive Order 14028 (2021) mandated SBOMs for software sold to the US federal government, which drove SBOM tooling from niche to standard. The CISA guidance on SBOM formats (SPDX, CycloneDX) is the reference.

gitleaks and trufflehog are the standard secret-scanning tools, and GitHub's own secret scanning (with push protection) reflects that finding secrets in history and blocking them at push is now a platform feature rather than an add-on.

The debate

Is scanning enough? No, and this is the central position. Scanning finds known-bad things, and SolarWinds, the xz backdoor and dependency confusion all bypassed source scanning. The durable controls are provenance (SLSA, signing) and elimination (dynamic secrets, workload identity), and scanning is the layer that catches what those missed. A team with only scanning has the visible half.

What is the highest-value single change? Making the pipeline usable, because a bypassed gate is not a control. In the worked example reachability analysis took CVE flags from 412 to a single digit and the skip rate from 68 percent to 4, and every other control was worthless while the pipeline was being skipped. Usability is a security property, and it is the one that determines whether the rest of the pipeline runs.

Dynamic secrets or rotation? Dynamic, where the secrets manager supports it, because a leaked short-lived credential is worthless within the hour and attributable to the request that leaked it. Rotation is the fallback for credentials that cannot be dynamic, and it is harder than it sounds because zero-downtime rotation needs a dual-validity window. The best outcome is no secret at all: workload identity for anything that can use it.

Is SLSA L3 realistic? For most organisations, L1 or L2 is the practical target, and L3's isolated-non-falsifiable-build requirement is significant work. The value ramps quickly: L1 (provenance exists) plus signing plus admission verification catches the arbitrary-image and unsigned-artifact cases, which are the common ones, and L3 addresses the sophisticated build-compromise case that is rarer. Chasing L3 before L1 is misordered.

How do you handle the false-positive problem? Reachability analysis for CVEs (is the vulnerable code path actually reached), EPSS for prioritising the rest (see CVSS vs EPSS vs KEV), and a hard rule that a gate blocking on non-exploitable findings will be bypassed. The pipeline's job is to surface the three things that matter, not the four hundred that do not, and a scanner that cannot distinguish them is worse than a slower one that can.

Admission control or shift-left? Both, and they are not substitutes. Shift-left catches issues cheaply and early as advisory; admission control is the enforcement point where "we scanned it" becomes "it cannot run." Most teams have shift-left and lack admission control, which means their scanning is advisory and bypassable, and the developer who references a public latest image proves it.

Follow-up Q&A

"Why isn't scanning source code enough for supply-chain security?"

Because the threats bypass source review. SolarWinds compromised the build system and injected a backdoor into legitimately-signed updates; the xz backdoor was in a dependency inserted over years by a trusted maintainer; dependency confusion pulls an attacker's package the resolver prefers. None of these appear in a pull request. The durable controls are provenance (proving an artifact was built from known source by a known builder, via SLSA and signing) and elimination of static credentials, with scanning catching what those miss.

"What is an SBOM and why does it matter?"

A Software Bill of Materials: an inventory of every dependency and version in an artifact. Its value is answering "am I affected" in minutes when a CVE drops, rather than days of grepping build files across services. In one case an SBOM query returned the four affected services in two minutes for a vulnerability that would previously have taken a day to scope. It is the artifact that makes a zero-day response tractable.

"How does dependency confusion work and how do you stop it?"

An attacker publishes a public package with the name of your internal one at a high version, and the resolver, preferring the highest version across all configured registries, pulls the attacker's. The fixes are a committed lockfile with integrity hashes (a swapped package fails verification), namespace scoping so internal packages come only from the private registry, reserving your internal names on the public registry, and a pull-through proxy configured never to fall back to public for internal namespaces. The lockfile is the most broadly effective.

"Dynamic secrets or rotation?"

Dynamic where possible: the application requests a credential created on demand and revoked in an hour, unique per request, so a leak is worthless quickly and attributable to the request that leaked it. Rotation is the fallback for credentials that cannot be dynamic, and it is harder than it sounds because you cannot atomically swap a credential across running instances, so you need a dual-validity window where both old and new work. The best outcome is no secret at all, via workload identity.

"What makes a security pipeline actually get used?"

Signal quality. A gate that flags 400 dependency CVEs, most unreachable, gets bypassed, and in one case 68 percent of PRs skipped the checks with a label. Reachability analysis took the flag count to a single digit and the skip rate to 4 percent, and only then were the real findings acted on. Usability is a security property: a noisy gate is a disabled gate, and a disabled gate is worse than none because it creates the belief that scanning is happening.

"Where is enforcement, versus advisory scanning?"

Admission control. Shift-left SAST, SCA and IaC scanning in the PR are advisory: a comment, and bypassable. Admission control at deploy (signature verification, an OPA or Kyverno policy) is where "we scanned it" becomes "it cannot run." Most teams have the shift-left half and lack admission control, so their scanning is advisory, and the developer who references a public redis:latest image directly is the proof: only the admission policy blocks it.

Common misconceptions

"Supply-chain security is dependency scanning." Scanning finds known-vulnerable dependencies. The build-system compromise (SolarWinds), the trusted-maintainer backdoor (xz) and dependency confusion all bypass it. Provenance and elimination are the durable controls.

"Deleting a leaked secret from git is remediation." It was public the moment it was pushed, and it is still in history. The credential must be rotated, and scanning must cover history, not just the working tree.

"A committed image is safe because we scanned it." A CVE can be disclosed after deployment, and an unsigned or arbitrary-registry image bypasses the scan entirely. Admission control and runtime re-scanning are what cover the gap.

"More scanning is more security." A noisy scanner gets bypassed, and a bypassed gate is worse than none. Signal quality (reachability, EPSS) determines whether the pipeline is used.

"SLSA L3 is the goal." L1 plus signing plus admission verification catches the common cases. L3 addresses the sophisticated build compromise and is significant work; chasing it before L1 is misordered.

Interview delivery note

Say this verbatim: "The supply-chain threats bypass source review: SolarWinds was the build system, xz was a trusted maintainer, dependency confusion is the resolver preferring the attacker's version. So the durable controls are provenance and eliminating static credentials, and scanning catches what those miss. And the pipeline only works if it is usable: a gate flagging 400 CVEs gets a skip label 68 percent of the time, and reachability analysis taking that to a single digit is what made the real findings get acted on." The threat framing and the usability insight.

The senior-versus-staff separator is usability as a security property. A senior engineer builds a comprehensive pipeline with SAST, SCA, secret scanning and image scanning. A staff engineer knows that a pipeline flagging 400 findings will be bypassed, that a bypassed gate is worse than none because it creates false assurance, and that reachability analysis and EPSS to surface the three findings that matter is what determines whether any of it works. Signal quality over coverage is the judgement.

The second signal is admission control as the enforcement point. Distinguishing advisory shift-left scanning from the deploy-time gate where "we scanned it" becomes "it cannot run," and noting that most teams lack the latter, shows you know where a scan becomes a control.

The Staff & Team Lead Interview Reference