Security triage (CVE playbook)

Goal

Triage a CVE finding (container scanning, dependency scanning, or SAST) and decide fix vs. dismiss with cited evidence.

Triple-source verification

For every distinct CVE, verify with three independent sources before any decision — a scanner or a single web page is not proof.

1. Installed == latest candidate. Run apt-cache policy <pkg> inside the same base image. If the installed version equals the repo candidate, no package bump helps.
2. Trivy fix[] empty? Extract from the promote-stage gl-container-scanning-report.json artifact.
3. Debian Security Tracker (security-tracker.debian.org/tracker/<CVE>): vulnerable / postponed vs fixed in <version> per suite (bookworm vs trixie differ).

Classification buckets

The buckets above are the container-scanning reading. For a SAST finding whose code ships but whose dangerous path is gated (a vendored dependency compiled into a binary, behind a config flag the deployment flavors keep off), NOT_APPLICABLE would claim the code is unreachable — it is reachable in the dev flavors and in any build made without the flag. Use MITIGATING_CONTROL there, and say in the comment where the code ships, where it is reachable, and which control keeps it out of the deployment.

Dismissal comment semantics

“awaiting upstream” is not an honest label — we are affected.

Use [affected-no-fix]:

We are affected by X in Y (present in the latest scan of <image>). No upstream fix exists (triple-source verified <date>). Risk accepted until a fix ships — the monthly rebuild surfaces it automatically when Debian publishes.

Every dismissal comment references the triage doc.

RESOLVED vs DISMISSED

• RESOLVED — the vulnerability is gone (package eliminated from the image, proven by absence from the latest scan artifact).
• DISMISSED — still present but consciously accepted.

Never conflate the two.

Bulk execution pattern

1. Build the decision set offline (dry-run). Classify every finding, save the mapping, print per-image/per-bucket counts, ASSERT the exact expected totals per bucket before touching the API.
2. Spot-check N random decisions against the source artifacts.
3. Execute via GraphQL with 0.15–0.2 s spacing, logging every HTTP result. Make the script resumable (append OK <uuid> lines and skip those on re-run). Raw logs are never committed.
4. Reconcile afterwards: re-count states via GraphQL and match the executed totals.

Reducing container CVEs (build-side playbook)

• Audit the runtime tool inventory before slimming an image. Three blind spots CI cannot see: entrypoint scripts, compose healthchecks, runtime subprocess use.
• Drop the build toolchain from runtime stages. node:22 ships build-essential/python3/make — switch the runtime stage to node:22-slim. For Python images with compiled wheels, use a multi-stage build.
• apt-get upgrade -y layers in every runtime image to pull current Debian security updates at build time.
• Runtime smoke tests on the PROMOTED images after each hardening MR: pull the tag, verify every tool the entrypoint/healthcheck needs, run the heavy-import set, check baked-in model files.

Monthly loop

• Pipeline schedule (cron 0 4 1 * * Europe/Paris) sets FORCE_IMAGE_REBUILD=true: rebuilds all 15 images, rescans, promotes, then the E2E suite validates the fresh images.
• Manual trigger:

  GITLAB_HOST=opensource.unicc.org glab ci run -b main \
    --variables 'FORCE_IMAGE_REBUILD:true'
  

• After each monthly rebuild: re-triage. Any finding with a fix that now applies = fixable. New findings = triple-source triage. Stale detected findings whose CVEs are absent from the new scans = resolve. Update the wave table in the triage doc.

Instance facts (GitLab 18.2.8-ee)

• REST dismiss/resolve endpoints are unreliable for findings without a Vulnerability entity.
• GraphQL is the only reliable path — mutation { securityFindingDismiss(input: {uuid, comment, dismissalReason}) { ... } }. For old findings whose raw record has aged out, use mutation { vulnerabilityDismiss(input: {id: "gid://gitlab/Vulnerability/<n>", comment, dismissalReason}) { ... } }.
• Dismissal reason enum: ACCEPTABLE_RISK | FALSE_POSITIVE | MITIGATING_CONTROL | USED_IN_TESTS | NOT_APPLICABLE.
• No auto-resolve: since GitLab 17.0, findings no longer seen by the latest default-branch scan stay detected forever until manually transitioned.
• SAST finding ids are not stable. An id is built from the rule id (with its revision suffix), the analyzer’s tracking signature and the location; an analyzer version bump changes the tracking signature and re-mints the id. A dismissed SAST finding therefore only stays dismissed while the analyzer keeps emitting that exact id — the durable state is “the analyzer no longer produces this finding at all”: fix the code, or exclude the path from scanning.
• SAST_EXCLUDED_PATHS overrides GitLab’s default excluded paths (spec, test, tests, tmp). Each comma-separated element is a gitignore-style pattern for the analyzer’s own pre-filter and a doublestar pattern for GitLab’s post-filter. Ruby File.fnmatch — under which a trailing /** does not match a file inside the directory — is the Secret Detection dialect, so a SAST pattern must not be copied into SECRET_DETECTION_EXCLUDED_PATHS as-is.

Related

• How to open a MR
• Internal → CVE-REMEDIATION.md
• Internal → security triage doc