Setting Retention Policies for Flag Audit Logs
This how-to is part of Building Audit Trails for Compliance. Your flag audit log grows forever, which is both a storage cost and a compliance liability — keeping records too long can violate data-minimization rules, deleting them too soon can fail an audit. This how-to sets a defensible retention policy: classify each record, enforce a retention window with tiered storage, keep the log immutable within the window, and produce the evidence that the policy is actually applied.
The trap most teams fall into is treating retention as a single number applied to the whole log — “keep everything for two years” — because that is the easiest lifecycle rule to write. It is also the one that fails both ways at once: two years is far too long for a record carrying an end user’s identifier under a data-minimization regime, and far too short for the payment-flag change a financial auditor will ask about in year five. A defensible policy is per-record, driven by why each record exists rather than by a convenient default. The work below is mostly about making that distinction cheap to compute and impossible to skip, so a record’s window is decided the moment it is written rather than argued about the day an auditor arrives.
Prerequisites
Step-by-Step Procedure
Step 1 — Classify records into retention classes
Map each record type to a retention class based on what compels keeping it — compliance evidence, security forensics, or operational history — and the shortest lawful maximum for any personal data it contains. The classification function must be deterministic and depend only on the record itself, never on wall-clock time or external lookups, because you will re-run it years later to decide whether an old record is eligible for deletion and it must return the same class it did on day one. Pin the class into the record at write time as an immutable field, and treat the function as the authority only for records written before that field existed — if the classifier’s logic drifts, a stored class protects you from silently re-shortening or extending windows on historical data.
Prefer namespaced flag keys as the classification signal, as in the ops.payments. and ops.auth. prefixes above, so a new payment flag inherits the compliance window without anyone remembering to tag it. The failure mode here is a record that should have been “compliance” defaulting to “operational” and getting purged after 90 days — a gap an auditor will find precisely because the surrounding records survived. When in doubt, a record escalates to the longer window, never the shorter.
# classify.py — retention class drives the window
RETENTION_DAYS = {"compliance": 2555, "security": 365, "operational": 90} # ~7y / 1y / 90d
def retention_class(record: dict) -> str:
if record["flag"].startswith(("ops.payments.", "ops.auth.")): return "compliance"
if record.get("security_relevant"): return "security"
return "operational"Step 2 — Tier storage by age within the window
Keep recent records in hot, queryable storage and age older ones into cheap archival storage, so retention does not mean paying hot-storage prices for years. The economics are stark: a seven-year compliance window at hot-storage rates can cost twenty to fifty times what the same records cost in an archival object-storage class, and the vast majority of that data is never read again after its first few weeks. Set HOT_DAYS to cover your realistic investigation and reporting horizon — 30 to 90 days is typical, long enough that an incident review or a monthly report never has to thaw an archive — and let everything older settle into cold storage untouched.
Tiering is a performance decision as much as a cost one. Cold archives (S3 Glacier, GCS Archive, and equivalents) trade retrieval latency measured in minutes or hours for a fraction of the price, which is fine for records you touch only when an auditor asks but disqualifying for the recent history your on-call engineers query during an incident. Drive the transition with the storage layer’s own lifecycle rules or TTL indexes rather than an application cron where you can — a bucket lifecycle policy that moves objects to a colder class at 30 days is declarative, runs even when your service is down, and is itself a piece of evidence you can show an auditor.
# tier.py — move to cold storage past the hot window
def tier_for(record_age_days: int, cls: str) -> str:
if record_age_days <= HOT_DAYS: return "hot"
if record_age_days <= RETENTION_DAYS[cls]: return "cold"
return "expired" # eligible for deletionStep 3 — Enforce deletion at window expiry
Delete records the moment their retention window ends — no sooner (audit risk) and no later (data-minimization risk) — using a scheduled, logged deletion job. Deletion is the step that turns a written policy into an enforced one, so treat it as a first-class, observable process rather than a fire-and-forget TTL. Every purge emits its own meta-record (retention_delete above) into a separate stream that is itself retained — you delete the audit record but keep the fact that you deleted it, including its id, class, and the window that authorized removal. That meta-log is what lets you answer “prove you deleted the records you were required to delete” without resurrecting the very data you were obliged to destroy.
Make the job idempotent and rate-bounded. A purge that scans, checks eligibility, and deletes should be safe to re-run after a crash, and it should cap how many records it removes per pass so a misclassification bug cannot wipe years of history in one run — an unexpectedly large deletion batch is a signal to halt and alert, not to proceed. Guard the eligibility check against clock skew, too: if the deletion host’s clock jumps forward, records can appear expired days early, so anchor “now” to a trusted time source and refuse to delete records whose computed age is implausibly close to the window boundary without a second confirmation pass.
# delete.py — logged, bounded deletion at expiry
def purge_expired():
for rec in store.scan():
if tier_for(age_days(rec), retention_class(rec)) == "expired":
store.delete(rec.id)
audit_meta_log("retention_delete", rec.id, rec_class=retention_class(rec))Step 4 — Produce a retention-compliance report
Generate a report showing, per class, the oldest retained record and the count deleted, so an auditor can verify the policy is enforced rather than merely written. The single most persuasive number is the oldest retained record per class: if it sits comfortably below the class window, the deletion job is demonstrably keeping up; if it creeps above the window, your purge has silently stalled and you have a live compliance gap. Pair that with the count deleted in a recent period, because a zero-deletion month for a class that should be aging records out is as suspicious as an over-window record — it usually means the job stopped running, not that nothing expired.
Run the report on a schedule and alert on its assertions, not just on demand for auditors. A retention policy fails quietly: the job that should purge expired records keeps not-running for weeks before anyone notices, and by then you have an over-retention finding with no way to undo the calendar. Treat oldest_days > window for any class as a paging-level alert, and archive each report run so the trend itself — steady deletions, oldest-record ages holding flat — becomes the evidence, not just the current snapshot.
# report.sh — evidence the policy runs
flagctl audit retention-report --json | jq '{
compliance: .classes.compliance.oldest_days,
operational_deleted_30d: .classes.operational.deleted_last_30d }'
# oldest_days must be <= the class window; deletions prove the job runsVerification
Confirm the policy is enforced both ways: no record exceeds its class window, and records within the window are intact and immutable. These two assertions are not redundant — the first proves you are not over-retaining, the second proves the records you did keep are still trustworthy evidence rather than something that could have been quietly edited. An auditor discounts a retained record they cannot prove is unaltered, so the immutability sample is what gives the retained data its evidentiary weight. Run both as exit-code assertions in a scheduled check so a regression trips CI or an alert rather than waiting to be discovered during a review.
# No record older than its class window; and a within-window record is unaltered
flagctl audit retention-report --assert no-record-exceeds-window # exit 0 = compliant
flagctl audit verify-immutability --sample 100 # checksums intactGotchas & Edge Cases
- A legal hold overrides the schedule. When litigation or an investigation places a hold, scheduled deletion must pause for the affected records. Check for an active hold before purging, and log the hold so the deviation from the normal window is itself auditable.
- Personal data mixed into operational records shortens the window. If an operational record incidentally contains personal data (a hashed user reference that is still identifiable), the data-minimization window may be shorter than the operational one. Classify by the most restrictive applicable rule, not the record’s nominal type.
- Backups outlive the window. Deleting from the live store while backups retain the record for a year violates the policy in spirit and often in law. Apply retention to backups too, or ensure backup rotation expires records within the same window. Where a backup necessarily outlives an individual record’s window — as with immutable, encrypted, whole-database snapshots — document that fact and its rotation period so the residual copy is a known, bounded exception rather than a silent one an auditor discovers for you.
- Shortening a window does not retroactively delete. Changing a class from seven years to one does not remove the six-year-old records already on disk; they simply become eligible for deletion on the next purge, and only if the job actually scans them. After any window reduction, run a one-off backfill purge and confirm the report’s oldest-record figure has dropped, or you will carry over-retained data under a policy that on paper says otherwise.
- A stalled deletion job is invisible without alerting. Unlike a failed write, a purge that stops running produces no error — it produces nothing at all, and the log simply keeps growing past its windows. The only defense is monitoring the absence of expected deletions: alert when a class that should be aging records out reports zero purges over a period longer than its normal cadence.
- Restoring from an old backup can resurrect deleted records. A disaster-recovery restore from a snapshot taken before a purge silently reintroduces records you were legally required to delete. Re-run the deletion job as a mandatory post-restore step, and treat “records reappear after a restore” as a data-minimization incident, not a routine recovery artifact.
Troubleshooting & FAQ
How do I pick the retention window?
Start from the compliance requirement that applies (SOC 2 evidence often ~1 year, financial records often ~7 years, personal data as short as lawful) and take the binding one per record class. Document the rationale — an auditor accepts a justified window far more readily than a round number with no basis.
Can I keep everything forever to be safe?
No — indefinite retention of personal data violates data-minimization principles under regimes like GDPR, turning “to be safe” into a liability. Keep operational data as briefly as useful, compliance evidence as long as required, and nothing personal beyond its lawful window.
Does deletion break the immutability guarantee?
No — immutability means a record cannot be altered within its life, not that it lives forever. Scheduled deletion at window expiry is a distinct, logged operation that is itself recorded. The audit trail stays trustworthy because deletions are policy-driven and evidenced, not silent edits.
How does a legal hold interact with the retention window?
A legal hold trumps the retention schedule in both directions: it pauses scheduled deletion for the records it covers, so they are kept past their normal window until the hold lifts. Check for an active hold before every purge and skip held records rather than deleting and restoring them. Record the hold and its scope in the meta-log so the deviation from the standard window is itself auditable, and re-evaluate held records for deletion the moment the hold is released.
What should I do when a GDPR erasure request targets a retained record?
An erasure request and a retention obligation can genuinely conflict — you may be required to keep a flag-change record as compliance evidence while also being asked to delete a data subject’s information from it. Resolve it by scope: erase or irreversibly pseudonymize the personal fields while retaining the operational record itself, so the change history survives without the identifying data. See the GDPR compliance checklist for how to structure records so this surgical erasure is possible.
How often should the deletion job run?
Match the cadence to your shortest window and your tolerance for over-retention: a daily purge is standard, keeping the maximum over-retention to under 24 hours past any record’s expiry. Running less often than daily lets records linger measurably past their window; running far more often mostly adds scan cost without changing your compliance posture. Whatever the cadence, alert when a scheduled run is missed, since a purge that silently stops is the most common way a retention policy fails.
Should the retention-compliance report itself be retained?
Yes — each report run is evidence that the policy was enforced on a given date, so archive them for at least as long as your longest record window. An auditor reviewing year five wants to see an unbroken trend of steady deletions and oldest-record ages holding below their limits, and that story only exists if you kept the historical reports rather than only the current snapshot.