General-purpose observability, built for debugging performance and errors, often doesn't capture what a security investigation actually needs — a system can be fully observable for engineering purposes and still leave you unable to answer "who accessed this, and when" after an incident.
Logs, with security-relevant detail
Security-useful logging goes beyond error tracking to capture: who performed an action (authenticated identity, not just a request ID), what resource was accessed or modified, and when — with enough context to reconstruct a sequence of events during an investigation. Authentication events (successful and failed logins), authorization decisions (especially denials), and administrative actions are consistently high-value to log.
Metrics for security signal
Beyond typical performance metrics, security-relevant metrics include failed authentication rate (a spike can indicate a brute-force attempt), unusual request volume from a single identity or IP, and rate of authorization denials (a spike can indicate probing for access it shouldn't have). Trending these over time surfaces anomalies that a single log entry wouldn't.
Traces for reconstructing attack paths
Distributed tracing, typically used for performance debugging across microservices, is equally valuable for reconstructing how a request moved through your system during a security investigation — which services a specific suspicious request touched, in what order, is exactly the kind of reconstruction an incident investigation needs.
What to instrument specifically for security
- Authentication and authorization events — every login attempt, every explicit access denial.
- Administrative and privileged actions — changes to permissions, configuration, or other users' accounts.
- Data access to sensitive resources, not just modification — read access to sensitive data is often as important to log as write access.
- Secret and credential usage — where feasible, logging when and by what a stored credential was actually used.
Centralization and retention
Security-relevant logs scattered across many systems, each with its own short retention window, make investigation far harder than necessary. Centralizing logs into a searchable system with a retention period long enough to cover realistic incident discovery timelines (often well past 30 days, since not every incident is discovered immediately) is foundational.
Alerting on the signal, not just collecting it
Collected but unreviewed logs and metrics provide little value during an active incident if nobody is alerted to the anomaly in near-real-time. Pairing security-relevant telemetry with defined alerting thresholds turns passive data collection into active detection.