https://scryops.dev/An independent publication on modern observability engineering. Trends, deep dives, how-tos, and open questions.enhttps://scryops.dev/feed-icon.pnghttps://scryops.dev/144144Sat, 03 Oct 2026 00:00:00 +0000https://scryops.dev/alert-body-template/https://scryops.dev/alert-body-template/Sat, 03 Oct 2026 00:00:00 +0000A copy-paste alert body that answers four questions before the on-call has to ask: what's broken, how fast it's getting worse, what's already been checked, and the first three steps.AlertingOn-CallSLOsPrometheusHow-tohttps://scryops.dev/guides/opentelemetry-overview/https://scryops.dev/guides/opentelemetry-overview/Sat, 03 Oct 2026 00:00:00 +0000OpenTelemetry is a single instrumentation layer that produces traces, metrics, and logs in a vendor-neutral format. This guide explains what each signal is for, how the SDK and Collector relate, and where to go next.OpenTelemetryObservabilityTracingMetricsLogshttps://scryops.dev/guides/runbook-authoring/https://scryops.dev/guides/runbook-authoring/Thu, 01 Oct 2026 00:00:00 +0000A runbook that's hard to follow under pressure isn't a runbook. It's a liability. Here's the anatomy of one that actually shortens incident response, and how to keep it true.On-CallAlertingSLOsReliabilityOperationsBest Practiceshttps://scryops.dev/qa/structured-logging-antipatterns/https://scryops.dev/qa/structured-logging-antipatterns/Thu, 01 Oct 2026 00:00:00 +0000Three patterns that look like structured logging but aren't, in C#, Java, Go and Python, and what to write instead.LogsStructured LoggingBest Practiceshttps://scryops.dev/qa/profiling-vs-tracing/https://scryops.dev/qa/profiling-vs-tracing/Thu, 01 Oct 2026 00:00:00 +0000Tracing tells you which path a request took and how long each hop took. Profiling tells you what your CPU was actually doing during those hops. They're complementary — use both.ProfilingTracinghttps://scryops.dev/qa/what-is-an-sli/https://scryops.dev/qa/what-is-an-sli/Thu, 01 Oct 2026 00:00:00 +0000An SLI is a quantitative measure of service behaviour from the user's perspective. Choose the metric that, when it degrades, users notice — not server-side proxies that feel measurable but don't map to user experience.SLOsReliabilityObservabilityhttps://scryops.dev/guides/data-sovereignty-and-residency/https://scryops.dev/guides/data-sovereignty-and-residency/Thu, 01 Oct 2026 00:00:00 +0000A system that spans continents produces telemetry that spans legal jurisdictions. Here's how to keep traces and logs where the law wants them, and still see your whole system.CompliancePrivacyGDPRMulti-CloudOpenTelemetryCollectorObservabilityhttps://scryops.dev/guides/structured-logging-machine-readable/https://scryops.dev/guides/structured-logging-machine-readable/Thu, 01 Oct 2026 00:00:00 +0000Logs were designed for humans grepping text files at 2am. Now they also have to feed query engines, correlation and anomaly detection. Here's what that changes about what you write, and which field names to use.LogsOpenTelemetryStructured LoggingAIObservabilityhttps://scryops.dev/guides/on-call-procedures/https://scryops.dev/guides/on-call-procedures/Thu, 01 Oct 2026 00:00:00 +0000A page is just the starting gun. What happens between the alert firing and the postmortem closing determines whether your team gets better or just gets tired.On-CallAlertingReliabilityObservabilityOperationshttps://scryops.dev/guides/logging-foundations/https://scryops.dev/guides/logging-foundations/Thu, 01 Oct 2026 00:00:00 +0000What logging is for, how it fits next to metrics and traces, and what to log, where and how. The mental model to have before touching a logging framework, and the starting point for the rest of the logging guides.LogsObservabilityOpenTelemetryhttps://scryops.dev/guides/log-levels-and-severity/https://scryops.dev/guides/log-levels-and-severity/Thu, 01 Oct 2026 00:00:00 +0000Log levels are the emotional register of your system's voice. How to use ERROR, WARN, INFO, DEBUG and TRACE consistently, how they map onto OpenTelemetry's severity numbers, and what each one costs you.LogsObservabilityBest Practiceshttps://scryops.dev/qa/ebpf-production-ready/https://scryops.dev/qa/ebpf-production-ready/Thu, 01 Oct 2026 00:00:00 +0000For profiling and network observability on a modern Linux kernel, yes. Cilium, Parca, Pixie and Tetragon all run in production. The caveats are kernel version and BTF support, the privileges the agent needs, and application-level tracing, which still trails what an SDK gives you.eBPFKubernetesProfilinghttps://scryops.dev/howtos/wire-trace-ids-into-logs/https://scryops.dev/howtos/wire-trace-ids-into-logs/Thu, 01 Oct 2026 00:00:00 +0000Logs and traces live in separate worlds until you connect them. Put the trace ID on every log line in .NET, Java, Go or Python, check where each runtime actually writes it, and make the field name a contract your backend can use.OpenTelemetryLogsTracingPythonHow-tohttps://scryops.dev/howtos/set-up-slo-burn-rate-alerts/https://scryops.dev/howtos/set-up-slo-burn-rate-alerts/Thu, 01 Oct 2026 00:00:00 +0000A step-by-step walkthrough: define an SLI, calculate your error budget, write Prometheus recording rules, and wire up multi-window burn rate alerts that page you before users notice.SLOsAlertingPrometheusGrafanaHow-tohttps://scryops.dev/howtos/instrument-java-service-opentelemetry/https://scryops.dev/howtos/instrument-java-service-opentelemetry/Thu, 01 Oct 2026 00:00:00 +0000Instrument a Spring Boot service with the OpenTelemetry Java agent or the Spring Boot starter: traces, metrics and logs with no code, then your own spans and metrics, the Micrometer bridge and log correlation. All verified against a local Collector.OpenTelemetryTracingObservabilityHow-tohttps://scryops.dev/howtos/instrument-dotnet-service-opentelemetry/https://scryops.dev/howtos/instrument-dotnet-service-opentelemetry/Thu, 01 Oct 2026 00:00:00 +0000Add OpenTelemetry to an ASP.NET Core service: traces, metrics and logs in one setup block, manual spans and metrics for business logic, Serilog, and the zero-code agent for services you can't change. All verified against a local Collector.OpenTelemetryTracingObservabilityHow-tohttps://scryops.dev/guides/ebpf-continuous-profiling/https://scryops.dev/guides/ebpf-continuous-profiling/Thu, 01 Oct 2026 00:00:00 +0000Profile every process on a node with one DaemonSet and no code changes. How eBPF profilers work, which tool to pick, a tested Parca deployment, and the privileges, storage and trace-linking limits you need to plan around.eBPFProfilingKuberneteshttps://scryops.dev/guides/otel-context-propagation/https://scryops.dev/guides/otel-context-propagation/Thu, 01 Oct 2026 00:00:00 +0000A distributed trace is only as complete as its weakest propagation link. One hop that drops the context and the trace splits in two. W3C Trace Context and Baggage, the propagator settings that matter, and the places context gets lost — between services and inside them — in .NET, Java, Go, Python and Node.OpenTelemetryTracingObservabilityBest Practiceshttps://scryops.dev/guides/sli-selection-by-service-type/https://scryops.dev/guides/sli-selection-by-service-type/Thu, 01 Oct 2026 00:00:00 +0000Availability and latency are the obvious SLIs. But they don't fit every service type. This guide provides SLI selection frameworks for APIs, data pipelines, batch jobs, storage systems, event-driven services, and more.SLOsReliabilityObservabilityMetricshttps://scryops.dev/guides/alert-correlation/https://scryops.dev/guides/alert-correlation/Thu, 01 Oct 2026 00:00:00 +0000A single failure in a distributed system can trigger dozens of alerts across every layer it touches. Correlation groups the symptoms back into one cause — so the on-call engineer sees a problem, not a storm.AlertingObservabilityReliabilityOn-CallAIOpshttps://scryops.dev/articles/opa-observability-governance/https://scryops.dev/articles/opa-observability-governance/Tue, 29 Sep 2026 00:00:00 +0000Every team has a tagging standard. Most of them live in a wiki, enforced by nobody, remembered by almost nobody, and invisible to the CI pipeline. Open Policy Agent fixes the root cause.ObservabilityComplianceCI/CDBest PracticesOperationshttps://scryops.dev/articles/sampling-strategy/https://scryops.dev/articles/sampling-strategy/Tue, 29 Sep 2026 00:00:00 +0000A flat 5% sampling rate sounds like a sensible trade between cost and coverage. It isn't. A random slice of your traffic is mostly the requests you'll never look at, and it throws away the rare ones you need at the same rate.TracingSamplingOpenTelemetryObservabilityhttps://scryops.dev/guides/slos-and-error-budgets/https://scryops.dev/guides/slos-and-error-budgets/Tue, 29 Sep 2026 00:00:00 +0000Service Level Objectives and error budgets give reliability a quantitative shape — a target, a budget for deviation, and burn rate signals that tell you when to stop shipping and start fixing.SLOsAlertingReliabilityObservabilityOn-Callhttps://scryops.dev/howtos/configure-collector-tail-sampling/https://scryops.dev/howtos/configure-collector-tail-sampling/Tue, 29 Sep 2026 00:00:00 +0000Move from flat probabilistic sampling to tail-based sampling in the OTel Collector. Keep every error and slow trace, cut health-check noise to 1%, and check that the Collector is doing what you think.OpenTelemetrySamplingCollectorTracingHow-tohttps://scryops.dev/howtos/enrich-logs-with-business-context-dotnet/https://scryops.dev/howtos/enrich-logs-with-business-context-dotnet/Tue, 29 Sep 2026 00:00:00 +0000A log line that says 'payment failed' tells you something broke. One that says 'payment failed, enterprise customer, checkout-v2 experiment' tells you what to do about it. Here's how to add that context to every log event in a .NET service, safely, with Serilog.LogsStructured LoggingOpenTelemetryBest PracticesHow-tohttps://scryops.dev/articles/alert-design-principles/https://scryops.dev/articles/alert-design-principles/Tue, 29 Sep 2026 00:00:00 +0000The alert fires. The on-call is up. Now what? If the answer is 'check the dashboard', the alert isn't finished. The alert body is where the fix starts, or where you lose an hour chasing context.AlertingOn-CallSLOsReliabilityObservabilityhttps://scryops.dev/guides/alert-severity-levels/https://scryops.dev/guides/alert-severity-levels/Tue, 29 Sep 2026 00:00:00 +0000The P0-P4 framework was built for a world of static thresholds. Here's how to reconnect it to SLO burn rates — so severity reflects actual user impact, not arbitrary lines.AlertingSLOsOn-CallReliabilityhttps://scryops.dev/qa/synthetic-monitoring-vs-rum/https://scryops.dev/qa/synthetic-monitoring-vs-rum/Sun, 27 Sep 2026 00:00:00 +0000Synthetic monitoring runs scripted tests on a schedule. RUM captures what real users actually experience. They answer different questions, and you need both.ObservabilityRUMMonitoringReliabilityhttps://scryops.dev/articles/distributed-tracing-dashboard-was-green/https://scryops.dev/articles/distributed-tracing-dashboard-was-green/Sun, 27 Sep 2026 00:00:00 +0000Metrics tell you how the crowd is doing. Logs tell you what one service saw. A trace tells you what one request went through, and at 2 a.m. that is usually the question you are asking.TracingObservabilityOpenTelemetrySamplingDebugginghttps://scryops.dev/howtos/scrub-pii-from-application-logs-dotnet/https://scryops.dev/howtos/scrub-pii-from-application-logs-dotnet/Sat, 26 Sep 2026 00:00:00 +0000Keep personal data out of your .NET logs before they leave the process: classify fields so the logger erases or pseudonymises them, scrub free text and exception messages in an OpenTelemetry processor, and prove nothing leaks.GDPRPrivacySecurityLogsComplianceOpenTelemetryHow-tohttps://scryops.dev/guides/log-context-enrichment/https://scryops.dev/guides/log-context-enrichment/Thu, 11 Jun 2026 00:00:00 +0000Enrichment turns isolated log records into connected business events. Here is the architecture that makes it work — static resource attributes, background-refreshed caches, and per-request scopes — without taxing the request path.LogsObservabilityOpenTelemetryStructured LoggingBest Practiceshttps://scryops.dev/guides/common-logging-pitfalls/https://scryops.dev/guides/common-logging-pitfalls/Thu, 11 Jun 2026 00:00:00 +0000The same logging mistakes turn up in every team and every stack: inconsistent field names, values buried in message strings, missing trace context, personal data and secrets in error logs, and loops that log the same thing ten thousand times. Here is where to look and what to fix.LogsStructured LoggingObservabilityBest Practiceshttps://scryops.dev/guides/async-logging/https://scryops.dev/guides/async-logging/Thu, 11 Jun 2026 00:00:00 +0000A synchronous log write makes the request thread wait on disk or network I/O. Async logging hands that work to a background thread, and quietly adds a queue that can fill up, drop records and lose them at shutdown. How to size it, watch it and flush it, with Serilog, the OpenTelemetry SDK and Python.LogsObservabilityOpenTelemetryReliabilityhttps://scryops.dev/howtos/set-up-log-based-alerting/https://scryops.dev/howtos/set-up-log-based-alerting/Wed, 10 Jun 2026 00:00:00 +0000Turn a LogQL query into a Grafana-managed alert rule that fires on error volume, a specific error type or a failing dependency. Covers the query, the rule settings that trip people up, routing to PagerDuty and Slack, and an end-to-end test.LogsObservabilityAlertingGrafanaHow-tohttps://scryops.dev/guides/compliance-observability/https://scryops.dev/guides/compliance-observability/Wed, 10 Jun 2026 00:00:00 +0000Regulated industries need observability too. A guide to building telemetry pipelines that satisfy GDPR, HIPAA, SOC 2, and PCI DSS requirements — covering data minimisation, retention mandates, audit trails, and what each framework actually requires.CompliancePrivacyGDPRSecurityObservabilityhttps://scryops.dev/guides/log-based-monitoring/https://scryops.dev/guides/log-based-monitoring/Wed, 10 Jun 2026 00:00:00 +0000Logs carry operational state at a resolution metrics can't match. Most teams only open them after something breaks. This guide covers how to query them continuously, turn them into metrics, and alert on what they surface without blowing up cardinality or cost.LogsObservabilityAlertingStructured Logginghttps://scryops.dev/guides/audit-trail-implementation/https://scryops.dev/guides/audit-trail-implementation/Wed, 10 Jun 2026 00:00:00 +0000An audit trail is not a log. It's a tamper-evident, time-ordered record of who did what, when, and why. Most teams build this wrong. Here is how to do it correctly using OpenTelemetry and append-only storage.ComplianceSecurityOpenTelemetryLogsPrivacyhttps://scryops.dev/articles/evolution-of-system-understanding/https://scryops.dev/articles/evolution-of-system-understanding/Sun, 07 Jun 2026 00:00:00 +0000From grepping one log file to querying wide, trace-linked events: how the questions we can ask a running system changed when monoliths split apart, and why OpenTelemetry had to exist.ObservabilityOpenTelemetryTracingPhilosophyhttps://scryops.dev/articles/observability-vs-monitoring/https://scryops.dev/articles/observability-vs-monitoring/Sun, 07 Jun 2026 00:00:00 +0000Monitoring tells you when something you predicted goes wrong. Observability lets you work out what's happening when it's something you didn't. You need both, and the gap between them is where incidents drag on.ObservabilityMonitoringPhilosophyCosthttps://scryops.dev/howtos/benchmark-sync-vs-channel-logging/https://scryops.dev/howtos/benchmark-sync-vs-channel-logging/Sun, 07 Jun 2026 00:00:00 +0000Async logging is supposed to take I/O off the request thread. Measure it: a sync-vs-channel benchmark under concurrent producers, in .NET, Go, or Python, and how to read the result.LogsPythonHow-tohttps://scryops.dev/guides/high-throughput-log-pipelines/https://scryops.dev/guides/high-throughput-log-pipelines/Sun, 07 Jun 2026 00:00:00 +0000At 1.5 million log events per second you cannot keep, batch, or ship everything the way you did at moderate scale. Content-aware sampling, OTel exporter tuning, Collector-side batching, and cheaper bytes on the wire.LogsSamplingOpenTelemetryCollectorOTLPhttps://scryops.dev/guides/high-throughput-logging/https://scryops.dev/guides/high-throughput-logging/Sun, 07 Jun 2026 00:00:00 +0000At hundreds of thousands of requests per second, the logging call itself becomes the bottleneck. Async channels, pooling, batching, and circuit breakers keep log I/O off the request thread.LogsReliabilityObservabilityhttps://scryops.dev/guides/distributed-logging/https://scryops.dev/guides/distributed-logging/Sun, 07 Jun 2026 00:00:00 +0000When a request crosses ten services, you get ten log streams that share nothing but a timestamp you can't trust. How to collect them on every node, carry the trace ID through, ship them through the OpenTelemetry Collector, and notice when lines go missing.LogsObservabilityOpenTelemetryCollectorKuberneteshttps://scryops.dev/guides/data-masking-in-telemetry/https://scryops.dev/guides/data-masking-in-telemetry/Sun, 07 Jun 2026 00:00:00 +0000Telemetry data is just as risky for PII as any database. Here's how to turn sensitive fields into safe, useful signals: hashing, tokenising, coarsening, and picking the right tool for the job.PrivacyOpenTelemetrySecurityObservabilityCollectorhttps://scryops.dev/guides/pii-in-telemetry/https://scryops.dev/guides/pii-in-telemetry/Tue, 26 May 2026 00:00:00 +0000Every OTel span that includes a customer email, shipping address, or payment token is a GDPR audit waiting to happen. The fix isn't application code; it's a Collector pipeline.OpenTelemetryPrivacyGDPRSecurityObservabilityCollectorhttps://scryops.dev/articles/alert-fatigue-is-an-observability-problem/https://scryops.dev/articles/alert-fatigue-is-an-observability-problem/Tue, 26 May 2026 00:00:00 +0000Every alert that fires is doing its job. That is the problem. The model is wrong, not the thresholds.AlertingSLOsOn-CallObservabilityhttps://scryops.dev/articles/what-is-observability-2-and-why-scryops/https://scryops.dev/articles/what-is-observability-2-and-why-scryops/Sun, 15 Mar 2026 00:00:00 +0000Observability 1.0 taught us to look backward. Observability 2.0 asks us to look forward. Most teams haven’t made that shift yet. That’s why I named this site after a medieval divination practice.ObservabilityOpenTelemetryAIPhilosophy