When your application runs across browsers, mobile devices, and cloud infrastructure, your team will face challenges. The question isn’t if you’ll have a production incident, but whether you’ll be able to see it happening in real time and debug it quickly.
Observability is the practice of making systems understandable through their outputs. It’s different from monitoring, which tells you when things are broken. Observability tells you why they’re broken.
The Three Pillars of Observability
Production observability rests on three complementary practices: distributed tracing, metrics collection, and structured logging. Each answers a different question:
- Distributed tracing shows you the path a request takes through your system.
- Metrics give you the aggregated health of components over time.
- Logs provide detailed context about specific events.
Together, they create a complete picture. A spike in error rates (metrics) points you to which service is failing. Traces show you which requests are affected and where they’re timing out. Logs show you the exact error message and stack trace.
Distributed Tracing Across Your Stack
Distributed tracing works by propagating a trace ID through every layer of your application. When a user loads your web app, a trace ID is generated. That same ID flows through your API calls, database queries, and mobile backend requests. This lets you reconstruct the entire journey of a single request.
Starting from the Browser
Browser instrumentation is often overlooked, but it’s where users first experience your application. Libraries like Sentry or Datadog’s RUM (Real User Monitoring) automatically capture page load times, resource timings, and JavaScript errors.
Here’s a minimal example of instrumenting a web application:
// Initialize Datadog RUM
import { datadogRum } from '@datadog/browser-rum';
datadogRum.init({
applicationId: 'YOUR_APP_ID',
clientToken: 'YOUR_CLIENT_TOKEN',
site: 'datadoghq.com',
service: 'web-frontend',
env: 'production',
sessionSampleRate: 100,
sessionReplaySampleRate: 20,
trackUserInteractions: true,
trackResources: true,
trackLongTasks: true
});
datadogRum.startSessionReplayRecording();This captures real user metrics, interactions, and errors. The trace context is automatically included in all outgoing requests via headers, so your backend can correlate the browser session with server-side activity.
Backend Trace Propagation
On your backend, you need to extract the trace context from incoming requests and propagate it to downstream services. The standard format is W3C Trace Context, which uses two headers: traceparent and tracestate.
Here’s how to implement trace propagation in a Node.js/Express application:
const express = require('express');
const tracer = require('dd-trace').init(); // Datadog APM
const app = express();
app.use((req, res, next) => {
// Extract trace context from headers
const traceId = req.get('x-trace-id') || generateTraceId();
const spanId = generateSpanId();
// Attach to request object for downstream use
req.traceId = traceId;
req.spanId = spanId;
// Set response headers so clients can correlate
res.set('x-trace-id', traceId);
next();
});
app.get('/api/users/:id', async (req, res) => {
const span = tracer.startSpan('get-user', {
childOf: req.spanId,
tags: {
'trace.id': req.traceId,
'user.id': req.params.id
}
});
try {
const user = await db.query('SELECT * FROM users WHERE id = ?', [req.params.id]);
res.json(user);
} catch (err) {
span.setTag('error', true);
span.log({ event: 'error', message: err.message });
res.status(500).json({ error: 'Internal server error' });
} finally {
span.finish();
}
});The key is that every service extracts the trace ID from the request and includes it in all downstream calls. This creates an unbroken chain from user action to database query.
Mobile App Instrumentation
Mobile apps need the same treatment. Libraries like Sentry for mobile or Datadog’s mobile SDK automatically capture crashes and errors, but you should also instrument key user flows.
// Flutter example with Sentry
import 'package:sentry_flutter/sentry_flutter.dart';
import 'package:http/http.dart' as http;
future void main() async {
await SentryFlutter.init(
(options) {
options.dsn = 'YOUR_SENTRY_DSN';
options.tracesSampleRate = 1.0;
options.environment = 'production';
},
appRunner: () => runApp(MyApp()),
);
}
class ApiClient {
Future<User> getUser(int userId) async {
final transaction = Sentry.startTransaction(
'get-user',
'http.client',
);
try {
final response = await http.get(
Uri.parse('https://api.example.com/users/$userId'),
headers: {
'x-trace-id': transaction.traceId,
'x-span-id': transaction.spanId,
},
);
transaction.setTag('http.status_code', response.statusCode);
return User.fromJson(jsonDecode(response.body));
} catch (e) {
transaction.throwable = e;
rethrow;
} finally {
await transaction.finish();
}
}
}Mobile trace context flows back to your backend through those same headers, creating end-to-end visibility from user tap to database result.
Structured Logging for Correlation
Logs are most useful when they’re structured. Instead of writing free-form text, emit logs as JSON with consistent fields. This lets you query and correlate logs across services.
Here’s a structured logging setup using Winston in Node.js:
const winston = require('winston');
const logger = winston.createLogger({
format: winston.format.combine(
winston.format.timestamp(),
winston.format.errors({ stack: true }),
winston.format.json()
),
defaultMeta: { service: 'api-server' },
transports: [
new winston.transports.File({ filename: 'error.log', level: 'error' }),
new winston.transports.File({ filename: 'combined.log' }),
],
});
// Log with correlation context
app.use((req, res, next) => {
req.log = logger.child({
traceId: req.traceId,
userId: req.user?.id,
endpoint: req.path,
});
next();
});
app.post('/api/orders', (req, res) => {
req.log.info('Order creation started', {
cartValue: req.body.total,
itemCount: req.body.items.length,
});
try {
const order = createOrder(req.body);
req.log.info('Order created successfully', {
orderId: order.id,
duration: Date.now() - req.startTime,
});
res.json(order);
} catch (err) {
req.log.error('Order creation failed', {
error: err.message,
stack: err.stack,
});
res.status(500).json({ error: 'Failed to create order' });
}
});Every log entry automatically includes the traceId. When you search for a specific trace ID in your log aggregation tool (Datadog, ELK, Splunk), all related logs surface together. This turns a pile of log files into a coherent story.
Metrics for Health at a Glance
Metrics are time-series data that show the health of your system. Common metrics include request latency, error rate, CPU usage, and queue depth. Unlike logs, which can be verbose and expensive to store, metrics are lightweight and designed for long-term retention.
Here’s how to emit custom metrics from your application:
const StatsD = require('node-dogstatsd').StatsD;
const client = new StatsD();
app.get('/api/users/:id', async (req, res) => {
const start = Date.now();
try {
const user = await db.query('SELECT * FROM users WHERE id = ?', [req.params.id]);
const duration = Date.now() - start;
// Emit latency metric
client.timing('api.get_user.duration', duration, ['endpoint:users']);
// Emit success counter
client.increment('api.get_user.success', 1, ['endpoint:users']);
res.json(user);
} catch (err) {
const duration = Date.now() - start;
// Emit error counter
client.increment('api.get_user.error', 1, [
'endpoint:users',
`error_type:${err.code}`
]);
res.status(500).json({ error: 'Failed to fetch user' });
}
});Metrics are tagged with context (service name, endpoint, error type). This lets you slice and dice data in your dashboard. You can ask questions like “What’s the 95th percentile latency for the users endpoint in production?” or “How many errors are we seeing per minute, broken down by error type?”
Building a Practical Alerting Strategy
Metrics are only useful if they drive action. Alerts should be specific, actionable, and low-noise. A good alert answers three questions: What is broken? Why does it matter? What should I do?
Here’s an example alert definition in Datadog:
name: High error rate on checkout API
type: metric alert
query: |
avg(last_5m):(
sum:api.checkout.error{*}
/ sum:api.checkout.requests{*}
) > 0.05
threshold: 0.05 (5%)
severity: critical
notify:
- team:payments
- @pagerduty
message: |
Checkout API error rate is above 5% ({{value}}%)
Service: {{host.name}}
Environment: {{env}}
Runbook: https://wiki.example.com/checkout-errors
Logs: https://app.datadoghq.com/logs?query=service:checkout%20status:errorNotice the alert includes a runbook link and a direct link to related logs. When on-call engineers get paged, they have context immediately. Bad alerts page you for every blip. Good alerts page you only when users are actually affected.
Bringing It Together: End-to-End Example
Let’s trace a real scenario. A user reports that checkout is slow. Here’s how observability helps you debug it:
- You search your dashboard for high checkout latency over the last hour.
- You find a spike at 14:32 UTC. You click through to the trace view.
- You see that 20% of checkout requests are taking 30+ seconds.
- You drill into a slow trace. The trace shows the request spent 28 seconds waiting on the payment processing service.
- You jump to logs filtered by service:payments. You see that the payment service was making database queries that took 25+ seconds each.
- You check metrics for the payments database. CPU was at 95%, and there’s a spike in slow queries.
- You check the database logs and see that a long-running query was holding locks.
- You kill the query, and latency returns to normal within seconds.
- You create an alert for database query duration and add an index to prevent recurrence.
That entire investigation takes minutes instead of hours because your traces, logs, and metrics are correlated and searchable.
Practical Implementation Checklist
Getting observability right takes time, but you don’t need to solve everything at once. Here’s a practical roadmap:
- Week 1: Set up centralized logging. Get logs from all services into one place.
- Week 2: Add trace IDs to logs. Correlate logs across services using a common ID.
- Week 3: Instrument your backend with distributed tracing. Capture spans for key operations.
- Week 4: Add browser and mobile instrumentation. Capture real user experience.
- Week 5: Define and deploy alerts. Focus on user-impacting metrics first.
- Week 6+: Iterate. Add more detailed tracing, improve alert accuracy, build dashboards for your team.
Start small. Pick one service and instrument it fully. Learn how to use your observability platform. Then expand to other services. The goal isn’t perfection; it’s visibility.
Considerations and Best Practices
Building observability requires careful attention to several factors:
- Sampling strategy matters. If you only capture 1% of traces, you’ll miss rare but important issues. Balance coverage with cost.
- Logging volume can become expensive. Focus on structured logs that provide signal, not noise.
- Alert on user impact. Internal metrics that don’t affect users create alert fatigue.
- Correlate your data. Logs, traces, and metrics need to reference each other to be useful.
- Plan for retention. Keep detailed traces for a few days, metrics for months, logs for weeks.
Observability is an investment. It costs money and engineering time upfront. But when a production issue hits, it pays for itself in hours saved and customer trust maintained.
Conclusion
Production systems are complex. They span frontend, mobile, backend, and infrastructure. With visibility into what’s happening, you can understand why it’s happening and fix it before users notice.
Start with the three pillars: distributed tracing to track requests, structured logging to capture context, and metrics to measure health. Correlate them so they reference each other. Build alerts that matter. Your future self will thank you when a production incident happens at 3 AM and you can resolve it in minutes instead of hours.
What’s the difference between observability and monitoring?
Monitoring tells you that something is broken. Observability tells you why it’s broken. Monitoring is passive (you check dashboards). Observability is active (you can ask questions of your system and get answers). You need both, but observability is what lets you debug production issues quickly.
Do I need to use the same vendor for tracing, logs, and metrics?
No. You can use Datadog for metrics, Grafana for dashboards, Sentry for error tracking, and ELK for logs. What matters is that they’re correlated. Each log and trace should include a trace ID so you can jump between tools. That said, using a single vendor simplifies setup and often costs less at scale.
How much will observability cost?
It depends on your scale. At small scale (under 1M requests per month), you can use free tiers of Sentry, Datadog, and Grafana. At medium scale, expect 500 to 2000 per month. At large scale, it can be thousands, but it’s usually cheaper than the engineer time you save on debugging. Many teams find the ROI positive within weeks.
What’s the right sampling rate for traces?
Start with 100% sampling in development and staging. In production, aim for 10-50% depending on traffic. Sample more aggressively for errors (capture 100% of error traces). Use tail-based sampling to capture traces that end up being slow or erroring, even if they weren’t sampled at ingestion time. Adjust based on what you learn about your system.
Can I add observability to an existing application without major refactoring?
Yes. Start with automatic instrumentation from your APM vendor. Libraries like Datadog APM or Sentry automatically capture errors and latency without code changes. Then gradually add custom instrumentation for business logic and critical paths. You don’t need to instrument everything at once.
