On a traditional server, when something fails, you always have the last resort: SSH, tail -f, and read. In serverless that resort doesn't exist. A request can cross API Gateway, a Lambda, an SQS queue, and another Lambda — and if your logs are loose strings written with console.log, reconstructing what happened is archaeology. I learned this the worst way: an intermittent production bug that took me days to locate because I couldn't follow a request end to end. This article is the logging system I've used since: structured logs, a correlation ID that travels with the request, and CloudWatch queries that answer questions in minutes.
Logs as data, not prose
The fundamental shift is to stop writing logs for humans and start writing them for machines. A prose log ("Error processing order 123 for user 456") is pleasant to read but impossible to query at scale. A structured log is a JSON object with consistent fields:
logger.error('order_processing_failed', {
orderId: order.id,
userId: user.id,
errorCode: err.code,
correlationId: ctx.correlationId,
durationMs: Date.now() - start,
});
CloudWatch Logs Insights understands JSON natively. With structured logs, "how many orders failed this week with this error code, and from which users?" stops being a heroic grep and becomes a query:
fields @timestamp, orderId, userId
| filter level = 'error' and errorCode = 'PAYMENT_TIMEOUT'
| stats count(*) by userId
| sort count(*) desc
I don't use a homemade logger: AWS Lambda Powertools for TypeScript already solves serialization and levels, and automatically adds the invocation context (request ID, function name, cold start). Reinventing that is wasted time.
The correlation ID: the thread that ties the system together
In an event-driven architecture, the problem isn't logging: it's correlating. The user's request triggers a Lambda that enqueues a message in SQS that's processed by another Lambda that writes to DynamoDB. Four different log groups, four different request IDs, no visible relationship between them.
My rule: the first entry point generates a correlationId (or adopts the one arriving in the X-Correlation-Id header), and that ID travels with everything. In SQS messages it goes inside the message attributes; in service-to-service calls, in a header; in every log, as a mandatory field. Powertools lets you inject it once into the logger and forget about it: every log from that invocation carries it.
With that, reconstructing the complete story of a request is a single Logs Insights query across all the log groups involved. What used to be an afternoon of archaeology is now thirty seconds.
💡 Observability isn't added when there's an incident: it's designed beforehand. The day something breaks in production, your logs already are what they are. Every field you didn't log is a question you can't answer.
Metrics without a hammer: EMF
For business metrics (orders processed, matchings generated, growing queues) I don't use CloudWatch API calls, which add latency and cost money per request. I use Embedded Metric Format: you write the metric as part of the JSON log with a special format, and CloudWatch extracts it asynchronously. Zero marginal cost on the hot path, and the metrics show up in dashboards and alarms like any other.
metrics.addMetric('MatchingCompleted', MetricUnit.Count, 1);
metrics.addMetric('MatchingDurationMs', MetricUnit.Milliseconds, elapsed);
The discipline I impose on myself: every Lambda emits at least one success metric and one failure metric with a business name, not a technical one. "Errors" tells me nothing at 3 a.m.; "PaymentTimeouts" does.
What I decided not to do
I didn't set up X-Ray everywhere. I tried it, and at my scale full distributed tracing was more noise than signal: most of my flows have two or three hops, and the correlation ID in structured logs covers them more than well enough. I reserve X-Ray for flows with more hops or when I suspect inter-service latencies the logs can't explain. It's an honest trade-off: less automatic visibility in exchange for less cost and less configuration to maintain.
I also don't centralize logs in an external tool. CloudWatch Logs Insights has a mediocre UX and queries billed per GB scanned, but adding a third party means another shipping pipeline, another contract, and another invoice. While the volume allows it, I prefer the pain I know. The day queries become truly slow or expensive, that decision gets revisited with data: GB scanned per month and minutes lost per incident.
The result
None of this is glamorous. It's three habits: JSON instead of prose, an ID that travels with the request, and business metrics embedded in the logs. But the operational difference is enormous: incidents went from "let me see if I can reproduce it" to "give me the correlation ID and I'll tell you what happened". In serverless you can't SSH into the server, so the system had better tell its own story.
