Goals After reading the article, you should: ✅ Be able to differentiate between Logs & Metrics ✅ Know the what and why of Cloudwatch Embedded metrics ✅ Gain a high-level understanding of what it takes to convert logs into metrics in AWS Content Introduction What is it? Why it matters Supported Platform Example - using AWS Lambda Client Libraries Conclusion Introduction If you use AWS, then there is a good chance you are using AWS Cloudwatch logs for your logging. Logs are very useful for specific events within your code between requests. However, if you want to gain high-level insights, you want to be looking at an aggregation of events at a point in time for a system rather than a specific event. This is where metrics come in. Now, the question becomes — how do you go from the logs to the metrics? If you are using AWS, then there is an easy way to achieve this, and that is using AWS Cloudwatch embedded metrics. The gist of it is that these are logs formatted in a specific format that AWS uses to convert these logs into metrics. Let’s go through the details. What Is It? AWS Cloudwatch embedded metrics are just logs in a specific format. It needs to be a valid JSON and contain the right details so AWS can convert these logs into metrics. Format: x-amzn-logs-format: json/emf Here is a sample of the format: { "_aws": { "Timestamp": 1574109732004, "CloudWatchMetrics": [ { "Namespace": "lambda-function-metrics", "Dimensions": [["functionVersion"]], "Metrics": [ { "Name": "time", "Unit": "Milliseconds" } ] } ] }, "functionVersion": "$LATEST", "time": 100, "requestId": "989ffbf8-9ace-4817-a57c-e4dd734019ee" } 📝 Helpful reference: AWS - Cloudwatch Embedded metrics specification Why It Matters Here are a few points of why I think AWS Cloudwatch embedded metrics matter. Metrics are actionable You go from logs (specific events) to high-level insights (metrics), and you can’t improve or analyze what you can’t see. These metrics can be used to define contracts such as SLAs, SLIs, and SLOs. AWS Cloudwatch embedded metrics make this process that much easier and simpler. Great Developer Experience (DX) This feature is an opt-in feature, meaning it’s optional. When you want to “enable” it, you just format your log in a specific way. No friction, no extra steps — It simply just works! In my opinion, that’s a big win. Shifted Operational Responsibility In this setup, AWS assumes almost all the responsibility for this feature and infrastructure. This reduces the operational burdens that you or your team is responsible for. To me, that’s another big win! Supported Platforms Using AWS Lambda, you don’t need any further configuration, it just works. If you are using AWS ECS or EKS, then the cloudwatch agent is required to make it work. - Cloudwatch embedded metrics are supported out of the box AWS Lambda - Requires the CloudWatch agent to send Cloudwatch embedded metrics AWS ECS and EKS Example - Using AWS Lambda Here is an example of how AWS Cloudwatch embedded metrics work with AWS Lambda. Again, this works out of the box. The steps: API gateway receives request API Gateway invokes Lambda Alias Lambda processes the event ⭐️ Lambda logs to cloudwatch logs Logs are automatically converted to Cloudwatch metrics Step 1-4 are pretty typical in an AWS Lambda setup, the big difference is that when you log in a certain format, AWS will pick that up and convert those logs into metrics. Step 4 is where the magic happens! You can format these logs manually or use a client library provided by AWS (recommended). Client Libraries To make it easier, AWS has provided client libraries to generate these formatted logs. It provides an API to help you create formatted logs that result in desired cloudwatch metrics based on the settings you provide. The client libraries: Node.js Python Java C# Example using the client library with Node.js: const { createMetricsLogger, Unit } = require("aws-embedded-metrics"); const sendMetrics = async () => { const metrics = createMetricsLogger(); metrics.putDimensions({ Service: "Aggregator" }); metrics.putMetric("ProcessingLatency", 100, Unit.Milliseconds); metrics.setProperty("RequestId", "422b1569-16f6-4a03-b8f0-fe3fd9b100f8"); // ... await metrics.flush(); }; await sendMetrics(); Conclusion So, to recap: Metrics provide high-level insights while logs provide details about specific events. AWS Cloudwatch embedded metrics convert your AWS Cloudwatch logs into AWS Cloudwatch metrics. When using AWS Lambda, embedded metrics are supported out of the box. When using AWS ECS & EKS, the cloudwatch agent will need to be configured. AWS provides client libraries to make it easier for you to manage the resulting embedded metrics. And... That’s all. I hope you found that helpful! If you learned something new, please share this article with a friend or co-worker 🙏❤️! (Thanks!)
