Observability improvements with Amazon CloudWatch

At AWS, we talk about new capabilities as they come, but it is also important to look at how they can be used together. Let’s do a quick recap for monitoring and observability with Amazon CloudWatch.

Logs support is now available in AWS Distro for OpenTelemetry (ADOT) and, as part of that, you can now send logs to CloudWatch.

AWS Lambda has introduced new controls to make it easier to search, filter, and aggregate Lambda function logs. You can now capture your Lambda logs in JSON format (without having to bring your own logging libraries) and control the log level from the function configuration.

Also, you can choose the CloudWatch log group to which Lambda sends its logs to. This allows you to aggregate logs from multiple functions in one place.

In the CloudWatch console, you can use Log Insights to query your logs. First, you select the log groups. It can be a log group where you send logs from all the components of your application running on Lambda functions, containers, EC2 instances, or on-premises.

To send logs, you can use the CloudWatch agent or the ADOT Collector (an AWS supported version of the upstream OpenTelemetry Collector).

To help you find what you’re looking for in the logs, you can start a Live Tail session on the selected log groups.

To focus the output of a Live Tail session, you can define filters that can match multiple IP subnets or HTTP status codes using regular expressions such as:

You can use natural language to query your logs (and your metrics, using Metrics Insight). Queries are generated by generative AI and depend on the data in your account.

This can help you quickly build advanced queries using new commands and features (such as the recently introduced multiple stats support).

When looking at the results of a log query, you can use patterns to help you find the needle in the haystack. You can also quickly compare the results with a previous period to see what changed and find if there is a new pattern that was not there yesterday or last week.

To automate some of these checks, you can enable anomaly detection on some log groups to surface anomalies found in your logs as they are processed during ingestion.

To help you optimize costs, CloudWatch now has a new log class for infrequent access logs at a reduced price. Log groups using the Infrequent Access log class can be queried using Logs Insight in single- and cross-account setups but have some limitations.

For example, you cannot use Live Tail or generate custom metrics via Embedded metric format (EMF).

You can now use CloudWatch to consolidate hybrid, multicloud, and on-premises metrics. You select and configure connectors that pull data from Prometheus, Amazon OpenSearch Service, Amazon RDS for MySQL / PostgreSQL, CSV files stored in Amazon S3, and Microsoft Azure Monitor.

Each connector is a Lambda function that CloudWatch invokes as needed to return metrics immediately.

Let me know how these new capabilities work for you!