Generated by Codex with GPT-5

What the post covers

AWS’s official Artificial Intelligence blog published Comprehensive observability for Amazon SageMaker AI LLM inference: From GPU utilization to LLM quality, a May 29, 2026 technical guide to monitoring hosted language models as both infrastructure workloads and probabilistic software components.

The post starts from a gap in conventional service monitoring. A normal endpoint can often be judged by familiar signals: request rate, error rate, latency, CPU load, memory pressure, and saturation. Those signals remain necessary for LLM inference, where variable token counts, GPU memory pressure, and traffic spikes complicate capacity planning. But they are not sufficient. An LLM endpoint can return HTTP 200 responses quickly while its answers quietly become less relevant, less accurate, less compliant, or less useful as the input distribution changes.

AWS therefore separates observability into two dimensions. “Quantity” monitoring tracks whether the serving system is healthy and cost-efficient. “Quality” monitoring tracks whether model outputs still meet the application’s requirements. The useful engineering move is not merely naming both categories. It is giving them separate metric paths and then bringing them back together in dashboards and alerts that let operators correlate infrastructure behavior with user-facing model behavior.

Two telemetry paths

The reference architecture uses three AWS services. Amazon SageMaker AI Inference Components host models, Amazon CloudWatch stores telemetry, and Amazon Managed Grafana visualizes and alerts on the signals. A single SageMaker endpoint can host multiple inference components, such as the post’s gpt-oss-20b and Qwen2.5-7B-Instruct examples. Each component keeps its own traffic routing, scaling policy, and metric attribution even when models share underlying infrastructure.

That attribution boundary matters. Shared serving capacity is economically attractive, but it is hard to operate if engineers cannot answer which model is creating latency, consuming GPU memory, or driving cost. SageMaker enhanced metrics supply instance-level, container-level, and per-GPU dimensions under the /aws/sagemaker/InferenceComponents/<model-name> namespace. The Grafana quantity dashboard surfaces request volume, latency, per-copy invocation distribution, GPU compute utilization, GPU memory utilization, used and free GPUs, instance count, and hourly cost by model.

Those panels help distinguish problems that look similar from the outside. Rising latency may come from traffic growth, poor load distribution, compute saturation, memory pressure, or an autoscaling policy that reacts too slowly. A shared endpoint may have spare capacity overall while one model remains constrained. Cost can rise because demand is growing or because GPUs are over-provisioned. Per-model and per-GPU telemetry makes those cases visible without forcing every model into a separate fleet.

Quality telemetry travels through a different namespace: /aws/sagemaker/inference-quality/<model-name>. AWS’s example publishes composite quality, safety, relevance, professional-tone, and evaluation-latency scores. Keeping this path separate from infrastructure metrics is a sensible design choice. SageMaker can publish serving telemetry automatically because it owns the endpoint runtime. Quality is application-specific: a code assistant, a support bot, and a retrieval-augmented research tool do not share the same definition of a good answer.

Quality as an operational signal

The quality dashboard treats model behavior as a time series rather than an occasional benchmark. That is the key idea in the post. Offline evaluation remains useful before deployment, but production traffic changes. Prompt distributions shift, real-world conditions evolve, and model or system updates can alter behavior. Quality degradation often arrives without a crash, a timeout, or an error spike.

AWS’s example evaluates sampled outputs with configurable rubrics and an LLM-as-judge pattern. It uses Claude Sonnet 4.6 through Amazon Bedrock as the evaluator, while allowing teams to substitute another evaluation system. The post calls out several operating constraints: the evaluator’s service terms must permit judging other models’ outputs, data-residency requirements must be satisfied, and the evaluator should be pinned to a specific version so scores remain comparable over time.

That last requirement is important. A quality score is only useful as a monitoring signal if its meaning remains stable enough for trends and thresholds to be interpreted. If the production model and the evaluator both drift at once, an alert may say less than it appears to say. Pinning the evaluator does not eliminate subjectivity, but it turns evaluator behavior into a controlled dependency rather than an invisible source of metric churn.

The Grafana dashboards attach configurable thresholds to quality signals and use Grafana Alerting plus Amazon SNS to route violations into existing incident workflows. Alerts are dimensioned by inference component, so a drop in one hosted model’s safety or relevance score can trigger focused triage. The post suggests integrating those notifications with tools such as Slack, PagerDuty, or OpsGenie.

This makes quality monitoring operational rather than decorative. A model-quality dashboard that is reviewed occasionally may help with product analysis. A quality signal that enters the same triage machinery as latency and saturation can support remediation decisions: investigate a prompt category, inspect a recent deployment, adjust a rubric, swap a model, or roll back a change.

Why the split matters

The architecture is deliberately modest. It does not propose a new distributed tracing system or a universal evaluation framework. It uses standard metrics plumbing and adds a clean contract: infrastructure health and output quality are different kinds of evidence, and both need to be observable over time.

The separation also keeps ownership clear. Serving infrastructure can expose request and GPU metrics with little application knowledge. Product, governance, and domain teams define quality rubrics that reflect the deployed use case. CloudWatch stores both streams, while Grafana presents views suited to different stakeholders: SREs looking for resource saturation and scaling failures, governance teams tracking safety and compliance thresholds, and product owners comparing models against user-facing quality goals.

AWS includes sample notebooks for configuring enhanced metrics, publishing custom quality metrics and alerts, and installing the Grafana dashboards. That turns the post from a conceptual checklist into a reproducible starting point, while leaving room for teams to replace the example metrics with domain-specific evaluations.

Takeaway

The broader engineering lesson is that an LLM endpoint is both a service and a changing behavioral system. Traditional observability answers whether the machinery is running. Continuous evaluation answers whether the machinery is still producing acceptable work. Neither view can stand in for the other.

For teams moving LLM applications into production, the practical pattern is to establish the two telemetry paths early. Measure latency, throughput, GPU allocation, and cost per model. Sample outputs and publish stable, application-specific quality scores. Pin evaluator dependencies, attach thresholds, and route quality failures into the same response process used for infrastructure incidents. The result is not perfect certainty about model behavior, but a much better operating posture: silent quality regressions become visible signals that can be correlated with the rest of the system.