Performance monitoring and metrics collection

1. Introduction

Imagine you're the admin of a website and every day a thousand users come by. In the logs you see that everything works, almost no errors (except someone sometimes forgets a password or messes up a captcha). "All good!" — you think.

And then someone opens a support ticket: the site is terribly slow, pages take 10 seconds to load. You dig into the logs — no errors! Great? No, because logs tell you what happened (or didn't), but they don't say how fast or slow it was, how many resources it took, or how that behavior changed as load increased.

This is where metrics come on stage — measurable characteristics of your application's operation. It's not just the number of errors, but average response time, memory usage, requests per second and other indicators that tell you about the system's health.

Comparison:
Logs — are "what happened".
Metrics — are "how well/poorly the system is working".
Trace — is "why the system behaves like this (in detail)".

What kinds of metrics exist, and what should you collect?

Main metric types:

Examples:

• Average and 95-th percentile of response time for GET requests.
• Number of users online right now.
• Memory usage (Private Bytes, Working Set).
• Frequency of errors like 500/503.
• DB queries per minute.

These metrics allow not only finding problems but also preventing them — because increased server load or a "creeping" response time can signal future outages.

2. How metrics collection works in .NET and the OpenTelemetry ecosystem

General architecture

In modern .NET (since .NET 6, especially in .NET 8/9) there is a standard metrics collection system based on OpenTelemetry.

Here's how it works:

1. Application code calls methods to increment counters, register gauges, record histograms.
2. OpenTelemetry Metrics SDK collects these metrics (in memory) and periodically exports them.
3. Metrics exporter sends them to the chosen monitoring system (Prometheus, Application Insights, Grafana Cloud, Datadog, etc.).
4. Monitoring backend aggregates, stores, visualizes, builds alerts and dashboards.

[Your application] 
       ⬇ 
 [Metrics collection (OpenTelemetry SDK)]
       ⬇
 [Metrics exporter (Prometheus, AI, Datadog, ...)]
       ⬇
 [Monitoring system/dashboards/alerts]

3. Practice: Basics of working with metrics in C#

Simple internal metrics: System.Diagnostics.Metrics

.NET provides a built-in metrics mechanism — System.Diagnostics.Metrics.

Main players: Meter, Counter<T>, Histogram<T>, ObservableGauge<T>.

Example: page visit counter

// Creating a Meter (usually one for the whole app)
using System.Diagnostics.Metrics;

static Meter meter = new Meter("MyCompany.MyApp", "1.0");

// Registering a counter
static Counter<long> homePageVisits = meter.CreateCounter<long>("HomePageVisits");

// Somewhere in a controller or service...
public void HomePageRequested()
{
    homePageVisits.Add(1);
    // Rest of the page handling code
}

Notes:

• Meter is a "factory" for metrics, with a unique name (application/company namespace).
• CreateCounter<long> creates a counter; increment via Add(1).

Example: measuring response time

static Histogram<double> pageLoadTime = meter.CreateHistogram<double>("PageLoadTimeMs");

// In the request handler:
public void OnRequest()
{
    var stopwatch = System.Diagnostics.Stopwatch.StartNew();

    // ...the request itself...

    stopwatch.Stop();
    pageLoadTime.Record(stopwatch.Elapsed.TotalMilliseconds);
}

Collecting gauges for dynamic values

A gauge is a metric that changes over time: number of connected users, current memory usage, etc.

static ObservableGauge<int> onlineUserGauge = meter.CreateObservableGauge(
    "OnlineUsers", 
    () => GetOnlineUserCount());

// Where GetOnlineUserCount is a method that returns the current value
static int GetOnlineUserCount()
{
    // Your real logic should be here!
    return ActiveUserList.Count;
}

In real life this works asynchronously: the app reports metric values and the exporter pulls them and pushes them out (for example, Prometheus scrapes the "/metrics" endpoint).

Adding metrics to a modern ASP.NET Core app

For ASP.NET Core a lot is available out of the box. Just add the package OpenTelemetry.Instrumentation.AspNetCore, and you'll get HTTP request metrics, response time, error counts, etc.

Example configuration in Program.cs:

using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics
            .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MyApp"))
            .AddAspNetCoreInstrumentation() // HTTP metrics
            .AddRuntimeInstrumentation()    // .NET CLR runtime metrics
            .AddProcessInstrumentation()    // process CPU/memory
            .AddMeter("MyCompany.MyApp")    // your metrics
            .AddPrometheusExporter();       // export to Prometheus
    });

var app = builder.Build();

app.MapGet("/", () => "Hello World!");

app.Run();

Now your app will expose metrics at /metrics, which can be scraped by Prometheus or other systems.

4. Practical examples of using metrics

Performance monitoring in real projects

We add metrics to find out:

• What average and peak RPS (requests per second) the API can handle?
• Where the bottlenecks are: one endpoint is 300 ms, another is 2000 ms?
• How much time is spent on DB calls? (add your own histograms)

Example: track DB query duration

static Histogram<double> dbQueryDuration = meter.CreateHistogram<double>("DbQueryDurationMs");

public async Task<List<Product>> GetProductsAsync()
{
    var sw = Stopwatch.StartNew();
    var result = await _db.Products.ToListAsync();
    sw.Stop();
    dbQueryDuration.Record(sw.Elapsed.TotalMilliseconds);
    return result;
}

Example: counting errors

static Counter<long> apiErrors = meter.CreateCounter<long>("ApiErrors");

public IActionResult SomeEndpoint()
{
    try
    {
        // some action
        return Ok();
    }
    catch (Exception)
    {
        apiErrors.Add(1);
        throw;
    }
}

Working with labels (tags) for metrics

It's important to group data by useful dimensions: endpoint, error type, user type, etc.

homePageVisits.Add(
    1, 
    KeyValuePair.Create<string, object>("UserType", "Admin"));

Or for a histogram:

dbQueryDuration.Record(
    sw.Elapsed.TotalMilliseconds, 
    KeyValuePair.Create<string, object>("QueryType", "GetProducts"));

Thanks to tags you can build graphs in Grafana not only for the whole app but also for specific segments.

5. Integration with Prometheus, Application Insights, Datadog, Grafana

Exporters and integration

• Prometheus — popular open-source monitoring, de-facto standard for cloud and Kubernetes.
• Application Insights — cloud integration for Azure.
• Datadog, Grafana Cloud — for professional infrastructures.

All these systems can collect metrics from .NET via OpenTelemetry exporters. Documentation on OTel exporters

Prometheus (steps):

1. Add the NuGet package: OpenTelemetry.Exporter.Prometheus.
2. Add .AddPrometheusExporter() to your metrics registration.
3. Configure Grafana datasource to Prometheus and build dashboards.

Useful links:

• OpenTelemetry .NET Metrics
• Azure Application Insights documentation
• Prometheus + .NET

6. Specifics, pitfalls and common mistakes

One common mistake — overly detailed tags. If you give tags too many unique values (for example, user/order IDs), the number of time series will explode — this will overload metric storage and increase costs (the so-called cardinality explosion). Keep tags coarse-grained.

Developers sometimes ignore System.Diagnostics.Metrics and ready-made tools, reinventing the wheel with logs and timers. As a result monitoring is less integrated and harder to maintain. Use standard tools and automatic instrumentation.

Another mistake — metrics are collected but not exported. Exporter configuration is mandatory: add, for example, .AddPrometheusExporter() and make sure the /metrics endpoint is available for scraping.

And finally, confusion between metric types: average response time is tracked with a Counter, while you should use Histogram — otherwise you won't see spikes and distribution. Counters are for counts; histograms are for time/size/distributions; gauges are for current states.

2

Task

C# SELF, level 64, lesson 3

Locked

Measuring Operation Execution Time Using a Histogram

Measuring Operation Execution Time Using a Histogram

Open