diff --git a/docs/core/diagnostics/debug-deadlock.md b/docs/core/diagnostics/debug-deadlock.md index cafe425a96f55..54c50220f53d8 100644 --- a/docs/core/diagnostics/debug-deadlock.md +++ b/docs/core/diagnostics/debug-deadlock.md @@ -1,15 +1,13 @@ --- -title: Debugging deadlock - .NET Core -description: A tutorial that walks you through debugging a locking issue in .NET Core. +title: Debugging deadlock - .NET +description: A tutorial that walks you through debugging a locking issue in .NET. ms.topic: tutorial -ms.date: 07/20/2020 +ms.date: 09/08/2026 --- -# Debug a deadlock in .NET Core +# Debug a deadlock in .NET -**This article applies to: ✔️** .NET Core 3.1 SDK and later versions - -In this tutorial, you'll learn how to debug a deadlock scenario. Using the provided example [ASP.NET Core web app](/samples/dotnet/samples/diagnostic-scenarios) source code repository, you can cause a deadlock intentionally. The endpoint will stop responding and experience thread accumulation. You'll learn how you can use various tools to analyze the problem, such as core dumps, core dump analysis, and process tracing. +In this tutorial, you'll learn how to debug a deadlock scenario. Using the provided example [ASP.NET Core web app](/samples/dotnet/samples/diagnostic-scenarios), you can cause a deadlock intentionally. The endpoint will stop responding and experience thread accumulation. You'll learn how to collect and analyze a process dump to identify the blocked threads, lock owners, and wait cycle. In this tutorial, you will: @@ -25,10 +23,9 @@ In this tutorial, you will: The tutorial uses: -- [.NET Core 3.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version +- A supported [.NET SDK](https://dotnet.microsoft.com/download/dotnet) - [Sample debug target - web app](/samples/dotnet/samples/diagnostic-scenarios) to trigger the scenario -- [dotnet-trace](dotnet-trace.md) to list processes -- [dotnet-dump](dotnet-dump.md) to collect, and analyze a dump file +- [dotnet-dump](dotnet-dump.md) to list processes and collect and analyze a dump file ## Core dump generation @@ -41,14 +38,18 @@ dotnet run To find the process ID, use the following command: ```dotnetcli -dotnet-trace ps +dotnet-dump ps ``` Take note of the process ID from your command output. Our process ID was `4807`, but yours will be different. Navigate to the following URL, which is an API endpoint on the sample site: `https://localhost:5001/api/diagscenario/deadlock` -The API request to the site will stop responding. Let the request run for about 10-15 seconds. Then create the core dump using the following command: +The API request to the site will stop responding. Let the request run for about 10-15 seconds. + +A dump is the recommended artifact for an existing deadlock because it preserves the current threads, lock owners, and wait cycle. If the deadlock is intermittent or you need to understand how it formed, start a contention and thread-time trace before reproducing it. For a Linux example, see [Capture deadlock formation](dotnet-trace-collect-linux-scenarios.md#capture-deadlock-formation). + +Create the core dump using the following command: ### [Linux](#tab/linux) @@ -259,7 +260,6 @@ The second thread is similar. It's also trying to acquire a lock that it already ## See also -- [dotnet-trace](dotnet-trace.md) to list processes - [dotnet-counters](dotnet-counters.md) to check managed memory usage - [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file - [dotnet/diagnostics](https://github.com/dotnet/diagnostics/tree/main/documentation/tutorial) @@ -267,4 +267,4 @@ The second thread is similar. It's also trying to acquire a lock that it already ## Next steps > [!div class="nextstepaction"] -> [What diagnostic tools are available in .NET Core](index.md) +> [What diagnostic tools are available in .NET](index.md) diff --git a/docs/core/diagnostics/debug-highcpu.md b/docs/core/diagnostics/debug-highcpu.md index 06e6c2a5ae658..2037644ac215b 100644 --- a/docs/core/diagnostics/debug-highcpu.md +++ b/docs/core/diagnostics/debug-highcpu.md @@ -1,15 +1,13 @@ --- -title: Debug high CPU usage - .NET Core -description: A tutorial that walks you through debugging high CPU usage in .NET Core. +title: Debug high CPU usage - .NET +description: A tutorial that walks you through debugging high CPU usage in .NET. ms.topic: tutorial -ms.date: 03/19/2026 +ms.date: 09/08/2026 --- -# Debug high CPU usage in .NET Core +# Debug high CPU usage in .NET -**This article applies to: ✔️** .NET Core 3.1 SDK and later versions - -In this tutorial, you'll learn how to debug an excessive CPU usage scenario. Using the provided example [ASP.NET Core web app](/samples/dotnet/samples/diagnostic-scenarios) source code repository, you can cause a deadlock intentionally. The endpoint will stop responding and experience thread accumulation. You'll learn how you can use various tools to diagnose this scenario with several key pieces of diagnostics data. +In this tutorial, you'll learn how to debug an excessive CPU usage scenario. Using the provided example [ASP.NET Core web app](/samples/dotnet/samples/diagnostic-scenarios), you can intentionally run CPU-intensive work and use metrics and platform-appropriate profiling tools to identify the expensive code. In this tutorial, you will: @@ -25,9 +23,9 @@ In this tutorial, you will: The tutorial uses: -- [.NET Core 3.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version. +- A supported [.NET SDK](https://dotnet.microsoft.com/download/dotnet). - [Sample debug target](/samples/dotnet/samples/diagnostic-scenarios) to trigger the scenario. -- [dotnet-trace](dotnet-trace.md) to list processes and generate a profile. +- [dotnet-trace](dotnet-trace.md) to collect CPU profiles and runtime traces. - [dotnet-counters](dotnet-counters.md) to monitor cpu usage. ## CPU counters @@ -171,15 +169,17 @@ Throughout the duration of the request, the CPU usage will hover around the incr At this point, you can safely say the CPU is running higher than you expect. Identifying the effects of a problem is key to finding the cause. We will use the effect of high CPU consumption in addition to diagnostic tools to find the cause of the problem. -## Analyze High CPU with Profiler +## Analyze high CPU with a profiler -When analyzing an app with high CPU usage, use a profiler to understand what the code is doing. `dotnet-trace collect` works on all operating systems, but safe-point bias and managed-only callstacks limit it to more general information than a kernel-aware profiler like ETW for Windows or `perf` for Linux. Depending on your operating system and .NET version, improved profiling capabilities might be available—see the platform-specific tabs that follow for detailed guidance. +When analyzing an app with high CPU usage, use a profiler to understand what the code is doing. `dotnet-trace collect` works on all operating systems, but safe-point bias and managed-only call stacks limit it to more general information than kernel-aware profiling through ETW on Windows or `perf_events` on Linux. Depending on your operating system and .NET version, improved profiling capabilities might be available. See the platform-specific tabs that follow for detailed guidance. ### [Linux](#tab/linux) +Prefer `dotnet-trace collect-linux` for the .NET-oriented Linux workflow. Use OneCollect `record-trace` when you need its lower-level scripting, filtering, or output controls, and use `perf` directly only when you need `perf.data`, perf-native analysis, or hardware performance counters. + #### Use `dotnet-trace collect-linux` (.NET 10+) -On .NET 10 and later, [`dotnet-trace collect-linux`](dotnet-trace.md#dotnet-trace-collect-linux) is the recommended profiling approach on Linux. It combines EventPipe with OS-level perf_events to produce a single unified trace that includes both managed and native callstacks, all without requiring a process restart. This requires root permissions and Linux kernel 6.4+ with `CONFIG_USER_EVENTS=y`. See [collect-linux prerequisites](dotnet-trace.md#prerequisites) for full requirements. +On .NET 10+, [`dotnet-trace collect-linux`](dotnet-trace.md#dotnet-trace-collect-linux) is the recommended Linux workflow. It retains .NET runtime and application event collection while adding kernel CPU samples, native call stacks, and selected Linux events through `perf_events`, all without requiring a process restart. This requires root permissions and Linux kernel 6.4+ with `CONFIG_USER_EVENTS=y`. See [collect-linux prerequisites](dotnet-trace.md#prerequisites) for full requirements. Ensure the [sample debug target](/samples/dotnet/samples/diagnostic-scenarios) is configured to target .NET 10 or later, then run it and exercise the high CPU endpoint (`https://localhost:5001/api/diagscenario/highcpu/60000`) again. While it's running within the 1-minute request, run `dotnet-trace collect-linux` to capture a machine-wide trace: @@ -191,11 +191,28 @@ Let it run for about 20-30 seconds, then press Ctrl+C or Enter [!div class="nextstepaction"] -> [Debug a deadlock in .NET Core](debug-deadlock.md) +> [Debug a deadlock in .NET](debug-deadlock.md) diff --git a/docs/core/diagnostics/debug-memory-leak.md b/docs/core/diagnostics/debug-memory-leak.md index 56cc9d217efc6..217b0f5f54843 100644 --- a/docs/core/diagnostics/debug-memory-leak.md +++ b/docs/core/diagnostics/debug-memory-leak.md @@ -2,13 +2,11 @@ title: Debug a memory leak tutorial description: Learn how to debug a memory leak in .NET. ms.topic: tutorial -ms.date: 11/13/2023 +ms.date: 09/08/2026 --- # Debug a memory leak in .NET -**This article applies to:** ✔️ .NET Core 3.1 SDK and later versions - Memory can leak when your app references objects that it no longer needs to perform the desired task. Referencing these objects prevents the garbage collector from reclaiming the memory used. That can result in performance degradation and an exception being thrown. This tutorial demonstrates the tools to analyze a memory leak in a .NET app using the .NET diagnostics CLI tools. If you're on Windows, you may be able to [use Visual Studio's Memory Diagnostic tools](/visualstudio/profiling/memory-usage) to debug the memory leak. @@ -27,7 +25,7 @@ In this tutorial, you will: The tutorial uses: -- [.NET Core 3.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version. +- A supported [.NET SDK](https://dotnet.microsoft.com/download/dotnet). - [dotnet-counters](dotnet-counters.md) to check managed memory usage. - [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file (includes the [SOS debugging extension](sos-debugging-extension.md)). - A [sample debug target](/samples/dotnet/samples/diagnostic-scenarios/) app to diagnose. @@ -141,6 +139,8 @@ Observe that the memory usage has grown to over 20 MB. By watching the memory usage, you can safely say that memory is growing or leaking. The next step is to collect the right data for memory analysis. +If you only need to compare managed heap composition or identify which object types are growing, start with [`dotnet-gcdump`](dotnet-gcdump.md), which collects less process state than a full dump. This tutorial uses `dotnet-dump` because the investigation continues from growing object types to the reference paths that keep those objects alive. + ### Generate memory dump When analyzing possible memory leaks, you need access to the app's memory heap to analyze the memory contents. Looking at relationships between objects, you create theories as to why memory isn't being freed. A common diagnostic data source is a memory dump on Windows or the equivalent core dump on Linux. To generate a dump of a .NET application, you can use the [dotnet-dump](dotnet-dump.md) tool. @@ -266,7 +266,7 @@ You can also delete the dump file that was created. ## See also -- [dotnet-trace](dotnet-trace.md) to list processes +- [dotnet-trace](dotnet-trace.md) to collect runtime performance traces - [dotnet-counters](dotnet-counters.md) to check managed memory usage - [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file - [dotnet/diagnostics](https://github.com/dotnet/diagnostics/tree/main/documentation/tutorial) @@ -275,4 +275,4 @@ You can also delete the dump file that was created. ## Next steps > [!div class="nextstepaction"] -> [Debug high CPU in .NET Core](debug-highcpu.md) +> [Debug high CPU in .NET](debug-highcpu.md) diff --git a/docs/core/diagnostics/debug-threadpool-starvation.md b/docs/core/diagnostics/debug-threadpool-starvation.md index b518aed76566c..6c1b685b25887 100644 --- a/docs/core/diagnostics/debug-threadpool-starvation.md +++ b/docs/core/diagnostics/debug-threadpool-starvation.md @@ -1,8 +1,8 @@ --- title: Debug ThreadPool Starvation -description: A tutorial that walks you through debugging and fixing a ThreadPool starvation issue on .NET Core +description: A tutorial that walks you through debugging and fixing a ThreadPool starvation issue on .NET. ms.topic: tutorial -ms.date: 04/19/2022 +ms.date: 09/04/2026 --- # Debug ThreadPool starvation @@ -36,8 +36,8 @@ The tutorial uses: Download the code for the [sample app](/samples/dotnet/samples/diagnostic-scenarios) and run it using the .NET SDK: ```dotnetcli -E:\demo\DiagnosticScenarios>dotnet run -Using launch settings from E:\demo\DiagnosticScenarios\Properties\launchSettings.json... +dotnet run +Using launch settings from /path/to/DiagnosticScenarios/Properties/launchSettings.json... info: Microsoft.Hosting.Lifetime[14] Now listening on: https://localhost:5001 info: Microsoft.Hosting.Lifetime[14] @@ -47,7 +47,7 @@ info: Microsoft.Hosting.Lifetime[0] info: Microsoft.Hosting.Lifetime[0] Hosting environment: Development info: Microsoft.Hosting.Lifetime[0] - Content root path: E:\demo\DiagnosticScenarios + Content root path: /path/to/DiagnosticScenarios ``` If you use a web browser and send requests to `https://localhost:5001/api/diagscenario/taskwait`, you should see the response `success:taskwait` returned after about 500 ms. This shows that the web server is serving traffic as expected. @@ -57,7 +57,7 @@ If you use a web browser and send requests to `https://localhost:5001/api/diagsc The demo web server has several endpoints which mock doing a database request and then returning a response to the user. Each of these endpoints has a delay of approximately 500 ms when serving requests one at a time but the performance is much worse when the web server is subjected to some load. Download the [Bombardier](https://github.com/codesenberg/bombardier/releases) load testing tool and observe the difference in latency when 125 concurrent requests are sent to each endpoint. ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskwait +bombardier https://localhost:5001/api/diagscenario/taskwait Bombarding https://localhost:5001/api/diagscenario/taskwait for 10s using 125 connection(s) [=============================================================================================] 10s Done! @@ -73,7 +73,7 @@ Statistics Avg Stdev Max This second endpoint uses a code pattern that performs even worse: ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/tasksleepwait +bombardier https://localhost:5001/api/diagscenario/tasksleepwait Bombarding https://localhost:5001/api/diagscenario/tasksleepwait for 10s using 125 connection(s) [=============================================================================================] 10s Done! @@ -147,7 +147,7 @@ If your app is running a version of .NET older than .NET 9, the output UI of dot The preceding counters are an example while the web server wasn't serving any requests. Run Bombardier again with the `api/diagscenario/tasksleepwait` endpoint and sustained load for 2 minutes so there's plenty of time to observe what happens to the performance counters. ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/tasksleepwait -d 120s +bombardier https://localhost:5001/api/diagscenario/tasksleepwait -d 120s ``` ThreadPool starvation occurs when there are no free threads to handle the queued work items and the runtime responds by increasing the number of ThreadPool threads. The `dotnet.thread_pool.thread.count` value increases rapidly to 2-3x the number of processor cores on your machine, and then further threads are added 1-2 per second until stabilizing somewhere above 125. The key signals that ThreadPool starvation is currently a performance bottleneck are the slow and steady increase of ThreadPool threads and CPU Usage much less than 100%. The thread count increase will continue until either the pool hits the maximum number of threads, enough threads have been created to satisfy all the incoming work items, or the CPU has been saturated. Often, but not always, ThreadPool starvation will also show large values for `dotnet.thread_pool.queue.length` and low values for `dotnet.thread_pool.work_item.count`, meaning that there's a large amount of pending work and little work being completed. Here's an example of the counters while the thread count is still rising: @@ -202,7 +202,7 @@ Once the count of ThreadPool threads stabilizes, the pool is no longer starving. Starting in .NET 6, ThreadPool heuristics were modified to scale up the number of ThreadPool threads much faster in response to certain blocking Task APIs. ThreadPool starvation can still occur with these APIs, but the duration is much briefer than it was with older .NET versions because the runtime responds more quickly. Run Bombardier again with the `api/diagscenario/taskwait` endpoint: ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskwait -d 120s +bombardier https://localhost:5001/api/diagscenario/taskwait -d 120s ``` On .NET 6 you should observe the pool increase the thread count more quickly than before and then stabilize at a high number of threads. ThreadPool starvation is occurring while the thread count is climbing. @@ -216,7 +216,7 @@ To eliminate ThreadPool starvation, ThreadPool threads need to remain unblocked Run Bombardier again to put the web server under load: ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskwait -d 120s +bombardier https://localhost:5001/api/diagscenario/taskwait -d 120s ``` Then run dotnet-stack to see the thread stack traces: @@ -303,15 +303,17 @@ There's one particular event that helps diagnosing thread pool starvation: the W Run Bombardier again to put the web server under load: ```dotnetcli -bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskwait -d 120s +bombardier https://localhost:5001/api/diagscenario/taskwait -d 120s ``` Then run dotnet-trace to collect wait events: ```dotnetcli -dotnet trace collect -n DiagnosticScenarios --clrevents waithandle --clreventlevel verbose --duration 00:00:30 +dotnet-trace collect -n DiagnosticScenarios --clrevents waithandle --clreventlevel verbose --duration 00:00:30 ``` +On .NET 10+ Linux, prefer the [`collect-linux` blocking configuration](dotnet-trace-collect-linux-performance.md#blocking-contention-and-threadpool-behavior) when its prerequisites are met. It records the same focused runtime signals together with native stacks and Linux scheduling context, which helps distinguish blocked workers from runnable workers that aren't receiving CPU. + That should generate a file named `DiagnosticScenarios.exe_yyyyddMM_hhmmss.nettrace` containing the events. This nettrace can be analyzed using two different tools: - [PerfView](https://github.com/microsoft/perfview/releases): A performance analysis tool developed by Microsoft for Windows only. @@ -390,7 +392,7 @@ public async Task> TaskAsyncWait() Running Bombadier to send load to the `api/diagscenario/taskasyncwait` endpoint shows that the ThreadPool thread count stays much lower and average latency remains near 500ms when using the async/await approach: ```dotnetcli ->bombardier-windows-amd64.exe https://localhost:5001/api/diagscenario/taskasyncwait +bombardier https://localhost:5001/api/diagscenario/taskasyncwait Bombarding https://localhost:5001/api/diagscenario/taskasyncwait for 10s using 125 connection(s) [=============================================================================================] 10s Done! @@ -402,3 +404,9 @@ Statistics Avg Stdev Max others - 0 Throughput: 98.81KB/s ``` + +## See also + +- [Diagnose performance issues in .NET applications](performance-diagnostics.md) +- [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md) +- [`dotnet-stack`](dotnet-stack.md) diff --git a/docs/core/diagnostics/dotnet-trace-collect-linux-performance.md b/docs/core/diagnostics/dotnet-trace-collect-linux-performance.md new file mode 100644 index 0000000000000..1ff21e343ac8e --- /dev/null +++ b/docs/core/diagnostics/dotnet-trace-collect-linux-performance.md @@ -0,0 +1,251 @@ +--- +title: Investigate Linux performance with dotnet-trace +description: Use dotnet-trace collect-linux to investigate CPU, memory, blocking, I/O, exception, and startup performance issues. +ms.date: 09/08/2026 +ms.topic: how-to +#Customer intent: As a .NET developer on Linux, I want to collect and analyze the right trace data to find the cause of a performance problem. +--- + +# Investigate Linux performance with `dotnet-trace collect-linux` + +`dotnet-trace collect-linux` records .NET runtime events together with Linux CPU samples, native call stacks, process activity, scheduling data, and kernel events. Use it when you need to correlate what a .NET application is doing with what is happening on the rest of the Linux machine. + +Start with [Diagnose performance issues in .NET applications](performance-diagnostics.md) to choose the primary workflow for the observed symptom. When that workflow calls for a Linux performance trace, this article explains how to select `collect-linux` data, move from a broad trace to a focused trace, and analyze the result. For complete command syntax and platform requirements, see the [`collect-linux` reference](dotnet-trace.md#dotnet-trace-collect-linux). + +For hands-on practice choosing among the configurations in this article, use the runnable [Linux performance investigation scenarios](dotnet-trace-collect-linux-scenarios.md). + +## Start with a short machine-wide trace + +When the cause is unknown, begin with a short, machine-wide trace that intentionally combines only the `dotnet-common` and `cpu-sampling` profiles. `dotnet-common` provides lightweight .NET runtime context, while `cpu-sampling` provides Linux CPU samples and native call stacks. Together they can show whether CPU is consumed by the application, the runtime, native or kernel code, or another process without enabling every high-volume event source. + +Machine-wide collection also preserves evidence that a process-only trace can miss: + +- CPU consumed by competing processes. +- Short-lived child processes. +- Work performed before the target process can be attached. +- Application, runtime, native-library, and kernel CPU in the same time range. + +The initial trace omits `thread-time` deliberately. Scheduler and context-switch events can make a machine-wide trace much larger. Add `thread-time` when blocking is already the symptom, or as a focused follow-up when the target is slow while using little CPU. + +```dotnetcli +sudo dotnet-trace collect-linux --profile dotnet-common,cpu-sampling +``` + +Start collection shortly before reproducing the problem, capture a representative 10-30 second interval, and stop collection after the symptom occurs. Keeping the first trace short bounds collection overhead and trace size while retaining enough context to choose the next investigation step. Press Enter or Ctrl+C to stop interactively, or add a duration such as `--duration 00:00:00:30`. The profile in the preceding command is the default configuration, shown explicitly. + +Resolve native and ReadyToRun frames as described in [Get symbols for native runtime frames](dotnet-trace.md#get-symbols-for-native-runtime-frames). + +Open the trace in a current version of PerfView and filter the result to the relevant process and time range. + +Use the first trace to choose a direction, and then collect only the detailed events required to answer the next question. High event rates increase trace size and collection overhead, so use a focused configuration from the start for long-running collection. + +## Understand the available data + +### CPU samples show what was running + +CPU sampling periodically records the stack running on each CPU. Use CPU samples to determine: + +- Which process consumed CPU. +- Which application method or native function used CPU. +- Whether work moved into the runtime, a native library, or the kernel. +- Whether another process competed with the target for CPU. + +CPU samples don't explain time spent sleeping or waiting because a thread that isn't running can't be sampled. + +### Thread-time data shows why threads weren't running + +Thread-time data records scheduling and context-switch activity. It distinguishes among: + +| Thread state | Interpretation | +| --- | --- | +| Running and accumulating CPU time | The thread is executing; use CPU stacks to find the expensive path. | +| Waiting in a lock, task, timer, sleep, or I/O call | The thread can't run until the operation completes. | +| Ready to run but receiving little CPU | Other runnable work is competing for CPU. | +| ThreadPool worker waiting synchronously | The worker can't process queued work and might contribute to starvation. | + +PerfView can label time spent not executing as *blocked time* or *off-CPU time*. This time is summed across threads and can exceed the wall-clock duration of the trace. + +### .NET runtime events describe runtime activity + +Runtime events provide details that CPU samples alone don't contain: + +| Runtime data | Questions it can answer | +| --- | --- | +| GC and allocation | Which types allocate? Where are they allocated? Why and how often does GC run? How long are pauses? | +| Exceptions | Which exceptions are thrown, even when caught? Which path repeatedly throws them? | +| Contention | Which lock acquisition path waits, and for how long? | +| Threading | Is the ThreadPool adding workers? Are workers blocked? | +| JIT and loader | Which methods are compiled and which assemblies load during startup? | + +The `dotnet-common` profile provides useful summary events. Detailed allocation, contention, exception, and JIT analysis usually requires a focused follow-up trace. + +### Linux events connect operations to callers + +Linux perf events can record syscalls, process lifecycle, scheduling, and other kernel activity. Event call stacks connect an operation such as `read`, `write`, `fsync`, or `execve` to the managed or native caller that initiated it. + +## Choose a focused follow-up trace + +The examples show only the event configuration. Add `--duration` or `--output` when appropriate. Use `--name` or `--process-id` only when the target process is already running. To trace startup, collect machine-wide and filter to the process during analysis. + +### General starting point + +This is the default configuration, written explicitly: + +```dotnetcli +sudo dotnet-trace collect-linux \ + --profile dotnet-common,cpu-sampling +``` + +### Allocation and GC + +```dotnetcli +sudo dotnet-trace collect-linux \ + --profile gc-verbose,cpu-sampling +``` + +### Blocking, contention, and ThreadPool behavior + +```dotnetcli +sudo dotnet-trace collect-linux \ + --profile thread-time,cpu-sampling \ + --clrevents threading+contention+waithandle \ + --clreventlevel Verbose +``` + +### First-chance exceptions + +This configuration includes exceptions that application code catches: + +```dotnetcli +sudo dotnet-trace collect-linux \ + --profile cpu-sampling \ + --clrevents exception \ + --clreventlevel Verbose +``` + +### Reads, writes, and durable flushes + +```dotnetcli +sudo dotnet-trace collect-linux \ + --profile thread-time,cpu-sampling \ + --perf-events "syscalls:sys_enter_read,syscalls:sys_exit_read,syscalls:sys_enter_write,syscalls:sys_exit_write,syscalls:sys_enter_fsync,syscalls:sys_exit_fsync" +``` + +### Startup and JIT + +Begin collection before launching the application: + +```dotnetcli +sudo dotnet-trace collect-linux \ + --profile cpu-sampling \ + --clrevents assemblyloader+loader+jit+threading \ + --clreventlevel Verbose +``` + +Profiles are presets, and presets for different data sources can be combined. For example, `cpu-sampling` and `thread-time` configure Linux collection while `--clrevents` configures the .NET runtime provider. + +Configurations don't merge when `--providers`, `--profile`, and `--clrevents` configure the same .NET provider. The precedence is: + +1. An explicit `--providers` entry. +1. The first selected profile that configures the provider. +1. `--clrevents`, only when nothing earlier configured `Microsoft-Windows-DotNETRuntime`. + +For example, if `dotnet-common` or `gc-verbose` configures the runtime provider, the tool prints a warning and ignores a supplied `--clrevents` list. Use kernel-only profiles such as `cpu-sampling` or `thread-time` with a focused `--clrevents`/`--clreventlevel` configuration. + +## Analyze the trace + +### 1. Select the symptom interval + +Open the trace in [PerfView](https://github.com/microsoft/perfview) and select the time interval in which the slowdown, memory growth, pause, or startup delay occurred. Analysis outside that interval can hide a short problem beneath otherwise normal behavior. + +### 2. Examine the whole machine + +Open **CPU Stacks** and compare processes: + +- Does the target own most CPU samples? +- Are other processes using the CPUs at the same time? +- Did many child processes appear? +- Is CPU spread across many threads or concentrated in one? + +Another process having many samples doesn't by itself prove that it slowed the target. External CPU competition becomes a supported explanation when its CPU use overlaps the slowdown, the machine is close to saturation, and runnable target threads receive less CPU than expected. A thread-time trace can confirm the last condition. + +### 3. Follow the target's CPU stacks + +Filter to the target process. Begin with methods that have high *exclusive* cost, where samples occurred directly in the method. Then inspect their *inclusive* callers until the stack reaches application code. + +If native frames remain unresolved, configure the [matching native symbols](dotnet-trace.md#get-symbols-for-native-runtime-frames) before relying on method names in those frames. + +Common directions include: + +- One application method dominates: investigate that algorithm or call site. +- GC methods dominate: inspect GC and allocation data. +- Exception helpers dominate: inspect exception events. +- Lock slow paths appear: inspect contention and thread-time data. +- Filesystem, socket, or syscall frames appear: inspect Linux events. +- `libclrjit` dominates early in the trace: inspect JIT and loader events. + +Inlining can remove small logical methods from physical stacks. Use source, disassembly, or a diagnostic no-inline build when samples reach only a broad surviving caller. + +### 4. Investigate memory and GC + +Use these PerfView views: + +1. **GCStats** to review collection generation, reason, frequency, pause time, promoted data, and heap size. +1. **GC Heap Alloc Ignore Free (Coarse Sampling)** to find dominant allocation types and their application callers. +1. **Events** filtered to `GC/AllocationTick`, `GC/Start`, and `GC/Stop` for exact timing and payload fields. + +Allocation traces show where objects are created, not why they remain alive. Use [`dotnet-gcdump`](dotnet-gcdump.md) or a [process dump](dotnet-dump.md) for retention roots. If process RSS grows while the managed heap remains flat, inspect native allocations and operating-system memory mappings. + +### 5. Investigate low-CPU latency + +Open thread-time stacks for the target and compare CPU time with blocked time. Look for: + +- Monitor, reader/writer lock, task wait, sleep, futex, timer, and I/O frames. +- ThreadPool worker starts, adjustments, and cooperative-blocking events. +- The application callback immediately above a wait. +- Runnable target threads receiving little CPU while other processes execute. + +Contention events can identify a waiting lock path and duration without showing what the owner was doing. Use a process dump to prove an existing deadlock and its lock-ownership cycle. + +Async work can resume on another thread, so a physical wait stack doesn't always preserve the logical initiating caller. Use activities, distributed tracing, or application instrumentation when that relationship is required. + +### 6. Investigate I/O + +Filter events to the relevant syscall and open its stacks. Follow `read`, `write`, `fsync`, or a socket operation back to application code. Thread-time data shows whether the operation blocked threads or occupied ThreadPool workers. + +Add block-device events only when the remaining question is storage-device latency. They are machine-wide and can be noisy. They aren't required when syscall stacks already identify an application issuing excessive small operations or forced flushes. + +### 7. Investigate exceptions and startup + +For exceptions, count first-chance events and inspect their stacks. A repeated throw site can reduce throughput even when every exception is caught and no error is logged. + +For startup, begin collection before launching the application. Correlate early CPU activity with: + +- JIT method events and `libclrjit` stacks. +- Assembly and module loading. +- Static initialization. +- File operations and process launches. + +Short-lived child processes can appear in process lifecycle events even when they don't run long enough to receive a CPU sample. + +## Know when to use another artifact + +See [Choose a different artifact when a performance trace can't answer the question](performance-diagnostics.md#choose-a-different-artifact-when-a-performance-trace-cant-answer-the-question) for artifact-selection guidance. In particular, use a native memory profiler for native allocation ownership, Linux `perf` for hardware performance counters, and source or disassembly when optimization removed a logical method from the physical call stack. + +## Manage collection overhead + +Trace overhead depends on event rate, enabled providers, stack capture, CPU count, and workload behavior. High-volume traces can lose events or perturb the application being measured. + +- Keep the initial trace short. +- Enable only the detailed events needed for the current question. +- Treat event totals as lower bounds when they disagree with application or operating-system counters. +- Repeat the capture with a narrower configuration when exact counts matter. +- Test production collection procedures and container limits before an incident. + +## See also + +- [Diagnose performance issues in .NET applications](performance-diagnostics.md) +- [Practice Linux performance investigations](dotnet-trace-collect-linux-scenarios.md) +- [`dotnet-trace` reference](dotnet-trace.md) +- [Debug high CPU usage](debug-highcpu.md) +- [Collect diagnostics in Linux containers](diagnostics-in-containers.md) diff --git a/docs/core/diagnostics/dotnet-trace-collect-linux-scenarios.md b/docs/core/diagnostics/dotnet-trace-collect-linux-scenarios.md new file mode 100644 index 0000000000000..94cdcccfa0343 --- /dev/null +++ b/docs/core/diagnostics/dotnet-trace-collect-linux-scenarios.md @@ -0,0 +1,268 @@ +--- +title: Practice Linux performance investigation scenarios +description: Use runnable performance scenarios to learn how to choose and analyze dotnet-trace collect-linux configurations. +ms.date: 09/04/2026 +ms.topic: tutorial +#Customer intent: As a .NET developer on Linux, I want hands-on examples that teach me how to select trace data and diagnose different performance symptoms. +--- + +# Practice Linux performance investigations with `dotnet-trace collect-linux` + +This tutorial uses one sample application to create CPU, memory, garbage collection, blocking, contention, I/O, startup, exception, process, and mixed-cause performance problems. Each group starts from an observed symptom and uses a different collection strategy or analysis pivot. Work through the scenarios without inspecting the implementation first if you want to practice diagnosing an unknown cause. + +For an explanation of the trace data and analysis views used in this tutorial, see [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md). + +## Prerequisites + +The tutorial requires: + +- Linux that meets the [`collect-linux` prerequisites](dotnet-trace.md#prerequisites). +- The [.NET 10 SDK](https://dotnet.microsoft.com/download/dotnet/10.0). +- The latest [`dotnet-trace`](dotnet-trace.md) global tool. +- A current version of [PerfView](https://github.com/microsoft/perfview) for analysis. +- The [performance scenarios sample](/samples/dotnet/samples/dotnet-trace-collect-linux-performance-scenarios/). + +From the sample directory, build the application and list its scenarios: + +```dotnetcli +dotnet build -c Release +dotnet run -c Release --no-build -- --list +``` + +Run workloads for 30 to 60 seconds so that the symptom is visible while you collect a trace. Each workload prints its process ID, symptom, and configured duration: + +```dotnetcli +dotnet run -c Release --no-build -- cpu-hotspot 45 +``` + +Use a second terminal for collection. Start with the [short machine-wide trace](dotnet-trace-collect-linux-performance.md#start-with-a-short-machine-wide-trace), which preserves competing processes, child processes, and work performed before a process can be attached. Start the workload shortly before collection unless a scenario specifically instructs you to start collection first. Open each trace in PerfView, select the interval in which the workload ran, and filter to the relevant process when the investigation doesn't require whole-machine context. + +## Establish a healthy baseline + +Run the control before diagnosing the intentionally unhealthy scenarios: + +```dotnetcli +dotnet run -c Release --no-build -- healthy 45 +``` + +Collect the default `dotnet-common,cpu-sampling` trace. CPU, memory, GC, and latency should remain stable, with no dominant anomalous stack, wait, pause, or event rate. Use this result to learn what ordinary runtime, scheduler, and process activity looks like instead of treating every event in a trace as a problem. + +## Diagnose CPU consumption + +CPU sampling answers what was executing. Use the [general starting configuration](dotnet-trace-collect-linux-performance.md#general-starting-point) for these scenarios. + +### Find a managed CPU hotspot + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- cpu-hotspot 45 +``` + +In **CPU Stacks**, filter to the sample process and find the application method with the greatest exclusive cost. The recursive `Fibonacci` method should own nearly all samples. This is the direct case in which CPU sampling identifies both the expensive method and its application caller. + +### Recognize an inlining attribution limit + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- inlining 45 +``` + +CPU samples reach the surviving application frame, but optimized helper methods and their logical callers might not appear as separate frames. The trace still localizes the expensive region, but source inspection, disassembly, or a diagnostic build with inlining disabled is required to divide cost among methods that optimization removed from the physical stack. + +### Follow managed code into native code + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- native-cpu 45 +``` + +The sample repeatedly copies native memory through the C runtime. Follow CPU stacks from the managed P/Invoke frame into `memcpy`. If the native function is unresolved, configure the [matching native symbols](dotnet-trace.md#get-symbols-for-native-runtime-frames) before assigning cost to an address. + +## Diagnose allocation and garbage collection + +When the initial trace shows significant GC activity, use the focused [allocation and GC configuration](dotnet-trace-collect-linux-performance.md#allocation-and-gc). Analyze it with the [memory and GC workflow](dotnet-trace-collect-linux-performance.md#4-investigate-memory-and-gc). + +### Correlate allocation, GC, and CPU + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- allocation-gc 45 +``` + +The focused trace should identify `System.String` as the dominant allocation type and reach the sample's string-building path. Correlate allocation timestamps with collections and CPU samples rather than assuming that visible GC activity is the only source of CPU cost. + +### Identify large object heap pressure + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- loh-gc 45 +``` + +Look for large `System.Byte[]` allocations, generation 2 collections, and large object heap data. This distinguishes a workload that allocates a modest number of large objects from one that creates a high count of small objects. + +### Attribute induced collections + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- induced-gc 45 +``` + +Inspect GC start events for the `Induced` reason and follow the application stack to `GC.Collect`. This proves that explicit collection, rather than allocation volume alone, is causing frequent stop-the-world pauses. + +### Separate managed retention from native growth + +Run the managed growth scenario: + +```dotnetcli +dotnet run -c Release --no-build -- managed-memory-growth 45 +``` + +The managed heap and sampled `System.Byte[]` allocations increase together. The trace identifies where objects are allocated, but it doesn't prove why they remain reachable. Use [`dotnet-gcdump`](dotnet-gcdump.md) for heap composition or [`dotnet-dump`](dotnet-dump.md) for retention roots. + +Then run: + +```dotnetcli +dotnet run -c Release --no-build -- native-memory-growth 45 +``` + +Process RSS rises while managed heap metrics remain nearly flat. That contrast rules out a managed retention problem. Switch to a native memory profiler or operating-system memory mapping data to attribute native allocations; collecting more managed allocation events won't recover that information. + +## Diagnose blocking, starvation, and contention + +Use the focused [blocking, contention, and ThreadPool configuration](dotnet-trace-collect-linux-performance.md#blocking-contention-and-threadpool-behavior) when an application is slow but consumes little CPU, or when runnable work appears to receive less CPU than expected. + +### Find sync-over-async ThreadPool starvation + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- sync-over-async 45 +``` + +Look for ThreadPool worker growth, cooperative-blocking events, and worker stacks waiting in `Task` methods. Together, these signals distinguish sync-over-async starvation from an application that is merely idle while awaiting asynchronous work. + +### Recognize an async causality boundary + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- async-delay 45 +``` + +Thread-time data proves that the operation spends most of its time waiting on timers with little CPU consumption. A physical wait stack might not preserve the logical caller that initiated an asynchronous operation. Pivot to activities, distributed tracing, or application instrumentation when the question is which request or business operation initiated the delay. + +### Distinguish reader and writer contention + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- lock-contention 45 +``` + +Compare reader and writer lock-acquisition stacks and inspect contention duration. The trace should show writer waits while readers repeatedly hold the lock, which is more actionable than a generic conclusion that the process is blocked. + +### Capture deadlock formation + +Start the focused blocking collection before launching this scenario. While collection is running, use another terminal: + +```dotnetcli +dotnet run -c Release --no-build -- deadlock 15 +``` + +Starting first preserves the opposing acquisition paths as the deadlock forms. Attaching after the process is already deadlocked can't reconstruct earlier lock events. Use a process dump to prove the current owners and complete lock cycle. + +### Find more than one cause + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- cpu-and-contention 45 +``` + +The same focused trace should retain a CPU-intensive application path and independent lock contention. Investigate each signal instead of stopping after the first plausible cause, especially when one finding doesn't explain every observed symptom. + +## Diagnose file I/O + +Use the focused [reads, writes, and durable flushes configuration](dotnet-trace-collect-linux-performance.md#reads-writes-and-durable-flushes) to collect syscall stacks together with thread-time data. + +### Find syscall amplification + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- tiny-writes 45 +``` + +Filter events to `write` and follow their call stacks to the one-byte application write. A high syscall rate for little useful data identifies batching as the likely optimization. Treat trace event counts as lower bounds if collection reports or external counters indicate event loss. + +### Connect synchronous I/O to unavailable workers + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- sync-io-threadpool 45 +``` + +Look for repeated `fsync` calls beneath ThreadPool worker file operations, then use thread-time data to see the resulting worker unavailability. This scenario requires both I/O events and scheduler context; either source alone gives an incomplete explanation. + +## Diagnose exceptions + +Run: + +```dotnetcli +dotnet run -c Release --no-build -- swallowed-exceptions 45 +``` + +Use the focused [first-chance exception configuration](dotnet-trace-collect-linux-performance.md#first-chance-exceptions). + +Count `InvalidOperationException` events and inspect their call stacks. First-chance events reveal repeated exceptions even though the application catches them and writes no error log. + +## Diagnose startup and process activity + +Startup and short-lived process work can finish before you attach to the target. Start machine-wide collection first. + +### Find JIT-dominated startup + +Begin the focused [startup and JIT collection](dotnet-trace-collect-linux-performance.md#startup-and-jit), then run: + +```dotnetcli +dotnet run -c Release --no-build -- jit-startup 15 +``` + +Correlate early CPU samples with JIT method events and `libclrjit` stacks. The workload becomes idle after generating many methods, so attaching after startup would miss the expensive phase. + +### Identify short-lived child processes + +Collect the default machine-wide trace first, then run: + +```dotnetcli +dotnet run -c Release --no-build -- process-churn 15 +``` + +Process lifecycle events identify repeated `/bin/true` launches even when individual children are too short-lived to receive CPU samples. Follow the parent process stack to the process-start path. + +### Prove external CPU competition + +Collect the default machine-wide trace while running: + +```dotnetcli +dotnet run -c Release --no-build -- cpu-competition 30 +``` + +Compare CPU across all processes and add thread-time data in a follow-up trace if needed. External competition is supported when competing processes consume the machine during the slowdown and runnable target threads receive less CPU than expected. A process-only trace would hide the competing workers. + +## Compare your conclusion with the source + +After writing down the evidence and conclusion for a scenario, inspect the [sample implementation](/samples/dotnet/samples/dotnet-trace-collect-linux-performance-scenarios/). If the trace doesn't contain the relationship required to prove the cause, identify the appropriate pivot tool rather than treating the expected implementation as evidence. + +## See also + +- [Diagnose performance issues in .NET applications](performance-diagnostics.md) +- [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md) +- [`dotnet-trace` reference](dotnet-trace.md) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index 447b4a905a3fd..4932c5c5360bc 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -1,7 +1,7 @@ --- title: dotnet-trace diagnostic tool - .NET CLI -description: Learn how to install and use the dotnet-trace CLI tool to collect .NET traces of a running process without the native profiler, by using the .NET EventPipe. -ms.date: 06/10/2026 +description: Learn how to use dotnet-trace to collect .NET application traces and Linux system-wide performance traces. +ms.date: 09/04/2026 ms.topic: reference ms.custom: sfi-ropc-nochange --- @@ -295,6 +295,8 @@ dotnet-trace collect Collects diagnostic traces using perf_events, a Linux OS technology. `collect-linux` enables the following additional features over [`collect`](#dotnet-trace-collect). +For a symptom-driven collection and analysis workflow, see [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md). + | Feature | `collect` | `collect-linux` | |------------------------------------------|-----------|-----------------------------------| | Supported OS | Any | Linux only, kernel version >= 6.4 | @@ -705,11 +707,17 @@ However, when you want to gain a finer control over the lifetime of the app bein ### Get symbols for native runtime frames -`collect-linux` captures native frames in callstacks. To resolve native method names for runtime libraries (such as `libcoreclr.so`), place the corresponding debug symbol files on disk beside the libraries. Without these symbols, native frames appear as unresolved addresses in the trace. +`collect-linux` captures native and ReadyToRun (R2R) frames in call stacks. [PerfView and TraceEvent 3.2.1 or later](https://github.com/microsoft/perfview/releases/tag/v3.2.1) can download and resolve symbols when you analyze the trace: + +- .NET native and R2R symbols are available from the Microsoft Symbol Server. +- Many Azure Linux native symbols are also available from the Microsoft Symbol Server. +- For native libraries from other Linux distributions, configure PerfView or TraceEvent with a local symbol path that contains the matching distribution symbol files. -`collect-linux` dynamically enables perf map generation for JIT-compiled code when the trace begins, so you don't need to restart any .NET processes. +In PerfView, open a stack view, select the unresolved module frames, and choose **Lookup Symbols**. If prompted, enable the Microsoft Symbol Server. TraceEvent applications can perform the same analysis-time lookup through `SymbolReader`. No symbol setup is required before collection for symbols available from the configured server or local symbol paths. -To download native runtime symbols, use [dotnet-symbol](./dotnet-symbol.md): +`collect-linux` dynamically enables perf map generation for JIT-compiled code, so you don't need to restart .NET processes. + +For an offline workflow, use [dotnet-symbol](./dotnet-symbol.md) before collection to place .NET native symbols beside the corresponding runtime libraries: 1. Install `dotnet-symbol`: @@ -727,6 +735,8 @@ To download native runtime symbols, use [dotnet-symbol](./dotnet-symbol.md): After you place the symbols, `collect-linux` resolves native method names when it collects the trace. +### Collect the trace + This example captures CPU samples for all processes on the machine. Any processes running .NET 10+ will also include some additional lightweight events describing GC, JIT, and Assembly loading behavior. ```output diff --git a/docs/core/diagnostics/eventpipe.md b/docs/core/diagnostics/eventpipe.md index 7e4b8ce8434c2..67843eda1d349 100644 --- a/docs/core/diagnostics/eventpipe.md +++ b/docs/core/diagnostics/eventpipe.md @@ -1,7 +1,7 @@ --- title: EventPipe Overview description: Learn about EventPipe and how to use it for tracing your .NET applications to diagnose performance issues. -ms.date: 03/19/2026 +ms.date: 09/04/2026 ms.topic: overview --- @@ -23,9 +23,9 @@ To learn more about the NetTrace format, see the [NetTrace format documentation] ## EventPipe vs. ETW/perf_events -EventPipe is part of the .NET runtime and is designed to work the same way across all the platforms .NET Core supports. This allows tracing tools based on EventPipe, such as `dotnet-counters`, `dotnet-gcdump`, and `dotnet-trace`, to work seamlessly across platforms. +EventPipe is part of the .NET runtime and is designed to work the same way across all the platforms .NET supports. This allows tracing tools based on EventPipe, such as `dotnet-counters`, `dotnet-gcdump`, and `dotnet-trace`, to work seamlessly across platforms. -However, because EventPipe is a runtime built-in component, its scope is limited to managed code and the runtime itself. Without other tracing tools, EventPipe events include stack traces with managed code frame information only. To get events from other unmanaged user-mode libraries, CPU sampling for native code, or kernel events, use OS-specific tracing tools such as ETW or perf_events. On Linux, the [perfcollect tool](./trace-perfcollect-lttng.md) helps automate using perf_events and [LTTng](https://en.wikipedia.org/wiki/LTTng). +However, because EventPipe is a runtime built-in component, its scope is limited to managed code and the runtime itself. Without other tracing tools, EventPipe events include stack traces with managed code frame information only. To collect events from other unmanaged user-mode libraries, CPU samples for native code, or kernel events, use platform tracing facilities and collectors. Examples include ETW on Windows, [`dotnet-trace collect-linux`](./dotnet-trace.md#dotnet-trace-collect-linux) or `perf` on Linux, and [OneCollect `record-trace`](https://github.com/microsoft/one-collect/tree/main/record-trace) on either platform. Starting in .NET 10, EventPipe on Linux can emit events as [user_events](https://docs.kernel.org/trace/user_events.html), enabling collection of managed events, OS/kernel events, and native callstacks in a single unified trace. This mode requires admin/root privileges and Linux kernel 6.4+. For more information, see [`dotnet-trace collect-linux`](./dotnet-trace.md#dotnet-trace-collect-linux). @@ -35,7 +35,7 @@ The following table is a summary of the differences between EventPipe and ETW/pe |Feature|EventPipe|EventPipe (user_events)|ETW|perf_events| |-------|---------|----------------------|---|-----------| -|Cross-platform|Yes|No (only on supported Linux distros)|No (only on Windows)|No (only on supported Linux distros)| +|Cross-platform|Yes|No (Linux only)|No (Windows only)|No (Linux only)| |Require admin/root privilege|No|Yes|Yes|Yes| |Can get OS/kernel events|No|Yes|Yes|Yes| |Can resolve native callstacks|No|Yes|Yes|Yes| diff --git a/docs/core/diagnostics/index.md b/docs/core/diagnostics/index.md index 6d9a59c4969a4..44762e45349c5 100644 --- a/docs/core/diagnostics/index.md +++ b/docs/core/diagnostics/index.md @@ -1,9 +1,9 @@ --- -title: Diagnostics tools overview - .NET Core -description: An overview of the tools and techniques available to diagnose .NET Core applications. -ms.date: 10/20/2023 +title: Diagnostics tools overview - .NET +description: An overview of the tools and techniques available to diagnose .NET applications. +ms.date: 09/08/2026 ms.topic: overview -#Customer intent: As a .NET Core developer I want to find the best tools to help me diagnose problems so that I can be productive. +#Customer intent: As a .NET developer, I want to find the best tools to help me diagnose problems so that I can be productive. --- # Diagnostics in .NET @@ -37,6 +37,8 @@ For most cases, whether adding logging to an existing project or creating a new [Metrics](metrics.md) are numerical measurements recorded over time to monitor application performance and health. Metrics are often used to generate alerts when potential problems are detected. Metrics have very low performance overhead and many services configure them as always-on telemetry. Exceptions are often recorded as metrics, and can be summarized to reduce the cardinality of the data. For more information, see [Exception summarization](diagnostic-exception-summary.md). +For a tutorial that instruments an application with the API, see [Measure performance using EventCounters](event-counter-perf.md). + ### Distributed traces [Distributed Tracing](./distributed-tracing.md) is a specialized form of logging that helps you localize failures and performance issues within applications distributed across multiple machines or processes. This technique tracks requests through an application correlating together work done by different application components and separating it from other work the application may be doing for concurrent requests. It is possible to trace every request and sampling can be optionally employed to bound the performance overhead. @@ -56,38 +58,15 @@ If debugging or observability is not sufficient, .NET supports additional diagno ## Diagnostics tools -.NET supports a number of [CLI tools](./tools-overview.md) that can be used to diagnose your applications. - -## .NET Core diagnostics tutorials - -### Debug a memory leak - -[Tutorial: Debug a memory leak](debug-memory-leak.md) walks through finding a memory leak. The [dotnet-counters](dotnet-counters.md) tool is used to confirm the leak and the [dotnet-dump](dotnet-dump.md) tool is used to diagnose the leak. - -### Debug high CPU usage - -[Tutorial: Debug high CPU usage](debug-highcpu.md) walks you through investigating high CPU usage. It uses the [dotnet-counters](dotnet-counters.md) tool to confirm the high CPU usage. It then walks you through using [Trace for performance analysis utility (`dotnet-trace`)](dotnet-trace.md) or Linux `perf` to collect and view CPU usage profile. - -### Debug deadlock - -[Tutorial: Debug deadlock](debug-deadlock.md) shows you how to use the [dotnet-dump](dotnet-dump.md) tool to investigate threads and locks. - -### Debug ThreadPool Starvation - -[Tutorial: Debug threadPool starvation](debug-threadpool-starvation.md) shows you how to use the [dotnet-counters](dotnet-counters.md) and [dotnet-stack](dotnet-stack.md) tools to investigate ThreadPool starvation. - -### Debug a StackOverflow - -[Tutorial: Debug a StackOverflow](debug-stackoverflow.md) demonstrates how to debug a on Linux. - -### Debug Linux dumps +.NET supports a number of [CLI tools](./tools-overview.md) that can be used to diagnose your applications. To automate a custom diagnostic workflow, use the [diagnostics client library](diagnostics-client-library.md) and . -[Debug Linux dumps](debug-linux-dumps.md) explains how to collect and analyze dumps on Linux. +## Diagnostics tutorials -### Measure performance using EventCounters +### Performance tutorials -[Tutorial: Measure performance using EventCounters in .NET](event-counter-perf.md) shows you how to use the API to measure performance in your .NET app. +Use [Diagnose performance issues in .NET applications](performance-diagnostics.md) to choose the recommended workflow for a performance symptom. The guide links to the applicable detailed tutorials and hands-on exercises. -### Write your own diagnostic tool +### Crash and dump tutorials -[The diagnostics client library](diagnostics-client-library.md) lets you write your own custom diagnostic tool best suited for your diagnostic scenario. For more information, see the [Microsoft.Diagnostics.NETCore.Client API reference](microsoft-diagnostics-netcore-client.md). +- [Debug a StackOverflow](debug-stackoverflow.md) demonstrates how to debug a on Linux. +- [Debug Linux dumps](debug-linux-dumps.md) explains how to collect and analyze dumps on Linux. diff --git a/docs/core/diagnostics/performance-diagnostics.md b/docs/core/diagnostics/performance-diagnostics.md new file mode 100644 index 0000000000000..d81edc8201a29 --- /dev/null +++ b/docs/core/diagnostics/performance-diagnostics.md @@ -0,0 +1,87 @@ +--- +title: Diagnose performance issues in .NET applications +description: Choose diagnostic data and tools based on the symptoms of a .NET performance problem. +ms.date: 09/08/2026 +ms.topic: conceptual +#Customer intent: As a .NET developer, I want to choose the right diagnostic tools and data to find the cause of a performance problem. +--- + +# Diagnose performance issues in .NET applications + +Performance investigations are most effective when you start with the observed symptom, collect the least expensive data that can distinguish likely causes, and then collect a more detailed artifact only when the evidence points to it. + +This article helps you choose among metrics, logs, distributed traces, stack snapshots, performance traces, GC dumps, and process dumps. For installation and complete command syntax, see [.NET diagnostic tools](tools-overview.md). + +## Start with the symptom + +Before collecting a large trace or dump, record: + +- The time range in which the problem occurred. +- The affected process, request, operation, or workload. +- CPU usage, memory usage, request rate, latency, and error rate. +- Whether the problem is continuous, intermittent, or limited to startup. +- Whether the machine, container, or only one process is resource constrained. + +Always-on [metrics](metrics.md), logs, and [distributed traces](distributed-tracing.md) are often the best sources for this first step. They have lower overhead than detailed performance traces and help identify the process and time window that require deeper investigation. + +## Choose a performance collector for the platform + +Many of the workflows in the next section require a trace of activity over time. Choose a collector that provides real operating-system CPU samples and native context when CPU or system activity matters, and add .NET runtime events when the question involves GC, allocation, exceptions, JIT, loading, or managed threading. + +| Environment | Preferred collection workflow | +| --- | --- | +| Windows development | Use the [Visual Studio Performance Profiler](/visualstudio/profiling/) for an interactive investigation. | +| Windows system-wide or production analysis | Use ETW through Windows Performance Recorder and Windows Performance Analyzer, or use PerfView. | +| .NET 10+ on Linux with the required kernel support | Use [`dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md) for one trace containing .NET events, Linux CPU samples, native stacks, scheduling, and selected Linux events. | +| Linux without the `collect-linux` prerequisites | Use Linux `perf` for CPU, native, and system profiling. Use [`dotnet-trace collect`](dotnet-trace.md#dotnet-trace-collect) separately when you need .NET runtime events or managed stack samples. | +| macOS | Use Xcode Instruments for CPU and native profiling. Use [`dotnet-trace collect`](dotnet-trace.md#dotnet-trace-collect) separately when you need .NET runtime events or managed stack samples. | +| Automated production or container collection | Use [`dotnet-monitor`](dotnet-monitor.md) for automated .NET diagnostics, and pair it with the platform profiler when the investigation requires operating-system or native context. | + +On Linux or Windows, use OneCollect [`record-trace`](https://github.com/microsoft/one-collect/tree/main/record-trace) when you need lower-level scripts, event selection, process or CPU filtering, or alternate output formats. On Linux, use `perf` directly when you specifically require `perf.data`, perf-native analysis, or hardware performance counters. [PerfCollect](trace-perfcollect-lttng.md) is the earlier Linux workflow and its runtime-event collection requires LTTng 2.12; it isn't the default fallback for `collect-linux`. + +`dotnet-trace collect` works on all operating systems and is useful for .NET runtime events and managed stack sampling. Its managed sampling and managed-only stacks aren't a replacement for a platform CPU profiler when you need unbiased CPU attribution, native frames, kernel activity, or competing-process context. + +## Use the recommended workflow for the symptom + +Use [`dotnet-counters`](dotnet-counters.md), your monitoring system, or [built-in .NET metrics](built-in-metrics.md) to confirm the symptom and identify the affected process and time range. Then use the platform workflow from the preceding section to collect the evidence described here. + +| Observed symptom | Recommended workflow | +| --- | --- | +| High CPU usage | Follow [Debug high CPU usage](debug-highcpu.md). Collect a CPU profile with the platform profiler, find the code with the greatest exclusive cost, and follow its callers back to application code. | +| Increasing managed memory | Confirm managed heap growth, collect a [`dotnet-gcdump`](dotnet-gcdump.md) to identify growing types, and collect a [`dotnet-dump`](dotnet-dump.md) when you need retention roots. Follow [Debug a memory leak](debug-memory-leak.md) for the dump workflow. If process RSS grows while the managed heap remains stable, use a native memory profiler instead. | +| Long or frequent GC pauses | Confirm pause time and collection frequency, then collect allocation and GC events to identify allocation types, callers, generations, collection reasons, and pause duration. On Linux, use the [`collect-linux` allocation and GC configuration](dotnet-trace-collect-linux-performance.md#allocation-and-gc). On other platforms, use ETW, the Visual Studio profiler, or `dotnet-trace collect --profile gc-verbose`. Use a GC dump or process dump only when the remaining question is why objects survive. | +| Slow work with low CPU usage | For a continuously stuck process, begin with repeated [`dotnet-stack`](dotnet-stack.md) snapshots. For an intermittent delay, collect scheduling or thread-time data with the platform profiler. On Linux, use the [`collect-linux` blocking configuration](dotnet-trace-collect-linux-performance.md#blocking-contention-and-threadpool-behavior). Use activities or application instrumentation when physical stacks don't preserve the logical async or request caller. | +| ThreadPool starvation | Follow [Debug ThreadPool starvation](debug-threadpool-starvation.md): confirm worker growth and queueing with metrics, use `dotnet-stack` for a continuous issue, and use a threading and wait trace for an intermittent issue. | +| Deadlock | Collect a process dump and follow [Debug a deadlock](debug-deadlock.md) to inspect lock owners and the wait cycle. To understand how an intermittent deadlock forms, start contention and scheduling collection before reproduction. | +| Slow file or network I/O | Start with dependency telemetry and logs. If the delay is inside the process or operating system, collect scheduling data and the relevant file, socket, or syscall events to connect blocked time and operations to callers. Use storage, network, database, or remote-service diagnostics when the delay is outside the process. | +| High exception rate | Confirm the rate with metrics or logs, then collect .NET first-chance exception events and call stacks to identify repeated throw sites, including caught exceptions. Use a dump when you need the state of one unhandled exception or crash. | +| Slow startup | Start the platform profiler before launching the process and include CPU, loader, JIT, file, and process activity. Add startup-specific application instrumentation when runtime and operating-system events don't identify the delayed logical operation. | +| Slow distributed request | Use distributed tracing to identify the service and dependency that own the latency, then collect a process trace from that service for the same interval. Add application instrumentation when the required business or async relationship isn't represented. | + +## Choose a different artifact when a performance trace can't answer the question + +Collecting more of the same data doesn't recover information that the artifact doesn't contain. Choose the artifact that records the relationship you still need to prove. + +| Information you need | Appropriate source | +| --- | --- | +| Why managed objects remain alive | GC dump or process dump | +| Exact finalizer queue and heap roots | Process dump | +| Existing deadlock ownership and lock cycle | Process dump | +| Logical async, request, or distributed causality | Activity, distributed tracing, or application instrumentation | +| Database query plans or remote-service internals | Database and dependency diagnostics | +| Native allocation ownership | Native memory profiler | +| Cache misses, branch prediction, IPC, or memory bandwidth | Hardware performance counters and platform profiler | +| Inlined methods or generated machine instructions | Source, disassembly, or a diagnostic build | + +## Practice the Linux workflows + +[Practice Linux performance investigations with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-scenarios.md) provides runnable CPU, memory, GC, blocking, contention, I/O, exception, startup, process, mixed-cause, and healthy-control exercises. Use it after the symptom table points to a Linux performance trace and you want hands-on practice selecting the focused configuration and interpreting the evidence. + +## See also + +- [.NET diagnostic tools](tools-overview.md) +- [Metrics collection](metrics-collection.md) +- [Dumps](dumps.md) +- [Collect dumps on crash](collect-dumps-crash.md) +- [Collect diagnostics in Linux containers](diagnostics-in-containers.md) +- [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md) diff --git a/docs/core/diagnostics/specialized-diagnostics-overview.md b/docs/core/diagnostics/specialized-diagnostics-overview.md index 16abe55f46313..882c522336e59 100644 --- a/docs/core/diagnostics/specialized-diagnostics-overview.md +++ b/docs/core/diagnostics/specialized-diagnostics-overview.md @@ -1,7 +1,7 @@ --- title: Specialized Diagnostics description: A guide to more advanced diagnostics support in .NET -ms.date: 05/19/2023 +ms.date: 09/04/2026 --- # Specialized diagnostics @@ -11,11 +11,11 @@ If debugging or observability is not sufficient, .NET supports additional diagno [Event Source](./eventsource.md) provides the ability to collect detailed diagnostic information about what's happening inside .NET processes. It includes telemetry information for the runtime, GC, libraries, and application code. -Event Source data can be collected in-process using the API or with external diagnostics tools such as [Visual Studio](/visualstudio/profiling), [dotnet-monitor](./dotnet-monitor.md), [dotnet-trace](./dotnet-trace.md), [PerfView](https://github.com/microsoft/perfview), and the [Perfcollect](./trace-perfcollect-lttng.md) scripts. Using the external tools to collect event source data in traces is commonly used for performance analysis. +Event Source data can be collected in-process using the API or with external diagnostics tools such as [Visual Studio](/visualstudio/profiling), [dotnet-monitor](./dotnet-monitor.md), [dotnet-trace](./dotnet-trace.md), [PerfView](https://github.com/microsoft/perfview), and [PerfCollect](./trace-perfcollect-lttng.md). On Linux, use [`dotnet-trace collect-linux`](./dotnet-trace.md#dotnet-trace-collect-linux) when runtime events must be correlated with native call stacks and kernel events. ### EventPipe -[EventPipe](./eventpipe.md) is a runtime component that can be used to collect tracing data, similar to ETW or LTTng. The goal of EventPipe is to allow .NET developers to easily trace their .NET applications without having to rely on platform-specific, OS-native components, such as ETW or LTTng. +[EventPipe](./eventpipe.md) is a runtime component that can be used to collect tracing data, similar to ETW or `perf_events`. The goal of EventPipe is to allow .NET developers to easily trace their .NET applications without having to rely on platform-specific, OS-native components, such as ETW or `perf_events`. EventPipe is the mechanism behind many of the diagnostic tools. It can be used for consuming events emitted by the runtime as well as custom events written with [EventSource](xref:System.Diagnostics.Tracing.EventSource). @@ -46,4 +46,5 @@ The same diagnostics tools that are used in non-containerized Linux environments ## See also - [Debug high CPU usage](./debug-highcpu.md) +- [Collect a Linux trace with dotnet-trace](./dotnet-trace.md#dotnet-trace-collect-linux) - [Collect a performance trace in Linux with PerfCollect](./trace-perfcollect-lttng.md) diff --git a/docs/core/diagnostics/tools-overview.md b/docs/core/diagnostics/tools-overview.md index 5ab847b2ccacb..9dc8eede27dce 100644 --- a/docs/core/diagnostics/tools-overview.md +++ b/docs/core/diagnostics/tools-overview.md @@ -1,9 +1,9 @@ --- title: .NET Diagnostic tools overview -description: An overview of the tools available to diagnose .NET Core applications. -ms.date: 06/8/2023 +description: An overview of the tools available to diagnose .NET applications. +ms.date: 09/04/2026 ms.topic: overview -#Customer intent: As a .NET Core developer I want to find the best tools to help me diagnose problems so that I can be productive. +#Customer intent: As a .NET developer I want to find the best tools to help me diagnose problems so that I can be productive. --- # .NET diagnostic tools @@ -24,7 +24,7 @@ ms.topic: overview ### dotnet-counters -[dotnet-counters](dotnet-counters.md) is a performance monitoring tool for first-level health monitoring and performance investigation. It observes performance counter values published via the API. For example, you can quickly monitor things like the CPU usage or the rate of exceptions being thrown in your .NET Core application. +[dotnet-counters](dotnet-counters.md) is a performance monitoring tool for first-level health monitoring and performance investigation. It observes performance counter values published via the API. For example, you can quickly monitor things like the CPU usage or the rate of exceptions being thrown in your .NET application. ### dotnet-dump @@ -40,7 +40,7 @@ The [dotnet-monitor](dotnet-monitor.md) tool is a way to monitor .NET applicatio ### dotnet-trace -.NET Core includes `EventPipe`, which exposes diagnostics data. The [dotnet-trace](dotnet-trace.md) tool allows you to consume interesting profiling data from your app that can help in scenarios where you need to root-cause apps running that are running slowly. +The [dotnet-trace](dotnet-trace.md) tool is a cross-platform .NET diagnostic tool that collects traces from running applications without using a native profiler. On Linux, it can also combine .NET runtime and application events with machine-wide CPU samples, native call stacks, and Linux kernel events collected through the `perf_events` facility. For a symptom-driven workflow, see [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md). ### dotnet-stack @@ -56,6 +56,10 @@ The [dotnet-stack](dotnet-stack.md) tool allows you to quickly print the managed ## Other tools +### OneCollect `record-trace` + +The [OneCollect `record-trace`](https://github.com/microsoft/one-collect/tree/main/record-trace) tool records system-wide performance traces on Linux and Windows. Use it for lower-level, scriptable control over event selection, process and CPU filtering, and trace output. For a .NET-oriented Linux workflow that configures runtime events and system profiling together, use [`dotnet-trace collect-linux`](dotnet-trace.md#dotnet-trace-collect-linux). + ### PerfCollect -[PerfCollect](trace-perfcollect-lttng.md) is a bash script you can use to collect traces with `perf` and `LTTng` for a more in-depth performance analysis of .NET apps running on Linux distributions. +[PerfCollect](trace-perfcollect-lttng.md) is the earlier bash-based workflow for collecting Linux CPU samples with `perf` and .NET runtime and EventSource events with LTTng. Prefer `dotnet-trace collect-linux` for new investigations when possible. PerfCollect remains documented for existing workflows, but its runtime-event collection depends on an older LTTng ABI. diff --git a/docs/core/diagnostics/trace-perfcollect-lttng.md b/docs/core/diagnostics/trace-perfcollect-lttng.md index 328e6c413f4cb..c6f17d6ef6930 100644 --- a/docs/core/diagnostics/trace-perfcollect-lttng.md +++ b/docs/core/diagnostics/trace-perfcollect-lttng.md @@ -2,13 +2,16 @@ title: Tracing .NET applications with PerfCollect. description: A tutorial that walks you through collecting a trace with perfcollect in .NET. ms.topic: tutorial -ms.date: 04/10/2025 +ms.date: 09/04/2026 --- # Trace .NET applications with PerfCollect **This article applies to: ✔️** .NET Core 2.1 SDK and later versions +> [!IMPORTANT] +> For .NET 10+ Linux investigations, prefer [Investigate Linux performance with `dotnet-trace collect-linux`](dotnet-trace-collect-linux-performance.md). PerfCollect is the earlier .NET Linux tracing workflow. Its .NET runtime event collection requires LTTng 2.12; on distributions with LTTng 2.13 or later, the LTTng portion must be disabled as described later in this article. + When performance problems are encountered on Linux, collecting a trace with `perfcollect` can be used to gather detailed information about what was happening on the machine at the time of the performance problem. `perfcollect` is a bash script that uses [Linux Trace Toolkit: next generation (LTTng)](https://lttng.org) to collect events written from the runtime or any [EventSource](xref:System.Diagnostics.Tracing.EventListener), as well as [perf](https://perf.wiki.kernel.org/) to collect CPU samples of the target process. @@ -65,7 +68,6 @@ For resolving method names of native runtime DLLs (such as libcoreclr.so), `perf > [!NOTE] > LTTng had a breaking change between versions 2.12 and 2.13. The .NET runtime currently supports version 2.12. If your Linux distribution has adopted 2.13 or later then we recommend disabling the LTTng portion of the perfcollect functionality. To do this add the option '-nolttng' to the perfcollect command-line and in step 3 do not set the DOTNET_EnableEventLog environment variable. -1. **[App]** Set up the application shell with the following environment variables - this enables tracing configuration of CoreCLR. 1. **[App]** Set up the application shell with the following environment variables - this enables tracing configuration of CoreCLR. ```bash diff --git a/docs/navigate/tools-diagnostics/toc.yml b/docs/navigate/tools-diagnostics/toc.yml index 747c01c7edf78..e0c60917410b4 100644 --- a/docs/navigate/tools-diagnostics/toc.yml +++ b/docs/navigate/tools-diagnostics/toc.yml @@ -392,6 +392,13 @@ items: - name: Overview displayName: diagnostics, instrumentation href: ../../core/diagnostics/index.md + - name: Diagnose performance issues + href: ../../core/diagnostics/performance-diagnostics.md + items: + - name: Investigate Linux performance + href: ../../core/diagnostics/dotnet-trace-collect-linux-performance.md + - name: Practice Linux performance investigations + href: ../../core/diagnostics/dotnet-trace-collect-linux-scenarios.md - name: Managed debuggers href: ../../core/diagnostics/managed-debuggers.md - name: Profiling tools