diff --git a/hugo/content/en/opentelemetry/config/hostname_tagging.md b/hugo/content/en/opentelemetry/config/hostname_tagging.md index 0f879615268..9444d7f9305 100644 --- a/hugo/content/en/opentelemetry/config/hostname_tagging.md +++ b/hugo/content/en/opentelemetry/config/hostname_tagging.md @@ -1,22 +1,97 @@ --- title: Hostname and Tagging +description: Configure consistent host identification and tagging for OpenTelemetry telemetry sent to Datadog. aliases: - /opentelemetry/collector_exporter/hostname_tagging further_reading: - link: "/opentelemetry/collector_exporter/" tag: "Documentation" text: "Setting Up the OpenTelemetry Collector" +- link: "/opentelemetry/mapping/hostname/" + tag: "Documentation" + text: "Mapping OpenTelemetry Semantic Conventions to Hostnames" --- {{< img src="opentelemetry/collector_exporter/hostname_tagging.png" alt="Hostname information collected from OpenTelemetry" style="width:100%;" >}} ## Overview -To extract the correct hostname and host tags, Datadog Exporter uses the [resource detection processor][2] and the [Kubernetes attributes processor][3]. These processors allow for extracting information from hosts and containers in the form of [resource semantic conventions][1], which is then used to build the hostname, host tags, and container tags. These tags enable automatic correlation among telemetry signals and tag-based navigation for filtering and grouping telemetry data within Datadog. +Datadog uses OpenTelemetry resource attributes to associate metrics, traces, and logs with hosts. Consistent host identification enables correlation across telemetry signals and host tag inheritance. + +For the hostname resolution order and the full list of supported resource attributes, see [Mapping OpenTelemetry Semantic Conventions to Hostnames][8]. + +## Hostname recommendations + +The right configuration depends on how you send telemetry to Datadog. Find your setup in the following table, then follow the linked section. + +| How you send telemetry | Recommendation | +|---|---| +| [OTLP ingestion by the Datadog Agent](#otlp-ingestion-by-the-datadog-agent) | Run an Agent on every host that generates telemetry. Omit hostname attributes, or set them to match the Agent hostname. | +| [DDOT Collector exporting directly to Datadog](#ddot-collector-exporting-directly-to-datadog) | Include the `infraattributes` processor in every pipeline. On hosts, enable `allow_hostname_override`. On Fargate, supply platform resource attributes instead. | +| [Collector exporting through a gateway](#collector-exporting-through-a-gateway) | Detect host information in the node-level Collector and preserve those resource attributes through the gateway. | +| [OpenTelemetry Collector with the Datadog Exporter sending directly to Datadog](#opentelemetry-collector-with-the-datadog-exporter-sending-directly-to-datadog) | Run a Collector on each host and use the `resourcedetection` processor with the detectors for your environment. | +| [Direct OTLP intake without an Agent or Collector](#direct-otlp-intake-without-an-agent-or-collector) | Set platform resource attributes instead of `host.name`. | + +### OTLP ingestion by the Datadog Agent + +Deploy the Datadog Agent on every host that generates OTLP telemetry. Sending telemetry from one host to an Agent on another host is not supported. For setup instructions, see [OTLP Ingestion by the Datadog Agent][11]. + +If incoming telemetry has no valid hostname attributes, Datadog uses the Agent hostname. If you set `host.name`, `host.id`, or another hostname attribute, make its value match the Agent hostname to avoid duplicate hosts. To override hostname resolution explicitly, set the `datadog.host.name` resource attribute to the Agent hostname. + +### DDOT Collector exporting directly to Datadog + +The DDOT Collector's `infraattributes` processor adds infrastructure attributes and tags to OTLP telemetry. Include it in every signal pipeline. The processor needs resource attributes that identify the source container. If infrastructure tags are missing, see [Infrastructure tags are missing from telemetry][10]. + +The rest of the configuration depends on whether the DDOT Collector runs on a host. + +#### Host-based deployments + +This applies to the DDOT Collector as a [DaemonSet][9], and on [Linux][14] and [Windows][15] hosts. + +Because the DDOT Collector runs inside the Datadog Agent, hostname attributes on incoming telemetry can resolve to a different name than the Agent's, which makes a single node appear as two hosts. Enable `allow_hostname_override` to use the Agent hostname instead: + +```yaml +processors: + infraattributes: + allow_hostname_override: true +``` + +#### Fargate sidecar deployments + +This applies to the DDOT Collector on [ECS Fargate][16] and [EKS Fargate][17], where the Datadog Agent runs as a sidecar container in the same task or pod as your application. + +Fargate does not support host-based deployments, so the host guidance above does not apply. Supply the platform resource attributes that `infraattributes` needs instead: + +- On ECS Fargate, add the ECS resource detector to your OpenTelemetry SDK to provide the `aws.ecs.task.arn` attribute. +- On EKS Fargate, add the EKS resource detector to your SDK, or add the `resourcedetection` processor with the `[env, eks]` detectors to your Collector configuration. + +### Collector exporting through a gateway + +This applies to gateway deployments of the OpenTelemetry Collector with the Datadog Exporter, and to the [DDOT Collector as a gateway on Kubernetes][18]. + +In a gateway deployment, the Collector that exports to Datadog does not run on the host that produced the telemetry. If host information is not attached before the data reaches the gateway, telemetry from many hosts can collapse onto the gateway's hostname, or each Collector pod can register as its own host. + +Detect host information in the node-level Collector, then configure the gateway to preserve those resource attributes instead of detecting them again. For the Datadog Exporter, use the **Kubernetes DaemonSet -> Gateway** configuration in [Datadog Exporter configuration](#datadog-exporter-configuration). If a gateway deployment reports the wrong host, see [Gateway collector not forwarding host metadata][19]. + +### OpenTelemetry Collector with the Datadog Exporter sending directly to Datadog -For more information, see the OpenTelemetry project documentation for the [resource detection][2] and [Kubernetes attributes][3] processors. +This applies to the [OpenTelemetry Collector with the Datadog Exporter][20] running on each host or as a Kubernetes DaemonSet. If your Collector forwards to a gateway, see [Collector exporting through a gateway](#collector-exporting-through-a-gateway) instead. -## Setup +Run a Collector on every host and add the `resourcedetection` processor with the detectors for your environment, as described in [Datadog Exporter configuration](#datadog-exporter-configuration). + +### Direct OTLP intake without an Agent or Collector + +This applies to [OTLP intake for serverless platforms][12], such as AWS Lambda, ECS Fargate, Azure Functions, and Cloud Run, and to [OTLP intake for managed platforms][13]. + +These ingestion paths do not use a host-based Agent or Collector. Set the cloud and platform resource attributes for your environment instead of relying on `host.name` for workload identification. Each platform requires a different set of attributes. + +
Host metadata sent to the OTLP intake endpoints does not populate the Infrastructure Host List.
+ +If you run the DDOT Collector as a sidecar on ECS Fargate or EKS Fargate rather than sending to an OTLP intake endpoint, see [Fargate sidecar deployments](#fargate-sidecar-deployments). + +## Datadog Exporter configuration + +The Datadog Exporter uses the [resource detection processor][2] and the [Kubernetes attributes processor][3] to collect host and container resource attributes. Add the appropriate processors to the relevant metrics, traces, and logs pipelines. {{< tabs >}} {{% tab "Host" %}} @@ -26,7 +101,7 @@ Add the following lines to your Collector configuration: ```yaml processors: resourcedetection: - # bare metal + # Bare metal detectors: [env, system] system: resource_attributes: @@ -46,16 +121,18 @@ processors: enabled: true host.cpu.cache.l2.size: enabled: true - # GCP - detectors: [env, gcp, system] - # AWS - detectors: [env, ecs, ec2, system] - # Azure - detectors: [env, azure, system] timeout: 2s override: false ``` +For cloud environments, replace `detectors` with the appropriate list: + +- Amazon EC2: `[env, ec2, system]` +- Amazon ECS on EC2: `[env, ecs, ec2, system]` +- Amazon ECS Fargate: `[env, ecs]` +- Google Cloud: `[env, gcp, system]` +- Azure: `[env, azure, system]` + {{% /tab %}} {{% tab "Kubernetes Daemonset" %}} @@ -292,7 +369,7 @@ processors: {{% /tab %}} {{< /tabs >}} -## Data collected +### Data collected | OpenTelemetry attribute | Datadog Tag | Processor | |---|---|---| @@ -352,11 +429,11 @@ processors: | `container.image.tag` | `image_tag` | `k8sattributes` | -## Full example configuration +### Full example configuration For a full working example configuration with the Datadog exporter, see [`k8s-values.yaml`][4]. This example is for Amazon EKS. -## Example logging output +### Example logging output ``` ResourceSpans #0 @@ -493,10 +570,22 @@ processors: from_attribute: ``` -[1]: https://opentelemetry.io/docs/specs/semconv/resource/ [2]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourcedetectionprocessor/README.md [3]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/k8sattributesprocessor/README.md [4]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/datadogexporter/examples/k8s-chart/k8s-values.yaml [5]: https://opentelemetry.io/docs/languages/js/resources/ [6]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/datadogexporter/examples/collector.yaml -[7]: https://docs.datadoghq.com/opentelemetry/schema_semantics/host_metadata/ +[7]: /opentelemetry/mapping/host_metadata/ +[8]: /opentelemetry/mapping/hostname/ +[9]: /opentelemetry/setup/ddot_collector/install/kubernetes_daemonset/ +[10]: /opentelemetry/troubleshooting/#infrastructure-tags-are-missing-from-telemetry +[11]: /opentelemetry/setup/otlp_ingest_in_the_agent/ +[12]: /opentelemetry/setup/otlp_ingest/serverless/ +[13]: /opentelemetry/setup/otlp_ingest/managed_platforms/ +[14]: /opentelemetry/setup/ddot_collector/install/linux/ +[15]: /opentelemetry/setup/ddot_collector/install/windows/ +[16]: /opentelemetry/setup/ddot_collector/install/ecs_fargate/ +[17]: /opentelemetry/setup/ddot_collector/install/eks_fargate/ +[18]: /opentelemetry/setup/ddot_collector/install/kubernetes_gateway/ +[19]: /opentelemetry/troubleshooting/#gateway-collector-not-forwarding-host-metadata +[20]: /opentelemetry/setup/collector_exporter/ diff --git a/hugo/content/en/opentelemetry/mapping/hostname.md b/hugo/content/en/opentelemetry/mapping/hostname.md index a519bc5abf1..39bdacdbba2 100644 --- a/hugo/content/en/opentelemetry/mapping/hostname.md +++ b/hugo/content/en/opentelemetry/mapping/hostname.md @@ -12,7 +12,7 @@ further_reading: OpenTelemetry defines certain semantic conventions for resource attributes related to hostnames. If an OpenTelemetry Protocol (OTLP) payload for any signal type has known hostname resource attributes, Datadog honors these conventions and tries to use its value as a hostname. The default hostname resolution algorithm is built with compatibility with the rest of Datadog products in mind, but you can override it if needed. -This algorithm is used in the [Datadog exporter][3] as well as the [OTLP ingest pipeline in the Datadog Agent][2] and [DDOT Collector][5]. When using the [recommended configuration][4] for the Datadog exporter, the [resource detection processor][1] adds the necessary resource attributes to the payload to ensure accurate hostname resolution. +This algorithm is used in the [Datadog exporter][3] as well as the [OTLP ingest pipeline in the Datadog Agent][2] and [DDOT Collector][5]. When using the [configuration recommendations for your ingestion path][4], the [resource detection processor][1] adds the necessary resource attributes to the payload for accurate hostname resolution. ## Conventions used to determine the hostname @@ -146,7 +146,7 @@ The following host names are deemed invalid and discarded: [1]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#resource-detection-processor [2]: /opentelemetry/interoperability/otlp_ingest_in_the_agent [3]: /opentelemetry/setup/collector_exporter/ -[4]: /opentelemetry/config/hostname_tagging/ +[4]: /opentelemetry/config/hostname_tagging/#hostname-recommendations [5]: /opentelemetry/migrate/ddot_collector/ [6]: https://github.com/DataDog/datadog-agent/tree/main/comp/otelcol/otlp/components/processor/infraattributesprocessor -[7]: https://github.com/DataDog/datadog-agent/tree/main/comp/otelcol/otlp/components/processor/infraattributesprocessor#expected-attributes \ No newline at end of file +[7]: https://github.com/DataDog/datadog-agent/tree/main/comp/otelcol/otlp/components/processor/infraattributesprocessor#expected-attributes diff --git a/hugo/content/en/opentelemetry/setup/ddot_collector/install/kubernetes_daemonset.md b/hugo/content/en/opentelemetry/setup/ddot_collector/install/kubernetes_daemonset.md index a0b087050b5..4407750cecc 100644 --- a/hugo/content/en/opentelemetry/setup/ddot_collector/install/kubernetes_daemonset.md +++ b/hugo/content/en/opentelemetry/setup/ddot_collector/install/kubernetes_daemonset.md @@ -694,6 +694,8 @@ service: {{% /tab %}} {{< /tabs >}} +Because the DDOT Collector runs inside the Datadog Agent, hostname attributes on incoming telemetry can resolve to a different name than the Agent's. The `infraattributes` processor can apply the Agent hostname instead. See [Hostname and Tagging][58] for the recommended configuration. + #### Key components To send telemetry data to Datadog, the following components are defined in the configuration: @@ -974,3 +976,4 @@ View metrics from the DDOT Collector to monitor the Collector health. [55]: /containers/datadog_operator [56]: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/ [57]: https://github.com/DataDog/helm-charts/blob/main/charts/datadog-operator/README.md +[58]: /opentelemetry/config/hostname_tagging/#ddot-collector-exporting-directly-to-datadog diff --git a/hugo/content/en/opentelemetry/setup/ddot_collector/install/kubernetes_gateway.md b/hugo/content/en/opentelemetry/setup/ddot_collector/install/kubernetes_gateway.md index 889d0fb7014..0b0a9ceb21f 100644 --- a/hugo/content/en/opentelemetry/setup/ddot_collector/install/kubernetes_gateway.md +++ b/hugo/content/en/opentelemetry/setup/ddot_collector/install/kubernetes_gateway.md @@ -484,6 +484,8 @@ If you set fullnameOverride, the gateway's Kubernetes service name The example configurations use insecure TLS for simplicity. Follow the [OTel configtls instructions][7] if you want to enable TLS. +In a gateway deployment, host information must be attached before telemetry reaches the gateway. For the recommended hostname configuration, see [Hostname and Tagging][12]. + ### Advanced configuration options {{< tabs >}} @@ -1371,3 +1373,4 @@ To view your gateway pods: [9]: http://github.com/kubernetes-sigs/metrics-server [10]: /containers/guide/cluster_agent_autoscaling_metrics/?tab=helm [11]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/datadogextension +[12]: /opentelemetry/config/hostname_tagging/#collector-exporting-through-a-gateway diff --git a/hugo/content/en/opentelemetry/setup/otlp_ingest/managed_platforms.md b/hugo/content/en/opentelemetry/setup/otlp_ingest/managed_platforms.md index 50e601be9f6..6d6ec928f06 100644 --- a/hugo/content/en/opentelemetry/setup/otlp_ingest/managed_platforms.md +++ b/hugo/content/en/opentelemetry/setup/otlp_ingest/managed_platforms.md @@ -20,6 +20,8 @@ Use this option when you run workloads on a managed platform where installing a
Host metadata sent to managed platform endpoints does not populate the Infrastructure Host List.
+For hostname recommendations across OpenTelemetry setups, see [Hostname and Tagging][18]. + Each endpoint supports the following signal paths: | Signal | Path | @@ -120,3 +122,4 @@ Sampling controls available in the Collector (tail-based sampling, probabilistic [15]: https://docs.mulesoft.com/monitoring/telemetry-exporter [16]: https://docs.retool.com/apps/guides/observability/performance-monitoring [17]: https://vercel.com/marketplace/datadog +[18]: /opentelemetry/config/hostname_tagging/#direct-otlp-intake-without-an-agent-or-collector diff --git a/hugo/content/en/opentelemetry/setup/otlp_ingest/serverless.md b/hugo/content/en/opentelemetry/setup/otlp_ingest/serverless.md index 466fbcefcee..165d5295497 100644 --- a/hugo/content/en/opentelemetry/setup/otlp_ingest/serverless.md +++ b/hugo/content/en/opentelemetry/setup/otlp_ingest/serverless.md @@ -40,6 +40,8 @@ The following configuration applies to all platforms. **Resource attributes**: Set platform-specific attributes with `OTEL_RESOURCE_ATTRIBUTES`. See each cloud provider tab below for required and optional attributes. +Use the platform attributes on this page for workload identification instead of relying on `host.name`. For hostname recommendations across OpenTelemetry setups, see [Hostname and Tagging][6]. + ```shell export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf" export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="{{< region-param key="otlp_trace_endpoint" >}}" @@ -212,3 +214,4 @@ The GCP Resource Detector SDK automatically populates: `cloud.account.id`, `clou [3]: /opentelemetry/setup/otlp_ingest/logs/ [4]: /opentelemetry/setup/otlp_ingest/metrics/ [5]: /opentelemetry/setup/otlp_ingest/managed_platforms/ +[6]: /opentelemetry/config/hostname_tagging/#direct-otlp-intake-without-an-agent-or-collector diff --git a/hugo/content/en/opentelemetry/setup/otlp_ingest_in_the_agent.md b/hugo/content/en/opentelemetry/setup/otlp_ingest_in_the_agent.md index 074b142e8ca..0420d348678 100644 --- a/hugo/content/en/opentelemetry/setup/otlp_ingest_in_the_agent.md +++ b/hugo/content/en/opentelemetry/setup/otlp_ingest_in_the_agent.md @@ -35,6 +35,8 @@ Read the OpenTelemetry instrumentation documentation to understand how to point
The supported setup is an ingesting Agent deployed on every OpenTelemetry-data generating host. You cannot send OpenTelemetry telemetry from collectors or instrumented apps running one host to an Agent on a different host. But, provided the Agent is local to the collector or SDK instrumented app, you can set up multiple pipelines.
+For hostname recommendations across OpenTelemetry setups, see [Hostname and Tagging][11]. + ## Enabling OTLP Ingestion on the Datadog Agent {{< tabs >}} @@ -393,3 +395,4 @@ env: [5]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver/config.md [6]: /agent/configuration/agent-configuration-files/ [10]: /opentelemetry/runtime_metrics/ +[11]: /opentelemetry/config/hostname_tagging/#otlp-ingestion-by-the-datadog-agent diff --git a/hugo/content/en/opentelemetry/troubleshooting.md b/hugo/content/en/opentelemetry/troubleshooting.md index 66d49701fce..de3d3aabcb5 100644 --- a/hugo/content/en/opentelemetry/troubleshooting.md +++ b/hugo/content/en/opentelemetry/troubleshooting.md @@ -53,7 +53,7 @@ Alternatively, you can override the hostname using the `datadog.host.name` attri - set(attributes["datadog.host.name"], "${NODE_NAME}") ``` -For more information on host-identifying attributes, see [Mapping OpenTelemetry Semantic Conventions to Hostnames][2]. +For more information on host-identifying attributes, see [Mapping OpenTelemetry Semantic Conventions to Hostnames][2]. For the recommended hostname configuration for your setup, see [Hostname and Tagging][9]. ### Unexpected hostnames with AWS Fargate deployment @@ -349,3 +349,4 @@ features: [6]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor#readme [7]: https://github.com/DataDog/datadog-agent/tree/main/comp/otelcol/otlp/components/processor/infraattributesprocessor#readme [8]: https://pkg.go.dev/go.opentelemetry.io/otel/sdk/resource#WithContainerID +[9]: /opentelemetry/config/hostname_tagging/#hostname-recommendations