fix(vector): fix internal_metrics typo and set sane 30s scrape interval

[saule1508]

Description

The Vector collector DaemonSet consistently consumes 4–6 GiB of RAM per node (and up to 10 GiB) on our cluster. By looking at the prometheus metrics, the source of the memory consumption is the component internal_metrics which generates upward of 3Mib/s (compared to 0.2 MiB/s forthe application logs).

This is because the generated vector.toml completely omits scrape_interval_secs, forcing Vector to poll its entire registry on its internal 1-second default

This PR corrects the typo in the key name (ScrapeIntervalSeconds instead of ScrapIntervalSeconds) and in the serialization tags (secs instead of seconds) and sets a sane, explicit default of 30 seconds inside the generator.

/cc jcantrill
/assign alanconway

Links

• Depending on PR(s):
• GitHub issue:
• JIRA:
• Enhancement proposal:

Summary by CodeRabbit

• Bug Fixes
  • Corrected the internal metrics scrape interval setting name.
  • Internal metrics now scrape every 30 seconds by default across generated configurations.
  • Prometheus output authentication explicitly uses the GET HTTP method.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: saule1508
Once this PR has been reviewed and has the lgtm label, please assign jcantrill for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 666aa0c6-b522-4f7a-aed4-4336672b3d64

📥 Commits

Reviewing files that changed from the base of the PR and between edd266b and 50797f3.

📒 Files selected for processing (5)

• internal/generator/vector/api/config_test.go
• internal/generator/vector/api/sources/internal_metrics.go
• internal/generator/vector/conf/complex.toml
• internal/generator/vector/conf/complex_http_receiver.toml
• internal/generator/vector/conf/container.toml

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

---

📝 Walkthrough

Walkthrough

The Vector internal metrics API now uses scrape_interval_secs and defaults to 30 seconds. Test fixtures and generated configuration examples use the same field and value.

Changes

Internal metrics scrape interval

Layer / File(s)	Summary
Update internal metrics API default internal/generator/vector/api/sources/internal_metrics.go	The field is renamed to ScrapeIntervalSeconds, its serialization tag becomes scrape_interval_secs, and the constructor sets it to 30.
Align configuration fixtures internal/generator/vector/api/config_test.go, internal/generator/vector/conf/*.toml	TOML and YAML fixtures, plus Vector configurations, use scrape_interval_secs = 30. The Prometheus authentication verb remains get.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 50797

The PR corrects the internal metrics configuration naming and sets an explicit 30-second scrape interval; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: jcantrill, vparfonov

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 2 files. (3 skipped: 3 …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the typo correction and the explicit 30-second scrape interval. It accurately summarizes the main changes.
Description check	✅ Passed	The description explains the memory issue, identifies the missing configuration and 1-second default, and summarizes the implementation. It includes reviewer and approver assignments and the required …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description explains the memory issue, identifies the missing configuration and 1-second default, and summarizes the implementation. It includes reviewer and approver assignments and the required Links section.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 2 files. (3 skipped: 3 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

Hi @saule1508. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[vparfonov]

/ok-to-test

[vparfonov]

@jcantrill fixing name of config parameter is totally correct scrape_interval_secs, but not sure about value in 30 seconds

[vparfonov]

/retest-required

[openshift-ci]

@saule1508: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[saule1508]

Hi @vparfonov, thank you for the comment ! Currently the scrape_interval_secs is omitted, and it default to one second (https://vector.dev/docs/reference/configuration/sources/internal_metrics/) but due to the volume of metrics computed in our openshift cluster collector nodes (which is not a specially big cluster) this generates massive memory consumption.
I think 30 seconds make more sense because anyway prometheus is going to scrape every 30 seconds in our environment and very likely in most environments.

[jcantrill]

@saule1508 do you have a graphic or other image to identifies the memory savings after this change?

[saule1508]

@jcantrill Spent the day measuring this properly. Test cluster, Logging 6.6, 7 collectors, external Loki, worker nodes have 128 CPU cores.

On this PR specifically:

vector_internal_metrics_cardinality = 359 per collector, so at the 1s default each collector emits ∼359 events/s.

• sum(rate(vector_component_received_bytes_total{component_id="internal_metrics"}[5m])): 2.7 MB/s → 105 KB/s (∼26×) after setting scrape_interval_secs = 30.

For scale, that 2.7 MB/s was more than all real log sources on the same collectors combined — application, audit and infrastructure inputs are each in the 100–500 KB/s range.

• sum(rate(vector_component_sent_bytes_total{component_type="prometheus_exporter"}[5m])) is unchanged across the change.

So the collector was producing ∼26× more metric snapshots than anything consumed, each additionally cloned through the generated add_nodename_to_metric remap. Our ServiceMonitors (instance, minimal-instance, telemetry-instance) all inherit the platform-default 30s scrape interval.

Correction to my original report: I opened this attributing our collector memory usage to internal metrics. I believe that was mostly wrong. Our actual memory problem is that collector memory scales with node core count, not log volume — ∼810 MB on 8-core nodes vs 5–6.5 GB on 128-core nodes in the same cluster, with memory.peak hitting 9.7 GiB against a 10 GiB limit.

Setting VECTOR_THREADS=8 dropped those 128-core collectors from ∼6 GB to under 1 GB. I'll report that separately once I've confirmed the mechanism — I suspect it's simply that the collector container has no CPU limit (cpu.max = max), so Vector sizes its runtime to all 128 host cores, in which case it may be fixable via spec.collector.resources.limits.cpu with no code change. Testing that tomorrow.

I do see a memory reduction after adding the scrape_interval_secs = 30, not as compelling as the one from the vector_threads. I will do a proper test and post a chart tomorrow.

I am not sure the value of 30s is the best, since the default scape interval is 30s it might make sense, or could be a bit lower, maybe 15s ? I can test other values if it helps. Will work on it tomorrow

[jcantrill]

Setting VECTOR_THREADS=8 dropped those 128-core collectors from ∼6 GB to under 1 GB. I'll report that separately once I've confirmed the mechanism — I suspect it's simply that the collector container has no CPU limit (cpu.max = max), so Vector sizes its runtime to all 128 host cores, in which case it may be fixable via spec.collector.resources.limits.cpu with no code change. Testing that tomorrow.

If the CLF defines no collector requests/limits, the operator defaults to the following values:

	DefaultRequestMemory = resource.MustParse("64Mi")
	DefaultRequestCpu    = resource.MustParse("500m")
	DefaultLimitMemory   = resource.MustParse("2048Mi")
	DefaultLimitCpu      = resource.MustParse("6000m")

This means it should at most consume 6 cores at 2G each. If you provide any values, it takes what is spec'd. Vector is agressive and will consume as much is available. If the default's are not being respected then that is a bug and should be logged accordingly

[saule1508]

@jcantrill

Following up on your comment about default limits, I have found strong evidence of a bug where the default CPU limit is not being applied.

On a test cluster, I examined a collector pod running on a 128-cores worker node.

• The spec has no CPU limit

oc describe command shows:

 oc describe pod -n openshift-logging instance-4wkgq
 ...
Command:
      sh
    Args:
      /usr/bin/run-vector.sh
    State:          Running
      Started:      Mon, 31 Aug 2026 12:41:59 +0000
    Ready:          True
    Restart Count:  4
    Limits:
      memory:  10Gi
    Requests:
      cpu:     200m
      memory:  2Gi

• But vector uses all cores

but when I look into the container at the number of tasks I have 177, so it will be 128 threads for sure

[root@instance-4wkgq bin]# ls /proc/1/task/ | wc -l
177

This confirms that the operator is not enforcing the expected default CPU limit of 6000m, causing Vector to consume excessive resources on high-core nodes.

• workaround and solution

I applied the following limit (via terraform)

      collector = {
        resources = {
          limits = {
            memory = "10Gi"
            cpu    = "8"
          }
          requests = {
            cpu    = "200m"
            memory = "2Gi"
          }
        }
      }

and now the threads are only 36 and the memory dropped to 500MB, in the collector pod:

[root@instance-nm8cg bin]# ls /proc/1/task/ | wc -l
36
[root@instance-nm8cg bin]# grep VmRSS /proc/1/status
VmRSS:    449756 kB

[saule1508]

@jcantrill do you want me to create a new issue regarding the default cpu limit not being respected ? I can also look at the code to see if I can find the error ?
At last I think I have solve my issue of memory by explicitly setting the cpu limit, which is great