feat: store CPU, GPU and RAM tracking methods in EmissionsData - #1388
Open
NoiceHax wants to merge 1 commit into
Open
feat: store CPU, GPU and RAM tracking methods in EmissionsData#1388NoiceHax wants to merge 1 commit into
NoiceHax wants to merge 1 commit into
Conversation
The tracking method picked for each device was only written to the log at setup time, so nothing downstream could tell whether CPU power came from RAPL, cpu_load or a TDP constant. ResourceTracker now writes ram_tracking_method, cpu_tracking_method and gpu_tracking_method into the tracker conf once the hardware setup is done, and EmissionsData carries them through to the CSV and the other outputs. The keys are also part of the hardware cache CONF_KEYS so a second tracker in the same process gets them back from the cached plan. The BoAmps mapper can now fill cpuTrackingMode and gpuTrackingMode, which it previously had to leave empty.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
CodeCarbon already logs which tracking method it picked for RAM, CPU and GPU at setup time, but that value stopped at the log line. Nothing downstream could tell whether CPU power came from RAPL, cpu_load or a TDP constant.
ResourceTracker._run_full_hardware_setupnow writesram_tracking_method,cpu_tracking_methodandgpu_tracking_methodinto the tracker conf once both setup futures resolve, andEmissionsDatacarries the three values through to the CSV and the other outputs. They go through_confrather than staying on theResourceTracker, because the cached setup path rebuilds state from the stored plan. That is also why the keys were added toCONF_KEYSinhardware_cache.py. The new dataclass fields default to"", so existing positional construction still works.The BoAmps mapper can now fill
cpuTrackingModeandgpuTrackingMode, which it had to leave empty before. The comment explaining why they were missing is gone with it.This covers the first bullet of the issue only. Sending the values to the API in the Experiment needs new columns on
RunBaseplus a carbonserver model and an alembic migration, so I left it out rather than mix two scopes in one PR. Happy to open a follow up.Related Issue
issue #1144
Motivation and Context
The tracking method changes how much you should trust a measurement, so it belongs next to the numbers it produced. Reading it back from a CSV weeks later is not possible today.
How Has This Been Tested?
Ran the package suite locally with
CODECARBON_ALLOW_MULTIPLE_RUNS=True pytest --ignore=tests/test_viz_data.py -m 'not integ_test' tests/: 629 passed, 19 skipped. The one failure,test_task_energy_with_live_update_interference, is a timing-sensitive energy sum assertion that fails the same way on a clean master checkout on my machine, so it is not from this change.New tests:
test_run_full_hardware_setup_stores_tracking_methods_in_conf,test_capture_and_apply_restore_tracking_methodsfor the cached path,test_offline_tracker_writes_tracking_methodsfor the CSV columns, and the BoAmps mapping test was split into a mapped case and an omitted case. I checked all of them fail before the source change and pass after.Two fixtures needed the new columns:
tests/test_data/emissions_valid_headers.csv, sincehas_valid_headerscompares the whole header set, and theEmissionsDatafixture intests/output_methods/test_file.py. On the second one, leaving the fields at""hits an existing problem in theupdatewrite path, wheredf[col].dtype.type(val)cannot cast an empty string on a column pandas read back as float. That already happens today withcloud_provider=""on private infrastructure, so I left it for a separate issue instead of widening this PR.Types of changes
AI Usage Disclosure
An AI agent wrote the code and this description while I directed it, and I reviewed and ran every change before submitting.
Checklist: