senkin: pin LightGBM threads and skip non-applicable datasets - #67
Open
benjaminfreyuu wants to merge 1 commit into
Open
senkin: pin LightGBM threads and skip non-applicable datasets#67benjaminfreyuu wants to merge 1 commit into
benjaminfreyuu wants to merge 1 commit into
Conversation
The 2026-08-14 run had senkin_tmp_train hit the 8h walltime on the CITE datasets and OOM/crash on the rest. Two causes: 1. LightGBM set no num_threads, so it defaulted to one thread per core the container *sees* (the whole node) while the job is cgroup-throttled to meta["cpus"] -- the threads oversubscribe and thrash. The original solution ran on Kaggle with dedicated cores so never hit this. Pin num_threads to meta["cpus"] (results-preserving; same class of fix as guanlab in openproblems-bio#59). 2. senkin has no applicability guard, so it ran on all 8 datasets including Multiome and the ADT->GEX swap, where it wastes hours and OOMs. Add an exit_non_applicable guard (via src/utils/exit_codes.py) so it only runs on the GEX->ADT CITE direction, and point the component test at bmmc_cite/normal. Verified: viash test passes (2/2) on the normal (GEX->ADT) direction; the guard correctly skips the swap direction. Note: pinning threads removes the oversubscription, but senkin runs 4 LightGBM models (134 targets x 5 folds) + 2 NNs in one job, which the original spread across separate multi-hour Kaggle sessions - it may still need a walltime above 8h or a lighter config on the full data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In
run_2026-08-14,senkin_tmp_trainhit the 8h walltime on the CITE datasets (exit 140) and OOM'd/crashed on the rest. The log shows all 8h spent inside LightGBM model 1 of 4 — it never even reaches the neural network.Two root causes:
LightGBM thread oversubscription. The params set no
num_threads, so LightGBM defaults to one thread per core the container sees (the whole node), while the job is cgroup-throttled tometa["cpus"]. The threads oversubscribe and thrash. The original solution (baosenguo/…-2nd-Place-Solution) ran on Kaggle with dedicated cores, so never hit this — its LightGBM config is otherwise identical. Same class of bug fixed for guanlab in Fix the component bugs surfaced by the full run #59.No applicability guard. senkin predicts protein (ADT) from RNA (GEX) but has no modality check, so it ran on all 8 datasets — including Multiome and the ADT→GEX swap — wasting hours and OOMing.
Fix
num_threads = meta["cpus"]on all four LightGBM param sets (results-preserving).exit_non_applicableguard (viasrc/utils/exit_codes.py) so senkin only runs on the GEX→ADT CITE direction; point the component test atbmmc_cite/normal.Verification
viash test src/methods/senkin_tmp/senkin_tmp_train/config.vsh.yaml→ 2/2 passed on the normal (GEX→ADT) direction; the guard correctly skips the swap direction.Note
Pinning threads removes the oversubscription, but senkin runs 4 LightGBM models (134 targets × 5 folds) + 2 NNs in one job — work the original spread across separate multi-hour Kaggle sessions. It may still need a walltime above 8h (or a lighter config) on the full data; that's a cluster/label decision I can't validate locally.