Conversation
Extend LABEL, BaseConformal, SCRIB, ClusterLabel, NeighborhoodLabel, and CovariateLabel to accept binary base models by re-presenting the single positive-class probability as a 2-class problem at the calibrate and forward boundaries. FavMac stays multilabel-only.
Reconcile binary-mode support with upstream's new scores.py scoring refactor (sunlabuiuc#1189 real APS/dynamic scoring, sunlabuiuc#1190 SCRIB fixes): expand binary y_prob to 2 columns before the true_class_*/all_class_* score functions in each method's calibrate and forward, on top of the new score_type/rng API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every
SetPredictorsubclass only accepted multiclass base models. A binary model emits a single positive-class probabilityy_probof shape(N, 1), but a conformal prediction set ranges over both labels{0, 1}and needs one column per class, so binary tasks were out entirely, and the binary metrics function couldn't score prediction sets at all.Fix
Re-present a binary model as a 2-class problem at the two boundaries where a raw
(N, 1)probability enters the pipeline: expand it to[P(y=0), P(y=1)](N, 2)before the non-conformity scoring incalibrate(for the threshold) and inforward(for the set), so the existing split-conformal machinery runs unchanged withK = 2. This sits on top of the newscores.pyscoring layer, so binary composes with anyscore_type(threshold or APS). Then wire the prediction-set metrics intobinary_metrics_fnso results are scorable.Changes
pyhealth/metrics/binary.py:binary_metrics_fnnow acceptsy_predsetand computes the conformal set metrics.pyhealth/metrics/prediction_set.py: addedPREDICTION_SET_METRICSandcompute_prediction_set_metric, a single dispatch point for set metrics.pyhealth/metrics/multiclass.py: delegates set-metric dispatch to the shared helper (removes duplicated block).pyhealth/calib/utils.py: addedbinary_to_2col(primitive) plusexpand_binary_cal(calibrate side, numpy) andexpand_binary_pred(forward side, numpy) which expand the probability before thescores.pyscore functions.pyhealth/calib/predictionset/{label,base_conformal,cluster/cluster_label,cluster/neighborhood_label,covariate/covariate_label}.py: accept binary in the mode gate, and expandy_probto 2 columns before thetrue_class_*/all_class_*score functions incalibrateandforward.pyhealth/calib/predictionset/scrib/__init__.py: accept binary in the mode gate; SCRIB keeps its own quicksearch scoring, so the expansion is an inline torch step (it thresholds the probability directly rather than going throughscores.py).tests/core/test_binary_label.py,tests/core/test_binary_predictionset.py: binary end-to-end coverage for all six methods and the helpers.tests/core/test_{cluster,covariate,neighborhood}_label.py: the "binary raises" tests now assert binary is accepted and that a genuinely unsupported mode still raises.Core decisions
y_probstays native(N, 1)- only the set is expanded.roc_auc, etc. are unaffected, andself.modestays honest ("binary") rather than being faked as multiclass.calibrateandforward. They're different data; only the thresholdself.tpasses between them, so each entry point must expand its own raw probability.scores.pylayer: expanding to(N, 2)before the score functions meansscore_type(threshold / APS) works for binary with no extra code.prediction_set.py, and the binary and multiclass metrics functions delegate to the shared helper (previously duplicated inmulticlass.py).FavMacis intentionally excluded. It is multilabel, and a binary task doesn't map to its cost-over-multiple-labels contract.