Skip to content

Dev to main release #64

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
arjunsridhar12345 merged 2 commits into from
Aug 6, 2026
Merged

Dev to main release #64

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
10230fa
feat: update the columns in the trials table from the acquisition sof…
arjunsridhar12345 Aug 6, 2026
1408e67
feat: add columns for anti bias intervention to trials table and qc p…
arjunsridhar12345 Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 52 additions & 7 deletions docs/trials_table_mapping.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ Columns are grouped by the raw source they map from.
| Trials column | Mapping |
| --- | --- |
| `auto_waterL` / `auto_waterR` | From `is_auto_reward_right`. `1` on the auto-responded side; `0` on the other side, when there was no auto-response (`None`), or when the trial is missing. |
| `anti_bias_left_water` / `anti_bias_right_water` | Boolean. `True` when the anti-bias algorithm delivered a water intervention to that side — i.e. `trial.metadata.extra.is_bias_water_intervention` is `True` **and** `is_auto_reward_right` points to that side (`False` → left, `True` → right). The anti-bias water uses the same auto-response channel as ordinary autowater, so the `is_bias_water_intervention` flag is what distinguishes it. `False` otherwise. |
| `anti_bias_lickspout_movement` | Signed horizontal displacement (mm, positive is rightward) the anti-bias algorithm moved the lickspouts on this trial: `trial.lickspout_offset_delta` when `trial.metadata.extra.is_bias_stage_intervention` is `True`, else `0.0`. |
| `bait_left` / `bait_right` | Boolean. `bait_right` is `True` if `p_reward_right == 1` and `is_auto_reward_right` is `None` or `False`. `bait_left` is `True` if `p_reward_left == 1` and `is_auto_reward_right` is `None` or `True`. |
| `response_duration` | `response_deadline_duration`. |
| `reward_consumption_duration` | `Trial -> reward_consumption_duration`. |
Expand All @@ -77,18 +79,58 @@ Columns are grouped by the raw source they map from.
| `base_reward_probability_sum` | If `type == "CoupledTrialGenerator"`, look at `reward_probability_parameters`. |
| `min_reward_each_block` | Present when `type == "CoupledWarmupTrialGenerator"` (has `min_block_reward`); otherwise `None`. |

### From `QuiescentPeriod.json` (`SoftwareEvents` stream)
### Trial period timing (the four period `SoftwareEvents` streams)

| Trials column | Mapping |
| --- | --- |
| `delay_start_time` | `timestamp`. |
| `start_time` | `timestamp` column. |
Each of `QuiescentPeriod.json`, `ResponsePeriod.json`,
`RewardConsumptionPeriod.json`, and `ItiPeriod.json` emits one event per trial at
the **start** of its period, and the periods run back-to-back (the AIND DF v2
trial structure):

```
go cue response ITI start next trial
| registered | |
quiescent | response | reward consumption | ITI | quiescent
|--------------->|------------------->|------------------->|-------------->|---------->
q[i] r[i] c[i] iti[i] q[i+1]
```

So each period's stop time is the next period's start time. All four streams are
aligned with `TrialOutcome` by position (a length mismatch is reported by
`_check_aligned`; a short stream pads with `NaN`).

### From `ITI_period.json` (`SoftwareEvents` stream)
Verified on
`864253_2026-07-29_11-50-18` (753 trials): all five streams have equal length,
`q[i] < r[i] < c[i] < iti[i] < q[i+1]` holds for every trial, the `SoundCard` go
cue falls within 1.2 ms of `r[i]` on every trial, and the realized period
durations track the configured ones (reward consumption ≈
`reward_consumption_duration`, `iti[i] → q[i+1]` ≈ `ITI_duration`).

| Trials column | Mapping |
| --- | --- |
| `stop_time` | `timestamp` column. Possible QC check: length should match `QuiescentPeriod.json`. |
| `quiescent_start_time` | `QuiescentPeriod` `timestamp`. |
| `quiescent_stop_time` | `ResponsePeriod` `timestamp` (the quiescent period ends where the response period begins). |
| `response_start_time` | `ResponsePeriod` `timestamp`. |
| `response_stop_time` | `RewardConsumptionPeriod` `timestamp`. |
| `reward_consumption_start_time` | `RewardConsumptionPeriod` `timestamp`. |
| `reward_consumption_stop_time` | `ItiPeriod` `timestamp`. |
| `ITI_start_time` | `ItiPeriod` `timestamp`. |
| `ITI_stop_time` | The **next** trial's `QuiescentPeriod` `timestamp`; `NaN` on the last trial of the session. |
| `delay_start_time` | `QuiescentPeriod` `timestamp` — the legacy name for `quiescent_start_time` (see the note below). |

There are no `start_time` / `stop_time` trial columns. NWB's `TimeIntervals`
requires a native `start_time` / `stop_time` per trial, so the pipeline derives
the trial extent when writing: `start_time` is `quiescent_start_time` and
`stop_time` is `ITI_stop_time`, falling back to `ITI_start_time` on the last
trial.

> **`delay` means `quiescent`.** The legacy `delay_*` columns describe the
> acquisition software's *quiescence period* — the lick-free interval preceding
> the go cue. `delay_start_time` is therefore the `QuiescentPeriod` timestamp and
> always equals `quiescent_start_time`, and `delay_duration` /
> `delay_beta` / `delay_min` / `delay_max` summarize
> `quiescence_period_duration`. Note `delay_duration` is the *configured*
> duration: each lick restarts the quiescent period, so the realized duration
> (`quiescent_stop_time - quiescent_start_time`) can be longer.

### From `HarpBehavior` (`PulseSupplyPort{0,1}`)

Expand Down Expand Up @@ -142,5 +184,8 @@ These were mapped during exploration but are no longer in scope:
| 2026-06-17 | `auto_waterL` / `auto_waterR` now encode no auto-response (`is_auto_reward_right` is `None`) and missing trials as `0` instead of `NULL`. The columns are non-nullable (`int`, default `0`). |
| 2026-06-20 | Added `reward_size_left` / `reward_size_right` (reward volume in uL) from `task_parameters.reward_size`, and `side_bias` from the per-trial `TrialMetrics` event (`bias` field). |
| 2026-06-20 | `reward_probabilityL` / `reward_probabilityR` now read the block probability from `trial.metadata.p_reward_left` / `p_reward_right` instead of the top-level per-trial `trial.p_reward_left` / `p_reward_right`. |
| 2026-07-27 | Added `anti_bias_left_water` / `anti_bias_right_water` (boolean anti-bias water interventions per side) and `anti_bias_lickspout_movement` (mm the anti-bias algorithm shifted the lickspouts) from `TrialOutcome`'s `trial.metadata.extra` (`is_bias_water_intervention` / `is_bias_stage_intervention`), `is_auto_reward_right`, and `lickspout_offset_delta`. These are also overlaid on the QC `side_bias.png` figure. |
| 2026-07-22 | `lickspout_position_x` / `y1` / `y2` / `z` now derive from the `HarpManipulator` `AccumulatedSteps` stream (microsteps → mm via the `InputSchemas.Rig` manipulator calibration, `full_step_to_mm / microstep_resolution`), sampled per trial via the closest sample in the `[start_time, stop_time)` window and re-referenced to the session-start position (displacement relative to session start, mm), replacing the static `InitialManipulatorPosition` software event. `Motor{i}` maps to `Axis(i + 1)` (X, Y1, Y2, Z). The rig and `AccumulatedSteps` streams are required when there are trials (`build` raises if either is missing). Column descriptions corrected from `um` to `mm`. |
| 2026-07-24 | `reward_size_left` / `reward_size_right` moved from session-level `task_parameters.reward_size` to per-trial `Trial.reward_size` (fields `.left` / `.right`). The columns are now nullable — `None` when the trial is missing. A missing `TaskLogic` stream no longer raises; session distribution columns are simply null. `min_reward_each_block` moved from `CoupledTrialGenerator` to `CoupledWarmupTrialGenerator`. |
| 2026-08-06 | **Breaking:** the trial `start_time` / `stop_time` columns are removed and replaced by one start/stop pair per task period: `quiescent_start_time` / `quiescent_stop_time`, `response_start_time` / `response_stop_time`, `reward_consumption_start_time` / `reward_consumption_stop_time`, and `ITI_start_time` / `ITI_stop_time`, read from the `ResponsePeriod` and `RewardConsumptionPeriod` streams in addition to `QuiescentPeriod` and `ItiPeriod`. Each period event marks its period's start, so each stop is the next period's start; `ITI_stop_time` is the next trial's `QuiescentPeriod` timestamp (`NaN` on the last trial). The two new streams are also checked for positional alignment with `TrialOutcome`. NWB's required native `start_time` / `stop_time` are now derived when writing (`quiescent_start_time` → `ITI_stop_time`, falling back to `ITI_start_time`), so the NWB trials table changes in two ways: the old `start_time` / `stop_time` columns are gone, and the native trial extent now ends at the *end* of the ITI rather than at its start. |
| 2026-08-06 | Confirmed and documented that the legacy `delay_*` columns describe the acquisition software's **quiescence period**: `delay_start_time` is the `QuiescentPeriod` timestamp (always equal to the new `quiescent_start_time`) and `delay_duration` / `delay_beta` / `delay_min` / `delay_max` summarize `quiescence_period_duration`. `delay_duration` is the *configured* duration — each lick restarts the quiescent period, so the realized `quiescent_stop_time - quiescent_start_time` can be longer. Column descriptions updated accordingly. |
62 changes: 50 additions & 12 deletions src/dynamic_foraging_processing/pipeline/_pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,16 @@
_DEFAULT_LEFT_LICK = LickSource("HarpBehavior", "DigitalInputState", "DIPort0")
_DEFAULT_RIGHT_LICK = LickSource("HarpBehavior", "DigitalInputState", "DIPort1")

#: Trials-table columns NWB models natively; every other column is added as an
#: extra trial column.
_TRIAL_TIME_COLUMNS = ("start_time", "stop_time")
#: Trials-table column NWB's required native ``start_time`` is taken from. The
#: trials table itself no longer carries trial ``start_time`` / ``stop_time``
#: columns — it carries one start/stop pair per task period instead — but
#: ``TimeIntervals`` requires both, so they are derived here.
_NWB_START_COLUMN = "quiescent_start_time"

#: Columns NWB's required native ``stop_time`` is taken from, in order of
#: preference: the end of the ITI, falling back to its start on the last trial
#: of the session (where the ITI end is unknown).
_NWB_STOP_COLUMNS = ("ITI_stop_time", "ITI_start_time")

#: Source repository recorded in the ``processing.json`` data process.
_CODE_URL = "https://github.com/AllenNeuralDynamics/dynamic-foraging-processing"
Expand Down Expand Up @@ -224,23 +231,54 @@ def _add_acquisition_table(nwb_file: pynwb.NWBFile, table: AcquisitionTable) ->
)

@staticmethod
def _add_trials(nwb_file: pynwb.NWBFile, trials: pd.DataFrame) -> None:
def _trial_extent(row: pd.Series) -> t.Tuple[float, float]:
"""Return NWB's required native ``(start_time, stop_time)`` for one trial.

The trials table has no trial start/stop columns of its own, so the
trial's extent is taken from its period bounds: it starts with the
quiescent period and ends with the ITI, falling back to the ITI start on
the last trial of the session (whose ITI end is unknown).

Parameters
----------
row : pandas.Series
One row of the trials table.

Returns
-------
tuple of (float, float)
The trial start and stop time (seconds).
"""
stops = [row[column] for column in _NWB_STOP_COLUMNS if pd.notnull(row[column])]
stop = stops[0] if stops else np.nan
return float(row[_NWB_START_COLUMN]), float(stop)

@classmethod
def _add_trials(cls, nwb_file: pynwb.NWBFile, trials: pd.DataFrame) -> None:
"""Add the trials table to the NWB file's native ``trials`` table.

``start_time`` / ``stop_time`` are modeled natively by NWB; every other
column is registered as an extra trial column (described by the matching
:class:`TrialConfig` field) and populated per row. The DataFrame index
Every trials-table column is registered as an extra trial column
(described by the matching :class:`TrialConfig` field) and populated per
row. NWB additionally requires a native ``start_time`` / ``stop_time`` per
trial, which are derived from the period columns (see
:meth:`_trial_extent`) rather than stored as columns. The DataFrame index
(named ``id``) is replicated as each trial's NWB ``id``. An empty table
(or one missing the required time columns) is skipped.
(or one missing the period columns the extent is derived from) is skipped.
"""
if trials.empty or any(col not in trials.columns for col in _TRIAL_TIME_COLUMNS):
required = (_NWB_START_COLUMN, *_NWB_STOP_COLUMNS)
if trials.empty or any(col not in trials.columns for col in required):
return
descriptions = TrialConfig.column_descriptions()
extra_columns = [col for col in trials.columns if col not in _TRIAL_TIME_COLUMNS]
for column in extra_columns:
for column in trials.columns:
nwb_file.add_trial_column(name=column, description=descriptions.get(column, column))
for row_id, row in trials.iterrows():
nwb_file.add_trial(id=int(row_id), **{column: row[column] for column in trials.columns})
start_time, stop_time = cls._trial_extent(row)
nwb_file.add_trial(
id=int(row_id),
start_time=start_time,
stop_time=stop_time,
**{column: row[column] for column in trials.columns},
)

# ------------------------------------------------------------------ #
# Writers
Expand Down
Loading
Loading