Skip to content

Array-API-native block_reduce, block_average and block_replicate off numpy - #1009

Open
mwcraig wants to merge 5 commits into
astropy:mainfrom
mwcraig:blocks-native
Open

Array-API-native block_reduce, block_average and block_replicate off numpy#1009
mwcraig wants to merge 5 commits into
astropy:mainfrom
mwcraig:blocks-native

Conversation

@mwcraig

@mwcraig mwcraig commented Sep 7, 2026

Copy link
Copy Markdown
Member

ccdproc.core.block_reduce, block_average and block_replicate are thin
wrappers around astropy.nddata, which starts by calling numpy.asanyarray
on its input. On a non-numpy array library that silently hands back a numpy
array (dask, jax) or fails outright when the data live on a device numpy
cannot reach (array-api-strict, and by extension cupy on a GPU). These were
three of the remaining array-API escapes in the escape baseline.

This adds ccdproc/_blocks.py, which does the same work using only array API
operations (reshape, permute_dims, repeat and slicing), and dispatches
to it for every namespace that is not numpy.

This is deliberately a stopgap: it exists so ccdproc's array-API work does not
have to wait for an astropy release. _blocks.block_reduce and
_blocks.block_replicate, minus the ccdproc dispatch, are the intended
starting point for astropy/astropy#15073; once astropy's own blocks.py is
array-API aware and that version is ccdproc's floor, this module and the
dispatch both go away. (block_average has no astropy counterpart -- it is
ccdproc's own thin wrapper -- so only the other two are candidates for the
lift.)

Dispatch policy

The numpy path is unchanged: if array_api_compat.is_numpy_namespace(xp) the
functions call astropy.nddata exactly as before, with the same arguments.
Everything else goes to _blocks. The CCDData rebuild and the
"following attributes were set ... will be ignored" warning behaviour are
untouched -- the native helpers are decorated with
astropy.nddata.support_nddata, as astropy's own are, so a CCDData
argument is unpacked and warned about identically on every backend.

block_size validation is done in pure Python and reproduces astropy's three
checks in astropy's order with astropy's messages verbatim; the tests assert
against astropy's live output rather than hard-coded strings so the two cannot
drift apart.

Behaviour differences

Results are identical to astropy's except for two dtype promotions, both of
integer/boolean input to the namespace's default real floating dtype:

  • block_replicate(..., conserve_sum=True) promotes before dividing, because
    array-api-strict rejects integer true division rather than promoting.
  • block_average promotes before averaging, because array-api-strict rejects
    a non-floating mean. numpy promotes on its own here, and jax and dask
    follow it, so without this an integer image averaged fine on three backends
    and raised on the fourth.

numpy returns float64 in both cases anyway, so these differ only for a
library whose default real dtype is not float64. Documented in
docs/array_api.rst and CHANGES.rst.

Verified backends

Run with the dev environment directly, CCDPROC_ARRAY_LIBRARY=<lib> (plus
JAX_ENABLE_X64=1 for jax); array-api-strict runs on its non-default
device1, which is where np.asarray raises.

pytest ccdproc/tests/test_blocks.py ccdproc/tests/test_ccdproc.py:

backend result
numpy 168 passed
jax 168 passed
dask 168 passed
array-api-strict (device1) 159 passed, 9 xfailed

Full suite, this branch vs. the merge base:

backend branch merge base
numpy 1045 passed, 5 skipped 975 passed, 5 skipped
array-api-strict 997 passed, 10 skipped, 43 xfailed 924 passed, 10 skipped, 46 xfailed
dask (escape enforcement) 1034 passed, 16 skipped not run
jax 1040 passed, 10 skipped not run

The deltas are fully accounted for: the 70 new tests in test_blocks.py, plus
the three formerly-xfailed array-api-strict tests now passing.

The three block_* lines are removed from
ccdproc/tests/array_escape_baseline.txt. Verified with a full dask run under
CCDPROC_LOG_ARRAY_ESCAPES=1 CCDPROC_ENFORCE_ESCAPE_BASELINE=1: no escapes
outside the baseline, no block_* sites in the escape log, and temporarily
restoring the three lines makes the ratchet report exactly those three as
"not hit this run" and nothing else.

Things worth a reviewer's eye

  • _nanfuncs._fill_doc gained a positional-only template argument so
    _blocks.py can reuse the docstring templating with its own parameter
    block. Backwards compatible; no _nanfuncs docstring changed.
  • core._block_namespace resolves the namespace as
    ccd.data if isinstance(ccd, CCDData) else ccd, matching sigma_func's
    convention. The old code used ccd.data unconditionally, which crashed for
    a bare array (np.ndarray.data is a memoryview) -- bare arrays now work.
    The narrowing is that a plain list passed with an explicit func used to
    reach astropy and now raises TypeError from array_namespace; nothing in
    the test suite or docs does that.
  • The promotion guard is xp.isdtype(dtype, ("integral", "bool")) rather than
    "not real floating", so complex input is not silently truncated to real.
  • With the array-api-strict xfail markers removed, the three existing block
    tests failed one step later on their own xp.zeros((4, 4), dtype=bool),
    which array-api-strict rejects; they now ask for dtype=xp.bool, as the
    rest of that file already does. The ccd._mask = ... TODO is untouched.
  • The first commit is green only on numpy: test_blocks.py ships with it and
    includes two tests of the core dispatch that need the second commit on
    non-numpy backends. Squashing the first two commits would fix that if you
    care about per-commit bisection on the backend jobs.

Part of #971.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F6f9L1GyMHrKnvLrrgWbWN

mwcraig and others added 4 commits September 7, 2026 15:27
astropy.nddata's block functions start with numpy.asanyarray, so on a
non-numpy array library they hand back a numpy array (dask, jax) or fail
outright when the data are on a device numpy cannot reach
(array-api-strict, cupy).

ccdproc/_blocks.py does the same work using only array API operations --
reshape, permute_dims, repeat and slicing -- so the result stays in the
caller's namespace and on the caller's device. block_size validation is
pure Python and reproduces astropy's three checks, in astropy's order,
with astropy's messages. Both functions are decorated with
astropy.nddata.support_nddata, as astropy's own are, so a CCDData
argument is unpacked and the "following attributes were set ... will be
ignored" warning is emitted identically.

The one deliberate divergence is dtype: block_replicate(conserve_sum=True)
promotes integer and boolean input to the namespace's default real
floating dtype before dividing, because array-api-strict rejects integer
true division rather than promoting. numpy returns float64 there anyway.

_nanfuncs._fill_doc gained a positional-only template argument so the new
module can reuse the docstring templating with its own parameter block.

Part of astropy#971.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6f9L1GyMHrKnvLrrgWbWN
ccdproc.core.block_reduce/block_average/block_replicate now resolve the
array namespace first and keep calling astropy.nddata only when it is
numpy; every other namespace gets ccdproc._blocks. The numpy branch is
the code that was there before, so numpy results are unchanged, and the
CCDData rebuild and the ignored-attribute warning are untouched.
block_replicate gained the xp= argument the other two already had.

With the escape gone, drop the three block_* lines from the array-escape
baseline and the three array-api-strict xfail markers on the block tests.
Those tests then failed one step later on xp.zeros(..., dtype=bool),
which array-api-strict rejects; they now ask for xp.bool, as the rest of
the file does.

Part of astropy#971.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6f9L1GyMHrKnvLrrgWbWN
numpy's mean promotes an integer array on its own, and jax and dask
follow it, but array-api-strict refuses a non-floating mean outright, so
an integer image averaged fine on three backends and raised on the
fourth. ccdproc._blocks.block_average now promotes integer and boolean
input to the namespace's default real floating dtype before reducing
with xp.mean, the same treatment block_replicate already had, and
core.block_average dispatches to it. The numpy path is unchanged and
still goes straight to astropy.nddata.

block_average has no astropy counterpart, so it lives here as ccdproc's
own thin wrapper; only block_reduce and block_replicate are candidates
for the upstream lift.

The docstring parameter template gained an empty-``extra`` form, since
block_average has no function-specific parameter between block_size and
xp.

Part of astropy#971.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6f9L1GyMHrKnvLrrgWbWN
@codecov

codecov Bot commented Sep 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.05%. Comparing base (9d18599) to head (fdaad17).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1009      +/-   ##
==========================================
+ Coverage   97.97%   98.05%   +0.07%     
==========================================
  Files           9       10       +1     
  Lines        1927     2005      +78     
==========================================
+ Hits         1888     1966      +78     
  Misses         39       39              
Flag Coverage Δ
dask 97.05% <94.31%> (-0.15%) ⬇️
jax 97.20% <94.31%> (-0.15%) ⬇️
numpy 97.80% <96.59%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Codecov flagged two untested branches in astropy#1009: the early return in
_block_namespace when the caller passes xp, and the int() failure
branch in _block_size that turns a NaN or infinite block size into
astropy's "must be integers" error.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F6f9L1GyMHrKnvLrrgWbWN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant