feat(spreads,indicators): typed sync+async Spreads and Indicators resources (#99)

[karlwaldman]

Closes #99.

What

• client.spreads and client.indicators on OilPriceAPI and AsyncOilPriceAPI, with typed pydantic models in the new oilpriceapi/metrics_models.py.
• spreads, 15 methods:
  • crack, crack_historical, crack_all, gasoil_crack
  • basis, basis_historical, basis_all
  • curve_structure, curve_structure_all
  • margin, margin_historical, margin_all
  • physical_premium, physical_premium_historical, physical_premium_all
• indicators, 10 methods:
  • fuel_switching, fuel_switching_historical
  • price_context(code, related_spreads=False)
  • storage_analytics, storage_analytics_all
  • annotations, annotations_batch
  • cftc_positioning, cftc_positioning_historical, cftc_positioning_all
• oilpriceapi/resources/_calculated_metrics.py is the single place that builds paths and params, validates arguments, and parses envelopes. The sync resources (resources/spreads.py, resources/indicators.py) and the async resources (async_resources.py) are thin wrappers over it, so the two clients cannot drift.
• Docs, README section, examples/spreads_indicators.py, and CHANGELOG under [Unreleased]. No version bump or tag.
• .gitignore has a blanket *.json, so this PR adds one exception, !tests/unit/fixtures/**/*.json, to commit the captured production bodies.

Premise verification

Checked against oilpriceapi-api origin/main 02196c516:

• Routes and controller: config/routes.rb scope 'spreads' / scope 'indicators' map to V1::SpreadsController. All 28 routes exist. Params come from the controller: type, crude, pair, commodity, index, gas, location, code, codes, spreads=related, start_date, end_date.
• Required vs conditional keys: read from app/services/calculated_metrics/*.rb to separate always-present keys from conditional ones (stale_flag, changes, .compact, {} blocks).
• Production probes: run 2026-09-13 with the paid test key. The key is read from env only and never written to a file.
  • Every route and the error paths were captured.
  • The 30 bodies in tests/unit/fixtures/calculated_metrics/ are verbatim captures.
• Corrections posted on [P1][Sprint 1][Parity] Add typed Spreads and market Indicators resources #99:
  • No support manifest or machine-readable capabilities file exists.
  • Node's typed resources do not match the wire (node#112).
  • These routes gate with 403 PREMIUM_REQUIRED, not 402.
  • congressional-trades is excluded because it has never returned data (api#8478).
  • Release deferred per instruction.

Wire shapes typed against (production, 2026-09-13)

• Envelope: {"status":"success","data":{...}}. Collections are nested under a named key: spreads, commodities, margins, premiums, locations.
• Errors: {"error":{"code","message","status","request_id","docs"}}.
  • 400 MISSING_PARAMETER
  • 401 UNAUTHORIZED
  • 404 DATA_NOT_AVAILABLE, with the valid values listed in the message
• crack:
  • {spread_type, crude_benchmark, value, unit, components:{crude|product|gasoline|diesel:{code,price,unit}}, timestamp, changes:{change_1d..}}
  • Adds data_stale / stale_warning only when stale, and only on single-product types (api#8477).
• crack/historical: {spread_type, crude_benchmark, period:{start,end}, coverage:{from,to,observations,complete}, data_revised_at, count, data:[{date,value,crude,gasoline,diesel}]}.
• gasoil-crack: {spread_type, name, value, unit, components:{product|crude:{code,contract_month,updated_at,price,unit}}, conversion:{barrels_per_tonne,basis,gasoil_usd_per_bbl}, timestamp, updated_at, data_stale?}.
• basis: {pair, spread_name, value, unit, components:{CODE:price}, signal, timestamp, percentile_1y, changes, negative_streak_days? (WAHA_HH only)}. History rows are {date,value,code_a,code_b}.
• curve-structure: {commodity, display_name, structure, severity, term_slope_pct, spreads:{m1_m3?,m1_m6,m1_m12?}, front_month:{price,contract}, back_month_6, curve_points, signal, timestamp}.
• margin: {index, name, margin_usd_bbl, crude_input:{code,price}, product_basket:{name:{yield_pct,price,code}}, signal, percentile_1y, changes, timestamp}. History rows are {date,margin,crude,revenue}.
• physical-premium: {commodity, name, premium, premium_pct, unit, components:{spot,futures:{code,price,contract}}, signal, elevated_streak_days, percentile_1y (null observed), timestamp}.
• fuel-switching: {oil_parity:{ratio_pct,threshold_pct,signal,parity_price,current_gas,headroom_pct}, components:{gas,crude}, energy_equivalent, historical_context ({} under 10 points), timestamp}.
• price-context:
  • {code, price, timestamp, context:{anomaly, anomaly_reason?, change_*?, high_52w?, low_52w?, percentile_1y?, percentile_5y?}, related_spreads?}
  • related_spreads entries are {name,value(float|str),signal,unit?,slope?}.
• storage-analytics: {location, name, current:{volume_mmbbl,utilization_pct,operational_capacity_mmbbl,data_date,timestamp}, draw_rate, seasonal ({} observed), anomalies, range_52w ({} observed), signal, trading_implication}.
• annotations: {code, price, timestamp, annotation_count, annotations:[{type,severity,message,...}]}. The batch form is {annotated:[...], total_codes, codes_with_annotations}.
• cftc-positioning:
  • {commodity, name, report_date, positioning:{speculative:{net,long,short,net_pct_of_oi}, commercial:{net}, open_interest}, signal, percentile_1y, week_change, timestamp}
  • BRENT, NATURAL_GAS, HEATING_OIL and GASOLINE have null long/short/OI.
  • History rows are {date,spec_net,open_interest,spec_net_pct_oi}. spec_net_pct_oi is always 0 on the wire (api#8476) and is preserved as sent.

Rules the models enforce

• A key the server always emits is required. A nullable one is Optional[...] with no default, so a body that drops it fails.
• Conditionally emitted keys are Optional[...] = None. None means "not sent", never false or zero.
• Values stay as sent: full float precision, units, nulls. Timestamps become tz-aware datetime; calendar dates become date.
• A malformed 200 raises OilPriceAPIError(code="MALFORMED_RESPONSE", raw_body=...). That covers a bad envelope, status != "success", a missing or mistyped field, a non-list collection, and non-JSON or empty bodies.
• Refused locally, before any request (ValueError):
  • blank or non-string selectors
  • dates not in YYYY-MM-DD form or not real calendar dates
  • start_date after end_date
  • a bare string, an empty list, blank codes or codes containing commas for annotations_batch
  • more than 20 codes, because the server silently annotates only the first 20

Review fix: local refusals raise ValidationError, not ValueError

The first push had 6 raw raise ValueError( for local input refusals. A builtin ValueError slips past the documented except OilPriceAPIError catch-all, which is the defect class #123 fixed. Every refusal now goes through one helper that matches _url._reject:

ValidationError(message=..., field=<arg name>, value=<offending value>, status_code=None)

• status_code=None, because no request was sent (fix(url,retry,errors): typed errors, a real 60s cap, no status on a local refusal (#123) #134).
• field is the argument name: pair, commodity, spread_type, code, codes, start_date or end_date. value is the input that was rejected.
• Dates still go through the shared format_date, which raises ValueError for older resources. These new methods catch that and re-raise it as ValidationError with the right field.
• No dual-base subclass. Every method in this PR is new, and no existing public method changed, so there is no earlier except ValueError caller to keep working.
• git diff origin/main -- oilpriceapi/ | grep -c '^+.*raise ValueError' gives 0.

The tests now check the exact type (type(error) is ValidationError), that it is an OilPriceAPIError, that status_code is None, and the expected field. That covers 17 cases each for sync and async, plus one test that the offending value is kept. Every case still asserts the transport was never called.

Red run with the old assertions still in place (the implementation was still raising ValueError):

E           ValueError: pair must be a non-empty string
E           ValueError: pair must be a non-empty string
E           ValueError: commodity must be a non-empty string
======================== 35 failed, 108 passed in 1.32s ========================

Red-capable proof after the fix. The helper was temporarily changed back to return ValueError(message), the tests were run, and the file was restored:

E           ValueError: pair must be a non-empty string
====================== 35 failed, 108 deselected in 0.83s ======================
RED
restored: True

Green after the fix: the module passes 143 tests. The full suite is 1372 passed / 3 failed / 65 skipped; the 3 failures are the same live test_demo_contract.py 429s. ruff, mypy (53 files) and validate_storefront_claims.py all pass.

TDD evidence

Red (tests written first, before any implementation)

============================= test session starts ==============================
platform darwin -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
rootdir: /Users/karlwaldman/code/py-99
configfile: pyproject.toml
plugins: cov-7.1.0, timeout-2.4.0, asyncio-1.4.0, anyio-4.15.1, respx-0.23.1
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 0 items / 1 error

==================================== ERRORS ====================================
_______ ERROR collecting tests/unit/test_spreads_indicators_resource.py ________
ImportError while importing test module '/Users/karlwaldman/code/py-99/tests/unit/test_spreads_indicators_resource.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/homebrew/Cellar/python@3.12/3.12.13_4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/unit/test_spreads_indicators_resource.py:34: in <module>
    from oilpriceapi.metrics_models import (
E   ModuleNotFoundError: No module named 'oilpriceapi.metrics_models'
=========================== short test summary info ============================
ERROR tests/unit/test_spreads_indicators_resource.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.16s ===============================

Green (new test module)

collected 142 items

tests/unit/test_spreads_indicators_resource.py ......................... [ 17%]
........................................................................ [ 68%]
.............................................                            [100%]

============================= 142 passed in 0.71s ==============================

The 142 tests drive the real sync and async clients through mocked httpx.Client.request / httpx.AsyncClient.request, asserting the path and params sent and field values from the captured bodies:

• 27 success cases × sync and async
• 401 / 402 / 403 / 404 / 400 / 429 mapping, including code, request_id, required_plan, remediation_url and retry_after
• 15 malformed-200 variants × sync and async, plus non-JSON and empty bodies
• no-data: an empty history window and an empty collection
• timeout typed, and a timeout recovered on retry
• 17 invalid-argument cases × sync and async, asserting the transport is never called

Mutation check (the suite fails when the guarantees are removed)

KILLED  | no model validation (model_construct) | 1 failed in 0.12s
KILLED  | no status check | 1 failed, 74 passed in 0.51s
KILLED  | no start>end check | 1 failed, 115 passed in 0.68s
KILLED  | no batch cap | 1 failed, 122 passed in 0.71s
KILLED  | nullable percentile gets default | 1 failed, 80 passed in 0.53s
KILLED  | non-JSON not wrapped (sync) | 1 failed, 97 passed in 0.63s
KILLED  | wrong related-spreads param | 1 failed, 19 passed in 0.21s
survivors: 0

Full suite, lint, types

=========== 3 failed, 1371 passed, 65 skipped, 30 warnings in 6.62s ============

• Baseline on clean origin/main 4bd2900 was 1229 passed / 3 failed / 63 skipped.
• +142 passed are the new tests; +2 skipped are the live tests, which skip without a key.
• The 3 failures are the pre-existing live tests/integration/test_demo_contract.py 429s, unchanged.

ruff check oilpriceapi/            -> All checks passed!
mypy oilpriceapi/ --ignore-missing-imports -> Success: no issues found in 53 source files
python scripts/validate_storefront_claims.py -> validated 74 public surfaces

Live smoke: run, passed

tests/integration/test_live_spreads_indicators.py ran against production on 2026-09-13 with the paid test key. It is read-only (GET only), spaces calls 1.1s apart, and treats a 429 as a skip.

tests/integration/test_live_spreads_indicators.py::test_spreads_latest_and_all PASSED [ 50%]
tests/integration/test_live_spreads_indicators.py::test_indicators PASSED [100%]

============================== 2 passed in 46.21s ==============================

Defects found and filed

• OilpriceAPI/oilpriceapi-api#8476: CFTC history spec_net_pct_oi is always 0 (integer division on value_units).
• OilpriceAPI/oilpriceapi-api#8477: the 3-2-1 composite crack never emits data_stale.
• OilpriceAPI/oilpriceapi-api#8478: congressional-trades is routed, has never returned data, and its 404 exposes QUIVER_API_KEY.
• OilpriceAPI/oilpriceapi-api#5913 (comment): history routes return an empty 200 for unknown selectors, and bad dates silently fall back to the default window.
• OilpriceAPI/oilpriceapi-api#3297 (comment): intermittent 500s from price-context, with request_ids.
• [P2] spreads and indicators resources are typed against a shape the API does not send, and omit required parameters oilpriceapi-node#112: Node spreads/indicators types do not match the wire, and required params cannot be passed.

Merge notes

This PR will conflict trivially with #143 (#100) and #144 (#101) in CHANGELOG.md, README.md, docs/reference/resources.md, async_resources.py and both clients. All of those conflicts are additive.

🤖 Generated with Claude Code

https://claude.ai/code/session_015ao5paex73xXvuM424Libo

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 2dbad09d-0cc0-4bd5-b7e2-01b3ed6eccad

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}