fix(ei): return the named collection EI methods promise (#107) by karlwaldman · Pull Request #139 · OilpriceAPI/python-sdk

karlwaldman · 2026-09-13T19:01:38Z

Fixes #107.

Did it still reproduce?

Yes, on current origin/main (7982b0b), confirmed two ways.

Live, GET https://api.oilpriceapi.com/v1/ei/rig_counts/by_basin with a Scale-tier key on 2026-09-13:

{"data":{"report_date":"2026-08-28","basins":[{"region":"permian","region_type":"basin","count":267,"week_over_week":0,"change_direction":"flat"}, ...]},"meta":{...}}

rig_counts.by_basin() is annotated List[Dict[str, Any]] and its docstring example is for basin in basins: basin['rig_count']. It returned the whole data object, so the documented loop iterated dict keys.

The audit found the same class of defect in 27 methods across all seven EI resources, sync and async — not just the three named in the issue. Every envelope below was captured from production on 2026-09-13, not inferred from the controller.

Envelope map — every shipped EI method

data is the top-level data key. Rig counts / oil inventories / OPEC / drilling productivity / forecasts use {data, meta}; well permits and frac focus use {status, data}.

`ei.rig_counts`

Method	Route	`data` shape	Was	Now
`list`	`GET /v1/ei/rig_counts`	bare list	list	no change
`get`	`GET /v1/ei/rig_counts/{id}`	object	object	no change
`latest`	`GET /v1/ei/rig_counts/latest`	`{id, report_date, source, last_updated, us_total, basins, top_states, drilling_type}`	object	no change (docstring corrected)
`by_basin`	`GET /v1/ei/rig_counts/by_basin`	`{report_date, basins:[…]}`	whole object	`basins` list
`by_state`	`GET /v1/ei/rig_counts/by_state`	`{report_date, states:[…]}`	whole object	`states` list
`historical`	`GET /v1/ei/rig_counts/historical`	`{region, start_date, end_date, records:[…]}`	whole object	`records` list

`ei.drilling_productivity`

Method	Route	`data` shape	Was	Now
`list`	`/v1/ei/drilling_productivities`	bare list	list	no change
`get`	`/v1/ei/drilling_productivities/{id}`	object	object	no change
`latest`	`…/latest`	`{id, report_month, source, last_updated, total_duc, basins}`	object	no change (docstring corrected)
`summary`	`…/summary`	`{report_month, total_duc_wells, average_oil_productivity, average_gas_productivity, basins, headline}`	object	no change (docstring corrected)
`duc_wells`	`…/duc_wells`	`{report_month, total_duc, by_basin:[…], declining_basins}`	whole object	`by_basin` list
`by_basin`	`…/by_basin`	`{basins:"all", months:[…]}` — `basins` is the echoed filter string, `months` is the collection	whole object	`months` list
`historical`	`…/historical`	`{basin, basin_name, records:[…]}`	whole object	`records` list
`trends`	`…/trends`	`{report_month, analysis_months, trends:[…], declining_duc_basins}`	whole object	`trends` list

`ei.forecasts`

Method	Route	`data` shape	Was	Now
`list`	`/v1/ei/forecasts`	bare list	list	no change
`get`	`/v1/ei/forecasts/{id}`	object	object	no change
`latest`	`…/latest`	`{id, report_date, source, last_updated, summary, forecasts}`	object	no change (docstring corrected)
`summary`	`…/summary`	`{report_month, forecasts, headline}`	object	no change (docstring corrected)
`prices`	`…/prices`	`{report_month, commodities:{brent:…, wti:…, natural_gas:…}}` — a mapping, never a list	whole object, typed `List[Dict]`	same value, retyped `Dict[str, Any]`
`production`	`…/production`	`{report_month, series:{…}}` — a mapping	whole object, typed `List[Dict]`	same value, retyped `Dict[str, Any]`
`historical`	`…/historical`	`{series_code, actuals:[…]}`	whole object	`actuals` list
`compare`	`…/compare`	`{series_code, month1, month2, comparison}`	object	no change

`ei.oil_inventories`

Method	Route	`data` shape	Was	Now
`list`	`/v1/ei/oil_inventories`	bare list	list	no change
`get`	`…/{id}`	object	object	no change
`latest`	`…/latest`	`{id, report_date, week_ending, source, last_updated, summary, inventories}`	object	no change (docstring corrected)
`summary`	`…/summary`	`{week_ending, inventories, headline}`	object	no change (docstring corrected)
`by_product`	`…/by_product`	`{week_ending, products:[…]}`	whole object	`products` list
`historical`	`…/historical`	`{product_type, location, records:[…]}`	whole object	`records` list
`cushing`	`…/cushing`	`{location, latest, history}`	object	no change

`ei.opec_production`

Method	Route	`data` shape	Was	Now
`list`	`/v1/ei/opec_productions`	bare list	list	no change
`get`	`…/{id}`	object	object	no change
`latest`	`…/latest`	`{id, report_month, publication_month, production_month, source, opec_total, countries, headline}`	object	no change (docstring corrected)
`total`	`…/total`	`{latest, history, trend}`	object	no change
`by_country`	`…/by_country`	`{report_month, publication_month, production_month, countries:[…], opec_total}`	whole object	`countries` list
`historical`	`…/historical`	`{country, country_name, records:[…]}`	whole object	`records` list
`top_producers`	`…/top_producers`	`{report_month, …, producers:[…], opec_total}`	whole object	`producers` list

`ei.well_permits`

Method	Route	`data` shape	Was	Now
`list`	`GET /v1/ei/well-permits`	`{well_permits:[…], meta}`	whole object	`well_permits` list
`get`	`…/{api_number}`	`{well_permit:{…}}`	the wrapper	the record
`latest`	`…/latest`	`{well_permits:[…], meta}`	whole object	no change — kept deliberately, see below
`summary`	`…/summary`	`{period_days, total_permits, by_state, top_operators, top_formations, by_permit_type, weekly_trend, last_updated, as_of, data_age_days, stale, as_of_basis, by_state_as_of, stale_states}`	object	no change (docstring corrected)
`by_state`	`…/by-state`	`{well_permits:[…], state, meta}`	whole object	`well_permits` list
`by_operator`	`…/by-operator`	`{well_permits:[…], operator_query, meta}`	whole object	`well_permits` list
`by_formation`	`…/by-formation`	`{well_permits:[…], formation_query, meta}`	whole object	`well_permits` list
`search`	`…/search`	`{well_permits:[…], meta}`	already correct	no change (helper refactored to share one implementation)

`ei.frac_focus`

Method	Route	`data` shape	Was	Now
`list`	`GET /v1/ei/frac-focus`	`{frac_focus_disclosures:[…], meta}`	whole object	`frac_focus_disclosures` list
`get`	`…/{upload_key}`	`{frac_focus_disclosure:{…}}`	the wrapper	the record
`latest`	`…/latest`	`{frac_focus_disclosures:[…], meta}`	whole object	no change — kept deliberately, see below
`summary`	`…/summary`	`{period_days, total_disclosures, by_state, top_operators, water_usage, monthly_trend, last_updated}`	object	no change (docstring corrected)
`by_state`	`…/by-state`	`{frac_focus_disclosures:[…], state, meta}`	whole object	list
`by_operator`	`…/by-operator`	`{frac_focus_disclosures:[…], operator_query, meta}`	whole object	list
`by_chemical`	`…/by-chemical`	`{frac_focus_disclosures:[…], chemical_query, meta}`	whole object	list
`search`	`…/search`	`{frac_focus_disclosures:[…], meta}`	whole object	list
`chemicals`	`…/{upload_key}/chemicals`	`{upload_key, api_number, well_name, operator, chemical_count, chemicals:[…], additives, cas_numbers, suppliers}`	whole object	`chemicals` list
`for_well`	`…/for-well/{api_number}`	`{api_number, frac_focus_disclosures:[…], count}`	whole object	`frac_focus_disclosures` list

ei.well_timeline() returns data as an object and is unchanged; it now calls the shared ei_data helper instead of its own copy of the unwrap line.

Counts: 27 methods changed (24 collections + 2 nested records + 1 shared search helper refactor), 2 retyped with no behaviour change, 29 recorded as no-change.

Public contract

The smallest compatible change, per method:

Collections now return the list the signature and the docstring example always promised. Code written against the documented contract was broken before and works now.
latest() on well permits and frac focus keeps returning the envelope object. Its annotation (Dict[str, Any]) was already truthful; changing it to a list would break callers and throw away the pagination/freshness counters the issue asks us to preserve. Only the docstring changed, to name the real keys.
prices() / production() return the same value as before. Only the annotation moved from List[Dict[str, Any]] to Dict[str, Any], because those endpoints return a mapping keyed by commodity / series code.

Breaking for callers who adapted to the bug rather than the docs. by_basin()["basins"], well_permits.list()["well_permits"] and well_permits.get(id)["well_permit"] must drop that subscript. Collection methods on well permits and frac focus no longer surface data["meta"] — that matches every other EI list(), which has always dropped meta. Called out in the CHANGELOG. No version bump; this sits under ## [Unreleased].

No second abstraction

oilpriceapi/resources/ei/_envelopes.py (ei_data, unwrap_ei_collection, unwrap_ei_object) is now the only place that knows the EI envelope shape. The if "data" in response: return response["data"] block repeated in 60+ EI methods is gone, and unwrap_well_permit_search_response keeps its public name, its MALFORMED_RESPONSE code and its exact error message while delegating to the shared helper — one implementation, not two. Its existing tests pass unchanged.

Failures are explicit: a 200 whose body is missing the named collection, or whose collection holds a non-record, raises OilPriceAPIError(code="MALFORMED_RESPONSE") with the raw body attached. An empty collection is an empty list, never an error; a missing collection is never a fabricated empty success.

Sync and async are identical

Every changed sync method has the identical change in oilpriceapi/async_resources.py. That is not a claim, it is pinned by test_async_ei_unwrapping_matches_sync_line_for_line, which reads both implementations with inspect.getsource and asserts each async method's trailing return expression matches its sync twin character for character, for all seven resources. Proven red-capable by reverting one async method to ei_data(response):

E       AssertionError: sync/async unwrapping drift: [('by_state', 'return unwrap_ei_collection( response, collection="states", subject="rig-count by-state" )', 'return ei_data(response)')]
FAILED tests/unit/test_ei_envelopes.py::test_async_ei_unwrapping_matches_sync_line_for_line[AsyncEIRigCountsResource-rig_counts]

Behavioural parity is covered too: every collection case runs through both clients and the results are asserted equal, and the public method sets of each sync/async pair must match.

Tests

tests/unit/test_ei_envelopes.py — 200 tests driving the real client transport. respx intercepts httpx, so the request leaves the client, the response body is the live-captured envelope, and the assertion is on what the resource hands back. Nothing stubs client.request; the issue calls this out specifically, and a resource-level mock cannot see the defect at all. respx>=0.20.2 added to the dev extra.

Coverage: valid collection, empty collection, missing collection key, malformed row, nested-record unwrapping, object-envelope passthrough (a regression guard against a broad global unwrap), 401 / 403 / 429, async equivalents, and both parity guards.

Red — the new test file against pre-fix `origin/main` sources

Run in a detached worktree at origin/main with only the new test file and _envelopes.py copied in:

$ python -m pytest tests/unit/test_ei_envelopes.py -q --no-cov
...
=================================== FAILURES ===================================
___________ test_named_collection_is_unwrapped[rig_counts.by_basin] ____________
E       AssertionError: rig_counts.by_basin must return the 'basins' list, got {'report_date': '2026-08-28', 'basins': [{'region': 'permian', 'region_type': 'basin', 'count': 267, 'week_over_week': 0, 'change_direction': 'flat'}]}
E       assert {'report_date...ek': 0, ...}]} == [{'region': '...eek': 0, ...}]

____________ test_named_collection_is_unwrapped[well_permits.list] _____________
E       AssertionError: well_permits.list must return the 'well_permits' list, got {'well_permits': [{'api_number': '05123534110000', 'state_code': 'CO', 'county': 'Weld', 'permit_number': 'P-1', 'permit_type': 'drill', 'permit_status': 'approved', 'permit_date': '2026-09-01', 'operator': {'name': 'Fixture Operating'}, 'well': {'name': 'Fixture 1H'}, 'location': {'lat': 40.1, 'lng': -104.7}, 'target': {'formation': 'Niobrara'}, 'provenance': {'source': 'cogcc'}}], 'meta': {'total_count': 1}}
E       assert {'well_permit...al_count': 1}} == [{'api_number...: 'P-1', ...}]

FAILED tests/unit/test_ei_envelopes.py::test_named_object_is_unwrapped[frac_focus.get]
FAILED tests/unit/test_ei_envelopes.py::test_named_object_is_unwrapped_async[well_permits.get]
FAILED tests/unit/test_ei_envelopes.py::test_named_object_is_unwrapped_async[frac_focus.get]
======================== 148 failed, 52 passed in 2.13s ========================

Green — same file on this branch

$ python -m pytest tests/unit/test_ei_envelopes.py tests/unit/test_ei_well_permits_resource.py -q --no-cov
tests/unit/test_ei_well_permits_resource.py ...............              [100%]

============================= 215 passed in 1.15s ==============================

Full suite

origin/main : 3 failed, 864 passed, 63 skipped in 5.71s
this branch : 3 failed, 1064 passed, 63 skipped in 10.38s

The 3 failures are identical on both sides and pre-existing: tests/integration/test_demo_contract.py makes live calls and gets HTTP 429 from this IP. Environmental, not code. The CI command excludes them:

$ pytest tests/ --ignore=tests/integration --ignore=tests/contract -m 'not slow' --no-cov -q
======================= 1060 passed, 14 skipped in 6.85s =======================

ruff check oilpriceapi/ — clean. mypy oilpriceapi/ --ignore-missing-imports — Success: no issues found in 48 source files.

CHANGELOG.md had two ## [Unreleased] headings, the lower one empty; tests/test_release_readiness.py asserts there is exactly one, so the empty duplicate is removed here.

🤖 Generated with Claude Code

https://claude.ai/code/session_015ao5paex73xXvuM424Libo

Every Energy Intelligence method typed `List[Dict[str, Any]]` returned `response["data"]`, but the EI controllers put their records under a *named* key inside `data`. `client.ei.rig_counts.by_basin()` returned `{"report_date": ..., "basins": [...]}` where the signature and the docstring example promised the basin list, so the documented `for basin in basins: basin["count"]` iterated dict keys. Verified live against https://api.oilpriceapi.com on 2026-09-13 with a Scale-tier key: 27 methods across seven EI resources are wrong the same way, in both the sync resources and `async_resources.py`. - `unwrap_ei_collection` / `unwrap_ei_object` / `ei_data` in `oilpriceapi/resources/ei/_envelopes.py` are now the single place that knows the envelope shape. The per-method `if "data" in response: return response["data"]` is gone, and `unwrap_well_permit_search_response` keeps its name and its error message but delegates to the shared helper instead of carrying a second copy. - A success body missing the named collection raises `OilPriceAPIError(code="MALFORMED_RESPONSE")` instead of handing back the envelope. An empty collection stays an empty list. - `ei.well_permits.get()` / `ei.frac_focus.get()` return the record rather than its `{"well_permit": ...}` wrapper. - `ei.forecasts.prices()` / `.production()` are typed `Dict[str, Any]`: both return a mapping keyed by commodity / series code, never a list. - `ei.well_permits.latest()` / `ei.frac_focus.latest()` deliberately keep returning the envelope so their pagination and freshness counters stay reachable; their docstrings now say so. - Docstring examples use the field names the API actually returns. Tests drive the real client transport (respx over httpx), not a stubbed resource, because the defect lives between the HTTP body and the return value. Valid, empty, missing-collection, malformed-row, 401/403/429 and async parity are covered, plus a source-level guard that every async EI method's return expression matches its sync twin character for character. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ao5paex73xXvuM424Libo

coderabbitai · 2026-09-13T19:01:51Z

Important

🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: aa42aa55-f76c-4b68-93b7-30372ca78f35

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Resolves the CHANGELOG conflict with #135 (streaming, #108), the only file that conflicted. Both PRs added an Unreleased entry; #135's landed in the stale mid-file `## [Unreleased]` block that this branch removes, so the streaming bullets move into the one canonical Unreleased section at the top of the file alongside the EI entries. Both entries are preserved verbatim and `## [Unreleased]` appears exactly once, as test_release_readiness.py requires. Code merged cleanly: async_resources.py auto-merged with #138's diesel `state_code` fallback, which touches a different resource class. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ao5paex73xXvuM424Libo

karlwaldman merged commit 7f2c3c5 into main Sep 13, 2026
7 checks passed

karlwaldman deleted the fix/ei-envelopes-107 branch September 13, 2026 19:31

karlwaldman mentioned this pull request Sep 13, 2026

chore(release): 1.15.0 #141

Merged

Navigation Menu

fix(ei): return the named collection EI methods promise (#107) - #139

fix(ei): return the named collection EI methods promise (#107)#139
karlwaldman merged 2 commits into
mainfrom
fix/ei-envelopes-107

karlwaldman commented Sep 13, 2026

Uh oh!

coderabbitai Bot commented Sep 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

karlwaldman commented Sep 13, 2026

Did it still reproduce?

Envelope map — every shipped EI method

ei.rig_counts

ei.drilling_productivity

ei.forecasts

ei.oil_inventories

ei.opec_production

ei.well_permits

ei.frac_focus

Public contract

No second abstraction

Sync and async are identical

Tests

Red — the new test file against pre-fix origin/main sources

Green — same file on this branch

Full suite

Uh oh!

coderabbitai Bot commented Sep 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

`ei.rig_counts`

`ei.drilling_productivity`

`ei.forecasts`

`ei.oil_inventories`

`ei.opec_production`

`ei.well_permits`

`ei.frac_focus`

Red — the new test file against pre-fix `origin/main` sources

coderabbitai Bot commented Sep 13, 2026 •

edited

Loading