fix: honor session cookies across HTTP client request paths - #2104
Open
Ayush7614 wants to merge 1 commit into
Open
fix: honor session cookies across HTTP client request paths#2104Ayush7614 wants to merge 1 commit into
Ayush7614 wants to merge 1 commit into
Conversation
Httpx send_request/stream now send outbound session cookies like crawl. Impit respects persist_cookies_per_session, keys the client cache by jar identity, and closes cached clients on cleanup. Httpx fingerprint headers are generated from a single profile so Accept and User-Agent stay consistent.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes inconsistent cookie handling across HTTP client request paths so that session cookies are reliably sent (and optionally persisted) whether requests go through crawler navigation (crawl) or handler-level calls (send_request/stream). It also makes Httpx’s fingerprint-derived headers internally consistent by sourcing Accept, Accept-Language, and User-Agent from a single generated fingerprint profile.
Changes:
- Httpx:
send_request/streamnow attach outbound session cookies via the shared request-building path (matchingcrawlbehavior). - Impit: implements
persist_cookies_per_sessionsemantics, caches clients by(proxy, cookie-jar identity), and adds client closing on cleanup/eviction. - Tests: adds coverage for cookie sending/persistence toggles, single-fingerprint headers, and Impit cleanup cache reset.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tests/unit/http_clients/test_http_clients.py | Adds unit coverage for session cookie sending in send_request/stream, persistence on/off behavior, single-fingerprint headers, and Impit cache cleanup. |
| src/crawlee/http_clients/_impit.py | Honors cookie persistence flag by using a resolved jar (shared vs copy), introduces client caching keyed by proxy + cookie jar identity, and closes cached clients on cleanup/eviction. |
| src/crawlee/http_clients/_httpx.py | Ensures handler-level requests attach session cookies and derives Accept/Accept-Language/User-Agent from one fingerprint profile. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+265
to
+267
| # Ephemeral jars (persist_cookies_per_session=False) must not pollute / thrash the LRU cache. | ||
| cacheable = cookie_jar is None or self._persist_cookies_per_session | ||
| cache_key = self._make_cache_key(proxy_url, cookie_jar) if cacheable else None |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
send_request/streamnow send outbound session cookies the same waycrawlalready did (via_build_request).persist_cookies_per_session(was accepted but ignored), caches clients by(proxy, cookie-jar identity), and closes cached clients on cleanup/eviction.Accept/Accept-Language/User-Agentare generated from a single fingerprint profile instead of two independentgenerate()calls that could mix browser profiles.Why
context.send_request()silently dropped session cookies under Httpx, breaking auth that worked for navigation. Impit's documentedpersist_cookies_per_session=Falsenever took effect because the session jar was always attached and mutated in place.Test plan
tests/unit/http_clients/test_http_clients.py— full file (82 passed)