fix: complete SessionError lifecycle and honor reclaim forefront

[Ayush7614]

Summary

• BasicCrawler SessionError path: honor error_handler replacement requests (same as regular errors), wrap handler exceptions in UserDefinedErrorHandlerError, and retire() the blocked session when rotations are exhausted.
• AdaptivePlaywrightCrawler: re-raise SessionError from the static sub-crawler instead of falling through to the browser with the same blocked session.
• Retries: reclaim_request now passes forefront=request.forefront so tiered-proxy priority retries actually stay at the front of the queue.

Why

error_handler could replace a request on normal failures but its return value was discarded for SessionError. Exhausted rotations left blocked sessions usable in the pool. Adaptive static SessionError was only logged, so a 403/blocked session could be reused for the browser fallback. Tiered proxies set request.forefront = True on retry, but reclaim always used the default forefront=False.

Test plan

• test_session_error_handler_can_replace_request
• test_session_retired_when_rotations_exhausted
• test_reclaim_uses_request_forefront_flag
• test_static_session_error_propagates_for_rotation
• Existing session rotation / error_handler tests still pass

[Copilot]

🟡 Not ready to approve

The SessionError path currently leaves requests in an inconsistent lifecycle state and misses retry/error tracking in one replacement branch, which can lead to incorrect persisted request metadata and statistics.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR completes the SessionError lifecycle in BasicCrawler/AdaptivePlaywrightCrawler and ensures retry reclaiming respects request queue priority (forefront) so tiered-proxy retries stay at the front of the queue.

Changes:

• Honor error_handler return values for SessionError, wrap handler exceptions consistently, and retire sessions when rotations are exhausted.
• Re-raise SessionError from the adaptive crawler’s static path to trigger session rotation instead of falling through to the browser.
• Pass forefront=request.forefront into reclaim_request so priority retries preserve queue ordering.

File summaries

File	Description
src/crawlee/crawlers/_basic/_basic_crawler.py	Updates retry reclaim behavior to honor forefront, and refines SessionError handling (rotation/retire + error_handler honoring).
src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py	Ensures static-path SessionError propagates to enable session rotation rather than browser fallback with the same session.
tests/unit/crawlers/_basic/test_basic_crawler.py	Adds unit tests covering SessionError error_handler replacement, session retirement on exhausted rotations, and forefront reclaim behavior.
tests/unit/crawlers/_adaptive_playwright/test_adaptive_playwright_crawler.py	Adds a unit test asserting static SessionError propagation triggers session rotation and prevents browser fallback.

Review details

Suppressed comments (1)

src/crawlee/crawlers/_basic/_basic_crawler.py:1504

• When session rotations are exhausted, the request is marked as handled without setting its final state to ERROR. This leaves failed requests in REQUEST_HANDLER state in storage, which diverges from the normal error path (where the request is set to RequestState.ERROR before marking handled).

            else:
                # Exhausted rotations: retire the blocked session so it is not reused from the pool.
                session.retire()
                await self._mark_request_as_handled(request)

• Files reviewed: 4/4 changed files
• Comments generated: 2
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.