Skip to content

fix(fetch): pass the correct baseurl to convert_to_md, fix undefined document in use_soup path - #1151

Open
Kumaresan-AI-Engineer wants to merge 1 commit into
ScrapeGraphAI:pre/betafrom
Kumaresan-AI-Engineer:fix/1142-fetch-node-baseurl
Open

fix(fetch): pass the correct baseurl to convert_to_md, fix undefined document in use_soup path#1151
Kumaresan-AI-Engineer wants to merge 1 commit into
ScrapeGraphAI:pre/betafrom
Kumaresan-AI-Engineer:fix/1142-fetch-node-baseurl

Conversation

@Kumaresan-AI-Engineer

Copy link
Copy Markdown

Summary

  • FetchNode's Chromium-loader path called convert_to_md(document[0].page_content, parsed_content) with the same HTML value for both the html and baseurl arguments, so html2text prepended the entire document to every relative link's href, inflating model input by ~74x on pages with many relative links. Now passes the actual source URL as baseurl.
  • The use_soup=True path had two related bugs in the same call: the html/url arguments were swapped, and parsed_content was referenced before assignment whenever cut=True (the default in every graph), raising UnboundLocalError.
  • The use_soup=True path also never assigned the document variable that handle_web_source unconditionally writes to state, so that path crashed with UnboundLocalError on every request, success or failure. Fixed by assigning document in both the success and failure branches.

Fixes #1142

Test plan

  • Added test_fetch_html_convert_to_md_uses_source_as_baseurl — asserts convert_to_md is called with (html, source) on the Chromium path.
  • Added test_fetch_html_use_soup_with_default_cut_does_not_raise — exercises use_soup=True with the default cut=True and asserts it no longer raises and calls convert_to_md with correct argument order.
  • uv run pytest tests/nodes/fetch_node_test.py passes.
  • uv run ruff check, uv run black --check, uv run isort --check-only pass on both changed files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant