refactor(dedupe): one definition of the location prefetch

[valentijnscholten]

Summary

• Which location relation to prefetch is a property of what Finding.get_locations() reads, not of any individual caller — but it was spelled out separately at every call site, and that drifts in both directions. The hash-recompute paths disagreed about it for months, and the batch dedupe loader prefetched the deprecated endpoints relation, which raises under V3_FEATURE_LOCATIONS on an instance migrated from endpoints.
• Adds location_prefetch_lookups(prefix="") to dojo/location/queries.py, next to vulnerability_id_prefetch, and routes build_candidate_scope_queryset and manage.py dedupe through it.
• The dedupe command's V3 branch prefetched locations — one hop — where get_locations() reads location.url, leaving a query per location reference. It now prefetches the full path like the other call sites, and its two near-identical select_related/prefetch_related blocks collapse into one.
• get_finding_models_for_deduplication is deliberately left alone: fix(dedupe): prefetch locations, not endpoints, in the batch dedupe loader #15508 fixes the crash there and re-baselines the perf query counts that change shifts. I measured that change independently while scoping this PR — +2 queries on each V3 step, and −104 on the second import — which matches fix(dedupe): prefetch locations, not endpoints, in the batch dedupe loader #15508's numbers, so re-deriving them here would only create a conflict. A TODO marks the one-line swap to the helper once it lands.
• unittests/test_dedupe_location_prefetch.py pins the helper (both flag states, the full location.url path, and the relation-prefix form) and asserts both build_candidate_scope_queryset modes use it. Queryset shape only — no fixtures, no timing. 6 tests pass, and test_deduplication_logic still passes at 85.
• The prefix argument exists for callers that page a model reaching the finding through a relation rather than paging Finding itself.

Extension points instead of a fork (second commit)

• Distributions that store extra hash fields or need extra scoping have maintained near-verbatim copies of this command, and a copy is what lets them drift — the vulnerability-id prefetch was fixed in a copy first and had to be fixed again elsewhere months later, and the location prefetch was fixed elsewhere and never reached a copy, leaving it prefetching the deprecated endpoint relation.
• Such a distribution can now subclass this command and override a hook: extra arguments, extra scope and its description, the hash generator, an extra recompute pass, the two batch-dedupe entry points, and product grading. Narrow on purpose — a stale override is a signature mismatch rather than silent drift.
• Behaviour is unchanged; every default does what the code did before. The scope build is unified into one queryset that filters down instead of a parser/no-parser if/else, which is what makes an extra scope filter composable.
• unittests/test_dedupe_command_hooks.py covers both halves: every hook has a working default, and a subclass's hooks are actually reached — including that --dedupe_only skips the extra hash pass, and that both batch paths and grading go through their hooks. This command had no tests at all before. 10 tests, verified locally.

[dryrunsecurity]

This pull request contains a critical finding where the sensitive file 'dojo/finding/deduplication.py' was modified by an author not on the allowed list. Although the issue is marked as failing, it is not currently blocking the merge.

🔴 Configured Sensitive Codepath Modified by Non-Allowed Author in dojo/finding/deduplication.py (drs_54782faa)

Vulnerability	Configured Sensitive Codepath Modified by Non-Allowed Author
Description	File 'dojo/finding/deduplication.py' matches configured sensitive codepath pattern 'dojo/finding/*.py' and was modified by '' (commit c81e08e) who is not in the allowed authors list.

We've notified @mtesauro.

---

Comment to provide feedback on these findings.

Report false positive: @dryrunsecurity fp [FINDING ID] [FEEDBACK]
Report low-impact: @dryrunsecurity nit [FINDING ID] [FEEDBACK]

Example: @dryrunsecurity fp drs_90eda195 This code is not user-facing

All finding details can be found in the DryRun Security Dashboard.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[valentijnscholten]

CI note: the remaining red is a dev-level failure, not this PR

After rebasing onto current dev, the only failing check is:

• test-user-interface / User Interface Tests (tests/notification_webhook_test.py …) → test_list_webhooks_page_loads fails in tearDown's assertNoConsoleErrors() with a SEVERE browser-console entry on the webhooks page ("Unit Tests Complete" is just the gate that rolls this up).

This isn't caused by this PR:

• This PR only changes dedupe/prefetch Python (dojo/finding/deduplication.py, dojo/location/queries.py, dojo/management/commands/dedupe.py) — it has no webhooks-page, template, or JS surface, so it can't produce a console error on /notifications/webhook.
• The same webhook UI batch is currently failing on unrelated PR docs(connectors): document the data-visibility warnings #15634, so it's not specific to this branch.
• The webhooks-page console error most likely comes from the recent dev refactor refactor(ui): remove the classic Bootstrap UI #15565 ("remove the classic Bootstrap UI"), which touched notification_webhook_test.py; the test also has prior flakiness history (751f757 "Stabilize flaky notification-webhook integration test").

Everything else is green (44 checks). This PR should go green once the dev-level webhooks console error is resolved. Happy to chase that root cause separately if useful, but it's out of scope for this dedupe refactor.