🚀 Bulk load share types - #1967
Draft
karlitschek wants to merge 3 commits into
Draft
karlitschek wants to merge 3 commits into
karlitschek wants to merge 3 commits into
Conversation
karlitschek
requested review from
enjeck and
juliusknorr
and
a lite review from Copilot
and removed request for
enjeck and
silverkszlo
August 6, 2026 13:14
AndyScherzinger
force-pushed
the
perf/noid/bulk-load-share-types
branch
from
September 15, 2026 08:59
d661748 to
dc334af
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
AndyScherzinger
force-pushed
the
perf/noid/bulk-load-share-types
branch
from
September 15, 2026 09:18
dc334af to
e0bb991
Compare
AndyScherzinger
marked this pull request as draft
September 15, 2026 22:07
AndyScherzinger
force-pushed
the
perf/noid/bulk-load-share-types
branch
from
September 15, 2026 22:24
e0bb991 to
e40a407
Compare
added 2 commits
September 16, 2026 00:32
The suite so far has been integration-only: everything under tests/api/ boots a real Nextcloud, so there was no way to exercise the app's pure logic without a server, and no test at all covered the functions that turn user input into file names. composer.json already declared a `test:unit` script pointing at tests/unit/phpunit.xml, but neither the config nor PHPUnit itself was present. This makes that script real: * tests/unit/phpunit.xml + bootstrap.php — no server, no database, no web server. `composer test:unit` works on a bare checkout. The bootstrap declares OC\Hooks\Emitter, which OCP\Files\IRootFolder extends but nextcloud/ocp does not ship, the same way tests/stubs/ocp.php already fills gaps for Psalm. * phpunit/phpunit and doctrine/dbal as dev dependencies. DBAL is needed because mocking OCP\IDBConnection reflects over IQueryBuilder, whose signatures reference Doctrine's types; the server provides it at runtime. * OCA\Notes\ is mapped in autoload-dev — the app relies on Nextcloud's own app autoloader, which is absent outside a server. * A separate phpunit-unit.yml workflow so these run on every pull request in seconds, independently of the server-backed test.yml. 120 tests covering NoteUtil (category-path normalisation including traversal attempts, title derivation, collision-safe file names, markdown stripping), NotesService (which files count as notes, the folder walk, titles from content), Note (title, category, excerpt, BOM and object-storage content handling), Util::retryIfLocked and ChunkCursor. Three tests are marked in their docblocks as characterization tests: they pin current behaviour that looks wrong so that a fix is a visible change rather than a silent one. No production code is touched by this commit. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
The characterization test claimed that every nested subcategory is dropped. That is only true while the parent's accumulated list is at least as long as the recursion's: the "+" union discards an entry whose index is already occupied, so a folder with more children than the parent has collected keeps the later ones. A folder 'Work' containing A, B and C therefore yields ['Work', 'Work/B', 'Work/C'] — 'Work/A' collides with 'Work' at index 0 and is lost, its siblings are not. Added as a second test case so the fix is verified against the real shape of the bug and not against a simpler mental model of it. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
AndyScherzinger
force-pushed
the
perf/noid/bulk-load-share-types
branch
from
September 15, 2026 22:32
e40a407 to
6acbb6c
Compare
AndyScherzinger
changed the title
AndyScherzinger
force-pushed
the
perf/noid/bulk-load-share-types
branch
2 times, most recently
from
September 15, 2026 22:57
2d8f0e5 to
bc67ab1
Compare
Note::getData() asks NoteUtil::getShareTypes() for the share types of every note it serialises, and that ran one IManager::getSharesBy() query per share type — eight per note. The web index endpoint loads the whole collection at once (chunkSize is 0 there), so rendering the note list issued eight queries times the number of notes: about 4000 for a user with 500 notes, purely to decide whether to draw the "shared" indicator dot. IManager::getSharesInFolder() answers for every file in a folder in one go, so the cost becomes one call per folder instead of eight per note. NoteUtil::loadShareTypes() preloads a whole tree that way and getShareTypes() reads from that cache, falling back to the old per-file lookup for the single-note endpoints where preloading a tree would cost more than it saves. This mirrors TagService::loadTags(), which already solves the same problem for favorites and is called from the same place. getSharesInFolder() only reports on a folder's direct children — passing $shallow = false is rejected by the server — so gatherNoteFiles() now also returns every folder it walked, and loadShareTypes() queries each one. The payload is deliberately unchanged. Shares are filtered against the same eight types the old code asked about and emitted in the same order, so `shareTypes` and `isShared` are identical to before; types the previous code never requested (TYPE_USERGROUP, the per-user half of a group share) stay unreported. A folder whose owner cannot be resolved disables the preload rather than caching an empty result, so a missing owner can never turn a shared note into an unshared-looking one, and the per-file fallback reports no shares instead of dereferencing that missing owner. getSharesInFolder() reports what one user has shared, so the cache only answers for notes owned by a user the preload queried as. A note owned by someone else — one shared into the notes folder, which mounts inside it — is looked up per file as before. Without that guard such a note reads as not shared for its recipient, which is a payload change and not a performance one. One getSharesInFolder() call costs about what serialising one note through the eight getSharesBy() calls it replaces costs, so the preload only pays off once a request serialises at least as many notes as the tree has folders. Which notes those are is known to Helper and not to getAll(): a chunked or pruned request returns a fraction of the collection, and the rest are emitted as bare ids that are never asked for their share types. getAll() therefore hands the walked folders back and Helper preloads for the notes it is about to serialise, so a small sync stays on the per-file path instead of walking the whole tree for a handful of notes. Measured on a dev instance with 59 notes in 14 folders and 12 shares, counting MySQL statements for GET /api/v1/notes, payload identical in every case for the owner and for both share recipients: whole collection 464 -> 194 chunkSize=50 403 -> 185 chunkSize=25 226 -> 160 chunkSize=10 121 -> 121 (preload declined) chunkSize=5 86 -> 86 (preload declined) sync of 3 changed 72 -> 72 (preload declined) Also drops the FIXME next to the hardcoded 15 and uses IShare::TYPE_SCIENCEMESH: the constant has existed since Nextcloud 26 and the app now requires 33. Covered by tests/unit/Service/NoteUtilShareTypesTest.php, which asserts that the query count follows the folder count and not the note count, that a preloaded result equals what the per-file path returns, and that the fallbacks still work. Assisted-by: Claude Code:claude-opus-5[1m] Signed-off-by: Andy Scherzinger <info@andy-scherzinger.de>
AndyScherzinger
force-pushed
the
perf/noid/bulk-load-share-types
branch
from
September 15, 2026 22:59
bc67ab1 to
62b532f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note::getData()asksNoteUtil::getShareTypes()for every note it serialises, and that ran onegetSharesBy()query per share type — eight per note — purely to decide whether to draw the "shared" indicator dot.NoteUtil::loadShareTypes()now preloads with onegetSharesInFolder()call per folder andgetShareTypes()reads from that cache, mirroringTagService::loadTags(). Because onlyHelperknows which notes a response will actually contain,getAll()hands the walked folders back andHelperpreloads for the notes it is about to serialise, declining when there are fewer of them than there are folders — so a chunked or incremental sync stays on the per-file path instead of walking the whole tree for a handful of notes. The cache also only answers for notes owned by a user the preload queried as, so a note shared into someone's notes folder still goes through the per-file path and the payload is unchanged for owner and recipient alike. The PR also adds a server-free PHPUnit suite undertests/unit/with its own CI workflow, which is what pins the query counts below.Measured on a dev instance with 59 notes in 14 folders and 12 shares, counting MySQL statements for
GET /api/v1/notes; the payload was identical in every row, for the owner and for both share recipients.chunkSize=50chunkSize=25chunkSize=10chunkSize=5🤖 AI (if applicable)