Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
29b749b
refactor: extract ModeHLL into its own files
Code0987 Sep 2, 2026
953a32b
refactor: extract ModeBitmap into its own files
Code0987 Sep 2, 2026
1e84ed7
refactor: extract ModeCounter into its own files
Code0987 Sep 2, 2026
6c9d4b3
refactor: extract ModeGeo into its own files
Code0987 Sep 2, 2026
ca97808
refactor: extract ModeZSet into its own files
Code0987 Sep 2, 2026
6bd1403
refactor: extract ModeSet into its own files
Code0987 Sep 2, 2026
7197443
refactor: extract ModeBloom into its own files
Code0987 Sep 2, 2026
ca868d7
refactor: extract ModeJSON into its own files
Code0987 Sep 2, 2026
4191f4d
refactor: extract ModeHash into its own files
Code0987 Sep 2, 2026
602b423
refactor: extract ModeList into its own files
Code0987 Sep 2, 2026
6861cd1
Merge feat/refactor-bitmap into feat/refactor-modes
Code0987 Sep 2, 2026
c5933b3
Merge feat/refactor-counter into feat/refactor-modes
Code0987 Sep 2, 2026
69231d8
Merge feat/refactor-list into feat/refactor-modes
Code0987 Sep 2, 2026
66e76cc
Merge feat/refactor-hash into feat/refactor-modes
Code0987 Sep 2, 2026
cce155f
Merge feat/refactor-json into feat/refactor-modes
Code0987 Sep 2, 2026
41356d1
Merge feat/refactor-bloom into feat/refactor-modes
Code0987 Sep 2, 2026
43d0f73
Merge feat/refactor-set into feat/refactor-modes
Code0987 Sep 2, 2026
5520ffb
Merge feat/refactor-zset into feat/refactor-modes
Code0987 Sep 2, 2026
c35a8ef
Merge feat/refactor-geo into feat/refactor-modes
Code0987 Sep 2, 2026
6aaeefc
fix: drop unused store imports and share owner-address error text
Code0987 Sep 2, 2026
76cda55
refactor: move mode logic into per-feature folders
Code0987 Sep 4, 2026
47e6531
fix: stop warmup before fanout close in join tests
Code0987 Sep 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ Apps: `client.DialTLS` with `pkg/tlsconfig.ClientFiles`. See [docs/OPERATIONS.md
| `pkg/set` | Exact set encode/decode for `ModeSet` |
| `pkg/zset` | Sorted-set encode/decode for `ModeZSet` |
| `pkg/geo` | Point index encode/haversine for `ModeGeo` |
| `pkg/listx` | Ordered list encode for `ModeList` |
| `pkg/list` | ModeList codec + owner/cluster helpers |
| `pkg/hashx` | Field-map encode for `ModeHash` |
| `pkg/counter` | int64 encode/add for `ModeCounter` |
| `pkg/jsonx` | Nested JSON path encode for `ModeJSON` |
Expand Down
57 changes: 57 additions & 0 deletions docs/design/2026-09-02-feature-folders.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Feature folders (same contract)

**Status:** approved (chat: this sounds good)
**Branch (later):** `feat/feature-folders`
**Date:** 2026-09-02

## Problem

`pkg/engine` is ~84 files. Most are per-mode (`list.go`, `list_apply.go`, `list_cluster.go`, tests, benches) sitting next to the cluster/Get/Put core.

Go still forbids `pkg/engine/list/` as `package engine` (one directory = one package). Unexported `ksRuntime` cannot be seen from another folder.

## Non-goals

- No API / proto / flag / version / fan-out / dirty-cache change
- Callers still use `engine.Engine.LPush` (not `list.LPush`)
- Not moving `Memory` methods out of `pkg/store` in this design (`lruItem` / dirty cache stay unexported there)
- Not exporting `ksRuntime`

## Contract

Unchanged public surface: `pkg/engine`, `pkg/store`, `pkg/client`. Codec import paths change (`pkg/listx` → `pkg/list`, same for the other `*x` packages).

## Approach

Each mode gets a top-level feature package. Engine keeps **thin public verbs + ApplyPut switch**. Mode logic lives in the feature package and talks to engine through a small **Host** interface defined **in the feature package** (so `pkg/list` does not import `pkg/engine` — no cycle). `*Engine` implements those interfaces.

```
pkg/list/ # package list — codec only (store imports this; no store import here)
codec.go
codec_test.go
engine_test.go # moved engine list tests (package list_test)
eng/ # package eng — owner write + GetOrLoad (imports store; engine imports eng)
host.go
local.go
cluster.go

pkg/engine/list.go # LPush/RPush/LPop/… only: validate, route, call list.*
pkg/engine/engine.go # ApplyPut: list.ApplyLPush(host, store, …)
pkg/store/list.go # unchanged (Memory receivers)
```

`pkg/engine` after a full roll-out: core (`engine.go`, `cluster.go`, `errors.go`, shared tests) plus **one file per mode**. Apply/cluster/mode tests leave.

Same map for the other modes (`pkg/hash`, `pkg/hll`, `pkg/bitmap`, `pkg/cms`, `pkg/topk`; `pkg/bloom` / `set` / `zset` / `geo` / `counter` already exist — add engine helpers there).

**First PR = List only** (this design). Other modes copy the pattern.

Rejected: `pkg/engine/list/` same package (illegal). Codec-only rename (does not shrink engine). Export `ksRuntime` so feature packages import engine (cycle + contract leak).

## Tests (already exist; keep them)

Move `pkg/engine/list_*test.go` into `pkg/list` as `package list_test` (still `engine.New()`, same asserts). Existing `pkg/store` list tests stay. `go test ./...`.

## Bench risk

Hot path is Get/Put, not List. Local Get-hit / StoreGetHit allocs/op must stay 15 / 2. No extra alloc on the Host interface if it is a concrete `struct { e *Engine; ks *ksRuntime }` passed by value — verify List micros if they exist.
38 changes: 38 additions & 0 deletions docs/design/2026-09-02-refactor-bitmap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Refactor ModeBitmap layout (no contract change)

**Status:** approved (chat: continue after HLL)
**Branch:** `feat/refactor-bitmap`
**Date:** 2026-09-02

## Problem

Bitmap store methods still live in `pkg/store/memory.go`. Engine Bitmap is one ~250-line file mixing public verbs, owner apply, and cluster hops.

## Non-goals

- No API / proto / flag / version / fan-out / hintID change
- No `bitmapx` algorithm change
- Not extracting other modes (HLL is a separate PR)

## Contract

Unchanged: `BitSet` / `BitGet` / `BitCount` / `BitPos` / `Delete(name)`, `FlagBitmap` + inbox `FlagBitmapSet`, snapshot fan-out, version under store mutex, present-bit, empty-until-delete.

## Approach

Same file layout as HLL / TopK / CMS:

| Before | After |
|--------|--------|
| `pkg/store/memory.go` B* | `pkg/store/bitmap.go` |
| `pkg/engine/bitmap.go` (all) | `bitmap.go` public verbs; `bitmap_apply.go` owner write; `bitmap_cluster.go` inbox/fan-out/GetOrLoad |

Rejected: rewrite bit packing; split `bitmapx` (150 lines is one file).

## Tests (already exist; keep them)

Existing `pkg/bitmapx`, `pkg/store` bitmap, `pkg/engine` bitmap unit + cluster + hint-after-down.

## Bench risk

No new Get/Peek logic. Local `BenchmarkEngineGetHit` / `BenchmarkStoreGetHit` allocs/op must stay 15 / 2.
40 changes: 40 additions & 0 deletions docs/design/2026-09-02-refactor-bloom.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Refactor ModeBloom layout (no contract change)

**Status:** approved (chat: continue)
**Branch:** `feat/refactor-bloom`
**Date:** 2026-09-02

## Problem

Bloom store methods still live in `pkg/store/memory.go`. Engine Bloom is one file mixing public verbs, owner write, and item fan-out.

## Non-goals

- No API / proto / version / fan-out change
- No `pkg/bloom` algorithm change
- Not extracting other modes (List / Hash / JSON stay on their own branches)

## Contract

Unchanged: BloomAdd ACK-only inbox (`FlagBloomAdd` item hint — hintID includes the item). BloomTest present-bit + local miss does not install from GetOrLoad. Merge-OR on handoff. Version stays on a live filter (OR, not LWW bump).

## Approach

Same file layout as List / Hash / JSON:

| Before | After |
|--------|--------|
| `pkg/store/memory.go` Bloom* | `pkg/store/bloom.go` |
| `pkg/engine/bloom.go` (all) | `bloom.go` public verbs; `bloom_apply.go` owner write + inbox apply; `bloom_cluster.go` owner forward |

Bloom-local error format strings live in a `const` block at the top of `bloom.go` (same text as before).

Rejected: switch BloomAdd to snapshot fan-out. Install-on-GetOrLoad. Split `pkg/bloom`.

## Tests (already exist; keep them)

Existing `pkg/bloom`, `pkg/store` bloom, `pkg/engine` bloom.

## Bench risk

No new Get/Peek logic. Local Get-hit / StoreGetHit allocs/op must stay 15 / 2.
38 changes: 38 additions & 0 deletions docs/design/2026-09-02-refactor-counter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# Refactor ModeCounter layout (no contract change)

**Status:** approved (chat: continue)
**Branch:** `feat/refactor-counter`
**Date:** 2026-09-02

## Problem

Counter store methods still live in `pkg/store/memory.go`. Engine Counter is one file mixing public verbs, owner write, and GetOrLoad.

## Non-goals

- No API / proto / Peer `CounterIncr` / version / fan-out change
- No `pkg/counter` algorithm change
- Not extracting other modes (HLL #40, Bitmap #41)

## Contract

Unchanged: `Incr` returns the new int64 (peer `CounterIncr` from non-owner). `CounterGet` present-bit. `FlagCounter` snapshot. Version under store mutex. Overflow → invalid argument, no wrap.

## Approach

Same file layout as HLL / Bitmap:

| Before | After |
|--------|--------|
| `pkg/store/memory.go` C* | `pkg/store/counter.go` |
| `pkg/engine/counter.go` (all) | `counter.go` public verbs; `counter_apply.go` owner write + install; `counter_cluster.go` GetOrLoad / snapshot |

Rejected: drop the Peer RPC (that is the return-*n* contract); split `pkg/counter` (39 lines).

## Tests (already exist; keep them)

Existing `pkg/counter`, `pkg/store` counter, `pkg/engine` counter unit + cluster.

## Bench risk

No new Get/Peek logic. Local Get-hit / StoreGetHit allocs/op must stay 15 / 2.
40 changes: 40 additions & 0 deletions docs/design/2026-09-02-refactor-geo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Refactor ModeGeo layout (no contract change)

**Status:** approved (chat: continue)
**Branch:** `feat/refactor-geo`
**Date:** 2026-09-02

## Problem

Geo store methods still live in `pkg/store/memory.go`. Engine Geo is one file mixing public verbs, owner write, and item fan-out.

## Non-goals

- No API / proto / version / fan-out / dirty-cache change
- No `pkg/geo` algorithm change
- Not extracting other modes (ZSet / Set / Bloom / List / Hash / JSON stay on their own branches)

## Contract

Unchanged: GeoAdd/GeoRem ACK-only inbox (`FlagGeoAdd` / `FlagGeoRem` item fan-out). GeoRem of missing is a no-op. Invalid lon/lat and NaN/negative radius rejected. Dirty `gCache` flushed on Get/Peek only.

## Approach

Same file layout as ZSet:

| Before | After |
|--------|--------|
| `pkg/store/memory.go` Geo* | `pkg/store/geo.go` (including `flushGeoValueLocked` / `insertGeoLocked`) |
| `pkg/engine/geo.go` (all) | `geo.go` public verbs; `geo_apply.go` owner write + inbox apply; `geo_cluster.go` owner forward / GetOrLoad |

Geo-local error format strings live in a `const` block at the top of `geo.go` (same text as before).

Rejected: rewrite dirty-cache / flush. Switch Geo to snapshot fan-out. Split `pkg/geo`.

## Tests (already exist; keep them)

Existing `pkg/geo`, `pkg/store` geo, `pkg/engine` geo unit + cluster.

## Bench risk

No new Get/Peek logic. Local Get-hit / StoreGetHit allocs/op must stay 15 / 2.
40 changes: 40 additions & 0 deletions docs/design/2026-09-02-refactor-hash.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Refactor ModeHash layout (no contract change)

**Status:** approved (chat: continue)
**Branch:** `feat/refactor-hash`
**Date:** 2026-09-02

## Problem

Hash store methods still live in `pkg/store/memory.go`. Engine Hash is one file mixing public verbs, owner write, item fan-out, and GetOrLoad.

## Non-goals

- No API / proto / version / fan-out / dirty-cache change
- No `pkg/hashx` algorithm change
- Not extracting other modes (List stays on `feat/refactor-list`)

## Contract

Unchanged: HSet/HDel ACK-only inbox (`FlagHashSet` / `FlagHashDel` item fan-out, not snapshot). HGet present-bit + field miss. Version via `nextVersion`. Dirty `hCache` flushed on Get/Peek only.

## Approach

Same file layout as List:

| Before | After |
|--------|--------|
| `pkg/store/memory.go` H* | `pkg/store/hash.go` (including `flushHashValueLocked` / `insertHashLocked`) |
| `pkg/engine/hash.go` (all) | `hash.go` public verbs; `hash_apply.go` owner write + inbox apply; `hash_cluster.go` GetOrLoad / owner forward |

Hash-local error format strings live in a `const` block at the top of `hash.go` (same text as before).

Rejected: rewrite dirty-cache / flush. Switch Hash to snapshot fan-out. Split `pkg/hashx`.

## Tests (already exist; keep them)

Existing `pkg/hashx`, `pkg/store` hash, `pkg/engine` hash unit + cluster.

## Bench risk

No new Get/Peek logic. Local Get-hit / StoreGetHit allocs/op must stay 15 / 2.
42 changes: 42 additions & 0 deletions docs/design/2026-09-02-refactor-hll.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Refactor ModeHLL layout (no contract change)

**Status:** approved (chat: start with HLL)
**Branch:** `feat/refactor-hll`
**Date:** 2026-09-02

## Problem

HLL store methods still live in `pkg/store/memory.go` (~2400 lines). TopK and CMS already have their own files. Engine HLL is one 190-line file mixing public API, owner apply, and cluster hops. `pkg/hllx.Merge` is exported but unused on the data path (install is LWW replace).

## Non-goals

- No API / proto / flag / version / fan-out / hintID change
- No `hllx` algorithm change
- No decoded cache / Get-Peek flush helper
- Not extracting other modes

## Contract

Unchanged: `HLLAdd` / `HLLCount` / `Delete(name)`, `FlagHLL` + inbox `FlagHLLAdd`, snapshot fan-out, version under store mutex, present-bit, empty-until-delete.

`pkg/hllx.Merge` becomes unexported (`merge`). Engine never called it. Package tests still cover per-register max.

## Approach

Standard Go file layout (one concern per file, same package):

| Before | After |
|--------|--------|
| `pkg/store/memory.go` HLL* | `pkg/store/hll.go` (same as `cms.go` / `topk.go`) |
| `pkg/engine/hll.go` (all) | `hll.go` public verbs; `hll_apply.go` owner write; `hll_cluster.go` inbox/fan-out/GetOrLoad |
| `pkg/hllx.Merge` | `merge` (unexported) |

Rejected: rewrite registers; split `hllx` (150 lines is one file); touch `ApplyPut` order.

## Tests (already exist; keep them)

Existing `pkg/hllx`, `pkg/store` HLL, `pkg/engine` HLL unit + cluster + hint-after-down. No new behavior tests.

## Bench risk

Hot path? No new Get/Peek logic. Extra file split must not add allocs. Local `BenchmarkEngineGetHit` / `BenchmarkStoreGetHit` allocs/op must stay 15 / 2.
40 changes: 40 additions & 0 deletions docs/design/2026-09-02-refactor-json.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Refactor ModeJSON layout (no contract change)

**Status:** approved (chat: continue)
**Branch:** `feat/refactor-json`
**Date:** 2026-09-02

## Problem

JSON store methods still live in `pkg/store/memory.go`. Engine JSON is one file mixing public verbs, owner write, inbox apply, and GetOrLoad.

## Non-goals

- No API / proto / version / fan-out / dirty-cache change
- No `pkg/jsonx` algorithm change
- Not extracting other modes (List / Hash stay on their own branches)

## Contract

Unchanged: JsonSet/JsonDel ACK-only inbox (`FlagJSONSet` / `FlagJSONDel`) then `FlagJSON` snapshot fan-out. JsonGet present-bit. Version under store mutex. Dirty `jCache` flushed on Get/Peek only.

## Approach

Same file layout as List / Hash:

| Before | After |
|--------|--------|
| `pkg/store/memory.go` J* | `pkg/store/json.go` (including `flushJSONValueLocked` / `insertJSONLocked`) |
| `pkg/engine/json.go` (all) | `json.go` public verbs; `json_apply.go` owner write + inbox apply; `json_cluster.go` owner forward / snapshot / GetOrLoad |

JSON-local error format strings live in a `const` block at the top of `json.go` (same text as before).

Rejected: rewrite dirty-cache / flush. Drop inbox and snapshot-only. Split `pkg/jsonx`.

## Tests (already exist; keep them)

Existing `pkg/jsonx`, `pkg/store` json, `pkg/engine` json unit + cluster.

## Bench risk

No new Get/Peek logic. Local Get-hit / StoreGetHit allocs/op must stay 15 / 2.
40 changes: 40 additions & 0 deletions docs/design/2026-09-02-refactor-list.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Refactor ModeList layout (no contract change)

**Status:** approved (chat: continue)
**Branch:** `feat/refactor-list`
**Date:** 2026-09-02

## Problem

List store methods still live in `pkg/store/memory.go`. Engine List is one file mixing public verbs, owner write, and GetOrLoad.

## Non-goals

- No API / proto / version / fan-out / dirty-cache change
- No `pkg/listx` algorithm change
- Not extracting other modes (HLL / Bitmap / Counter stay on their own branches)

## Contract

Unchanged: LPush/RPush ACK-only inbox; LPop/RPop Peer `ListPop`; FlagList snapshot; version under store mutex; dirty `lCache` flushed on Get/Peek only.

## Approach

Same file layout as HLL / Bitmap / Counter:

| Before | After |
|--------|--------|
| `pkg/store/memory.go` L* | `pkg/store/list.go` (including `flushListValueLocked` / `insertListLocked`) |
| `pkg/engine/list.go` (all) | `list.go` public verbs; `list_apply.go` owner write + inbox apply; `list_cluster.go` GetOrLoad / snapshot |

List-local error format strings live in a `const` block at the top of `list.go` (same text as before).

Rejected: rewrite dirty-cache / flush (JSON/Hash share that pattern; leave it). Split `pkg/listx` (166 lines).

## Tests (already exist; keep them)

Existing `pkg/listx`, `pkg/store` list, `pkg/engine` list unit + cluster.

## Bench risk

No new Get/Peek logic. Local Get-hit / StoreGetHit allocs/op must stay 15 / 2.
Loading
Loading