Batch no-run container reads during decoding
perfloop/roaring · Chatty I/O
https://perfloop.ai/t/oss/case_bc0tc0yh01
Verdict
VERIFIED · settled 2026-11-14
What happened: The paired measurements met the required improvement.
Hypothesis
A counting-reader probe serialized 16,384 singleton high-key containers and decoded them with Bitmap.ReadFrom through an ordinary io.Reader. It completed the round trip and recorded 16,388 direct Read calls for 163,848 bytes. This check establishes call multiplicity only; it does not measure syscall or storage time.
Bitmap.ReadFrom wraps a reader that is not internal.ByteInput in ByteInputAdapter. ByteInputAdapter.Next allocates the requested slice and invokes io.ReadAtLeast. readFrom calls Next once for each container after the descriptive and offset headers. Thus a no-run bitmap with N low-cardinality containers generates N small reader operations per decode in addition to headers; the parser admits up to 65,536 containers and a singleton array payload is two bytes. The cost is only plausible when the supplied reader has no lower coalescing layer and Read reaches a file, socket, or other storage boundary; ByteInput callers and already-buffered readers do not establish the same cost.
For the no-run format, keycard determines every array or bitmap payload length. Read exact, bounded groups of whole container payloads after the offset table and install sub-slices in each container, so the parser does not overread a caller's stream. Reproduce the counting-reader test with unbuffered file and short-read readers, then benchmark sparse container counts; the expected confirmation is reader or syscall calls falling from N plus a constant per decoded bitmap to one per bounded group while output and post-bitmap reader position remain unchanged.
Change to test: Special-case no-run data in roaringArray.readFrom: calculate payload lengths from keycard and issue exact bounded Next calls for contiguous whole-container groups. Slice each returned group into its containers and keep every group boundary at a serialized container boundary to preserve stream position and limit retained backing storage.
Where it lives
perfloop/roaring · roaring.go
Evidence
Sparse file decode, 1024 containers · 15 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
2380732 |
68213 |
−96.9% (−2307313) |
−2729158 to −2269449 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
read-calls/op |
1028 |
5 |
−99.5% (−1023) |
−1023 to −1023 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
B/op |
56411 |
56412 |
+1 |
−2 to +4 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
2054 |
1031 |
−49.8% (−1023) |
−1023 to −1023 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Sparse short decode, 1024 containers · 15 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
2451285 |
161343 |
−93.3% (−2287721) |
−2578422 to −2202116 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
read-calls/op |
1058 |
42 |
−96% (−1016) |
−1016 to −1016 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
B/op |
56411 |
56410 |
−1 |
−3 to +8 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
2054 |
1031 |
−49.8% (−1023) |
−1023 to −1023 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Sparse file decode, 4096 containers · 15 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
9490869 |
196067 |
−98% (−9298623) |
−9711485 to −8969137 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
read-calls/op |
4100 |
5 |
−99.9% (−4095) |
−4095 to −4095 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
B/op |
217214 |
217215 |
−1 |
−12 to +3 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
8198 |
4103 |
−50% (−4095) |
−4095 to −4095 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Sparse short decode, 4096 containers · 16 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
9757099 |
564351 |
−94.2% (−9191077) |
−10648485 to −8953624 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
read-calls/op |
4226 |
162 |
−96.2% (−4064) |
−4064 to −4064 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
B/op |
217214 |
217220 |
−2.5 |
−11 to +19 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
8198 |
4103 |
−50% (−4095) |
−4095 to −4095 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Sparse file decode, 16384 containers · 16 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
38033530 |
722036 |
−98.1% (−37322140) |
−58586528 to −35275675 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
read-calls/op |
16388 |
5 |
−100% (−16383) |
−16383 to −16383 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
B/op |
868618 |
868645 |
+24 |
−56 to +42 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
32775 |
16392 |
−50% (−16383) |
−16383 to −16383 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Sparse short decode, 16384 containers · 16 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
38761396 |
2066487 |
−94.6% (−36658963) |
−38590719 to −35326061 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
read-calls/op |
16898 |
642 |
−96.2% (−16256) |
−16256 to −16256 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
B/op |
868624 |
868582 |
−40 |
−50 to +71 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
32775 |
16392 |
−50% (−16383) |
−16384 to −16383 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Run-format decode, 6500000 values · 31 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
10136 |
10119 |
−23 |
−378 to +484 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
14006 |
14006 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
207 |
207 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Weather dataset UnmarshalBinary · 16 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
2490793 |
1996033 |
−20.2% (−501906) |
−691606 to −282120 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
B/op |
9150283 |
9130448 |
−0.2% (−19836) |
−19856 to −19812 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
6916 |
4360 |
−37% (−2556) |
−2556 to −2556 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Weather dataset bufio · 17 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
6644608 |
3260322 |
−51.1% (−3395067) |
−3843454 to −2925255 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
B/op |
9140666 |
9120855 |
−0.2% (−19839) |
−19921 to −19787 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
6716 |
4160 |
−38.1% (−2556) |
−2556 to −2556 |
loss ≤ 0.05 × |baseline median| |
PASSED |
US census dataset UnmarshalBinary · 17 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
189852 |
147337 |
−22.2% (−42088) |
−60243 to −36907 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
B/op |
140109 |
139525 |
−0.4% (−584) |
−585 to −582 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
5642 |
3621 |
−35.8% (−2021) |
−2021 to −2021 |
loss ≤ 0.05 × |baseline median| |
PASSED |
US census dataset bufio · 17 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
231691 |
185485 |
−19.9% (−46211) |
−70849 to −26156 |
gain > 0; gain > 0.05 × |baseline median|; arm medians separated |
PASSED |
B/op |
130507 |
129938 |
−0.4% (−568) |
−571 to −566 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
5442 |
3421 |
−37.1% (−2021) |
−2021 to −2021 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Small empty UnmarshalBinary · 32 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
89.84 |
90.65 |
+1.05 |
−3.74 to +4.4 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
48 |
48 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
1 |
1 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Small array UnmarshalBinary · 17 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
210.2 |
206.1 |
−4.7 |
−17.2 to +9.6 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
104 |
104 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
8 |
8 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Small bitmap UnmarshalBinary · 28 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
3140 |
3128 |
−19 |
−117 to +74 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
8301 |
8301 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
8 |
8 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Small mixed UnmarshalBinary · 43 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
3552 |
3550 |
+1 |
−84 to +145 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
8417 |
8417 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
12 |
12 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Small empty file · 18 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
7205 |
7232 |
+44 |
−186 to +117 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
0 |
0 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
0 |
0 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Small empty bufio · 18 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
4428 |
4463 |
+35.5 |
−92 to +109 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
0 |
0 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
0 |
0 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Small array file · 18 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
17150 |
17162 |
+46.5 |
−626 to +642 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
56 |
56 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
7 |
7 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
In-memory decode, 6500 values · 18 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
3155 |
3111 |
−19.5 |
−317 to +127 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
8301 |
8301 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
8 |
8 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
In-memory decode, 65000 values · 33 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
3167 |
3118 |
−48 |
−215 to +148 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
8301 |
8301 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
8 |
8 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
In-memory decode, 650000 values · 51 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
4109 |
4126 |
+10 |
−170 to +179 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
8817 |
8817 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
27 |
27 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Two bitmap containers, three-byte reads · 29 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
30974 |
29749 |
−3.6% (−1128) |
−3634 to −428 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
16555 |
16555 |
0 |
−1 to +1 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
10 |
9 |
−10% (−1) |
−1 to −1 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Array and bitmap containers, three-byte reads · 18 sample pairs
| metric | baseline | candidate | paired median change | confidence range | required | result |
|---|---|---|---|---|---|---|
ns/op |
15659 |
15032 |
−659 |
−2867 to +98 |
loss ≤ 0.05 × |baseline median| |
PASSED |
B/op |
8353 |
8353 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
allocs/op |
10 |
10 |
0 |
0 to 0 |
loss ≤ 0.05 × |baseline median| |
PASSED |
Checks: 6 of 6 passed. Verification: no defect found.
Timeline
2026-08-15· Case opened2026-09-15· PR opened