Keep failed upload batches recoverable across later sync cycles

perfloop-oss/parseable · RETRY AMPLIFICATION

https://perfloop.ai/t/oss/case_z6d5mrf8cp

Verdict

VERIFIED · settled 2026-11-11

What happened: The check fails before this change and passes with it.

Hypothesis

One persistent file upload error can keep healthy files out of later staging-upload attempts. Their abandoned claims are aged out only after a cycle reaches cleanup, but retrying the faulty file returns before cleanup. The native check reproduced a healthy file remaining unpublished beyond the nominal claim lifetime. Production fault frequency and affected batch sizes are unknown.

process_parquet_files claims the whole selected batch before spawning its tasks. collect_upload_results removes only the reported I/O-error path's claim and returns on the first error. Its join-error branch also returns without releasing the other claims. upload_files_from_staging propagates collection and snapshot-publication errors before cleanup_uploaded_staged_files, the only active-claim age sweep. retry_pending_staged_file_cleanup handles a different set: files already published and awaiting local deletion. If one file keeps failing and no other stream reaches cleanup, the unpublished siblings' claims never reach that sweep.

The stored native check, `bash tests/perfloop_upload_checks/run.sh perfloop_upload_retry`, failed the recovery predicate at revision 23586aa in checkProperty call_0JkCVtEr4dFXjqWynihcE3nQ (exit 101). It used two real, one-row Parquet files through the public server and LocalFS backend. A pipe-backed filesystem fault let the healthy remote upload validate before the sibling returned BrokenPipe. The sibling then returned PermissionDenied on 12 subsequent scheduled cycles. After 360 seconds the healthy remote modification time was unchanged, its staging file remained, and the stream had zero published descriptors. Clearing the fault allowed both files to publish and be deleted locally. The reduced input is healthy_slot=0, observe_seconds=360. The search stopped at this first role assignment; it did not complete the other assignment or exercise a task panic. Earlier constructor and metadata-filename fixture failures were checker errors, not counterexamples.

This is not a claim that every error blocks recovery forever. A successful empty batch can reach the sweep; an isolated join or snapshot error can therefore recover after claims age out. The independent proof target is claim ownership across successful collection, validation rejection, file I/O failure, task join failure and failed snapshot publication, including later cycles with a persistently failing sibling. Replaying this two-file witness is the smallest next check. A native fault-injection seam or another checker layout is provisional, not an additional frozen implementation requirement. Preserve prop_tcts5wky3d and prop_jw048gb1m8; partial publication is not prescribed.

The inspected upstream implementations do not own this recovery change: https://github.com/parseablehq/parseable/pull/1770 adds a guard for stream deletion, not upload batches; https://github.com/parseablehq/parseable/pull/1821 changes schema coordination; https://github.com/parseablehq/parseable/pull/1541 changes provider bounds; and https://github.com/parseablehq/parseable/pull/1852 changes periodic local-sync concurrency. Merged https://github.com/parseablehq/parseable/pull/1844 is the accepted revision and leaves these owners unchanged. The history available locally was shallow, the public commits API read was refused, and the contribution guide's https://logg.ing/community venue was inaccessible, leaving named history/archive gaps. This is distinct from the already assigned stream scheduling, field-stat scans, regex compilation, validation HEAD removal and manifest-delta work.

Change to test: Establish and repair batch claim recovery so still-staged, unpublished siblings remain retryable after collection errors without depending on the faulty file or an unrelated stream completing cleanup. Cover the join-error and snapshot-publication exits while preserving validation-failure retention without a descriptor, and publication before field statistics or local deletion.

Where it lives

perfloop-oss/parseable · src/sync.rs

Evidence

The check fails before this change and passes with it: `Staging-upload attempts with terminating file tasks release the claims of still-staged, unpublished files after file I/O, task join, or snapshot-publication errors, making them eligible on the next sync cycle; validation rejection keeps its staged file and adds no descriptor, and successful batches publish before field statistics or local deletion. Within a stream lifetime, retries of a validated staged file increment its storage-byte and billing metrics once, retaining the totals through snapshot publication.`

Checks: 2 of 2 passed. Verification: no defect found.

Timeline