Skip to content

WAL

HestiaStore supports opt-in write-ahead logging per index. WAL is the main feature to enable when you need local crash recovery stronger than flush/close boundaries alone.

For staged rollout, use WAL Canary Runbook. For longer-term replication design work, see WAL Replication and Fencing Design.

Enable WAL

IndexConfiguration<String, String> conf = IndexConfiguration
    .<String, String>builder()
    .identity(identity -> identity
        .name("orders")
        .keyClass(String.class)
        .valueClass(String.class))
    .wal(wal -> wal
        .durability(WalDurabilityMode.GROUP_SYNC))
    .build();

WAL is disabled by default through IndexWalConfiguration.EMPTY. Use wal(wal -> wal.disabled()) to explicitly disable it in an override.

Choose a durability mode

  • ASYNC: lowest write latency, weakest durability guarantee
  • GROUP_SYNC: batched fsync behavior with a balanced latency/durability trade-off
  • SYNC: fsync on each write, strongest durability and highest write overhead

Choose the mode from your durability target first, then tune performance around that choice.

Corruption and recovery policy

  • TRUNCATE_INVALID_TAIL: startup truncates a broken tail and continues
  • FAIL_FAST: startup stops when corruption is detected
  • Recovery validates global LSN monotonicity across WAL segments
  • Invalid wal/checkpoint.meta content aborts recovery

WAL directory layout

Inside the index directory:

  • wal/format.meta
  • wal/checkpoint.meta
  • wal/*.wal

WAL segment files are named as <20-digit-base-lsn>.wal.

Runtime ownership

Internally, WAL is now split into smaller collaborators while keeping the public WalRuntime contract stable:

  • WalMetadataCatalog handles format.meta, checkpoint.meta, temp-file promotion, and segment discovery.
  • WalRecoveryManager handles replay, invalid-tail handling, and checkpoint clamp behavior on open.
  • WalSegmentCatalog handles active-segment rotation, retained-byte tracking, and checkpoint cleanup.
  • WalWriter and WalSyncPolicy handle append-path durability behavior for ASYNC, SYNC, and GROUP_SYNC.

Architecture details live in WAL Runtime.

Tooling

WalTool supports:

  • verify for integrity checks
  • dump for record-level diagnostics

Use the standalone wal-tools-<version>.zip distribution on the operator host or diagnostic workstation. The distribution contains:

wal-tools-<version>.zip
wal-tools-<version>.zip.sha256

Example installation:

sha256sum -c wal-tools-<version>.zip.sha256
unzip wal-tools-<version>.zip -d /opt/hestiastore/wal-tools

Concrete example:

VERSION=1.2.3
RELEASE_DIR=/srv/releases/hestiastore
INSTALL_DIR=/opt/hestiastore/wal-tools

cd "$RELEASE_DIR"
sha256sum -c "wal-tools-${VERSION}.zip.sha256"
unzip -o "wal-tools-${VERSION}.zip" -d "$INSTALL_DIR"

Example usage:

/opt/hestiastore/wal-tools/bin/wal_verify /path/to/index/wal
/opt/hestiastore/wal-tools/bin/wal_dump /path/to/index/wal

Concrete examples:

/opt/hestiastore/wal-tools/bin/wal_verify /srv/hestia/indexes/orders/wal
/opt/hestiastore/wal-tools/bin/wal_verify /srv/hestia/indexes/orders/wal --json
/opt/hestiastore/wal-tools/bin/wal_dump /srv/hestia/indexes/orders/wal --json

JSON output is available through --json for both commands.

Exit codes:

  • 0: success
  • 1: usage or runtime failure
  • 2: verify found WAL issues

Operating signals

Monitor these first:

  • wal().syncFailureCount()
  • wal().corruptionCount()
  • wal().truncationCount()
  • wal().retainedBytes()
  • wal().checkpointLagLsn()
  • wal().pendingSyncBytes()
  • wal().syncAverageNanos()

When retained WAL exceeds maxBytesBeforeForcedCheckpoint, the write path applies forced checkpoint behavior and backpressure until retained WAL drops.

Structured log events include:

  • event=wal_recovery_start
  • event=wal_recovery_invalid_tail
  • event=wal_recovery_tail_repair
  • event=wal_recovery_drop_newer_segments
  • event=wal_recovery_checkpoint_clamp
  • event=wal_recovery_complete
  • event=wal_checkpoint_cleanup
  • event=wal_retention_pressure_start
  • event=wal_retention_pressure_cleared
  • event=wal_sync_failure
  • event=wal_sync_failure_transition

Metrics exposed by runtimeMonitoring().snapshot().wal()

  • throughput: wal().appendCount(), wal().appendBytes()
  • durability: wal().syncCount(), wal().syncFailureCount(), wal().durableLsn()
  • corruption and recovery: wal().corruptionCount(), wal().truncationCount()
  • retention and checkpointing: wal().retainedBytes(), wal().segmentCount(), wal().checkpointLsn(), wal().checkpointLagLsn()
  • pending work: wal().pendingSyncBytes(), wal().appliedLsn()
  • sync latency and batch sizing: wal().syncTotalNanos(), wal().syncMaxNanos(), wal().syncAverageNanos(), wal().syncBatchBytesTotal(), wal().syncBatchBytesMax(), wal().syncAverageBatchBytes()