Operations & Production Deployment

Edit this page on GitHub

Production deployment, tuning, and troubleshooting for sirix-core and the sirix-rest-api server. This document focuses on the operational surface — JVM flags, cache budgets, OS limits, observability, backups — rather than on API usage. For API documentation, see REST API and JSONiq API; for storage-format internals, see Architecture.

Status. Sirix is currently at 1.0.0-alpha5. The wire format is on BinaryEncodingVersion.V0; bumps are stamped into the page header and rejected on read with a clear “version not known” error. There is no migration tool yet — when V1 is introduced, a one-shot upgrader will ship alongside.

1. Supported environment

2. Mandatory JVM flags

Sirix uses Foreign Function & Memory (FFM), the Vector API, preview features, and several JDK-internal exports that must be opened. These flags are not optional — omission produces IllegalAccessError at startup.

--enable-preview
--enable-native-access=ALL-UNNAMED
--add-modules=jdk.incubator.vector
--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED
--add-opens=jdk.compiler/com.sun.tools.javac=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED

The same set is applied in the project’s Gradle build and in the REST API CI workflow.

Notes:

• jdk.unsupported/sun.misc is required only because of a transitive net.openhft/zero-allocation-hashing dependency. Sirix code itself no longer uses sun.misc.Unsafe directly.
• The jdk.compiler opens are needed when using the Brackit query stack with ahead-of-time AST compilation; they are harmless when not exercised.

3. Heap sizing and GC choice

Sirix is built around off-heap MemorySegment-allocated page memory. The on-heap budget covers (a) the JVM and Brackit’s query state, (b) per-thread buffers and caches, (c) intermediate query result objects, and (d) on-heap references to off-heap pages held by transactions. A typical production sizing:

Defaults inside the Gradle :test JVM are -Xms5g -Xmx12g — not because tests need 12 GB, but because they pre-touch the heap (AlwaysPreTouch) to make GC behavior comparable across runs.

GC

The reference GC is ZGC with always-pretouch and large-pages, configured in the project’s Gradle test JVM as:

-XX:+UseZGC
-XX:+AlwaysPreTouch
-XX:+UseLargePages
-XX:+UseStringDeduplication
-XX:+HeapDumpOnOutOfMemoryError
-XX:ReservedCodeCacheSize=1000m
-XX:EliminateAllocationArraySizeLimit=1024

Z is preferred because:

• Sirix’s hot path is largely off-heap, so old-gen pressure is dominated by long- lived caches, not transient objects. Z’s region-based collector handles this well.
• Sirix expects sub-second pause budgets; G1’s 50–200 ms pauses on a 12 GB heap with deep object graphs are too disruptive.

Generational ZGC (-XX:+ZGenerational) is supported but currently commented out in the build because some workloads regress versus single-gen Z; benchmark before flipping it on.

Direct memory

-XX:MaxDirectMemorySize should be at least 1 GB. Sirix uses direct buffers for FFI (LZ4), file-channel reads, and certain serialization paths.

Other flags worth knowing

• -XX:-UseJVMCICompiler — workaround for a Graal JIT speculation bug (oracle/graal#13387) that caused 27% wall-clock regressions on certain query shapes.
• -Xlog:gc*=debug:file=gc.log for production GC tracing.
• -Ddisable.single.threaded.check=true — disables a single-threaded-access check in some legacy code paths; needed for the parallel path.

4. Cache budgets

Sirix’s BufferManager is a multi-tier cache. The defaults are computed as fractions of the memory budget (the off-heap allocator’s max segment size), and can be overridden via system properties.

Set explicit byte counts when you know your working set:

-Dsirix.cache.recordPage=8589934592   # 8 GB
-Dsirix.cache.recordPageFragment=3221225472   # 3 GB
-Dsirix.cache.page=536870912   # 512 MB

Initial sizing log line (look for it in startup output):

INFO  io.sirix.access.Databases - Initializing global BufferManager with memory budget: 16 GB
INFO  io.sirix.access.Databases -   - RecordPageCache: 8589934592 bytes (8192 MB) (default: 25% of budget)
INFO  io.sirix.access.Databases -   - RecordPageFragmentCache: 3221225472 bytes (3072 MB) (default: 12.5% of budget)
INFO  io.sirix.access.Databases -   - PageCache: 1073741824 bytes (1024 MB) (default)

5. Native libraries

libsirix_lz77.so

A bundled native LZ77 decoder for Linux x86_64. Embedded as a JAR resource at /native/linux-x86_64/libsirix_lz77.so and extracted to a temp file at the first decode call.

• If present: ~2× decompression throughput versus the pure-Java fallback.
• If absent or platform mismatch: falls back to SirixLZ77Codec pure-Java decoder, which is correct but slower.
• Override: -Dsirix.lz77Codec.native.disable=true forces pure-Java for A/B testing.

To rebuild from source: ./gradlew :sirix-core:buildNativeLz77 (requires gcc on PATH). The build step is no-op when gcc is missing — the JAR ships only the prebuilt .so.

LZ4 (FFM)

The default FFILz4Compressor invokes the system liblz4.so.1 via FFM. On modern Linux distros this is in apt install liblz4-1 / dnf install lz4 and present by default. macOS: brew install lz4. Windows: build / install liblz4.dll.

If liblz4 is unavailable the constructor throws at first compress/decompress. Page writes succeed only when the compressor is functional; there is no runtime fallback for LZ4 (unlike LZ77).

6. OS-level requirements

7. Observability

The sirix-rest-api server exposes Prometheus-format metrics at GET /metrics via Micrometer. Wired in MetricsHandler.kt.

Currently exported:

Sirix-internal metrics (active transaction count, page cache hit/miss/evict, commit queue depth, GC pause attribution) are not yet exported through the Prometheus registry. A ResourceSession.activeTrxCount() accessor exists for in-process diagnostics; bridging it through Micrometer is on the production- readiness backlog. For now the recommended approach is JFR (-XX:StartFlightRecording) plus the Sirix logback appender at INFO level.

For the embedded-library use case (no REST), Sirix logs cache initialization, storage allocator decisions, and ClockSweeper progress at INFO. Logger names:

• io.sirix.access.Databases — startup, BufferManager init.
• io.sirix.cache.BufferManagerImpl / io.sirix.cache.ShardedPageCache — cache lifecycle.
• io.sirix.cache.ClockSweeper — eviction sweeps (PostgreSQL bgwriter pattern).
• io.sirix.cache.LinuxMemorySegmentAllocator — off-heap allocator events.
• io.sirix.access.Databases$Databases — close/cleanup warnings.

8. Backup and restore

Sirix has no streaming or incremental backup tool. Resource directories are self-contained; the operational pattern is:

1. Stop the writer for the resource (close any active NodeTrx). Read-only transactions can continue.
2. cp -a or rsync -a --inplace the resource directory to the backup target. Sirix’s append-only page format means this is consistent without additional coordination.

3. Verify the backup by opening it as a read-only resource:

   try (var db = Databases.openJsonDatabase(backupPath);
        var session = db.beginResourceSession("...");
        var rtx = session.beginNodeReadOnlyTrx()) { /* ... */ }
   

Restoring is a directory move/copy back; no replay is required.

Caveats:

• Hot backup (writer running) is not safe — the in-flight Transaction Intent Log can leave the on-disk image inconsistent. Wait for wtx.commit() / wtx.close() first.
• Snapshot-based backups via filesystem snapshots (LVM, ZFS) are safe iff the snapshot is atomic across all files of the resource. ext4 + LVM is fine; per- file snapshots are not.

A point-in-time recovery is possible via Sirix’s revision system: open the resource at the desired revision number or timestamp via session.beginNodeReadOnlyTrx(revision) / session.beginNodeReadOnlyTrx(Instant). No external tool needed.

9. Supported workloads

10. Known limitations and operational caveats

1. Single-writer-per-resource. A second beginNodeTrx() on a resource with an active writer throws after a 5-second tryAcquire timeout. Plan for serialised writes; do batch ingestion in one writer.

2. Brackit dependency at 1.0-SNAPSHOT. Sirix currently depends on io.sirix:brackit:1.0-SNAPSHOT. A tagged release is pending; until then, reproducible builds require pinning a specific Brackit commit hash via local Maven install.

3. No on-disk format migration tool. BinaryEncodingVersion.V0 is the only shipping version. When V1 lands, an upgrader will ship; today, opening a resource written by an incompatible Sirix version raises IllegalStateException: <n> not known.

4. HOT index does not isolate historical revisions on reads. A read-only transaction at revision N opening a HOT index sub-tree may observe the latest committed state of the index rather than the state at revision N. Not blocking the typical analytical use case where the index reflects the most recent commit.

5. Auto-commit features are in flight. Production should currently use synchronous commits via wtx.commit() and avoid the AfterCommitState.KEEP_OPEN_ASYNC path until a single design lands on main.

6. Large-scale (Chicago-scale) ingestion tests are not in CI. The reference 3.6 GB Chicago dataset is not in CI; large-scale ingestion regressions are caught manually by removing the @Disabled annotation and running locally on a machine with ≥ 16 GB RAM.

7. No automated crash-recovery test. kill -9 mid-commit, partial fsync, torn writes — these scenarios are believed to be safe given the commit-file + UberPage swap protocol, but a fault-injection harness has not been built.

11. Quick-start: launch the REST API server

java \
  --enable-preview \
  --enable-native-access=ALL-UNNAMED \
  --add-modules=jdk.incubator.vector \
  --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED \
  --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \
  --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED \
  --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED \
  --add-opens=jdk.compiler/com.sun.tools.javac=ALL-UNNAMED \
  --add-opens=java.base/java.lang=ALL-UNNAMED \
  --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
  --add-opens=java.base/java.io=ALL-UNNAMED \
  --add-opens=java.base/java.util=ALL-UNNAMED \
  -Xms4g -Xmx8g \
  -XX:+UseZGC -XX:+AlwaysPreTouch -XX:MaxDirectMemorySize=1g \
  -Dsirix.cache.recordPage=4294967296 \
  -Dsirix.cache.recordPageFragment=1610612736 \
  -jar bundles/sirix-rest-api/build/libs/sirix-rest-api-1.0.0-alpha5-fat.jar \
  -conf bundles/sirix-rest-api/src/main/resources/sirix-conf.json

/metrics will be available on the configured port immediately; database directories are created lazily under the path configured in sirix-conf.json.

12. Where to look when something is wrong

13. Further reading

• Architecture — page format, versioning, transaction model.
• Features — the user-facing feature matrix.
• REST API — endpoint reference.
• JSONiq API and Function Reference — query layer.
• Cost-based optimizer design (sirix repo).
• Native image build (sirix repo).
• Roadmap (sirix repo).