Architecture
SirixDB versions JSON and XML trees. Commits create revisions whose unchanged pages are shared with earlier revisions through copy-on-write storage. Application-supplied valid-time paths add a second time axis to transaction-time history; see Bitemporal Queries.
The implementation is in the SirixDB source. Storage and query costs depend on the data, versioning strategy, cache state and workload; this page describes the design rather than a performance guarantee.
Node-Based Document Model
SirixDB decomposes each document into a tree of fine-grained nodes. Each node has a stable 64-bit nodeKey that never changes across revisions. Nodes are linked via parent, child, and sibling pointers, enabling direct structural navigation; resolving a pointer can require storage reads.
Field names are stored once in an in-memory dictionary and referenced by 32-bit keys, saving space when the same field appears thousands of times.
Why Node Storage Matters
Nodes can be selected by key, traversed with cursors and compared across revisions. Updates version affected record-page fragments and their ancestor pages rather than requiring the application to maintain a full copy of every document. See transactional cursor examples.
Current Sirix main uses OBJECT and ARRAY roots, scalar value kinds for array items, and fused OBJECT_NAMED_* kinds for object fields. A named array/object field directly owns its children; separate OBJECT_KEY nodes are a legacy encoding. See the NodeKind definitions. Navigation follows stored pointers between nodes.
Why Bitemporal Storage?
SirixDB retains committed resource revisions and exposes them through temporal queries.
Query any point in history
A customer claims they were charged the wrong price. Your current database shows today’s price. What was the price at the moment of their order?
let $catalog := jn:open('shop', 'products', xs:dateTime('2024-01-15T15:23:47Z'))
return $catalog.products[?$$.sku eq "SKU-12345"].price
The query reads the stored revision selected for that timestamp. Applications remain responsible for capturing source data and valid-time values correctly.
Structured diffs between any two points in time
A production outage started at 2:00 AM. What configuration changes were made since midnight?
let $midnight := jn:open('configs', 'production', xs:dateTime('2024-01-15T00:00:00Z'))
let $incident := jn:open('configs', 'production', xs:dateTime('2024-01-15T02:00:00Z'))
return jn:diff('configs', 'production', sdb:revision($midnight), sdb:revision($incident))
Returns a structured JSON diff showing exactly what was inserted, deleted, updated, and replaced — with node-level precision.
Revision-Scoped Indexes
Secondary indexes are versioned with their resource. A read transaction selects a revision before using its data and indexes. Timestamp selection resolves a stored revision; it does not reconstruct missing source data or infer valid time.
Copy-on-Write with Path Copying
When a transaction modifies data, SirixDB doesn’t rewrite existing pages. Instead, it copies only the modified page and its ancestor path to the root. All unchanged pages are shared between the old and new revision via pointers. This is the same principle used in persistent data structures and ZFS.
A commit writes fragments for affected record pages plus new index pages on their ancestor paths to the revision root. Fragment contents and reconstruction costs depend on the versioning strategy.
Physically, each resource is stored in two append-only logical devices (files). LD₁ stores page content (IndirectPages, RecordPages, NamePages) followed by a RevisionRootPage at the end of each revision’s data. LD₂ stores revision metadata, including timestamps and offsets pointing to the corresponding RRP in LD₁. The layout below is schematic; the FileChannelWriter defines the current format and durability barriers.
The commit protocol writes data before publishing the revision through the UberPage. Page references carry content hashes. SirixDB uses shadow paging rather than a separate redo WAL; durability depends on the commit protocol and storage barriers. See the async commit design for publication and failure semantics.
Page Structure
Each resource is organized as a trie of pages. The RevisionRootPage is the entry point for a single revision, branching into subtrees for data, indexes, and metadata.
UberPage — The root entry point. Published after the revision data is written. Contains a reference to the IndirectPage tree that addresses all revisions.
IndirectPages — Fan-out nodes (512 references each) that form the trie structure. Borrowed from ZFS, they store checksums in parent references for data integrity. Unused slots are tracked with a bitset for compact storage.
RevisionRootPage — Entry point for a single revision. Stores the author, timestamp, and optional commit message. Branches to the data trie, path summary, name dictionary, and index pages.
RecordPages — Leaf pages storing up to 1024 nodes each. These are the pages that get versioned by the sliding snapshot algorithm.
Sliding Snapshot Versioning
SirixDB doesn’t just copy entire pages on every change. It versions RecordPages at a sub-page level, storing only changed records. The sliding snapshot algorithm, developed by Marc Kramis, avoids the trade-off between read performance and write amplification that plagues traditional approaches.
| Strategy | Reads to reconstruct | Write cost per revision | Write spikes? |
|---|---|---|---|
| Full | 1 page | Entire page (all records) | No |
| Incremental | At most N fragments since last full dump | Changed records; a full page every N revisions | Yes (periodic full dump) |
| Differential | 2 pages (full + diff) | All changes since last full dump | Yes (growing deltas + full dump) |
| Sliding Snapshot | At most N fragments | Changed + expired records | No |
The sliding snapshot uses a window of size N, configurable per resource; see the revisionsToRestore configuration. Changed records are always written. Records older than N revisions that haven’t been written are carried forward. This guarantees that at most N page fragments need to be read to reconstruct any page — regardless of total revision count.
For details, see Marc Kramis’s thesis: Evolutionary Tree-Structured Storage: Concepts, Interfaces, and Applications.
Transaction Model
SirixDB uses a single-writer, multiple-reader (SWMR) concurrency model per resource. At most one write transaction can be active on a resource at any time, while multiple read-only transactions can remain open on committed revisions.
Read transactions are bound to a specific revision and see an immutable snapshot — they never observe uncommitted changes and never block writers. Close transactions when finished so their resources can be released.
Uncommitted changes are held in an in-memory transaction intent log (TIL). A commit writes changed pages and publishes a new revision. Rollback discards the uncommitted logical state; spill or pre-flush work may already have written unreachable bytes.
Because each resource is an independent log-structured store, different resources can have independent writers, while sharing process memory and I/O resources.
Auto-commit and the asynchronous pre-flush
Long-running write transactions can bound memory and commit latency in two ways, and the distinction matters:
- Synchronous auto-commit (after every N modifications and/or every T seconds) runs the full commit protocol: it creates a real revision with its own commit record, exactly like an explicit
commit(), and the transaction stays open. - Asynchronous pre-flush (
AfterCommitState.KEEP_OPEN_ASYNC_FLUSH) is not a commit. The transaction intent log takes an O(1) snapshot (freezing its current entries), and a background thread writes the snapshot’s leaf pages to the data file through shadow references while the writer keeps working on a fresh log generation. No revision, no commit record, nothing becomes visible — if the process crashes before the next real commit, the pre-flushed bytes are unreachable dead space in the append-only file, exactly like a rolled-back transaction.
At the next real commit, leaf pages that were not modified after the snapshot are reused by their already-written disk offsets (and content hashes) — the bulk of the commit’s I/O has already happened off the commit path. Pages that were modified after the snapshot are copied on write into the current log generation; the new copy is authoritative, the earlier flush is superseded, and a forwarding link guarantees every reference resolves to the new version rather than the stale one. Superseded flushes stay behind as unreachable dead bytes — a deliberate write-amplification trade-off on write-hot pages in exchange for low-latency commits.
Two invariants hold unconditionally: every revision has a commit record (there is no code path that creates a revision without one — the async pre-flush creates no revision at all), and only reachable pages carry authority — page hashes embedded in parent pages cover exactly what a revision can read, never dead space.
Asynchronous durable commits
A third mode, AfterCommitState.KEEP_OPEN_ASYNC_COMMIT, is the middle ground between the two: every threshold crossing produces a real revision — durable, chained, queryable — but the expensive half of the commit protocol runs on a background thread. The writer thread serializes the epoch’s pages and hands off; the background thread performs the durability barriers (data force, index-catalogue fsync, DSYNC uber-page beacon writes) and only then publishes the revision to readers (durable-before-visible — a reader can never open a revision that a crash could still lose). Commits are pipelined with depth 1: the next epoch is built on the pending revision while the previous one hardens, and backpressure stalls the writer only if the device falls behind. A crash loses at most the in-flight epoch plus the current working set — versus everything since the last real commit with the pre-flush, and at most one batch with synchronous auto-commits.
Secondary Indexes
Name, path and CAS document indexes are integrated into the resource’s versioned page structure. These indexes are part of the transaction and version with the data — the index at revision 42 always matches the data at revision 42. See Features for other index APIs and their limits.
Path Summary
With path-summary building enabled, a resource maintains a compact path summary — a tree of all unique paths in the document. Each unique path gets a path class reference (PCR), a stable integer ID. Nodes in the main data tree reference their PCR, enabling efficient path-based lookups.
The data tree uses one node per named field: name fields are
OBJECT_NAMED_STRING nodes and age fields are OBJECT_NAMED_NUMBER nodes.
The fused users array owns the two user objects directly. Its path summary
retains both the field-name step and the array step (/users/[]). Index entries
point to the fused field nodes shown above.
Index Types
| Index | Key | Use case |
|---|---|---|
| Name | Field name hash → node keys | Find all nodes named "email" regardless of path |
| Path | PCR → node keys | Find all nodes at path /users/[]/name |
| CAS | (PCR + typed value) → node keys | Find all users where age > 30 on path /users/[]/age |
CAS (content-and-structure) indexes combine paths with typed values and support range queries. These three document index types are stored in Height Optimized Tries (HOT) within the same versioned page structure.
For the JSONiq API to create and query indexes, see the Function Reference.
Further Reading
- Evolutionary Tree-Structured Storage — Marc Kramis’s thesis describing the sliding snapshot algorithm
- Flexible Secure Cloud Storage — Sebastian Graf’s PhD dissertation on the trie-based page structure and formal verification of the sliding snapshot algorithm
- HOT: A Height Optimized Trie Index — the trie structure used for SirixDB’s secondary indexes
- SirixDB on GitHub — source code and detailed
docs/ARCHITECTURE.md - REST API documentation — HTTP interface for SirixDB
- JSONiq API — query language guide