sirix

SirixDB

SirixDB - The Bitemporal Database System

Query any revision as fast as the latest

CI Build Status Maven Central PRs Welcome All Contributors

🟢 Live Demo · Why SirixDB · Docs · Website · Discord · Forum · Web UI

Status: 1.0.0-beta — usable today and actively developed. The on-disk format and public APIs are stabilizing toward a 1.0 release; feedback from real use is exactly what we're looking for.

The Problem

You update a row in your database. The old value is gone.

To get history, you bolt on audit tables, change-data-capture, or event sourcing. Now you have two systems: one for current state, one for history. Querying the past means replaying events or scanning logs. Your “simple” audit requirement just became an infrastructure project.

Git solves this for files—but you can’t query a Git repository. Event sourcing preserves history—but reconstructing past state means replaying from the beginning.

The Solution

SirixDB is a database where every revision is a first-class citizen. Not an afterthought. Not a log you replay.

// Query revision 1 - instant, not reconstructed
session.beginNodeReadOnlyTrx(1)

// Query by timestamp - which revision was current at 3am last Tuesday?
session.beginNodeReadOnlyTrx(Instant.parse("2024-01-15T03:00:00Z"))

// Both return the same thing: a readable snapshot, as fast as querying "now"

This works because SirixDB uses structural sharing with sub-page versioning. Unchanged pages are shared between revisions via copy-on-write — and versioning continues below the page: a commit writes page fragments containing only the changed records, and the sliding-snapshot algorithm guarantees any page is reconstructible from at most N fragments. Block-level COW (ZFS-style) copies a whole page when one byte in it changes; delta-based systems make reads replay ever-growing diff chains. SirixDB pays neither cost. Revision 1000 doesn’t store 1000 copies—it stores the current state plus pointers to shared history.

The result:

Bitemporal: Two Kinds of Time

Most databases (if they version at all) track one timeline: when data was written. SirixDB tracks two:

Why does this matter?

January 15: You record "Price = $100, valid from January 1"
January 20: You discover the price was actually $95 on January 1

After correction, you can ask:
  "What did we THINK the price was on Jan 16?"  →  $100 (transaction time)
  "What WAS the price on Jan 1?"                →  $95  (valid time)

Both questions have correct, different answers. Without bitemporal support, the correction destroys the audit trail.

Core Properties

Performance

History is not a tax. Reading an old revision is a direct page lookup, not a replay — any revision reads as fast as the latest, and session-open cost is flat regardless of how much history exists (0.18 ms at 10,000 revisions).

A few measured receipts (we benchmark against ourselves and publish the losses, methodology in WHY-SIRIX.md and the linked comparison docs):

Every fast path is fail-closed: a kernel only runs when the optimizer can prove the query’s shape matches what it emits, and a differential suite requires byte-identical output against the general path. Wrong-but-fast is a bug class, not a setting.

How Versioning Works

Logical page structure of a resource with 3 revisions
Logical page structure of a resource with 3 revisions — read-only transactions (RTX) can open any revision, while a single write transaction (WTX) appends to the latest.

SirixDB stores data in a persistent tree structure where revisions share unchanged pages and nodes. Traditional databases overwrite data in place and use write-ahead logs for recovery. SirixDB takes a different approach:

Physical Storage: Append-Only Log

All data is written sequentially to an append-only log. Nothing is ever overwritten.

Physical Log (append-only, sequential writes)
┌────────────────────────────────────────────────────────────────────────┐
│ [R1:Root] [R1:P1] [R1:P2] [R2:Root] [R2:P1'] [R3:Root] [R3:P2'] ...    │
└────────────────────────────────────────────────────────────────────────┘
     t=0      t=1     t=2      t=3      t=4       t=5       t=6    → time

Logical Structure: Persistent Trie

Each revision has a root node in a trie. Unchanged pages are shared via references.

  [Rev 1]           [Rev 2]         [Rev 3]
     │                 │               │
     ▼                 ▼               ▼
  [Root₁]           [Root₂]         [Root₃]
   │   │             │   │           │   │
   │   └──────────┐  │   └────────┐  │   └─────────┐
   ▼              ▼  ▼            ▼  ▼             ▼
┌──────┐        ┌──────┐        ┌──────┐        ┌──────┐
│  P1  │        │  P2  │        │ P1'  │        │ P2'  │
└──────┘        └──────┘        └──────┘        └──────┘
 Rev 1          Rev 1+2         Rev 2+3          Rev 3
                (shared)        (shared)

Rev 2 modified only P1 (writing P1’) and still shares P2 with Rev 1; Rev 3 modified only P2 (writing P2’) and still shares P1’ with Rev 2 — matching the physical log above.

Page Versioning Strategies

SirixDB supports multiple strategies for storing page versions, configurable per resource:

┌──────────────────────────────────────────────────────────────────────────────┐
│ FULL: Each page stores complete data                                         │
│                                                                              │
│   Rev1: [████████]  Rev2: [████████]  Rev3: [████████]                       │
│         (full)            (full)            (full)                           │
│                                                                              │
│   + Fast reads (no reconstruction)                                           │
│   - High storage cost                                                        │
├──────────────────────────────────────────────────────────────────────────────┤
│ INCREMENTAL: Diffs from previous revision + periodic full snapshots          │
│                                                                              │
│   Rev1: [████████]  Rev2: [Δ←1]  Rev3: [Δ←2]  Rev4: [████████]               │
│         (full)       (diff)       (diff)       (full snapshot)               │
│                                                                              │
│   Rev5: [Δ←4]  Rev6: [Δ←5]  Rev7: [████████]  ...                            │
│         (diff)       (diff)       (full snapshot)                            │
│                                                                              │
│   Full snapshot written every N revisions (N = configurable window)          │
│   + Bounded read cost (max N-1 diffs between full snapshots)                 │
│   + Compact diffs (each diff is against previous revision only)              │
│   - Read cost grows linearly within each window                              │
├──────────────────────────────────────────────────────────────────────────────┤
│ DIFFERENTIAL: Diffs from reference snapshot + periodic full snapshots        │
│                                                                              │
│   Rev1: [████████]  Rev2: [Δ←1]  Rev3: [████████]  Rev4: [Δ←3]               │
│         (full)       (diff)       (full snapshot)    (diff)                  │
│                                                                              │
│   Rev5: [Δ←3]  Rev6: [████████]  Rev7: [Δ←6]  ...                            │
│         (diff)       (full snapshot)    (diff)                               │
│                                                                              │
│   Full snapshot every N revisions; diffs reference the last snapshot         │
│   + Bounded read cost (max 1 diff to apply)                                  │
│   - Diffs grow larger as they diverge from last snapshot                     │
├──────────────────────────────────────────────────────────────────────────────┤
│ SLIDING SNAPSHOT: Incremental diffs within a sliding window of size N        │
│                                                                              │
│   Rev1: [████████]  Rev2: [Δ←1]  Rev3: [Δ←2]  Rev4: [Δ←3 + R1 copy]          │
│         (full)       (diff)       (diff)       (diff + out-of-window         │
│                                                 records from Rev1)           │
│         ◄──────── window N=3 ──────────►                                     │
│                      ◄──────── window N=3 ──────────►                        │
│                                                                              │
│   As the window slides forward, records from pages that fall out of          │
│   the window are copied into the newest diff page, ensuring any              │
│   revision can be reconstructed from at most N page fragments.               │
│                                                                              │
│   + Bounded read cost (max N page fragments to combine)                      │
│   + No unbounded diff growth (out-of-window data is always rescued)          │
│   = Best balance of storage vs read performance                              │
└──────────────────────────────────────────────────────────────────────────────┘

When you modify data:

  1. Only the affected pages are copied and modified (copy-on-write)
  2. Unchanged pages are referenced from the new revision
  3. The old revision remains intact and queryable

Storage cost: O(changed pages) per revision, not O(total document size).

Read performance: Opening a revision is O(1) by revision number or O(log R) by timestamp (binary search over R revisions). Each page read requires combining at most N page fragments, where N is the snapshot window size (configurable, default 3). Tree traversal to locate a node is O(log nodes), same as querying the latest revision.

Quick Start

Platform support: Linux, macOS, and Windows are CI-tested: gating lanes run the core, query, and Kotlin test suites on ubuntu-latest, macos-latest, and windows-latest on every pull request. All platforms run the same Umbra-style frame-slot allocator (platform-specific reserve/commit plumbing: POSIX mmap, Windows VirtualAlloc). Linux additionally gets native binaries and the Docker images; on macOS and Windows both the native JVM and Docker (via Docker Desktop) work. One known limitation: crash-recovery re-initialization with MEMORY_MAPPED storage is unsupported on Windows (see docs/KNOWN_LIMITATIONS.md).

Using the CLI (Native Binaries)

SirixDB provides two CLI tools, both available as instant-startup native binaries:

Binary Module Description
sirix-cli sirix-kotlin-cli Full-featured CLI for database operations
sirix-shell sirix-query Interactive JSONiq/XQuery shell

Build native binaries with GraalVM:

# Build both CLIs as native binaries (requires GraalVM with native-image)
./gradlew :sirix-kotlin-cli:nativeCompile  # produces: sirix-cli
./gradlew :sirix-query:nativeCompile       # produces: sirix-shell

# Or run via JAR
./gradlew :sirix-kotlin-cli:run --args="-l /tmp/mydb create"

sirix-cli: Database Operations

The -l option specifies the database path. Each database can contain multiple resources.

Create a database and store JSON:

sirix-cli -l /tmp/mydb create json -r myresource -d '{"name": "Alice", "role": "admin"}'

Query your data:

sirix-cli -l /tmp/mydb query -r myresource

Run a JSONiq query:

# $$ is bound to the document root, so access fields directly
sirix-cli -l /tmp/mydb query -r myresource '$$.name'

Update and create a new revision:

sirix-cli -l /tmp/mydb update -r myresource '{"team": "engineering"}' -im as-first-child

Query a previous revision:

sirix-cli -l /tmp/mydb query -r myresource -rev 1

View revision history:

sirix-cli -l /tmp/mydb resource-history myresource

sirix-shell: Interactive Query Shell

The interactive shell provides a REPL for JSONiq/XQuery queries. A query can span multiple lines — an empty line executes it; exit with Control-D:

$ sirix-shell
Enter query string (terminate with Control-D):
sirix > 1 + 1

Query result
2
Enter query string (terminate with Control-D):
sirix > jn:store('mydb','resource','{"key": "value"}')

Query result

Enter query string (terminate with Control-D):
sirix > jn:doc('mydb','resource').key

Query result
"value"

Using the REST API

Start SirixDB and its bundled OAuth2 provider (Keycloak) with Docker:

git clone https://github.com/sirixdb/sirix.git
cd sirix
docker compose up

This starts two services: the SirixDB REST server on http://localhost:9443 (plain HTTP in the default local configuration — terminate TLS in a proxy for anything public) and a Keycloak instance that is auto-seeded with two demo users — admin/admin (full access) and viewer/viewer (read-only).

Check the server is up (this endpoint needs no auth):

curl http://localhost:9443/health   # -> {"status":"UP"}

All endpoints are OAuth2-protected. Obtain a bearer token from the server’s /token endpoint, then use it on subsequent requests:

# 1. Get an access token (the server proxies to Keycloak)
TOKEN=$(curl -s -X POST http://localhost:9443/token \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin","grant_type":"password"}' | jq -r .access_token)

# 2. Store a JSON resource
curl -X PUT http://localhost:9443/mydb/myresource \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"Alice","role":"admin"}'

# 3. Read it back
curl http://localhost:9443/mydb/myresource \
  -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/json"

For local development you can skip Keycloak entirely: start the server with auth.mode=none (or docker run -e SIRIX_AUTH_MODE=none ...) and every request runs as an all-permissions admin user — the server logs a loud warning. Container memory is tunable via SIRIX_XMS/SIRIX_XMX/SIRIX_MAX_DIRECT/SIRIX_JAVA_OPTS env vars (defaults fit a laptop).

→ docs/QUICKSTART.md walks through the whole loop — create, query, commit, time-travel read, diff — with verified, copy-pasteable commands. See the REST API documentation for the full endpoint reference.

Security note: The bundled Keycloak realm, demo users, client secret, and the self-signed TLS certificate under bundles/sirix-rest-api/src/main/resources/ are for local development only. Before any public deployment, generate your own certificate, rotate the client secret, and create real users with strong passwords. See docs/operations.md.

Using the MCP Server (for AI Agents)

SirixDB ships a native Model Context Protocol server, so AI agents (Claude, Cursor, Windsurf, or any MCP client) can talk to it directly. Because every revision is copy-on-write, agents get O(1) disposable snapshots, time-travel reads, and structural diffs for free — branch, experiment, then discard or promote, with a human-reviewable diff. It is read-only by default and includes access control, output sanitization, and an audit log.

# Build a self-contained launcher (creates build/install/sirix-mcp/bin/sirix-mcp)
./gradlew :sirix-mcp:installDist

Register it with your MCP client (e.g. Claude Desktop / Cursor mcp_servers.json):

{
  "mcpServers": {
    "sirixdb": {
      "command": "/path/to/sirix/bundles/sirix-mcp/build/install/sirix-mcp/bin/sirix-mcp",
      "args": ["--database-path", "/path/to/data"]
    }
  }
}

Add --read-write to the args to allow mutations (read-only is the default). See docs/MCP_SERVER_DESIGN.md for the full tool reference.

As an Embedded Library

<dependency>
  <groupId>io.sirix</groupId>
  <artifactId>sirix-core</artifactId>
  <version>1.0.0-beta7</version>
</dependency>

// Gradle (Kotlin DSL)
implementation("io.sirix:sirix-core:1.0.0-beta7")

var dbPath = Path.of("/tmp/mydb");

// Create database and resource
Databases.createJsonDatabase(new DatabaseConfiguration(dbPath));
try (var database = Databases.openJsonDatabase(dbPath)) {
    database.createResource(ResourceConfiguration.newBuilder("myresource").build());

    // Insert JSON data (creates revision 1)
    try (var session = database.beginResourceSession("myresource");
         var wtx = session.beginNodeTrx()) {
        wtx.insertSubtreeAsFirstChild(JsonShredder.createStringReader("{\"key\": \"value\"}"));
        wtx.commit();
    }

    // Update creates revision 2 (revision 1 remains unchanged)
    try (var session = database.beginResourceSession("myresource");
         var wtx = session.beginNodeTrx()) {
        wtx.moveTo(2);  // Move to the "key" node
        wtx.setStringValue("updated value");
        wtx.commit();
    }

    // Read from revision 1 - still accessible
    try (var session = database.beginResourceSession("myresource");
         var rtx = session.beginNodeReadOnlyTrx(1)) {
        rtx.moveTo(2);
        System.out.println(rtx.getValue());  // Prints: value
    }
}

Time-Travel Queries

SirixDB extends JSONiq/XQuery (via Brackit) with temporal axis and functions.

Access by Revision Number or Timestamp

(: Open specific revision :)
jn:doc('mydb','myresource', 5)

(: Open by timestamp - returns revision valid at that instant :)
jn:open('mydb','myresource', xs:dateTime('2024-01-15T10:30:00Z'))

Temporal Axis Functions

Navigate a node’s history across revisions:

(: Single-step navigation :)
jn:previous($node)       (: same node in the previous revision :)
jn:next($node)           (: same node in the next revision :)

(: Boundary access :)
jn:first($node)          (: node in the first revision :)
jn:last($node)           (: node in the most recent revision :)
jn:first-existing($node) (: revision where this node first appeared :)
jn:last-existing($node)  (: revision where this node last existed :)

(: Range navigation - returns sequences :)
jn:past($node)           (: node in all past revisions :)
jn:future($node)         (: node in all future revisions :)
jn:all-times($node)      (: node across all revisions :)

(: With includeSelf parameter :)
jn:past($node, true())   (: include current revision :)
jn:future($node, true()) (: include current revision :)

Example: iterate through all versions of a node:

for $version in jn:all-times(jn:doc('mydb','myresource').users[0])
return {"rev": sdb:revision($version), "data": $version}

Diff Between Revisions

(: Structured diff between any two revisions :)
jn:diff('mydb','myresource', 1, 5)

(: Diff with optional parameters: startNodeKey, maxLevel :)
jn:diff('mydb','myresource', 1, 5, $nodeKey, 3)

For adjacent revisions, jn:diff reads directly from stored change tracking files. For non-adjacent revisions it computes the diff.

If hashes are enabled, you can also detect changes via hash comparison:

(: Find which revisions changed a specific node - requires hashes enabled :)
let $node := jn:doc('mydb','myresource').config
for $v in jn:all-times($node)
let $prev := jn:previous($v)
where empty($prev) or sdb:hash($v) ne sdb:hash($prev)
return sdb:revision($v)

Bitemporal Queries

Query both time dimensions (see Bitemporal: Two Kinds of Time above for why this matters).

Configuring Valid Time Support

Configure a resource with valid time paths to enable automatic indexing and dedicated query functions:

// Configure resource with valid time paths
var resourceConfig = ResourceConfiguration.newBuilder("employees")
    .validTimePaths("validFrom", "validTo")  // specify your JSON field names
    .buildPathSummary(true)
    .build();

database.createResource(resourceConfig);

// Or use conventional field names (_validFrom, _validTo)
var resourceConfig = ResourceConfiguration.newBuilder("employees")
    .useConventionalValidTimePaths()
    .build();

Via REST API, use query parameters when creating a resource:

# Custom valid time field names
curl -X PUT "http://localhost:9443/database/resource?validFromPath=validFrom&validToPath=validTo" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{"name": "Alice", "validFrom": "2024-01-01T00:00:00Z", "validTo": "2024-12-31T23:59:59Z"}]'

# Use conventional _validFrom/_validTo fields
curl -X PUT "http://localhost:9443/database/resource?useConventionalValidTime=true" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{"name": "Bob", "_validFrom": "2024-01-01T00:00:00Z", "_validTo": "2024-12-31T23:59:59Z"}]'

When valid time paths are configured, SirixDB automatically creates CAS indexes on the valid time fields for optimal query performance.

Valid Time Query Functions

(: Get records valid at a specific point in time :)
jn:valid-at('mydb','myresource', xs:dateTime('2024-07-15T12:00:00Z'))

(: True bitemporal query: combine transaction time and valid time :)
(: "What records were known on Jan 20 and valid on July 15?" :)
jn:open-bitemporal('mydb','myresource',
    xs:dateTime('2024-01-20T10:00:00Z'),   (: transaction time - opens revision :)
    xs:dateTime('2024-07-15T12:00:00Z'))   (: valid time - filters via index :)

(: Extract valid time bounds from a node :)
let $record := jn:doc('mydb','myresource')[0]
return {
  "validFrom": sdb:valid-from($record),
  "validTo": sdb:valid-to($record)
}

Transaction Time Functions

(: Transaction time: what did the database look like at a point in time? :)
jn:open('mydb','myresource', xs:dateTime('2024-01-15T10:30:00Z'))

(: Get the commit timestamp of current revision :)
sdb:timestamp(jn:doc('mydb','myresource'))

(: Open all revisions within a transaction time range :)
jn:open-revisions('mydb','myresource',
        xs:dateTime('2024-01-01T00:00:00Z'),
        xs:dateTime('2024-06-01T00:00:00Z'))

Revision Metadata Functions

(: Get revision number and timestamp :)
sdb:revision($node)              (: revision number of this node :)
sdb:timestamp($node)             (: commit timestamp as xs:dateTime :)
sdb:most-recent-revision($node)  (: latest revision number in resource :)

(: Get history of changes to a specific node :)
sdb:item-history($node)          (: all revisions where this node changed :)
sdb:is-deleted($node)            (: true if node was deleted in a later revision :)

(: Author tracking (if set during commit) :)
sdb:author-name($node)
sdb:author-id($node)

(: Commit with metadata :)
sdb:commit($doc)
sdb:commit($doc, "commit message")
sdb:commit($doc, "commit message", xs:dateTime('2024-01-15T10:30:00Z'))

Merkle Hash Verification (Optional)

When enabled in resource configuration, SirixDB stores a hash for each node computed from its content and descendants. Use this for:

sdb:hash(jn:doc('mydb','myresource'))           (: root hash :)
sdb:hash(jn:doc('mydb','myresource').users[0])  (: subtree hash :)

See Query documentation for the full API.

Web Interface

The SirixDB Web GUI provides visualization of revision history and diffs:

git clone https://github.com/sirixdb/sirixdb-web-gui.git
cd sirixdb-web-gui
docker compose -f docker-compose.demo.yml up

Open http://localhost:3000 (login: admin/admin)

Architecture

JSON Tree Encoding

SirixDB shreds JSON into a typed node tree where each node has a stable key across revisions:

How JSON is encoded as a node tree
A JSON document and its internal tree representation — each node carries a stable key (nodeKey) for identity tracking across revisions.

Document Storage and Path Summary

When JSON is stored, SirixDB also builds a path summary — a compact trie capturing all unique paths in the document. This powers the path and CAS indexes:

Document tree and path summary
Left: the document tree. Right: the path summary trie with stable path class records (PCR) used for indexing.

On-Device Layout

Logical device layout of a resource
Physical layout on disk — data is split across two logical devices (LD₀ for metadata offsets, LD₁ for page data), written sequentially per revision.

Storage Model

Database (directory)
└── Resource (single JSON or XML document with revision history)
    └── Revisions (numbered 1, 2, 3, ...)
        └── Pages (variable-size blocks containing node data)

Key Design Decisions

Aspect Design Trade-off
Write pattern Append-only No in-place updates; simpler recovery; larger storage footprint
Consistency Single writer per resource No write conflicts; readers never blocked
Index updates Synchronous Queries always see current indexes
Node IDs Stable across revisions Enables tracking node identity through time

Indexes

Three secondary index types, all updated synchronously inside the writing transaction — queries never see a stale index:

Two interchangeable storage backends sit behind every index type, selected per resource via ResourceConfiguration (useHOTIndexes() / useRBTreeIndexes()):

Backend Structure Notes
HOT (default) Height-Optimized Trie over off-heap leaf pages cache-friendly, SIMD partial-key search, fewer levels
RBTree red-black-tree records in the standard page trie traditional, stable

Like the rest of the engine, indexes are fully versioned: opening an index at revision N returns the index state as of N — never a later commit’s. RBTree indexes inherit this from the standard page-versioning trie (the same copy-on-write pages as the document tree); for the HOT backend it is verified directly across point and range reads, session close/reopen, and a concurrent pinned-reader-vs-writer (see HOTMultiVersionInvariantsTest).

Projection indexes (experimental, analytical)

A fourth, columnar index type accelerates analytical queries — aggregates, filtered counts, group-bys, count-distinct — over homogeneous record sets. A projection index extracts the declared fields of every record under a root path into compact column-oriented leaf pages (1024 rows per leaf: frame-of-reference numerics, per-leaf string dictionaries, presence bitmaps), which the vectorized executor scans with SIMD kernels instead of walking the document tree. Persisted leaves are bit-packed to roughly 5% of their in-memory size, so the on-disk tax over the versioned document store is ~10%.

Create one with JSONiq — the resource must be created with a path summary (buildPathSummary(true)):

(: store a record set :)
jn:store('mydb', 'sales.jn', '[
  {"age": 30, "active": true,  "dept": "Eng",   "city": "NYC"},
  {"age": 45, "active": false, "dept": "Sales", "city": "LA"},
  {"age": 52, "active": true,  "dept": "Eng",   "city": "NYC"}
]')

(: project (age, active, dept, city) over the top-level array :)
let $doc := jn:doc('mydb', 'sales.jn')
let $stats := jn:create-projection-index($doc, '/[]',
    ('/[]/age', '/[]/active', '/[]/dept', '/[]/city'),
    ('long', 'boolean', 'string', 'string'))
return {"revision": sdb:commit($doc)}

Nested roots and nested columns are paths too — e.g. a record set under '/wrapper/records/[]' with a column '/wrapper/records/[]/address/city', or a descendant pattern '//records/[]' spanning sibling subtrees. Missing fields are tracked per row in presence bitmaps, so sparse data stays correct.

There is no separate scan function — eligible queries route through the projection automatically once it is installed (compile through SirixCompileChain.createWithJsonStore(store, session) to get the analytical executor). Plain JSONiq does it:

(: full-column aggregates — served from the numeric column, no tree walk :)
let $doc := jn:doc('mydb', 'sales.jn')
return {"sum": sum(for $r in $doc[] return $r.age),
        "min": min(for $r in $doc[] return $r.age),
        "max": max(for $r in $doc[] return $r.age)}

(: filtered count — conjunctive predicate over the age + active columns :)
let $doc := jn:doc('mydb', 'sales.jn')
return count(for $r in $doc[] where $r.age > 40 and $r.active return $r)

(: single- and multi-key group-by — dictionary-encoded group columns :)
let $doc := jn:doc('mydb', 'sales.jn')
for $r in $doc[]
let $d := $r.dept, $c := $r.city
group by $d, $c
return {"dept": $d, "city": $c, "count": count($r)}

(: count-distinct — answered from the union of per-leaf dictionaries :)
let $doc := jn:doc('mydb', 'sales.jn')
return count(for $r in $doc[] let $d := $r.dept group by $d return $d)

The index is written into the session’s transaction — sdb:commit($doc) persists it, like the other index-creation functions. Projection definitions are catalogued in the resource’s index set exactly like path/CAS/name indexes, so a resource can carry several projections side by side (each in its own storage sub-tree), and queries discover them through the revision-scoped catalog and page layer — after re-opening a database, analytical queries use persisted projections automatically (decoded once per revision into a bounded in-memory cache, sub-second per ~10M rows), with no re-creation call needed. Because discovery is revision-scoped, uncommitted builds are invisible to other sessions, rollbacks need no compensation, and time-travel queries only ever see projection data that was current at their revision. Update transactions maintain projections incrementally (wired through the index-controller listener lifecycle, like the other index types): changes are attributed to their records as they happen, and at commit time only the touched leaves are patched — updated records are re-extracted in place, deleted records drop out, and new records append to the tail — so the same catalogued projection keeps serving across updates with no re-creation call. Changes the incremental path cannot attribute exactly (subtree moves, removing a record-set array itself, unresolvable structure, or more dirty records per transaction than -Dsirix.projection.maxIncrementalRecords, default 100 000, where a rebuild is cheaper) fall back to invalidation: the persisted columns are marked stale inside the transaction, queries at later revisions transparently use the regular pipeline, and re-running jn:create-projection-index rebuilds under the same definition; calling it with a different shape creates an additional projection. Uncommitted state is servable too: an executor constructed over an open write transaction (new SirixVectorizedExecutor(wtx, threads)) answers unpredicated aggregates, group-bys and count-distinct from the transaction’s own state — pending maintenance is applied on read (read-your-writes) and the leaves are read through the transaction log, uncached, so committed readers keep their isolated snapshots. The full function family matches the other index types: jn:find-projection-index($doc, $rootPath, $fields) returns a projection’s definition id (or -1), and jn:drop-projection-index($doc[, $idx-no]) drops one or all projections (tombstoning the stored columns so a later same-shape re-creation rebuilds instead of serving leftovers). Current limits: column types are long, boolean, and string (floating-point columns are rejected rather than silently degraded); columns are resolved by trailing field name, which must be unique and unambiguous under the record set; queries that the projection cannot serve exactly (unrepresentable values, non-covered predicates, ambiguous projection selection) fall back to the regular pipeline automatically, so results are always identical with or without the index.

Correctness & Formal Verification

A versioned storage engine is only useful if old revisions are exactly what was written. We take correctness seriously and treat it as a first-class, reviewable artifact:

The aim isn’t Coq-grade proof; it’s that every behavioral claim about the storage engine is stated precisely and guarded by a test.

Comparison with Alternatives

Feature SirixDB Postgres + Audit Git + JSON Event Sourcing Datomic
Query past state Direct page access Scan audit log Checkout + parse Replay events Direct segment access
Storage overhead O(changes) O(all writes) O(file × revs) O(all events) O(changes)
Granularity Node-level Row-level File-level Event-level Fact-level
Bitemporal Built-in Manual No Manual Built-in
Embeddable Yes No Yes Varies No
Query language JSONiq/XQuery SQL None Varies Datalog

Building from Source

git clone https://github.com/sirixdb/sirix.git
cd sirix
./gradlew build -x test

Requirements:

JVM flags (required for running):

--enable-preview
--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED
--add-exports=java.base/sun.nio.ch=ALL-UNNAMED
--add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED

Build native binaries (requires GraalVM):

./gradlew :sirix-kotlin-cli:nativeCompile  # sirix-cli
./gradlew :sirix-query:nativeCompile       # sirix-shell
./gradlew :sirix-rest-api:nativeCompile    # REST API server

Project Structure

bundles/
├── sirix-core/          # Core storage engine and versioning
├── sirix-query/         # Brackit JSONiq/XQuery integration + sirix-shell
├── sirix-rest-api/      # Vert.x REST server
├── sirix-kotlin-cli/    # Command-line interface (sirix-cli)
├── sirix-kotlin-api/    # Kotlin coroutine-based API
├── sirix-mcp/           # Model Context Protocol server for AI agents
├── sirix-examples/      # Runnable usage examples
└── sirix-benchmarks/    # JMH and scale benchmarks

Use Cases

Community

Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines, and please review our Code of Conduct.

For security vulnerabilities, see SECURITY.md.

Contributors

SirixDB is maintained by Johannes Lichtenberger and the open source community.

The project originated from Treetank, a university research project by Dr. Marc Kramis, Dr. Sebastian Graf and many students.


Ilias YAHIA
💻
BirokratskaZila
📖
Andrei Buiza
💻
Bondar Dmytro
💻
santoshkumarkannur
📖
Lars Eckart
💻
Jayadeep K M
📆

Keith Kim
🎨
Theofanis Despoudis
📖
Mario Iglesias Alarcón
🎨
Antonio Nuno Monteiro
📆
Fulton Browne
📖
Felix Rabe
📖
Ethan Willis
📖

Erik Axelsson
💻
Sérgio Batista
📖
chaensel
📖
Balaji Vijayakumar
💻
Fernanda Campos
💻
Joel Lau
💻
add09
💻

Emil Gedda
💻
Andreas Rohlén
💻
Marcin Bielecki
💻
Manfred Nentwig
💻
Raj
💻
Moshe Uminer
💻

Sponsors

Support SirixDB development on Open Collective.

License

BSD 3-Clause License

This site is open source. Improve this page.