Insights & Updates

The Consensus Problem

Most distributed databases rely on consensus algorithms like Raft or Paxos. These protocols ensure consistency by electing a leader, replicating writes, and requiring majority agreement before commits. The tradeoff, complexity and latency. Every write must travel through the leader, wait for quorum acknowledgment, and handle leader elections during failures. As clusters grow, consensus overhead increases, and scaling becomes painful.

CameoDB takes a different approach, a leaderless mesh. No single point of failure, no leader elections, no quorum waits. Instead, data distribution and routing are handled through consistent hashing and a Kademlia DHT. Each node knows where data belongs, and operations route directly to the owner. The result, linear scalability with minimal coordination overhead.

Consistent Hashing: Data Distribution Without Coordination

At the heart of CameoDB's distribution is a consistent hash ring. Each node is assigned virtual node tokens, and each shard maps to a specific position on the ring. When you write a document with a routing key, the system hashes the key using XXH3, finds the position on the ring, and determines which node owns that shard.

// Routing decision algorithm
hash = xxh3_64(routing_key)
shard_id = ConsistentRing.get_owner(hash)
node_id = shard_assignments[shard_id].owner
decision = if node_id == local_node
    RoutingDecision::Local
  else
    RoutingDecision::Remote { node_id, peer_addr }

This means deterministic routing without coordination. The same routing key always maps to the same shard and node, regardless of which node receives the request. Add a new node to the cluster, and the ring rebalances automatically. Remove a node, and data redistributes to neighbors. No central coordinator required.

Kademlia DHT: Peer Discovery Without Configuration

How do nodes find each other? CameoDB uses a Kademlia DHT built on libp2p. Each node maintains a routing table of known peers, organized by XOR distance from its own node ID. When a node joins, it contacts bootstrap nodes and discovers the network through iterative lookups.

The DHT provides three critical capabilities:

Actor-Based Remote Execution

CameoDB uses the Kameo actor framework for distributed execution. Each node runs a `NodeOrchestrator` actor that manages local shards, and these actors can communicate remotely over libp2p. When the router decides an operation belongs on a remote node, it uses Kameo's remote messaging to forward the request.

// Remote call path
orchestrator_name = format!("orchestrator-{}", target_node_id)
remote_ref = RemoteActorRef::lookup(orchestrator_name).await
result = remote_ref.ask(&ClientOp::Search { ... }).await

Remote actors are registered with stable names like `orchestrator-{node_id}` and `shard-{shard_id}`, making them discoverable across the cluster. The same `ClientOp` message type is used locally and remotely, so operation semantics are consistent regardless of where they execute.

Scatter-Gather: Fan-Out Without Bottlenecks

For queries without a routing key, CameoDB uses scatter-gather broadcast. The router asks the cluster coordinator for known peers, selects up to a fanout limit, and executes the query in parallel across local and remote nodes.

// Broadcast search algorithm
peers = coordinator.get_known_peers()
selected = peers.take(broadcast_fanout_limit)

local_result = handle_client_op(op)  // Local search
remote_results = parallel(selected, |peer| {
    remote.ask(peer, op, timeout)
})  // Fan-out to remotes

merged = top_k_merge(local_result, remote_results, limit)

Results are merged using score-aware top-K aggregation, allowing higher-scoring remote hits to displace weaker local results. Bounded concurrency ensures the system doesn't overwhelm itself with parallel requests, and per-call timeouts prevent stragglers from blocking the response.

Event-Driven State Management

Cluster metadata is persisted without polling or background tasks. All state transitions occur on membership events, `PeerDiscovered` when a node joins, `PeerLost` when a node leaves. The cluster coordinator maintains a simple state machine with three states, Active (all nodes present), Degraded (some nodes missing), Failed (below quorum).

Metadata is written to `metadata.redb` inline with state changes, providing crash-safe persistence without separate background processes. On boot, nodes load the persisted snapshot and reconcile with actual cluster state, logging discrepancies for operational visibility.

The Tradeoffs

Leaderless architecture isn't magic. It makes different tradeoffs than consensus-based systems:

The Takeaway

CameoDB's leaderless mesh trades consensus complexity for deterministic routing and automatic distribution. Consistent hashing ensures data lands in predictable places, Kademlia DHT handles peer discovery without configuration, and actor-based remote execution provides clean distributed semantics. The result is a database that scales linearly, handles failures gracefully, and keeps operational complexity low. No Raft, no Paxos, just math and networking doing what they do best.

The Inverted Index Advantage

Traditional databases struggle with text search. A LIKE query with wildcards scans every row, O(n) complexity that destroys performance as data grows. Full-text search engines use inverted indexes, mapping terms to document IDs for O(log n) lookups regardless of dataset size.

CameoDB integrates Tantivy, a Rust-based search library inspired by Lucene. Tantivy provides inverted indexes, term dictionaries, positional data for phrase queries, and BM25 relevance scoring. Combined with redb's ACID KV storage, CameoDB delivers hybrid search, fast point lookups via redb, rich full-text queries via Tantivy.

Index-Only Storage Strategy

Most search engines duplicate data, storing complete documents in the index for retrieval. This bloats index size and slows writes. CameoDB takes a different approach, index-only storage.

// Tantivy stores only indexed fields (no STORED flag)
schema = SchemaBuilder::new()
    .add_text_field("title", TEXT | STORED)
    .add_text_field("content", TEXT)  // No STORED
    .build()

// redb stores complete JSON documents
redb.insert("doc:123", full_json_document)

Tantivy stores only the `id` field and indexed fields without the STORED flag. Complete JSON documents live exclusively in redb. When you search, Tantivy returns matching document IDs and scores, then we batch-fetch the full documents from redb. This split-storage strategy means smaller indices, faster search performance, and zero data duplication.

Field-Level Control and Schema Evolution

CameoDB provides fine-grained control over field indexing through schema configuration. Each field can be configured with:

Schema evolution is automatic. When new fields appear in incoming documents, CameoDB infers their types and adds them to the schema. Field fingerprints track schema changes, enabling cache invalidation and distributed synchronization without manual schema migrations.

Query Capabilities: Beyond Simple Search

Tantivy supports sophisticated query patterns that go beyond keyword matching:

// Phrase queries with proximity
query = "database systems"~2  // Within 2 words

// Range queries on numeric fields
query = price:[100 TO 500]

// Boolean combinations
query = (title:rust OR title:go) AND year:[2020 TO 2024]

// Boosting for relevance tuning
query = title:rust^3 OR body:rust

Phrase queries use positional indexes to find terms in specific order with configurable proximity slop. Range queries work on numeric and date fields. Boolean operators enable complex logic with AND, OR, NOT. Field boosting allows you to weight certain fields higher in relevance scoring.

BM25 Relevance Scoring

Tantivy uses BM25, the industry-standard relevance scoring algorithm. BM25 considers term frequency (how often terms appear in a document), inverse document frequency (how rare terms are across the corpus), and document length normalization. This means documents matching rare terms score higher, and longer documents aren't unfairly penalized.

CameoDB returns scores alongside search results, enabling your application to implement custom ranking logic, result re-sorting, or confidence thresholds for filtering low-quality matches.

Segment Merging and Performance

Tantivy stores indexes in immutable segments. As documents are added, new segments are created. Periodic merging combines smaller segments into larger ones, reducing the number of segments that must be searched during queries. This improves read performance while maintaining write throughput.

CameoDB's Supervised Smart Commits control when segments are committed to disk. High-volume workloads trigger smart commits at adaptive thresholds (500-8000 operations), while low-volume patterns use supervised commits after 5 seconds of inactivity. This balances performance with durability guarantees.

The Takeaway

Tantivy provides the full-text search engine that powers CameoDB's hybrid architecture. The index-only storage strategy keeps indices lean while redb provides fast document retrieval. Schema evolution, phrase queries, BM25 scoring, and segment merging deliver production-grade search capabilities without the complexity of managing a separate search service. The result, rich search functionality integrated seamlessly with ACID-compliant storage.

The Hybrid Storage Philosophy

Most databases choose one storage model, relational tables for ACID guarantees, or search engines for rich query capabilities. CameoDB combines both in a single atomic transaction, delivering ACID-compliant key-value storage alongside full-text search without sacrificing consistency or performance.

The hybrid architecture leverages two complementary storage engines, redb for durable KV storage with ACID guarantees, and Tantivy for full-text inverted indexes. Every write operation touches both engines atomically, ensuring data consistency across storage and search.

Atomic Write Sequence

Each write operation follows a strict seven-step sequence to guarantee atomicity across both storage engines:

// 1. Generate sequence ID (AtomicU64, lock-free)
seq_id = counter.fetch_add(1)

// 2. Begin redb transaction (ACID isolation)
txn = kv.begin_write()

// 3. Write to WAL (durability before application)
wal.insert(seq_id, serialized_op)

// 4. Write to data table (complete JSON document)
data.insert(id, json_blob)

// 5. Update Tantivy index (in-memory buffer)
writer.add_document(id_only_doc)

// 6. Commit redb transaction (fsync if configured)
txn.commit()

// 7. Signal supervisor for smart commit
supervisor.reset_timer()

The sequence ID ensures monotonic ordering. The WAL (write-ahead log) guarantees durability before the operation is applied. The data table stores the complete JSON document. Tantivy receives only the indexed fields. The redb transaction commits with optional fsync for maximum durability. Finally, the supervisor is signaled for smart commit management.

ACID Guarantees Across Engines

CameoDB provides full ACID guarantees across the hybrid storage system:

Multi-Tenant Architecture

CameoDB supports multiple isolated indices within a single storage instance. Each index operates as an independent tenant with its own data tables, search indices, and sequence counters. This enables multi-tenancy without resource contention between indices.

Data isolation is complete, per-index `data_{index}` and `wal_{index}` tables in redb, per-index `indices/{index_name}/` directories for Tantivy. Sequence counters are independent per index, enabling parallel write scaling without lock contention.

Recovery and Consistency

On startup, CameoDB performs consistent hybrid recovery. The system reads uncommitted records from the redb WAL and replays them into the Tantivy search index. This ensures that any operations committed to redb but not yet flushed to Tantivy are recovered, maintaining consistency between the two storage engines.

The recovery process is automatic and requires no manual intervention. It handles crash scenarios, power failures, and unclean shutdowns, guaranteeing that the search index eventually reflects all committed writes.

Performance Characteristics

The hybrid architecture delivers balanced performance across operations:

The Takeaway

CameoDB's hybrid storage architecture combines the strengths of two specialized engines. redb provides ACID-compliant KV storage with fast point lookups and range queries. Tantivy provides rich full-text search with inverted indexes and relevance scoring. The atomic write sequence guarantees consistency across both engines. The result is a database that delivers both transactional integrity and search capabilities in a single system, without the complexity of managing separate storage and search infrastructure.

Performance Improvements

v0.2.2 brings significant performance gains across the board. Batch write throughput improved by 40% through optimized smart commit thresholds and reduced mutex contention. Search queries are now 25% faster thanks to improved Tantivy segment merging strategies.

New Query Operators

This release adds powerful new query operators for more flexible data retrieval:

// IN operator for set membership
query = status: IN [active pending review]

// Range comparisons with inclusive/exclusive
query = score:{100 TO 500}  // Exclusive
query = score:[100 TO 500]  // Inclusive

// Date comparisons
query = created_at:>2024-01-01
query = created_at:<=2024-12-31

The IN operator enables efficient set-based filtering without multiple OR clauses. Range queries now support both inclusive [] and exclusive {} syntax for precise control. Date comparisons use natural date formats for better readability.

Enhanced Distributed Stability

The distributed mesh now handles network partitions and node failures more gracefully. Improved Kademlia DHT routing reduces lookup latency by 30%. Peer discovery is more robust with exponential backoff and connection retry strategies.

Cluster state reconciliation now includes conflict resolution for concurrent writes. The system maintains a conflict-free replicated data type (CRDT) approach for metadata, ensuring eventual consistency across all nodes.

Expanded Platform Support

v0.2.2 adds official support for additional platforms:

API Enhancements

The HTTP API now includes comprehensive OpenAPI 3.0 documentation. New endpoints for index management, schema inspection, and health monitoring provide better operational visibility. Batch operations now support up to 10,000 documents per request.

Bug Fixes

This release includes fixes for 15 reported issues:

• Fixed memory leak in tiered cache release during multi-shard operations
• Resolved race condition in supervised commit timer reset
• Fixed Tantivy index corruption on crash recovery with large documents
• Corrected BM25 scoring for documents with multiple term matches
• Fixed schema evolution edge case with nested JSON objects
• Resolved connection pool exhaustion under high concurrent load
• Fixed distributed query timeout handling in scatter-gather
• Corrected WAL replay order for consistent recovery

Upgrade Path

Upgrading from v0.2.0 or v0.2.1 is straightforward. Stop the server, replace the binary, and restart. The storage format is backward compatible, no migration required. For v0.1.x and earlier, data reindexing is required, follow the migration guide in the documentation.

Get v0.2.2

Download the latest stable release from the downloads page. Binary packages are available for Linux, macOS, and Windows. Source code is available on GitHub under the FSL-1.1-ALv2 License, with Apache 2.0 and MIT licensing for client interfaces.

The ACID Challenge in Leaderless Systems

Traditional databases achieve ACID guarantees through consensus algorithms like Raft or Paxos. These protocols elect a leader to serialize writes, ensuring consistency across replicas. But consensus comes with costs, leader election latency, quorum requirements, and limited scalability.

CameoDB takes a different approach. Instead of global consensus, it provides ACID guarantees at the storage engine level through redb's transactional model. Each node maintains ACID compliance for local writes, while the distributed layer handles eventual consistency across the cluster. This tradeoff delivers local transactional integrity with global scalability.

Atomicity: All-or-Nothing Writes

Atomicity ensures that a transaction is either fully applied or not applied at all. CameoDB guarantees atomicity through redb's MVCC (multi-version concurrency control) transaction system. Every write operation occurs within a redb transaction, and if any step fails, the entire transaction rolls back.

// Atomic write with rollback on failure
txn = db.begin_write()
result = match execute_write_sequence(txn) {
    Ok(()) => txn.commit(),
    Err(e) => txn.abort(),  // Rolls back all changes
}

The seven-step write sequence (sequence ID, WAL, data table, Tantivy index, commit) executes atomically. If the Tantivy index update fails, the redb transaction aborts, leaving the WAL and data table untouched. No partial state, no orphaned writes.

Consistency: Invariants Across Storage

Consistency ensures that the database transitions from one valid state to another. CameoDB maintains consistency through schema validation and cross-engine synchronization. The storage engine enforces type constraints, field requirements, and relationship invariants.

The hybrid architecture maintains consistency between redb and Tantivy. The atomic write sequence ensures that both storage engines reflect the same logical state after each commit. The recovery process replays uncommitted WAL records into Tantivy, guaranteeing eventual consistency after crashes.

Isolation: Concurrent Operation Safety

Isolation ensures that concurrent transactions don't interfere with each other. CameoDB provides isolation through redb's MVCC system. Readers operate on snapshots, seeing a consistent view of the database at a point in time, unaffected by concurrent writes.

The system supports multiple isolation levels:

Writes are serialized through the storage engine's transaction manager. Concurrent writes to the same document are serialized by redb's locking mechanism, preventing lost updates and race conditions.

Durability: Surviving Failures

Durability ensures that committed transactions survive failures. CameoDB provides durability through the write-ahead log (WAL) and configurable fsync behavior. Every write operation is first recorded in the WAL before being applied to the data tables.

// WAL ensures durability before application
wal.insert(seq_id, serialized_op)  // Durable
data.insert(id, json_blob)       // Applied
writer.add_document(...)         // Indexed
txn.commit()                     // With optional fsync

The WAL survives crashes, power failures, and unclean shutdowns. On recovery, CameoDB replays uncommitted WAL records to restore the database to a consistent state. Configurable fsync allows you to choose between maximum durability (fsync on every commit) or higher throughput (fsync on smart commit intervals).

Distributed Consistency Model

CameoDB's distributed layer provides eventual consistency across the cluster. Writes are routed to the node that owns the shard based on consistent hashing. Each node maintains ACID compliance for its local writes. Changes propagate to other nodes through the gossip-based peer discovery and state synchronization.

This model means that read-your-writes consistency requires using routing keys. If you write a document with a routing key, subsequent reads with the same key route to the same node, ensuring you see your write. Without routing keys, reads may return stale data from other nodes until gossip synchronization completes.

Conflict Resolution

In a leaderless system, concurrent writes to the same document on different nodes can create conflicts. CameoDB uses a conflict-free replicated data type (CRDT) approach for metadata, ensuring eventual consistency without coordination. For user data, the system relies on the application layer to handle conflicts through version vectors or application-specific resolution strategies.

The sequence ID provides a total order for writes within a single node. Combined with timestamps and node IDs, the system can detect and report conflicts to the application layer for resolution.

The Tradeoffs

CameoDB's transaction model makes specific tradeoffs:

The Takeaway

CameoDB delivers ACID guarantees through redb's transactional storage engine, providing strong consistency at the node level. The distributed layer provides eventual consistency across the cluster, enabling linear scalability without consensus overhead. The result is a database that combines transactional integrity for local operations with global scalability for distributed workloads. Use routing keys for read-your-writes consistency, or embrace eventual consistency for maximum throughput.