New RocksDB Binding for Node.js

The Node.js ecosystem has never had a great RocksDB story untill now.

There are wrappers built on top of the old LevelDB abstractions. There are abandoned forks. There are bindings that expose a fraction of what RocksDB can actually do, without transaction support, without proper iterators, without TypeScript. If you wanted to use RocksDB from JavaScript in a serious way, you ended up writing your own glue code or compromising on what the engine could give you.

We needed something better for Harper 5.0, and we figured the Node.js community probably did too. So we built rocksdb-js from scratch: a native C++ binding with a TypeScript API, full transaction support, lazy range iterators, a built-in transaction log system, and prebuilt binaries for every major platform. It is open source under Apache 2.0.

This post is a walkthrough of what the library does, why we made the design decisions we did, and where the community can take it from here.

‍

The Basics

Install it:

npm install @harperfast/rocksdb-js

‍

Open a database and start working with it:

import { RocksDatabase } from '@harperfast/rocksdb-js';

const db = RocksDatabase.open('/path/to/db');

// Write some data
for (const key of ['a', 'b', 'c', 'd', 'e']) {
  await db.put(key, `value ${key}`);
}

// Read it back
console.log(await db.get('b')); // 'value b'

// Range query with lazy evaluation
for (const { key, value } of db.getRange({ start: 'b', end: 'd' })) {
  console.log(`${key} = ${value}`);
}

db.close();

‍

If you have used lmdb-js, this should feel familiar. That was intentional. Kris Zyp has maintained lmdb-js for years, and it has seen significant adoption. We wanted rocksdb-js to have a similar developer experience so that switching between storage engines does not mean learning an entirely new API.

But under the hood, the two libraries are doing very different things for very different workloads.

‍

Why RocksDB, and Why a New Binding

Harper had been using LMDB, and it was excellent for read-heavy workloads. It uses memory-mapped I/O, has extremely fast reads, and is battle-tested. Harper has run on LMDB for years, and the performance our enterprise customers see on read-heavy use cases speaks for itself.

RocksDB excels in a different space. Its LSM-tree architecture with continuous compaction efficiently handles write-heavy workloads with large, heterogeneous record sizes. It does not suffer from the fragmentation issues that can emerge with LMDB under sustained heavy writes. And its transaction model is more robust, with real isolation levels that matter for concurrent workloads.

Harper 5.0 supports both engines. LMDB remains available for existing deployments, and RocksDB is now the default for new workloads, providing better write optimization and stronger transaction guarantees. 

Building a new binding rather than wrapping an existing one came down to control. We needed direct access to RocksDB's transaction API, its column families, and its statistics interface. None of the existing Node.js bindings exposed what we needed, and the ones that came closest were either unmaintained or built on abstraction layers that added overhead we did not want.

‍

Transactions

This is the feature that drove us to build rocksdb-js in the first place. Real transactions with real isolation:

await db.transaction(async (txn) => {
  const user = await txn.get('user:1');
  await txn.put('user:1', { ...user, lastLogin: Date.now() });
  await txn.put('audit:login', { userId: 1, timestamp: Date.now() });
});
// Both writes commit atomically, or neither does.

‍

rocksdb-js supports both optimistic and pessimistic transactions. Optimistic transactions (the default) detect conflicts at commit time. Pessimistic transactions detect conflicts immediately, which is useful when you know contention is likely:

const db = RocksDatabase.open('/path/to/db', { pessimistic: true });

‍

For Harper, this unlocks patterns that were difficult to implement safely with LMDB. Complex multi-record updates, conditional writes, read-modify-write cycles, all with proper isolation guarantees that hold up under concurrent access from worker threads.

‍

Range Queries and Lazy Iteration

Range queries return an iterable, not an array. The iteration is lazy, meaning records are only read from disk as you consume them:

// Only reads records between 'user:100' and 'user:200'
for (const { key, value } of db.getRange({ start: 'user:100', end: 'user:200' })) {
  processUser(value);
}

// Array-like methods also work and are evaluated lazily
const activeUsers = db.getRange({ start: 'user:', end: 'user:~' })
  .filter(({ value }) => value.active)
  .map(({ value }) => value.name);

‍

This matters for large datasets. You do not load 10,000 records into memory to find the 50 you care about. The iterator reads from the RocksDB snapshot as you advance through it, keeping memory usage proportional to what you actually process.

‍

The Native Layer

rocksdb-js is a C++ binding that communicates with RocksDB via the Node-API (N-API). It is built directly on the modern Node-API (N-API), avoiding the overhead of legacy abstraction layers like node-addon-api, giving us more direct control over the native interface.

Compiling RocksDB from source takes 10 to 30 minutes, depending on the platform. That is not a great developer experience, so we built an automated pipeline for prebuilds. When you npm install, the package downloads a precompiled RocksDB binary for your platform:

• macOS (arm64 and x64)
• Linux (arm64 and x64 for both glibc and musl C runtimes)
• Windows (arm64 and x64, both the statically and dynamically linked C runtimes)

With prebuilds available, the full install and compile drops to about 12 seconds. Once the prebuild is cached, subsequent builds take around 2 seconds.

rocksdb-js is designed for parallelism and fully supports worker threads, which is how Harper handles concurrent HTTP requests. It works reliably.

‍

Transaction Logs

This is one area where rocksdb-js goes beyond what you would typically find in a key-value store binding. The library includes a built-in transaction log system that records changes in sequenced log files:

const db = RocksDatabase.open('/path/to/db', {
  transactionLogsPath: '/path/to/db/transaction_logs',
  transactionLogMaxSize: 16 * 1024 * 1024, // 16MB per file
  transactionLogRetention: '3d',            // Keep 3 days
});

‍

For Harper, this is the foundation of the replication system. The transaction log drives data synchronization between nodes, real-time messaging over WebSockets and MQTT, and durability. Having this integrated directly into the storage binding rather than bolted on as a separate system is a significant architectural advantage.

But this capability is not Harper-specific. Any application that needs change tracking, event sourcing, or replication can use it.

‍

Why Open Source

rocksdb-js is Apache 2.0 licensed. It powers Harper 5.0, but it was designed from the start to be a standalone, general-purpose library.

The reasoning is straightforward. We have maintained lmdb-js as an open-source library for years, and the value the community gets from having a high-performance, well-maintained key-value store in the Node.js ecosystem is clear. rocksdb-js fills a long-standing gap. The Node.js ecosystem deserves a RocksDB binding that is actively maintained, properly documented, fully featured, and not tied to a legacy abstraction layer.

Open-sourcing it also means the community can shape its direction. There are open issues and areas where contributions would make a real difference. The TypeScript API is comprehensive, and community input will help prioritize where to expand next. Benchmark coverage will get broader. And we know there are use cases we have not thought of.

‍

What We Need From the Community

If you are using RocksDB in Node.js today (or want to), here is where contributions and feedback would be most valuable:

Try it and file issues. The API covers the core surface area, but RocksDB is a massive engine with many configuration knobs. If something does not work the way you expect, or a configuration option you need is not exposed, open an issue.

Benchmarks. We have a benchmark suite and a separate stress test suite for verifying stability under extreme conditions, but more coverage across different workloads, record sizes, and concurrency levels would help everyone make better decisions about when to use RocksDB versus LMDB versus something else.

Custom stores. rocksdb-js supports a custom store interface that lets you override default database interactions. This is how Harper extends the binding for its own needs. If you have a use case that does not fit the default behavior, this is the extension point.

Platform testing. The prebuilds cover the major platforms, but edge cases exist. ARM on Linux and musl-based distros. Any testing and bug reports help.

‍

Get Started

npm install @harperfast/rocksdb-js

The repo is at github.com/HarperFast/rocksdb-js. The README has full API documentation. The library is licensed under Apache 2.0.

‍

If you are building on Harper 5.0, rocksdb-js is already there. If you are building something else entirely, it is yours to use.

Harper 5.0: https://github.com/HarperFast/harper

Join the Discord: https://discord.com/invite/VzZuaw3Xay

‍