Intro: Real-Time Message Capabilities
As a distributed database, Harper has long had the intrinsic ability to publish and deliver real-time data as it is written to our database. This functionality is essential for replicating data across a distributed cluster of Harper nodes. Harper replication is built on a publish/subscribe mechanism, allowing nodes to have real-time message delivery of transactions as they happen, to maintain data synchronization. This functionality is at the heart of Harper and is a robust, battle-hardened, and well-tested message delivery system.
Historically, this has been treated as an internal mechanism in Harper. However, we have increasingly seen the opportunity to leverage these capabilities, and provide convenient interfaces for users to tap into real-time data delivery.
Traditionally databases have functioned as a passive responder to queries. This means that building real-time functionality into applications often involves the integration of a multitude of complex and distinct servers, combining databases with message queue brokers and applications and all the difficulties of spanning these integrations, including authentication, authorization, message translation, notifications and more.
However, we believe the future of databases is real-time. Building real-time applications shouldn't require an extensive integration effort with multiple products. By exposing Harper's notification capabilities, applications can be built on Harper, easily perform traditional queries, and subscribe to real-time data notifications. By adding APIs to subscribe to our data, an entirely new realm of possibilities are unleashed with applications that can retrieve data instantly and monitor it in real-time. And these interfaces then provide the backbone for additional message routing in combination with data notification delivery.
Harper is built with real-time message capabilities. In this post, we wanted to lay out what you can do with these capabilities and the new interfaces and functionality that will be coming in the future to simplify, expand, and accelerate real-time communication.
Harper Custom Function with Stream Subscriptions
Harper’s custom functions enable applications to deliver real-time messages to interested clients using WebSockets.
WebSockets is a powerful transport protocol that facilitates access from web applications and distributed devices and can traverse proxies and layer 7 gateways. Our custom function template demonstrates a simple JSON-based protocol, but numerous application protocols can be layered on top of WebSockets and built from our custom function template, including MQTT, AMQP, and more.
Overview
With this Custom Function, Harper connects to a WebSocket server to facilitate real-time event publication via data subscriptions.
Setup
To set up this Custom Function, follow the steps at the top of the README. Essentially, clone the code, configure the config.json values, and restart.
How It Works
- Use a plugin to create a WebSocket server (@fastify/websocket)
- Use Harper's built-in streaming capabilities (https://docs.harperdb.io/docs/clustering)
- Allow clients to connect and subscribe to topics (schemas, tables, records)
- Deliver messages to connected clients based on their subscriptions
Doing the Same with MQTT
WebSockets is a transport protocol commonly used in web applications, while MQTT is an application protocol utilized for real-time messaging, often in Internet of Things (IoT) devices. MQTT is frequently layered on top of WebSockets to facilitate pub/sub across web applications and HTTP gateways. The above Custom Function design can also be adapted to build a real-time MQTT service with similar fanout capabilities. MQTT topic subscriptions can directly align with subscriptions in this custom function, and MQTT publish commands can map to record insertions or direct message publication within Harper streams.
Coming Capabilities: Real-Time Resource API
For our next release of Harper (4.2) we have built a new interface for interacting directly with databases in real-time. We call this the Resource API; a JavaScript interface that provides a standard interface for CRUD operations and subscribing to records, tables, and queries.
One of the significant benefits of the Resource API is that it provides a consistent path convention for addressing and locating resources and records across both RESTful interface and pub/sub topics. This means that pub/sub topics are directly aligned with database tables and records, and pub/sub protocols can be used to interface directly with the data in the database. There is no extra overhead for routing data between database and message queue.
The Resource API also builds on the custom functions of Harper to facilitate a highly extensible framework for defining application specific logic for CRUD operations as well as handling subscription and publishing requests, facilitating business logic for fine-grained control of access, data aggregation, transformation, and data structures.
MQTT Server Integration
The Resource API also provides the foundation for optimally integrated real-time protocols. We have built an MQTT server that will be included in the release, and other protocols have a direct path to implementation including AMQP, Server-Sent Events, and custom WebSocket protocols.
With the path and topic alignment, by using MQTT, you can easily choose topics that directly map to tables and records. Publishing to a topic publishes to the corresponding table record by primary key, and subscribing allows you to listen for any data changes or publishes on a given record, regardless of whether it was initiated through MQTT or through other database operations like SQL or custom function actions.
MQTT “retain” messages are particularly well-aligned with database backing, since publishing retain message directly maps to updating database records, and a subscribing to record will use the current record state as the retained message. This is the most optimal mechanism for using MQTT in Harper since it so naturally aligns with database access, and allows clients to connect and subscribe to data, and automatically get the current state of the data with maximum integration with other database interactions. This greatly simplifies reliable access without requiring quality of service acknowledgements and session tracking, and optimizing fast access to the latest data (over old data).
Structured Data
Harper is a database, not just a generic broker, and therefore highly adept at handling structured data. Data can be published and subscribed in all supported structured/object formats, including JSON, CBOR, and MessagePack, and the data will be stored and handled as structured data. This means that different clients can individually choose which format they prefer, both for inbound and outbound messages. One client could publish in JSON, and another client could choose to receive messages in CBOR.
Working with messages as structured data in a database greatly expands the way data can be processed. Messages that come into the database can easily be indexed, transformed, aggregated and joined with other data. Working with data at a structured level is just as powerful for message handling as it is for traditional database interactions.
Extensible Protocol Support
Harper’s upcoming release has an extensible plugin architecture, designed to support the additional new protocols, including other real-time protocols that can be built to facilitate more application specific protocols needs. Harper 4.2 includes protocols:
- HTTP - Ubiquitous protocol for web applications, and with REST conventions has powerful and flexible CRUD capabilities (but not a direct real-time protocol itself)
- WebSockets - A bi-directional transport protocol that uses HTTP upgrade mechanism to work within HTTP/web constraints. As a transport protocol, it can be used as an alternative to direct TCP for application protocols like MQTT.
- MQTT - A highly efficient pub/sub protocol designed for extremely lightweight clients with minimal client overhead (designed for IoT). MQTT supports different QoS levels that can employ server tracking to ensure message delivery.
And additional protocols can be built with the plugin system:
- AMQP - A highly robust and powerful messaging and pub/sub protocol that is designed for optimal delivery across a broad range of messages and applications
- Server Sent Events - A very simple protocol that is very easy to use in web applications with direct native browser support. This is a single directional push protocol, and is extremely efficient for single subscriptions in web applications.
We are currently load testing our MQTT server, where it is showing excellent performance and throughput characteristics. Harper is a highly concurrent, multi-threaded server, and the subscription and message delivery system is built on the fastest JavaScript serialization/deserialization library available along with the shared memory mapping capabilities of LMDB, facilitating exceptional speed as records and messages move across threads and are delivered to clients. We will post results as we finish our load testing efforts on more expansive networks.
The Future is Real-Time!
Harper is a distributed database designed for efficient, rapid real-time data distribution. It provides immediate real-time access to data and is expanding with flexible, powerful methods for accessing real-time data and messaging through extensible APIs and standards-based protocols, including MQTT.
%20(1).png){kind=icon}
.png){kind=icon}
.png){kind=icon}