Data Relay and Persistent Queue of Events

eXtremeDB Transaction Logging includes two features beyond providing persistence to in-memory databases.

Data Relay is the open replication mechanism for eXtremeDB. Through High Availability and Cluster, eXtremeDB can replicate transactions between two eXtremeDB instances. But, it is sometimes desirable to replicate data from eXtremeDB to a 3rd party system that eXtremeDB knows nothing about. For example, when used in a call management system, upon the completion of a call data is replicated from eXtremeDB to an Oracle-based customer account management system, e.g. to decrement the customer’s minutes, prepaid balance or other metrics. In a streaming trading application, eXtremeDB data is replicated to a SQL server-based customer account management system to reflect position changes in portfolios. In networking gear, changes to quality-of-service parameters need to be replicated to non-DBMS technology stack components.

To facilitate this need, eXtremeDB provides the Data Relay feature, which gives the programmer the means to create a pipe or file into which eXtremeDB writes transaction data and markers that can be read by the application. The pipe method is streaming, whereas the file method is batch (processed after the log file has been closed). The application programming interface (API) is straightforward: there is a function to establish the pipe or file (mco_translog start), a function to read (mco_translog_iterate), and a function to stop logging (mco_translog_stop). Pipes can be instantiated and torn-down dynamically, providing complete flexibility.

Persistent Queue of Events is an extension of Data Relay.  If a database schema defines events, the event handlers may be registered and fired during the reading of the log file or pipe in addition to or instead of iterating the database objects themselves. Utilization of the Persistent Queue of Events is, by its nature, asynchronous handling of events, but unlike the core eXtremeDB asynchronous event handling, there is no possibility to miss an event because event data are written to the pipe or file and handled in turn. Further, this feature can be used to limit the amount of data written to the pipe/file by providing an event mask to indicate that only data associated with certain events (or all events) is written (versus all transaction data being written, which is the default). Persistent Queue of Events can also be the foundation upon which you can build a publish-and-subscribe architecture.