eXtremeDB embedded database was developed to be a powerful tool for the professional developer, offering adaptable architecture, powerful run-time features, and unmatched developer flexibility. Please use the following links to learn more about the features in the DBMS proven to be the fastest hybrid persistent and in-memory database system for edge and cloud.
Adaptable Database Architecture
Powerful Run-Time Features
Unmatched Developer Flexibility
The eXtremeDB database system, feature by feature:
Tiny Footprint (Approximately 200K)
For IoT Edge/embedded systems: Small code size and minimal overhead (database system overhead for metadata is as small as 15% of managed data volume) means less RAM is required; eXtremeDB’s streamlined design permits a lower cost CPU. As a result, your design can use less expensive hardware, resulting in lower manufacturing costs. Or, use the extra CPU speed to offer a snappier user experience than your competition, and use the extra memory to manage more data at the same cost as competing products.
For IoT Server/Big Data systems: Small code size translates to a short execution path, meaning better performance at any clock speed than less efficient database systems. RAM is more expensive than any other storage medium. By using RAM efficiently, eXtremeDB can pay for itself (i.e. license costs are offset by a reduction in hardware costs).
In-Memory or On-Disk Storage
In addition to the core eXtremeDB in-memory database system, McObject’s eXtremeDB offers hybrid storage: on a table-by-table basis, tables can be designated for in-memory or on-disk storage (with flexible caching). Choose the best storage medium based on performance, persistence, cost and form factor.
Core In-Memory Database System (IMDS) Design
eXtremeDB was designed to be an in-memory database system (IMDS) from the beginning (in-memory database capability was not a a jerry-rigged afterthought). This is important because the optimization goals of in-memory and persistent database systems are diametrically opposed. An in-memory database system eliminates disk and file I/O, cache management and other sources of latency. By working with data directly in main memory, eXtremeDB avoids the overhead of making multiple data copies and transfers inherent in disk-based DBMSs. Databases can be created in shared memory, enabling concurrent access by multiple processes. Read about the performance advantages of IMDS technology in the white papers offered from this site.
Columnar Layout for Time Series Data
Traditional DBMSs bring rows of data into the CPU cache for processing. But time series data – such as financial trades and quotes, and IoT sensor data – is naturally columnar, and handled more efficiently by a column-based layout. eXtremeDB for HPC stores time series data with a columnar layout, and “normal” data with a conventional row-based layout. The result is higher performance, resulting from a database system that best leverages I/O and CPU cache speed and avoids costly (in performance terms) fetches from main memory. Click here to read more.
eXtremeDB’s transactions support the ACID (Atomic, Consistent, Isolated and Durable) principles, which safeguard data integrity by guaranteeing that updates will complete together or the database will roll back to a pre-transaction state. For real-time embedded systems, eXtremeDB also enables the developer to prioritize transactions.
eXtremeDB Transaction Logging adds persistence to in-memory databases by writing database changes into a transaction log on persistent media. Logging may be set to different levels of transaction durability, allowing system designers to make intelligent trade-offs between performance and risk of unrecoverable transactions. See the options for in-memory and persistent databases here.
When an eXtremeDB transaction is started, it can be assigned one of 5 priority levels and will be queued accordingly by the eXtremeDB transaction manager. It should be used in concert with real-time operating system task priorities (and care should be taken not to create a priority inversion).
The Time-To-Live (TTL) mechanism facilitates automatic deletion of objects according to TTL policies. Two TTL policies are supported: MaxCount and MaxTime. The former sets an object count threshold, while the latter sets an object age threshold. Both policies can be set for a single class at the same time. Learn more in our online documentation.
Optimistic or Pessimistic Concurrency Control
Another important tool for scalability is eXtremeDB’s optional Multi-Version Concurrency Control (MVCC) transaction manager, which is available as an alternative to the “pessimistic” database locking of the original eXtremeDB MURSIW (MUltiple Reader, SIngle Writer) transaction manager. MVCC can dramatically improve scalability and performance, especially in applications with on-disk or hybrid (in-memory and on-disk) database storage; many tasks or processes concurrently modifying the database (versus read-only); and in multi-core systems. Conversely, for IoT Edge/embedded systems with few concurrent tasks, the much simpler MURSIW transaction manager is usually superior.
eXtremeDB improves on typical Least Recently Used (LRU) cache policies by allowing applications to influence how long certain pages remain in cache, to minimize retrieval overhead for objects used in time-sensitive tasks. When considering whether to remove a page from the cache, eXtremeDB’s LRU algorithm examines the cache priority property set for the page; the higher the priority, the longer the page remains linked to the LRU list (stays in cache). A caching priority of zero is the default.
Deterministic Rule-Based SQL Optimizer
Cost-based SQL optimizers must collect sample data, generate statistics and analyze thousands — sometimes hundreds of thousands — of possible execution plan steps; these requirements make cost-based optimizers’ operation CPU-intensive and unpredictable. In contrast, eXtremeSQL (eXtremeDB’s SQL interface) uses a highly efficient and predictable rule-based optimizer.
RESTful Web Services Support
eXtremeDB’s provides Web services using the popular REST (Representational State Transfer) protocol. The REST server can be accessed using any language that supports HTTP and JSON, which includes but is not limited to C/C++, Python, Java and C#. Note that the REST server is simply an HTTP server that serves JSON content, and thus conforms to the HTTP and JSON standards. Learn more in our documentation.
High Availability and Clustering
Committed to 99.999% (five nines) uptime or better? eXtremeDB High Availability (HA) edition ensures continuous database operation even in the face of hardware or software failure. eXtremeDB HA supports both synchronous (2-safe) and asynchronous (1-safe) replication, with automatic failover. eXtremeDB HA is an active/passive (aka master/slave) architecture; replicas are read-only.
Distributed architectures based on eXtremeDB Cluster are active/active; every database instance serves as a master. Any process on any node can update its local database, and Cluster efficiently replicates changes to other nodes. Clustering can dramatically increase available net processing power, lower system expansion costs (by enabling use of low-cost “commodity” hardware,) and maximize scalability and reliability for data-intensive applications.
McObject’s eXtremeDB Data Relay technology facilitates seamless, fine-grained data sharing between real-time systems based on eXtremeDB, and external systems such as enterprise DBMSs. As part of the eXtremeDB Transaction Logging module, Data Relay helps developers by simplifying the code that “looks inside” database transactions for changes that should be relayed. It also guarantees maximum efficiency by eliminating the CPU-intensive task of monitoring database activity. Sharing of data can be either synchronous or asynchronous.
Active Replication Fabric for IoT
eXtremeDB Active Replication Fabric is McObject’s implementation for data exchange between “Internet of Things” (IoT) edge devices and database storage on remote systems (e.g. public/private cloud servers). Active Replication Fabric has features to accommodate intermittent or unreliable connections, and low bandwidth connections. Read more here.
Binary Schema Evolution
In addition to RDBMS-like CREATE TABLE and ALTER TABLE capability, an eXtremeDB embedded database system can save a database as a binary image and then restore it with a changed schema. This can accomplish design changes more quickly, using less memory and storage, than the alternative approach using XML import/export. Learn more.
eXtremeDB’s persistent storage can exploit multi-disk (solid state or spinning) configurations with its support for RAID-like data striping and data mirroring. Striping accelerates performance when working with two or more disks by parallelizing access to data. Mirroring provides continuous backup by replicating data onto separate disks (if disk A fails, identical records are available from disk B).
eXtremeDB is available for use on all major workstation and server platforms: x32- and x64-bit versions of Windows and Linux, Linux on POWER8, Solaris Sparc, Solaris x86_64, HP-UX (Itanium), and AIX.
eXtremeDB is also available for all major embedded platforms: VxWorks, INTEGRITY, QNX, ThreadX, eCos, FreeRTOS and too many more to mention. Also x86, PowerPC, ARM, MIPS, etc.
eXtremeDB source code is available. The minimum requirement to build eXtremeDB for any platform is a 32-bit CPU and a decent quality C compiler.
Distributed Query Processing
This capability enables partitioning of an eXtremeDB for HPC database and for eXtremeSQL to distribute query processing across multiple servers, CPUs and/or CPU cores. Performance is accelerated – dramatically, in some cases – via parallel execution of database operations and by harnessing the capabilities of many servers, CPUs and CPU cores. Learn more, or see a Distributed Database Contrast and Compare chart.
Graphical Dashboard (xPanel) (click here for more information)
xPanel consists of several graphical interfaces that simplifies eXtremeDB configuration and monitoring.
The xSQL Configuration Tool facilitates creating new, and modifying existing, xSQL JSON-formatted configuration files.
The Feed Handler Configuration Tool simplifies the process of establishing application settings that implement connections to financial market feeds.
The SQL System Analyzer collects and visualizes SQL processing statistics.
The eXtremeDB Performance Monitor provides a comprehensive, real-time view of various factors affecting the performance of eXtremeDB-based systems.
The SQL Query Console provides the capability to execute ad hoc SQL statements and view the results.
The Tracer displays the contents of a trace-log-file that tracks the runtime activity of eXtremeDB modules.
Run-length encoding (RLE) compression can be applied to columnar data (i.e. fields defined as the ‘sequence’ data type). In McObject’s tests, activating this feature reduced storage space requirements by 75% and increased speed in reading the database by 21%. eXtremeDB also includes a feature for compressing non-columnar data.
eXtremeDB protects your database. Cyclic Redundancy Check (CRC) on the database page level detects any unauthorized modification to stored data, while AES encryption employs a user-provided cipher to prevent access or tampering. Secure Sockets Layer (SSL) and Transport Layer Security (TLS) is supported in all communications (client/server, HA and Cluster).
eXtremeDB for HPC delivers on-chip analytics, for breakthrough efficiency working with market data. On-chip analytics is implemented in software, and therefore should not be confused with the practice of optimizing analytics using FPGAs.
On-chip analytics is also substantially different from in-memory computing. While in-memory computing uses main memory as DBMS storage, to eliminate various kinds of latency, on-chip analytics optimizes data flows to and from the CPU and maximizes the amount of relevant data held in CPU cache, to reduce latency-inducing transfers between CPU cache and main memory.
In eXtremeDB for HPC, on-chip analytics can be used independent of the product’s in-memory database system (IMDS) capability.
On-chip analytics in eXtremeDB for HPC has three components:
- The “sequence” data type implements columnar layout for data elements. Sequences are ideal for representing time series such as tick streams.
- A rich library of vector-based statistical functions, to accelerate analysis of these sequences/columns.
- A pipelining technique is used to combine these functions into assembly lines of processing, with output of one function becoming input for the next, eliminating data transfers between CPU cache and main memory for interim data sets (data sets between computation steps).
In other words, pipelining keeps time series data in CPU cache while it is being processed by multiple functions, slashing latency by eliminating transfers between the CPU and main memory.
Learn more about On-Chip Analytics, including practical examples.
Lua for Stored Procedures
Lua is a very elegant, popular and easy-to-adopt scripting language, with extensive grammar that supports operator overloading, encapsulation, inheritance, polymorphism and more. Lua’s sophisticated and blazingly fast dynamic Just-In-Time compiler (LuaJIT) and small footprint makes it a great procedural language to develop complex database user-defined functions and stored procedures for the eXtremeSQL server. Lua-based procedures run in the context of the SQL server and therefore minimize client-server inter-process communication and attendant network overhead, and fully utilize the multi-core nature of modern hardware.
Similar to triggers in a relational database management system, this feature enables eXtremeDB to notify an application when something “of interest” in the database changes. It is available in synchronous and asynchronous modes.
Built-in Feed Handlers
Thomson Reuters and Vela feed handlers: these plug & play feed handlers mean that data loading is seamless, enabling clients to ingest market data feeds into their databases quickly and efficiently. Clients are now also able to build their own data loaders very quickly.
With databases constantly increasing in size, a complete backup/restore can be very time consuming. Incremental backup automatically detects the changes in a database, and only backs those up, speeding up that process. This enhancement dramatically improves the operational efficiency of eXtremeDB.
Time Series Analysis
Time series analysis is a specialized sub-domain of data management, with special rules. Applications that crunch time series data (including market data) benefit from a DBMS tailored to the task.
C, C++, C# Java and Python APIs
eXtremeDB provides the developer with multiple application programming interfaces (APIs). Access eXtremeDB using a fast, native navigational API consisting of C/C++ functions. This is provided as both a type-safe and intuitive application-specific API (functions are generated based on the data design), and as a uniform data access (UDA) API for a consistent interface across all projects. eXtremeDB’s Java Native Interface (JNI) and C# native interface offer the fastest possible DBMS solutions in these languages, and the ability to access eXtremeDB while working entirely with “plain old” Java and C# objects. eXtremeSQL implementation includes JDBC and ODBC drivers for interoperability, or a more succinct and easier to use proprietary API. The Python wrapper provides convenient classes, similar to the Java API, for low-level database access, as well as eXtremeSQL access.
B-Tree, R-Tree, Patricia Trie, KD-Tree, Trigram and Hash Indexes
eXtremeDB provides a wide range of database indexes, to boost application performance and minimize footprint. eXtremeDB offers R-trees for geospatial data, Patricia tries for IP/telecom, KD-trees for multi-dimensional data and Query-by-Example (QBE), B-trees, Trigram index for fuzzy search, hash indexes and more. For in-memory databases, rather than storing duplicate data, indexes contain only a reference to data, keeping memory requirements to an absolute minimum.
Vector-based Statistical Functions
eXtremeDB for HPC provides an extensive library of vector-based statistical functions for financial calculations. Functions execute over one or more sequences of data (time series data, such as trades and quotes or IoT sensor data), cutting latency by maximizing the CPU cache use. Functions can be pipelined to form an assembly line of operations on sequences, to perform statistical/quantitative analysis. Download the white paper Pipelining Vector-Based Statistical Functions for In-Memory Analytics.
eXtremeDB includes hooks that enable developers to provide a desired character sorting sequence (collation) for data stored as text, including collation that supports a particular language or combination of languages. A single application can support search and other text-processing functions in multiple languages.
Wide Range of Supported Data Types
eXtremeDB supports a wide range of data types — including structures, arrays, boolean, binary, decimal, vectors and BLOBs — for maximum coding efficiency. Data can be stored in the same complex form in which it is used in the application, or as normalized relations.
Unmatched Developer Support
McObject technical support engineers are database and real time application experts who answer customer questions in detail. Whether embedded systems beginners with basic configuration questions or seasoned developers looking for hints on optimization, customers get prompt, informative answers and continued follow-up to speed production and get the most from eXtremeDB.
Compared to self-developed (‘homegrown’) data management, eXtremeDB offers a proven solution that slashes months from development, QA and ongoing support. Why take on the complexity and cost of developing a database management system from scratch, when an off-the-shelf solution with >30 million deployments in the field meets your requirements? Read customer testimonials and see a partial client list.
Designed To Prevent Database Corruption
The eXtremeDB native API is type-safe: errors in data-typing are caught at compile time, to eliminate database corruption. In addition, the eXtremeDB runtime implements many verification traps and consistency checks. After application debugging, the optimized version of the eXtremeDB runtime can be used, removing traps and internal checks, and restoring valuable clock cycles.
In addition, cyclic redundancy check is built into the backup-and-restore feature for in-memory databases, executing automatically when a file is loaded to ensure the database was written in its entirety when saved, and has not been corrupted.
See the on demand webinar: Eliminating Database Corruption