“eXtremeDB is well-designed to solve the unique development challenges of data management for real-time embedded software.” — Schneider Electric
eXtremeDB Technical FAQs
We hope that you find the following information helpful. If you have any further questions please call us at 425-888-8505 or email us at [email protected].
What performance can I expect from eXtremeDB?
The performance of the eXtremeDB runtime depends on many variables such as the processor and memory used, data access pattern, etc. In general, as an in-memory database, eXtremeDB operates many times faster than a conventional on-disk database management system, even if the disk-based DBMS is used with a RAM disk (this is discussed in detail in the white paper “Main Memory vs. RAM-disk Databases: A Linux-based Benchmark”. The link will trigger a download).
McObject’s policy is to assist our customers with the evaluation and benchmarking of eXtremeDB on the platform of interest. We are always ready to construct a reasonable test for your particular environment and application.
What performance can I expect from eXtremeSQL?
Like that of the eXtremeDB core runtime, the SQL module’s performance varies from one environment to another. All things being equal, though, eXtremeSQL gains a performance advantage through its use of a highly efficient and predictable rule-based SQL query optimizer. In contrast, most SQL DBMSs rely on a cost-based optimizer that must collect sample data, generate statistics and analyze thousands — sometimes hundreds of thousands — of possible execution plans; these requirements make cost-based optimizers’ operation CPU-intensive and unpredictable.
What is the largest database that eXtremeDB can manage?
In 32-bit systems, an eXtremeDB in-memory database can contain as many objects as you can create in the available memory, with a user-configurable number of database connections (the default is 64). Note that a 32-bit operating environment is constrained by the amount of memory that can be addressed in a 32-bit pointer, usually 2GB or 3GB. So the practical in-memory database size limitation is typically imposed by the operating system, not eXtremeDB. An eXtremeDB on-disk database on 32-bit systems is limited to 4GB per database file by default, but there can be any arbitrary number of files that collectively represent an eXtremeDB database. The eXtremeDB source code can be recompiled for larger file support, but it increases the storage overhead imposed by eXtremeDB.
64-bit eXtremeDB on a 64-bit OS overcomes the 2- or 3-GB in-memory database size limitation. More details can be found on our Technical Specifications page.
What is the difference between the “direct pointer (DPTR)” and “offsets (OFFS)” versions of eXtremeDB?
The DPTR (direct pointer) version of eXtremeDB is most often used when an in-memory database is created in conventional (“local” or “heap”) memory. This is suitable for operating systems with a single-process, multi-task architecture (e.g. VxWorks 5.x) or where the application’s requirements are possibly multi-threaded but do not include concurrent access from multiple processes. The DPTR library uses pointer arithmetic to locate objects in memory. Shared memory can be used in this circumstance, but it can be difficult to map the shared memory segment to the same starting address within each process. The OFFS version of eXtremeDB places the database in shared memory that need not be mapped to the same local address space of each process. This approach is better suited when there will be multiple processes accessing the database concurrently on systems like Linux, Unix and Windows. The OFFS library locates objects in memory by calculating an offset from the beginning of the mapped memory. Calculating an offset takes approximately 5% longer than pointer arithmetic.
Does eXtremeDB support transactions?
Yes. eXtremeDB transactions support the ACID properties (ACID refers to the principles of Atomicity, Consistency, Isolation and Durability). eXtremeDB in-memory database transactions are durable in the sense that transactions are durable so long as the database itself is durable. Durability can be strengthened by periodically saving the in-memory database and, optionally, through the introduction of transaction logging. It should be noted that all databases are durable only as long as the media that they are stored in is durable, i.e. if a SSD or HDD fails, any database stored on it is no longer durable.
Does eXtremeDB support multi-threaded concurrent access (simultaneous readers and writers)?
Yes. eXtremeDB is fully re-entrant and thread-safe. The default multiple reader, single writer (MURSIW) transaction manager coordinates concurrent access, and concurrency requires no explicit action on the part of the developer other than to use eXtremeDB’s programming interface for starting, committing and aborting transactions. eXtremeDB’s optional multi-version concurrency control (MVCC) transaction manager eliminates pessimistic database locking and can dramatically improve scalability and performance when multiple tasks or processes are concurrently modifying the database (versus read-only), especially on multi-core hardware. A more detailed discussion of the two transaction managers, with benchmark results comparing their performance, is available here.
What programming interfaces are offered with eXtremeDB?
The most popular interface for working with eXtremeDB in embedded/IoT systems is its fast, native navigational application programming interface (API) consisting of C/C++ functions. This is provided as both a type-safe and intuitive application-specific API (functions are generated based on the data design), and as a uniform data access (UDA) API for a consistent interface across all projects.
eXtremeSQL includes a Java Database Connectivity (JDBC) driver and also supports ODBC, both of which expand eXtremeDB’s ability to share data with systems ranging from mainframes to servers and desktop applications, and enable developers to create new eXtremeDB-based applications using an open and familiar database API.
In addition, eXtremeDB’s Java Native Interface (JNI), C# native interface, and Python interface offer the fastest possible DBMS solutions in these languages, and the ability to access eXtremeDB while working entirely with “plain old” Java, C# and Python objects.
What is the “application-specific API” described above, and why would I want to use it?
With its native C/C++ API, eXtremeDB creates a unique programming interface for each database, derived from the data definition language (DDL), or database schema. This approach offers many advantages in areas including type safety, programming ease, application maintainability, determinism (predictability) and even learning curve. These benefits are described in a white paper titled “SQL or Navigational Database APIs: Which Best Fits Embedded Systems?” available for download.
Are duplicate/unique/compound/descending keys (indexes) supported?
Yes. eXtremeDB supports all of the above index types, as well as hash-based indexes, user-defined Object Identifiers (OIDs) and system-generated Auto-Ids (similar to auto-increment or serial data types in other DBMSs). eXtremeDB also offers Patricie Trie, R-Tree, Trigram and KD-Tree indexes.
I am confused with the cursor behavior; could you explain the navigation rules?
An eXtremeDB tree index is a tree-like structure in the application’s memory that contains a collection of keys. A key can be a simple one (based on a single field) or a compound one that contains more than one field. The tree-index stores the keys in sorted order. In order to support sorting, the tree-index must have a compare function. Regardless of the key type (simple or compound), the compare function must be able to compare two keys and determine whether the first key is less (compare returns -1), equal (0) or greater (1) than the second key. The tree-index algorithm sorts the keys based on this comparison. It is important to note that the tree-index algorithm sorts the keys based on their relative weight, determined from the compare function results, rather than on the key’s value. For more details, see the “Index Search and Navigation” section of the eXtremeDB User’s Guide.
Does eXtremeDB have a data definition language?
Yes. eXtremeDB has a data definition language (DDL) and a database dictionary. The native DDL looks very much like C++. It is created with a text editor and processed by our schema processor—which generates .h and .c files that contain the programming interface for the database—and is compiled and linked with the rest of the application code. eXtremeDB also supports databases defined using the SQL DDL (e.g. CREATE TABLE, CREATE INDEX, etc), and databases defined using Java or C# class definitions. A database defined in any language can be used with any other language.
Does eXtremeDB support complex data structures?
Yes. There is no theoretical limit to the complexity of the data you can define with eXtremeDB’s DDL. eXtremeDB supports C/C++ structs, nested structs, fixed and variable length arrays of any supported data type (including structs), and BLOBs.
What is the code size of eXtremeDB?
eXtremeDB’s footprint varies by CPU and compiler, but is approximately 200K (less if features/capabilities are removed from the database system source code). eXtremeDB extensions add to the footprint.
How much overhead does eXtremeDB introduce for its own metadata?
The overhead imposed by the eXtremeDB core runtime varies based on the application data layout, the number of indexes used and other factors. eXtremeDB can add as little as 20% to the size of the raw data for its own pointers and other meta data required to maintain indexes and other references to the data.
What options for persistence are offered by eXtremeDB?
eXtremeDB offers a number of persistence options. First of all, the core eXtremeDB has functions to write a database image to a stream such as a socket, a file or a pipe, and to load a database from a previously saved image. These operations run within the context of a transaction, assuring that a transaction-consistent image of the database is written (or read). In simple terms, you can capture a ‘snapshot’ of an in-memory database, on demand.
Second, the eXtremeDB Transaction Logging Edition runtime captures database changes and writes them to a file known as a transaction log. In the event of a hardware or software failure, the eXtremeDB runtime can recover the database using the last snapshot and the subsequent log. McObject’s benchmark tests confirm that the eXtremeDB database system with transaction logging provides a significant speed advantage over traditional “on-disk” database storage.
Third, eXtremeDB High Availability Edition maintains a master database and one or more synchronized replica databases within separate address spaces, with failover (i.e. an active/passive model). Typical hardware configurations include:
– Multiple processes or threads within the same hardware instance
– Two or more boards in a chassis with a high-speed bus for communication
– Separate address spaces on boards connected via industry standard communication media and protocol such as RS-485/RS-232, Control Area Network (CAN) or Ethernet
– Two or more controllers connected via a proprietary communication media and/or protocol
– Separate computers on a LAN
Fourth, the eXtremeDB Cluster feature manages databases across multiple hardware nodes, enabling two or more servers to share the workload. In an eXtremeDB Cluster deployment, every database instance serves as a “master” (i.e., and active/active model). Any process on any node can update its local database, and McObject’s clustering software replicates the changes to other nodes.
Fifth, eXtremeDB supports in-memory databases created in non-volatile memory (NVRAM). The runtime recovers the in-memory database after failure, provided that the in-memory image is in battery-backed RAM which retains its contents in the absence of system power (this includes newer generation of NVRAM that employ super capacitors in lieu of batteries).
Finally, eXtremeDB offers on-disk data storage alongside in-memory storage in a single database system. By specifying on a table-by-table basis which of a database’s records will be stored in memory and which will be cached in memory but ultimately saved to persistent storage, developers can take advantage of the most cost-effective and physical space-conserving approaches to data storage (byte-for-byte, disk storage can be cheaper than memory, and can also take less physical space).
Under what circumstances does the eXtremeDB runtime call malloc?
The eXtremeDB core runtime does not call malloc. Instead, eXtremeDB implements its own, highly optimized memory managers. Use of these custom memory managers improves performance, reliability and even portability. For an in-depth look at eXtremeDB’s approach to memory management, see this Dr. Dobb’s Journal article by McObject’s co-founder/CTO.
How difficult is it to port eXtremeDB to different operating systems?
eXtremeDB is designed and implemented to be highly portable. Currently, eXtremeDB runs on Win32 and Win64 platforms, Linux, many Unix platforms, and various embedded operating systems including VxWorks, INTEGRITY and QNX, and target CPUs including x86, PowerPC, ARM, MIPS, and others. The list of supported platforms is constantly expanding. Learn more.
What compilers are used to build eXtremeDB?
The eXtremeDB runtime can be built with the Microsoft Visual C/C++ compiler, gcc and all ANSI C compliant compilers. The eXtremeDB Data Definition Language compiler is currently built with Microsoft Visual Studio for Windows and gcc for Unix/Linux platforms.
Can eXtremeDB operate without an operating system?
Yes, in single-threaded mode. In this scenario, the eXtremeDB runtime does not use any operating systems services, or the C runtime library.
Can eXtremeDB operate without a file system?
eXtremeDB’s core in-memory database system design does not require any file system. If a file system is present, to allow data persistence through a system crash, eXtremeDB provides backup functionality via its stream-oriented “save” and “load” interfaces. eXtremeDB can backup databases not only to a file, but also to any stream that is available in the operating environment, such as a pipe or a socket.
What capabilities are present in eXtremeDB for application/database debugging?
eXtremeDB is distributed in two versions: “debug” and “release.” In debug mode, eXtremeDB provides excellent debugging capabilities, with multiple traps throughout the eXtremeDB code that facilitate detection of application errors. For example, if an incorrect pointer is passed into eXtremeDB, chances are that a fatal exception will be raised by the eXtremeDB library before the pointer is used, which makes it possible to examine the call stack and find the source of the corruption.
In addition, eXtremeDB’s native, application-specific C/C++ API is type-safe. Methods that are generated to provide access to a certain object type always take a reference to that data type as a parameter. If a mistake is made, a compiler error will be generated. In release mode, all the above traps are removed from the eXtremeDB runtime code, and many functions inside the runtime are inlined. That can significantly increase the performance of the eXtremeDB -based application. (The software quality benefits of a type-safe API are discussed in this Linux Journal article by McObject’s co-founder/CEO).
What options does eXtremeDB provide for database replication?
eXtremeDB High Availability (HA) implements database replication. The HA edition offers master-slave (active/passive) replication, with one “main” node and one or more synchronized replicas, with fail-over. eXtremeDB HA is a popular choice when the principal goal of database replication is to maximize system up time.
In eXtremeDB Cluster deployments, every node serves as a master (active/active). The feature Cluster also has a “local tables” option that exempts specified eXtremeDB tables from cluster-wide replication, to accelerate processing at the local node. Local tables are shareable, when desired, through a scatter/gather mechanism. eXtremeDB Cluster provides fault-tolerance; it also dramatically increases available net processing power, reduces costs by enabling system expansion via low-cost (i.e. “commodity”) servers, enables more evenly balanced workloads, and delivers more scalable data management.
A third option is the eXtremeDB Data Relay technology included with eXtremeDB Transaction Logging. Data Relay enables selective data sharing between real-time systems based on eXtremeDB, and external systems such as enterprise DBMSs. It “opens up” eXtremeDB’s transaction buffer; for every object affected by a transaction, the buffer provides a coded value that indicates whether each operation was an insert, update, or delete. Data Relay provides a database cursor that iterates over objects in the buffer and extracts the coded value. Based on this reading, the application can take the next step of drilling down into the object, analyzing changes and determining whether to replicate the change to an enterprise system.