DBMS Features for High Performance Computing
The HPC database system optimized for cloud, analytics and financial data management.
Adaptable Database Architecture
Powerful Run-Time Features
Unmatched Developer Flexibility
The eXtremeDB for HPC features for cloud, analytics and financial systems, feature-by-feature:
Tiny Footprint (as low as 200K)
Small code size and minimal overhead (database system memory footprint is as small as 15% of managed data volume) means less RAM is required; eXtremeDB’s streamlined design permits a lower cost CPU. As a result, your design can use less expensive hardware, resulting in lower manufacturing costs. Or, use the extra CPU speed to offer a snappier user experience than your competition, and use the extra memory to manage more data at the same cost as competing products.
Optional On-Disk or Hybrid Storage
In addition to the core eXtremeDB in-memory data storage, McObject’s eXtremeDB adds hybrid storage: certain record types can be designated for on-disk storage, with flexible caching. Choose the best storage medium based on performance, persistence, cost and form factor.
Core In-Memory Database System (IMDS) Design
As an in-memory database system (IMDS), eXtremeDB gives your application speed without rewrites or expensive new hardware. It eliminates disk and file I/O, cache management and other sources of latency. By working with data directly in main memory, eXtremeDB avoids the overhead of data duplication and transfer inherent in disk-based DBMSs. Databases can be created in shared memory, enabling concurrent access by multiple processes. Read about the performance advantages of IMDS technology in the white papers offered from this site.
Columnar Layout for Time Series Data
Traditional DBMSs bring rows of data into the CPU cache for processing. But financial data – such as trades and quotes – is naturally columnar, and handled more efficiently by a column-based layout. eXtremeDB stores time series data with a columnar layout, and “normal” data with a conventional row-based layout. The result is higher performance, resulting from a database system that best leverages the CPU cache speed and avoids costly (in performance terms) fetches from main memory. Click here to learn more.
ACID Transactions
eXtremeDB’s transactions support the ACID (Atomic, Consistent, Isolated and Durable) principles, which safeguard data integrity by guaranteeing that updates will complete together or the database will roll back to a pre-transaction state. eXtremeDB also enables the developer to prioritize transactions.
Transaction Logging
eXtremeDB Transaction Logging adds recoverability by writing database changes into a transaction log on persistent media. Logging may be set to different levels of transaction durability, allowing system designers to make intelligent trade-offs between performance and risk of unrecoverable transactions. Click here to read more.
Time-to-Live
The Time-To-Live (TTL) mechanism facilitates automatic deletion of objects according to TTL policies. Two TTL policies are supported: MaxCount and MaxTime. The former sets an object count threshold, while the latter sets an object age threshold. Both policies can be set for a single class at the same time. Learn more in our online documentation.
Multi-Version Concurrency Control (MVCC)
Another important tool for scalability is eXtremeDB’s optional Multi-Version Concurrency Control (MVCC) transaction manager, which is available as an alternative to the “pessimistic” database locking of the original eXtremeDB MURSIW (MUltiple Reader, SIngle Writer) transaction manager. MVCC can dramatically improve scalability and performance, especially in applications with on-disk or hybrid (in-memory and on-disk) database storage; many tasks or processes concurrently modifying the database (versus read-only); and in multi-core systems. Click here to read more.
Cache Prioritization
eXtremeDB improves on typical Least Recently Used (LRU) cache policies by allowing applications to influence how long certain pages remain in cache, to minimize retrieval overhead for objects used in time-sensitive tasks. When considering whether to remove a page from the cache, eXtremeDB’s LRU algorithm examines the cache priority property set for the page; the higher the priority, the longer the page remains linked to the LRU list (stays in cache). A caching priority of zero is the default.
Deterministic Rule-Based SQL Optimizer
Cost-based SQL optimizers must collect sample data, generate statistics and analyze thousands — sometimes hundreds of thousands — of possible execution plans; these requirements make cost-based optimizers’ operation CPU-intensive and unpredictable. In contrast, eXtremeSQL (eXtremeDB’s SQL interface) uses a highly efficient and predictable rule-based optimizer.
RESTful Web Services Support
eXtremeDB provides Web services using the popular REST (Representational State Transfer) protocol. The REST server can be accessed using any language that supports HTTP and JSON, which includes but is not limited to C/C++, Python, Java and C#. Note that the REST server is simply an HTTP server that serves JSON content, and thus conforms to the HTTP and JSON standards. Learn more in our online documentation.
High Availability and Clustering
Committed to 99.999% uptime or better? eXtremeDB High Availability (HA) module ensures continuous database operation even in the face of hardware or software failure. eXtremeDB HA supports both synchronous (2-safe) and asynchronous (1-safe) replication, with automatic failover.
In distributed architectures based on eXtremeDB Cluster, every database instance serves as a master. Any process on any node can update its local database, and Cluster efficiently replicates changes to other nodes. Clustering can dramatically increase available net processing power, lower system expansion costs (by enabling use of low-cost “commodity” hardware), and maximize scalability and reliability for data-intensive applications.
Open Replication (eXtremeDB Data Relay)
McObject’s eXtremeDB Data Relay technology facilitates seamless, fine-grained data sharing between embedded systems based on eXtremeDB, and external systems such as enterprise DBMSs. As part of the eXtremeDB Transaction Logging edition, Data Relay helps developers by simplifying the code that “looks inside” database transactions for changes that should be relayed. It also guarantees maximum efficiency by eliminating the CPU-intensive task of monitoring database activity. Sharing of data can be either synchronous or asynchronous.
Binary Schema Evolution
eXtremeDB can save a database as a binary image and then restore it with a changed schema. This can accomplish design changes more quickly, using less memory and storage, than the alternative approach using XML import/export. Learn more.
Database Striping/Mirroring
eXtremeDBs persistent storage can exploit multi-disk (solid state or spinning) configurations with its support for RAID-like data striping and data mirroring. Striping accelerates performance when working with two or more disks by parallelizing access to data. Mirroring provides continuous backup by replicating data onto separate disks (if disk A fails, identical records are available from disk B).
Multi-Platform
eXtremeDB is available for use on x32- and x64-bit versions of Windows and Linux, Linux on POWER8, Solaris Sparc, Solaris x86_64, HP-UX (Itanium), and AIX. Learn more.
Distributed Query Processing
This capability enables eXtremeDB to partition a database and distribute query processing across multiple servers, CPUs and/or CPU cores. Performance is accelerated – dramatically, in some cases – via parallel execution of database operations and by harnessing the capabilities of many host computers. Learn more.
Rapid Development in SQL and Python
Developers can access eXtremeDB’s most powerful capabilities – including pipelining vector-based statistical functions – from the industry standard SQL database programming language and Python scripting language. Using these languages alongside eXtremeDB’s dynamic database definition language (DDL) capability, developers can implement their ideas quickly, and optimize rapidly by testing changes to code, database tables and indexes.
C/C++, SQL, Java and Rust APIs
eXtremeDB provides the developer with multiple application programming interfaces (APIs). Access eXtremeDB using a fast, native navigational API consisting of C/C++ functions. This is provided as both a type-safe and intuitive application-specific API (functions are generated based on the data design), and as a uniform data access (UDA) API for a consistent interface across all projects. eXtremeDB’s SQL implementation includes an ODBC API for interoperability, a more succinct and easier to use proprietary API, and a type 3, version 4 JDBC driver. eXtremeDB’s Java Native Interface (JNI) offers the fastest possible DBMS solution in Java, and the ability to access eXtremeDB while working entirely with “plain old” Java objects. Rust is blazingly fast and memory-efficient: with no runtime or garbage collector, it can power performance-critical services and run on embedded devices. It is syntactically similar to C++ and easily integrates with other languages. C# can be used with the ODBC .NET Managed Provider.
Graphical Dashboard (xPanel) (click here for more information)
xPanel consists of several graphical interfaces that simplifies eXtremeDB configuration and monitoring.
Web Service Connectors provide RESTful web service connections to the following xPanel tools:
The xSQL Configuration Tool facilitates creating new, and modifying existing, xSQL JSON-formatted configuration files.
The Feed Handler Configuration Tool simplifies the process of establishing application settings that implement connections to financial market feeds.
The SQL System Analyzer collects and visualizes SQL processing statistics.
The eXtremeDB Performance Monitor provides a comprehensive view of various factors affecting the performance of eXtremeDB-based systems.
The SQL Query Console provides the capability to execute ad hoc SQL statements and view the results.
The Tracer displays the contents of a trace-log-file that tracks the runtime activity of eXtremeDB modules
Market Data Compression
Run-length encoding (RLE) compression can be applied to columnar data (i.e. fields defined as the ‘sequence’ data type). In McObject’s tests, activating this feature reduced tick data to one-quarter of its pre-compression size, and increased speed in reading the database by 21%. (eXtremeDB for HPC also includes a feature for compressing non-columnar data.)
Security Features
eXtremeDB protects your database. Cyclic Redundancy Check (CRC) on the database page level detects any unauthorized modification to stored data, while AES encryption employs a user-provided cipher to prevent access or tampering. Secure Sockets Layer (SSL) is supported in all communications (client/server, HA and Cluster).
In-database Analytics
eXtremeDB for HPC delivers in-database analytics, for breakthrough efficiency working with market data. In-database analytics are implemented in software, and therefore should not be confused with the practice of optimizing analytics using FPGAs.
In-database analytics also differ substantially different from in-memory computing. While in-memory computing uses main memory as DBMS storage, to eliminate various kinds of latency, in-database analytics optimize data flows to and from the CPU and maximizes the amount of relevant data held in CPU cache, to reduce latency-inducing transfers between CPU cache and main memory.
In eXtremeDB for HPC, in-database analytics can be used independent of the product’s in-memory database system (IMDS) capability.
In-database analytics with eXtremeDB for HPC has three components:
- The “sequence” data type implements columnar layout for data elements. Sequences are ideal for representing time series such as tick streams.
- A rich library of vector-based statistical functions, to accelerate analysis of these sequences/columns.
- A pipelining technique is used to combine these functions into assembly lines of processing, with output of one function becoming input for the next.
Pipelining keeps market data in CPU cache while it is being processed by multiple functions, slashing latency by eliminating transfers between the CPU and main memory.
Learn more about In-database Analytics, including practical examples.
Lua, Python and C++ for Stored Procedures
Lua is a very elegant, popular and easy-to-adopt scripting language, with extensive grammar that supports operator overloading, encapsulation, inheritance, polymorphism and more. Lua’s sophisticated and blazingly fast dynamic Just-In-Time compiler (LuaJIT) and small footprint makes it a great procedural language to develop complex database user-defined functions and stored procedures for the eXtremeDB SQL server.
Lua, Python or C++ based stored procedures run in the context of the SQL server and therefore minimizes client-server inter-process communication and attendant network overhead, and fully utilizes the multi-core nature of modern hardware.
Built-in Feed Handlers
Thomson Reuters and Vela feed handlers: these plug & play feed handlers mean that data loading is seamless, enabling clients to ingest market data feeds into their databases quickly and efficiently. Clients are now also able to build their own data loaders very quickly.
Incremental Backup/Restore
With tick databases constantly increasing in size, a complete backup/restore can be very time consuming. eXtremeDB for HPC, v.7.1 and higher, automatically detects the changes in a database, and only backs those up, speeding up that process. This enhancement dramatically improves the operational efficiency of eXtremeDB. Learn more in our online documentation.
Sharding
Shard-per-thread: sharding is a technique to scale performance through horizontal distribution of data. New for eXtremeDB, shard-per-thread is a very efficient and simpler method of scaling on a multi-core server. eXtremeDB also scales efficiently across multiple physical and virtual servers.
Time Series Analysis
Time series analysis is a specialized sub-domain of data management, with special rules. Applications that crunch time series data (including market data) benefit from a DBMS tailored to the task.
eXtremeDB performs time series analysis with record-setting performance through two capabilities: columnar data layout and pipelining vector-based statistical functions.
B-Tree, R-Tree, Patricia Trie, KD-Tree and Hash Indexes
eXtremeDB provides a wide range of database indexes, to boost application performance and minimize footprint. eXtremeDB offers R-trees for geospatial data, Patricia tries for IP/telecom, KD-trees for multi-dimensional data and Query-by-Example (QBE), B-trees, hash indexes and more. For in-memory databases, rather than storing duplicate data, indexes contain only a reference to data, keeping memory requirements to an absolute minimum.
Source code available
Available source code provides the ultimate transparency for developers using eXtremeDB. Use it for porting, learning, or to add a needed function. Contact us to learn more.
Vector-based Statistical Functions
eXtremeDB provides an extensive library of over 150 vector-based statistical functions for financial calculations. Functions execute over one or more sequences of data (typically time series data, such as trades and quotes), cutting latency by maximizing the CPU cache use. Functions can be pipelined to form an assembly line of operations on sequences, to perform statistical/quantitative analysis.
Custom Collations
eXtremeDB includes hooks that enable developers to provide a desired character sorting sequence (collation) for data stored as text, including collation that supports a particular language or combination of languages. A single application can support search and other text-processing functions in multiple languages.
McObject technical support engineers are database and real time application experts who answer customer questions in detail. Whether embedded systems beginners with basic configuration questions or seasoned developers looking for hints on optimization, customers get prompt, informative answers and continued follow-up to speed production and get the most from eXtremeDB.
Proven Solution
Compared to self-developed (‘homegrown’) data management, eXtremeDB offers a proven solution that slashes months from development, QA and ongoing support. Why take on the complexity and cost of developing a database management system from scratch, when an off-the-shelf solution meets your requirements?
Designed To Prevent Database Corruption
The eXtremeDB native API is type-safe: errors in data-typing are caught at compile time, to eliminate database corruption. In addition, the eXtremeDB runtime implements many verification traps and consistency checks. After application debugging, the optimized version of the eXtremeDB runtime can be used, removing traps and internal checks, and restoring valuable clock cycles.
In addition, cyclic redundancy check is built into the backup-and-restore feature for in-memory databases, executing automatically when a file is loaded to ensure the database was written in its entirety when saved, and has not been corrupted.
Broad platform support
eXtremeDB operates on multiple platforms including all major server and real-time operating systems (RTOSs), and can even run without an OS in “bare bones boards” configurations. It is designed to minimize reliance on OS-specific features, in order to simplify the task of porting to new operating systems.