HPC FAQs

Specialized features include an optional “sequence” data type that implements column-based handling of time series data (typically trades, quotes, etc.), and a library of vector-based statistical functions that execute over sequences and can be pipelined to form an assembly line of processing on the market data. Used together, these features maximize the efficiency of the CPU cache in managing market data. Columnar handling ensures that the largest amount of time-critical data (sequence elements) is brought into the CPU with each fetch, and avoids flooding the CPU cache with irrelevant data. Pipelining of vector-based functions causes intermediate result sets used in processing to remain in CPU cache rather than being output as temporary tables. This eliminates the overhead of back-and-forth trips between the CPU cache and main memory.

McObject launched the eXtremeDB product in 2001. The initial product, a highly efficient in-memory database system with a C/C++ language interface, targeted embedded systems such as telecom and networking equipment, industrial control, aerospace, defense and consumer electronics. Since that introduction, eXtremeDB has gained features including optional persistent storage, high availability and clustering sub-systems, 64-bit support, new programming interfaces, and much more. eXtremeDB is used by hundreds of companies and deployed over 30 million run-time applications and devices. These customers include many financial technology providers who, over the years, identified eXtremeDB as their ideal solution while researching high performance database systems. With feedback from these users – and from our own research identifying the significant challenges in handling market data – McObject developed eXtremeDB for HPC (formerly Financial Edition). It incorporates all capabilities and modules in the existing eXtremeDB product family, and adds features such as the columnar data handling and vector-based statistical functions discussed above.

At its core, eXtremeDB for HPC is an in-memory database system (IMDS): records are stored in main memory, eliminating caching, disk and file I/O, and other significant storage-related latency found in “traditional” on-disk DBMSs. As a result, eXtremeDB operates many times faster than a conventional on-disk database management system, even if the disk-based DBMS is used with a RAM disk (this is discussed in detail in the white paper “Main Memory vs. RAM-disk Databases: A Linux-based Benchmark”, available for download here).

However, eXtremeDB for HPC also enables the developer to selectively add persistent storage, so that a given database design can incorporate anywhere from zero to 100% persistent storage. When used with persistent storage, it  provides caching to keep frequently-accessed records in memory. It also includes features such as cache prioritization to minimize retrieval overhead for objects used in time-sensitive tasks, as well as database striping to accelerate performance, and mirroring to support continuous backup, when working with two or more disks.

Yes. It also implements many SQL extensions to support eXtremeDB-specific features, including the ability to pipeline vector-based statistical functions from SQL. Compared to other SQL databases, eXtremeSQL lends a performance advantage through its highly efficient and predictable rule-based SQL query optimizer. In contrast, most SQL DBMSs rely on a cost-based optimizer that must collect sample data, generate statistics and analyze thousands — sometimes hundreds of thousands — of possible execution plans; these requirements make cost-based optimizers’ operation CPU-intensive and unpredictable.

eXtremeDB also supports ODBC, which expands its ability to share data with systems ranging from mainframes to servers and desktop applications, and enables developers to create applications using an open and familiar database API.

Developers can access eXtremeDB’s most powerful capabilities – including pipelining vector-based statistical functions – from the industry standard SQL database programming language and Python scripting language. Using these languages alongside eXtremeDB for HPC’s dynamic database definition language (DDL) capability, developers can implement their ideas quickly, and optimize rapidly by testing changes to code, database tables and indexes.

Application development (including the use of pipelining and other specialized features) is supported in C/C++, Java and Python. The native, navigational C/C++ API is type-safe and specific to a given project/database design. When coding in C or C++, applications can access the database using this API as well as standard SQL/ODBC. eXtremeDB for HPC’s SQL implementation also provides a type-3, version 4 JDBC driver for development in Java. Its Java Native Interface (JNI) native interface offers the fastest possible DBMS solutions in Java, and affords developers the experience of working entirely with “plain old” Java objects.  C# can be used with the ODBC .NET Managed Provider.  See list of key features to learn more.

eXtremeDB for HPC is available for use on x32- and x64-bit versions of Windows and Linux, Linux on POWER8, Solaris Sparc, Solaris x86_64, HP-UX, and AIX.  

eXtremeDB supports distributed query processing (aka horizontal partitioning) in which queries are divided between shards (instances of the eXtremeDB database server) that execute on multiple servers, CPUs and/or CPU cores. Performance is accelerated by parallel execution of database operations and by harnessing the capabilities of many host computers rather than just one.

eXtremeDB’s code path is very short: a “footprint” of approximately 200K for the core database system points to McObject’s unrelenting focus on eliminating even small sources of latency. If trading organizations will co-locate their IT with exchanges to eliminate network communication distance and its accompanying latency, they’d do well to consider another type of ‘distance’ – software code execution path – which imposes its own latency.  (Please use the following link to download McObject’s CEO’s article on just this topic in Wall Street & Technology magazine.)  Or, read more about eXtremeDB offers a low latency solution.

Many database systems include one or two of the performance-enhancing features mentioned so far, such as in-memory data storage, columnar data handling, fast native APIs as well as high performance SQL, and a short execution path. But you’d be hard-pressed to find all of them in one database system, and solutions offering some of these features may also lack “real” database capabilities such as ACID transactions. In contrast, eXtremeDB offers all of them plus a rich assortment of additional development and run-time features (our key features list is a good place to see the breadth of eXtremeDB’s capabilities).

Equally important, with new NoSQL solutions and databases cropping up so frequently, McObject’s eXtremeDB is a proven technology. McObject can point to a track record of 20 years and over 30,000,000 successful deployments of the database system, in demanding financial technology as well as in mission- and safety-critical applications ranging from aerospace systems to smart grid, defense, consumer electronics, industrial control and telecom/networking infrastructure.  View a partial customer list.

eXtremeDB has been the subject of multiple STAC-M3 benchmark implementations, overseen and audited by the Securities Technology Analysis Center (STAC®). An overview of eXtremeDB’s record-setting performance on the benchmarks, and a link to download the STAC reports, is available here.

Another test of performance, from a partnership between McObject, Kove and Fultech Consulting, implemented complex risk analysis and reporting on a highly compressed timescale. It measured portfolio exposure of 50,000 accounts, each with 3,278 positions, using a 10-year historical value-at-risk (VaR) model, with a target processing time of ten minutes. The technology stack incorporating eXtremeDB for HPC (formerly Financial Edition,) performed the approximately 428 billion calculations in less than 8.5 minutes, handling approximately 13.5 terabytes of data, reduced to 3.5 terabytes of Kove® XPD™ L2 storage, to demonstrate continuous, sustained, deterministic and random financial model processing of approximately 950 million calculations per second and 65 terabytes per hour. Read more.

The documentation includes the eXtremeDB User’s Guide and Reference Guide. Separate addenda are offered for key functions such as High Availability and SQL.

We strongly recommend you start the evaluation process by reading eXtremeDB – Getting Started.  It walks through installation, basic concepts, and first steps in using the core product. Install the software using the provided installation program. Build the samples according to the readme file. Spend some time reading through the User’s Guide and, as different features are discussed, refer to the appropriate samples. Write your own, or expand the provided samples, to exercise your new knowledge. Contact us if you need assistance with the evaluation process. We will make sure that you get answers to all your questions in a timely manner, and will gladly assist you with writing benchmark tests.

McObject also offers a demonstration program based on eXtremeDB for HPC to illustrate how to use the sequence data type (columnar storage for time series data) and creating a pipeline of functions from eXtremeDB’s library of vector-based functions for statistical analysis, with SQL. The program loads a database with 10 years of daily trading history (open, close, high, low, volume) for the NYSE, then executes a complex query to calculate the relative-strength indicator (RSI) and identify stocks for which A) the RSI moved above 50, then below 50 but not less than 45, then above 50 on the last trading day (a bullish signal) and, B) the RSI moved below 50 then above 50 but not more than 55, then below 50 on the last trading day (a bearish signal). It is not necessary to download an eXtremeDB evaluation software package to run the demonstration program; it is entirely self-contained.  Learn more about the demo.

McObject offers an annual technical support and software maintenance contract. This entitles the developer to unlimited technical support through any channel (telephone, email, etc.), free updates (version number updates to the right of the decimal point, e.g. from 7.1 to 8.0) and free upgrades (version number updates to the left of the decimal point, e.g. from 7.1 to 8.0).

Yes. While eXtremeDB is not open source, McObject’s philosophy is to make the source code available and affordable for developers.  Contact us to learn more.