Time Series and Columnar Data

The sequence data type

Some data lends itself to columnar storage, such as time series. eXtremeDB includes support for ordered data (time series) as well as a library of vector-based analytics functions and SQL extensions to support SQL-style operations over time series data. eXtremeDB combines conventional row-oriented storage with column-oriented storage for data declared as a sequence.

A sequence is an unbounded array of eXtremeDB-supported scalar data elements (i.e. [u]int[1234], float and double). In the database schema, you name and define the sequence type and possibly specify whether it is ordered (descending or ascending). Multiple sequences are grouped together as elements of a class, which effectively forms a time series. Sequences are a columnar organization of data.

Furthermore, sequences are stored in tiles which, in turn, are sized to also take advantage of L1 cache. In other words, an entire sequence will often be loaded into the L1 data cache, eliminating multiple fetches from RAM across the much slower front-side bus.

(Please see page Using Sequences for details on usage and implementation of sequences.)

Design goals

Representative application domains include (but are not limited to) ticker plants, algorithmic trading, quantitative analysis, matching engines, and risk management. So the eXtremeDB design goals are driven by the requirements of these target applications, which are typically high transaction rate programs with:

• low latency requirements,
• high volume data streams and, potentially,
• complex analytical processing.

To these goals was added the need to provide an easy-to-use approach to database application development and to give the developer the maximum possible configurability and control over the performance and impact of the database runtime within their applications. So, with these overriding requirements, the guiding design principles include:

• Minimize resource requirements — essentially memory. Object sizes are kept small—the overhead that eXtremeDB introduces is low and, to a degree, controllable from the application
• Keep code size low
• Eliminate extra layers of code by closely integrating database storage and the host application language. Target applications often have many very small database transactions rather than large ones. That means that the operation to obtain data from an object, given a pointer or reference to the object, has to be very fast, otherwise the overhead cost (such as the cost of sending a “message”) could become prohibitive. The eXtremeDB data access methods make it possible to reference database objects at nearly the same speed as that of program variables
• Provide native support for dynamic data structures such as time series, variable-length strings, lists and trees. eXtremeDB extends the C/C++ language by supporting dynamic data in a very efficient (fast), safe (transactional), and compact (memory) way. Likewise, the Java API provides native Java language “data-enabling” and the .NET API integrates with the C# language and enhances LINQ query performance.

 

---