Low Latency Database Technology for Big Data, Capital Markets and IoT
How did NSE.IT – the 100%-owned IT consulting subsidiary of India’s National Stock Exchange – achieve sub-millisecond latency in its algorithmic trading solution? Download the case study (PDF)
Today’s IoT and capital markets depend heavily on applications’ underlying approach to managing information. But conventional relational database management systems (RDBMSs) cannot deliver the speed, reliability and flexibility these applications demand.
McObject’s eXtremeDB for HPC provides an alternative: a low latency database system for time series data such as that found in financial applications and the Internet of Things. Built on a core in-memory database system (IMDS) design with advanced features to deliver high performance, scalability and reliability, eXtremeDB for HPC is optimized for handling time series data such as trades and quotes, sensor data, and all other types of streaming data. Hybrid storage (in-memory and/or persistent) supports management of both real-time, streaming data and historical data with a single database architecture.
Choosing a time series database for analytics, tick database, risk management
For streaming data, eXtremeDB delivers low latency database management via a highly efficient in-memory database system (IMDS) design that removes the I/O, cache management, data transfer and other sources of DBMS latency.
For historical or OLTP data, offers a wide array of performance-enhancing features, such as pre-warming the cache, cache prioritization, and many more.
When deployed as an embedded database system, its in-process architecture eliminates costly (in performance terms) inter-process communication. When deployed as a client/server database system, eXtremeDB offers tremendous scalability.
In addition, eXtremeDB is optimized with internal transaction and memory managers to provide maximum efficiency working with multi-threaded applications on multi-processor systems. Support for clustering accelerates processing by enabling multiple servers to share the workload.
Scalable low latency database processing. Time series data management systems frequently require large volumes of information to be available continuously. eXtremeDB excels in this capability. McObject’s benchmark demonstrates nearly linear scalability of the 64-bit eXtremeDB, as database size grows to 1.17 Terabytes (15.54 billion rows) on a 160-core Linux server.
Specialized features. The product’s features for handling time series data include support for columnar data layout, which improves efficiency in handling time-series data such as ticks and quotes. System designers can combine a columnar layout with traditional rows in a single database design, to optimize run-time efficiency. A rich library of vector-based statistical functions cuts latency in time series analytics and maximizes efficiency of L1/L2 cache use. GUI-based database performance monitoring enables the user to view key metrics such as transaction time and throughput when fine-tuning to reduce latency.
Interoperability. eXtremeDB ’s fast, native C/C++ API interoperates fully with its SQL API. The native interface is ideal for time-sensitive operations while the SQL API (with its JDBC & ODBC support) permits higher level access and interfacing with external systems. eXtremeDB Data Relay enables open, highly selective replication between a system’s low latency database and external systems such as enterprise DBMSs.
Persistence and availability. Applications must ensure data integrity, persistence and availability. eXtremeDB for HPC achieves these goals through ACID transactions, optional transaction logging, and selective volatile or persistent storage via the product’s hybrid storage capability. High availability support enables deployment of non-stop fault-tolerant systems based on multiple, synchronized database copies, with application-directed failover.