Get ultra-fast financial analytics pipelining vector-based statistical functions
Financial analytics seek to accelerate the task of finding meaningful patterns in market data. As such, in-memory database systems (IMDSs) are an irreplaceable tool for this job. But IMDSs are not new – in fact, they’re almost the “new normal” for financial analytics. The eXtremeDB for HPC data management solution moves analytics beyond the status quo, and into higher profitability for trading organizations via a time-based advantage.
eXtremeDB for HPC offers two key features for a competitive edge in financial analytics: columnar data layout to overcome traditional DBMSs’ (and IMDSs’) shortcomings in dealing with time series data (including market data), and the programming technique of pipelining vector-based statistical functions to process sequences (columns).
These features build on eXtremeDB’s already-fast IMDS architecture and move the focus for optimizing streaming financial analytics into the CPU cache, where columnar layout and pipelining ensures priority-handling of the most time-critical data.
What is pipelining, and how does it reduce latency?
Pipelining is the programming technique in eXtremeDB that accelerates processing by combining the database system’s vector-based statistical functions into assembly lines of processing for market data, with the output of one function becoming input for the next. Calculations are pipelined in order to keep data within CPU cache during its transformation by multiple functions. Without pipelining, interim results from each function would be transferred back and forth between CPU cache and main memory, imposing significant latency due to the relatively lower-bandwidth front side bus (FSB) or quick path interconnect (QPI) between the two.
Throughput between main memory and CPU cache is 3x to 4x slower than the CPU can process data. Traditional DBMSs traverse this bottleneck frequently (twice per function), with the CPU handing off results of each step of a multi-step calculation to temporary tables in main memory.
In contrast, pipelining vector-based statistical functions with eXtremeDB for HPC avoids these hand offs, keeping interim results in CPU cache to reduce database latency.