Features for a Faster Database

McObject has specialized in database management systems since 2001.  We compiled this list of features you might want to search for if fast data management is a priority for your next project.

eXtremeDB was designed from the beginning to maximize database speed, flexibility and reliability for professional developers.  We hope you find this list of value.

In-memory database systems (IMDSs) offer superior performance and the possibility of very small RAM, CPU and storage demands. IMDSs (or RAM databases) boost speed and reduce footprint by eliminating logical and physical disk I/O, multiple data copies, and irrelevant tasks, such as caching.

In contrast, on-disk databases (HDD/SSD) cache frequently requested data in memory, for faster access, but write database updates, insertions and deletes through the cache to be stored to disk. On the plus side, byte-for-byte, disk storage can be cheaper than RAM, and can also take less physical space.

A hybrid database like eXtremeDB can be all-in-memory, all-persistent, or have a mix of in-memory tables and persistent tables with a simple database schema.

The right index can boost lookup speed logarithmically as well as reduce RAM and CPU demands. While the B-Tree is the best known index, many others can be more efficient in specific circumstances, such as geospatial/mapping and telecom/networking applications.  Less well-known indexes include Hash table, R-Tree, Patricia trie, Trigram and others.

Persistent databases keep recently-used records in cache, so they can be accessed without physical I/O. But managing the cache is itself logic that requires substantial memory and CPU cycles, so even a “cache hit” underperforms an in-memory database.  In-memory databases do not cache data.

As a hybrid database, eXtremeDB can be used all in-memory, all persistent or in combination.  Specifying one set of data as transient (managed in memory), while choosing persistent storage for other record types, requires a simple database schema declaration.

Normally, a database cache starts out empty and fills up as data is written to and/or fetched from the database. This is called a “cold” cache, and while the cache is cold, every search request will result in a cache-miss. Some database systems, including eXtremeDB, allow you to save the cache before shutting down a database, and to reload (or “pre-warm”) the cache on restart of the database.

A few database systems, including eXtremeDB, provide some control over the least-recently-used (LRU) algorithm that is used to determine what data to eject from the cache when it is necessary to bring new data into an already-full cache. Generically, this is called cache prioritization.

Pipelining is the technique in eXtremeDB that accelerates processing by combining the database system’s vector-based statistical functions into assembly lines of processing for time series data, with the output of one function becoming input for the next. Calculations are pipelined in order to keep data within CPU cache during its transformation by multiple functions. Without pipelining, interim results from each function’s transformation would be transferred back and forth between CPU cache and main memory, imposing significant latency.

Threads running in parallel often contend for system resources, actually reducing overall performance. Achieving multi-core’s promised linear performance gains hinges on resolving this issue.  This Webinar, Multi-core and Embedded Software: Optimize Performance by Resolving Resource Contention looks at two such conflicts — contention in updating a shared data store, and threads vying for access to the C/C++ memory manager — in-depth, and addresses them with techniques that can provide the basis to solve other multi-core resource conflicts.

Database pages can be stored in a compressed format and decompressed when accessed by the application. This can maximize available storage space, which could be important for an in-memory database, or even a persistent database stored on a relatively small SSD.

Run-length encoding (RLE) compression can be applied to time series data (i.e. fields defined as the ‘sequence’ data type). In McObject’s tests, activating this feature reduced storage space requirements by 90% and increased speed in reading the database by 21%.  Compression and RLE are separate features that can be used individually.

Data can also be compressed between server and device in eXtremeDB’s Active Replication Fabric for IoT networks. Operating in low-bandwidth networks requires techniques to compress the network traffic. To solve this problem for all network components of eXtremeDB at once, the compression is implemented at the system abstraction layer (SAL).

Relational databases typically store data in tables consisting of rows and columns, and transfer data between storage, main memory and CPU cache by organizing the data into pages and transferring whole pages of rows.

Time series data benefits from organizing data in a column-wise manner, wherein a database page contains only values of a single column of data.  In other words, with a columnar approach, tables’ columns (instead of rows) are the basic unit for constructing the database pages used for DBMS input/output.

eXtremeDB implements columnar data layout for fields of type ‘sequence’. Sequences can be combined to form a time series, ideal for working with tick streams, historical quotes, IoT sensor data and other sequential/chronological data.

eXtremeDB is proven, reliable database management, used in systems that cannot afford to fail from nuclear power plants to military aircraft to financial technology (FinTech).

eXtremeDB sets the standard in database management system reliability.  Industry leaders around the world depend on eXtremeDB for reliable embedded database management.

 Find eXtremeDB in over 30,000,000 deployments worldwide.

McObject has been helping industry leaders optimize their data management strategy since 2001.  See a partial list of customers.