Will The Real In-memory Database Please Stand Up
From the in-memory database system experts at McObject
A summary of the white paper by McObject, “Will The Real In-Memory Database Please Stand Up?” on the advantages of a real in-memory database (IMDS) for data management.
When you hatch a database management system, it will, by design and implementation, be either an in-memory database system or an on-disk database management system. The choice affects the fundamental optimization strategies that will be baked into the database system code. To optimize an on-disk database is to minimize disk I/O, so its designers and developers will use extra CPU cycles and extra memory if doing so will reduce or eliminate I/O. Conversely, IMDSs by definition eliminate all disk I/O; their optimization is all about delivering the highest performance at a given level of processing power, thus reducing demand for CPU cycles is a key objective. And since memory is storage for an IMDS, reducing memory overhead (i.e. RAM consumed for anything other than storing data) is also a key objective.
Those optimizations are diametrically opposed. So it follows that you cannot, 5, 10, 15 or 20 years after an on-disk DBMS was designed and developed, suddenly turn it into an in-memory database system and expect the same performance or efficient memory use as a database system written to be in memory in the first place. The on-disk design goals mentioned above were baked in a long time ago, and while you will end up with something that is faster than the original, it won’t be as fast or as efficient in its use of memory as a true IMDS, created from scratch with the appropriate set of design goals.
When it comes to evolving a product to meet market demands, an in-memory database vendor has an advantage. They can always choose to use more CPU cycles and/or memory in order to add a feature. So an IMDS can evolve into a hybrid in-memory/on-disk database system and the on-disk implementation can be every bit as good as the database system that was originally written to be on-disk. In other words, it’s easy to add ingredients to your pie (CPU cycles and memory consumption). But once they’re in the pie mix, you can’t really take those ingredients out, short of a rewrite.
A well-designed hybrid database system, based on a true in-memory database, is more efficient than one that has bolted on in-memory capability to a persistent database system.
The Port Authority Building in Antwerp is a fascinating example of architectural design, but not the most efficient use of resources.
“Those optimizations are diametrically opposed. So it follows that you cannot, 5, 10, 15 or 20 years after an on-disk DBMS was designed and developed, suddenly turn it into an in-memory database system and expect the same performance or efficient memory use as a database system written to be in memory in the first place.”
Co-founder and CEO, McObject
Learn more about eXtremeDB
eXtremeDB is a database development tool that is ideally suited for systems that are safety-critical or have stringent constraints. The type-safe native C API, comprehensive target-side debugging capabilities, host-side real-time diagnostic tools, optimized data layouts, integrated small footprint embedded web-server, and dozens of supported toolchains all ensure maximum flexibility for developers and minimize time-to-market.
Hybrid data storage
Unlike other IMDS, eXtremeDB can combine the strengths of on-disk and in-memory database systems. In other words, eXtremeDB databases can be all-in-memory, all-persistent, or have a mix of in-memory tables and persistent tables. This unparalleled flexibility enables developers to tailor data management in order to optimize applications for speed and persistence. System developers can make intelligent trade-offs between performance, cost-efficiency, power consumption, and physical space-conserving data storage hardware.
Webinars for Professional Developers
Watch to on-demand Webinars, hosted by experts, about proven database management system practices. Watch “Multi-Core & Embedded Software: Optimize Performance by Resolving Resource Contention“. Or, “Embedded Databases: Make or Break Technology Choices for High Performance Applications” and others.
Watch Embedded Databases: Building In Always On High Availability
This Webinar highlights the issue of operational continuity: how can a database system survive the failure of the software or hardware environment in which it operates?
Watch What Makes a Database System ‘In-Memory’?
Join McObject CEO Steve Graves to explore this topic, including the limitations (and burden) of database caching; data transfer and duplication; volatility and recoverability, and more. Gain ideas and techniques for building better, faster software.
Watch Eliminating Database Corruption
Why corruption occurs and provides strategies to prevent it, focusing on hidden dangers – like storage device settings that can undermine data consistency – as well as more recognizable risks, such as passing wrongly typed data to a database run-time.
Review our list of Webinars
Articles for Professional Developers
- Change Data Capture in Embedded Databases Embedded Computing Design
- Industrial Internet of Things (IIoT) Database Usage in Rail Systems insight.tech
- The Importance of Distributed Databases for the Internet of Things Embedded Software Engineer – ESE Kongress edition, page translates
- Comparing Optimistic and Pessimistic Concurrency, Embedded Computing Design
See a list of articles
White Papers for Professional Developers
We have been testing, improving on, and retesting our software from the beginning in 2001 in order to provide our clients with the best possible data management solutions. Read “Database Persistence, Without The Performance Penalty” and more.
Review our research
Some applications require higher data durability than in-memory storage provides. What if DRAM could be made persistent? AgigA Tech’s AGIGARAM non-volatile DIMM (NVDIMM) delivers that capability. McObject benchmarked the eXtremeDB In-Memory Database System using AGIGARAM as storage, including “pulling the plug” mid-execution, and comparing the NVDIMM to transaction logging as a solution for data durability/recoverability. This paper presents the benchmark tests and results.