Will The Real In-Memory Database Please Stand Up?
A summary of the research paper by the in-memory database experts at McObject, “Will The Real In-memory Database Please Stand Up?” on the advantages of a real in-memory database (IMDS) for data management.
When you hatch a database management system, it will, by design and implementation, be either an in-memory database system or an on-disk database management system. The choice affects the fundamental optimization strategies that will be baked into the database system code. To optimize an on-disk database is to minimize disk I/O, so its designers and developers will use extra CPU cycles and extra memory if doing so will reduce or eliminate I/O. Conversely, IMDSs by definition eliminate all disk I/O; their optimization is all about delivering the highest performance at a given level of processing power, thus reducing demand for CPU cycles is a key objective. And since memory is storage for an IMDS, reducing memory overhead (i.e. RAM consumed for anything other than storing data) is also a key objective.
Those optimizations are diametrically opposed. So it follows that you cannot, 5, 10, 15 or 20 years after an on-disk DBMS was designed and developed, suddenly turn it into an in-memory database system and expect the same performance or efficient memory use as a database system written to be in-memory in the first place. The on-disk design goals mentioned above were baked in a long time ago, and while you will end up with something that is faster than the original, it won’t be as fast or as efficient in its use of memory as a true IMDS, created from scratch with the appropriate set of design goals.
When it comes to evolving a product to meet market demands, an in-memory database vendor has an advantage. They can always choose to use more CPU cycles and/or memory in order to add a feature. So an IMDS can evolve into a hybrid in-memory/on-disk database system and the on-disk implementation can be every bit as good as the database system that was originally written to be on-disk. In other words, it’s easy to add ingredients to your banana bread (CPU cycles and memory consumption). But once they’re in the bread dough, you can’t really take those ingredients out, short of a rewrite.
For more information about in-memory database performance, review the white paper, Gaining an Extreme Performance Advantage.
Read more about in-memory database management systems in the white paper, In-Memory Database Systems: Myths and Facts.
A well-designed hybrid database system, based on a true in-memory database, is more efficient than one that has bolted on in-memory capability to a persistent database system.
The Port Authority Building in Antwerp is a fascinating example of architectural design, but not the most efficient use of resources.