Please refer to the Data Compression page for a discussion of the advantages of in-memory and persistent database compression.
In-Memory Database Compression
In-memory database compression is enabled by setting the
MCO_DB_INMEMORY_COMPRESSIONflag in the
mco_db_params_t.mode_maskparameter passed to
mco_db_open_dev(). (Note that this flag does not apply to disk-based or “direct pointers” databases).
There are three parameters controlling compression which are also set in the
compression_level: Integer values of 0-9. Higher values provide better compression at the cost of speed. For the default LZ algorithm only values of 0 and 1 are currently used. (The
zlibimplementation accepts values of 0-9: 0 - means no compression, varying sizes of objects are truncated by their real size; 1 - (default) means the standard compression level.)
compression_mask: a bitmap of the kinds of pages to be compressed. (By default only the header page of an object is compressed.)
expected_compression_ratio: used to allocate the page map. The virtual database space cannot be larger than the physical size * the compression ratio. The default value is 10. An element of the page map has a size of 8 bytes, so compression requires extra memory of size
mem_size/page_size*expected_compression_ratio*8for the virtual to physical page map. Also
mem_size/page_size*4bytes are required for the physical to virtual page map.
In-memory compression and encryption use the same page hash table, in which parameters are controlled by the
mco_db_params_t.page_hash_bundles. The compressor allocates buffers of size
max_active_pages * page_hash_bundles * db_max_connection * mem_page_sizefor decompressing accessed pages.
Note that in-memory compression can be used only for pure in-memory databases (i.e. no disk manager is used). So if this mode is specified when the
mcovtdsklibrary is used, the function
mco_db_load()will return error code
Persistent Database Compression
To enable compression, set the
MCO_FILE_OPEN_COMPRESSEDflag in the
dev.file.flagsfield of the file device specification. For example:dev.type = MCO_MEMORY_FILE; dev.assignment = MCO_MEMORY_ASSIGN_PERSISTENT; sprintf( dev.dev.file.name, FILE_PREFIX "%s.dbs", dbName ); dev.dev.file.flags = MCO_FILE_OPEN_COMPRESSED;
If this flag is not set, compression will not be performed and the file will function as a normal file. (Note that it is possible to use compression for some databases / files and not use it for other files).
Two additional parameters can be used to modify the compression process. These are the
disk_max_database_sizeelements of the
mco_db_params_tstructure passed to
disk_max_database_sizevalue determines the size of the memory allocation bitmap. If it is not set (0 or
INFINITE_DATABASE_SIZE), then the file size limit is 1Tb (which requires a 2Gb bitmap).
compression_levelis an integer value of 0..9 that specifies the level of compression. A value of 0 disables compression. Values of 1 and above specify different compression strategies as follows:
1 : The default strategy; means conditional compression based on the following criteria:
- Require 25% compression rate or not worth compressing
- Stop if no compression in the first 1Kb
- Stop history lookup if a match of 128 bytes is found
- Lower the ‘good match’ size by 10% at each iteration
2 – 9: Correspond to increasingly complex unconditional compression (always compress) strategies which have the following criteria:
- Continue even if there is a low rate of compression
- Stop history lookup if a match of 128 bytes is found
- Increasing attempts to find a good match with higher compression level numbers
Note that eXtremeDB implements LZ compression in two special file system libraries:
mcofu98ziplogwhich are only available on Unix systems like Linux, MacOS and Solaris. The application developer chooses one of these implementations by simply linking one or the other of these file system libraries in place of the standard filesystem library (
mcofu98zipimplementation uses compression only for random access I/O operations, and consequently is used only for compressing the database file (and not for compressing a transaction log file). It compresses the main database file (
dbname.dbs) and creates two additional files. Because the size of an updated page in the database file can be changed, the pages have to be reallocated. So, a page map file (
dbname.pagemap) is built to provide indirect access to the pages and a memory allocation bitmap file (
dbname.bitmap) to manage the space within the file.
mcofu98ziplogimplementation is used for compressing transaction log files. It uses the same compression algorithm as
mcofu98zip, but it overrides only the sequential I/O methods. As compression is applied sequentially and pages are not overwritten, it is not necessary to maintain extra page maps and allocation map. So only the compressed log file is produced, no additional files.
Note that it is not possible to provide compression both for database and transaction log files in a single application, because only one compression implementation (either
mcofu98ziplog) can be linked into the application.
Further, note that CRC checking may not be enabled simultaneously with compression because compression implements its own CRC.
Data Compression for IoT Communications
Data compression for IoT communications is enabled by setting the
compression_levelelement of structure
mco_iot_comm_paramst_t. The default level of
0means no compression. As explained in the Data Compression page, the communications between each server-device pair will be compressed only if both sides (server and device) have
0, and all communicating components must use the same compression level value.
Also explained in the Data Compression page is the formula used to determine the compression ratio. The
mco_iot_conn_get_stat()API can be called to retrieve the statistics for a connection; elements
recv_compression_ratioof returned structure
mco_iot_connection_stat_tshow the compression efficiency for incoming and outgoing communications.