Modular implementation

In embedded systems there is a vast variety of storage and flash devices—ranging from NOR/NAND raw chips to managed eMMC, SD cards, and RTOS-provided filesystems. Each device type comes with its own constraints: page sizes, erase-before-write rules, wear characteristics, latency profiles, and error correction schemes.

eXtremeDB/rt addresses this diversity with a persistent storage stack. Persistence is implemented in the mcovtdsk module (target/mcovtdsk/), which provides a uniform abstraction layer and communicates with storage back-ends via struct mco_file_t (see include/mco.h).

By isolating device-specific details behind mco_file_t and mco_xflash_media_t, the database core remains unchanged regardless of the storage technology underneath. This design:

• Supports diverse hardware without modifications to the kernel.
• Simplifies integration with new storage technologies.
• Provides flexibility to migrate between managed and raw flash depending on cost, performance, and reliability.

This modularity ensures the same database kernel can operate consistently across a wide spectrum of embedded hardware configurations.

Managed Flash

For managed flash devices, the underlying module is mcoxfile (target/mcovtdsk/mcoxfile/). The important point is that mcoxfile supports access through either a filesystem or a block device. To achieve this, it relies on filesystem wrappers located in target/sal/fsys/mcof*. Regardless of whether a file system or a block device is used, the interface between mcovtdsk and mcoxfile is always mco_file_t.

Raw (Unmanaged) Flash

Raw flash media is handled by the mcoxflash module (/target/mcovtdsk/mcoxflash/). This module implements a software Flash Translation Layer (FTL) and serves as a bridge between mcovtdsk and a flash-specific interface that includes:

• erase-before-write operations
• bad block management
• wear leveling
• low-level page/block addressing

In this case, mcoxflash communicates with the underlying unmanaged flash driver (or a flash simulator via struct mco_xflash_media_t (also defined in include/mco.h). Detailed descriptions of the fields and callbacks in this structure are included in the source comments.

Using a flash simulator in early stages for development and testing is often the best practice in embedded development, as it allows teams to test persistence behavior, validate wear-leveling and bad block handling logic before committing to hardware runs.

Modular Implementation Overview

• mcovtdsk always interacts with storage through mco_file_t, ensuring a consistent interface.
• mcoxfile adapts mco_file_t to managed flash, supporting file system or block device access.
• mcoxflash adapts mco_file_t to raw flash, using mco_xflash_media_t to drive erase/ program/read primitives.

This separation of persistence logic from media-specific details is the key to portability and long-term maintainability in embedded systems.

Key API Definitions

mco_file_t

is_reentrant

Non-zero if the file supports concurrent access by multiple threads (i.e., pread/pwrite are thread-safe).

is_transactional

Indicates whether the file implements transactional semantics (e.g., supports commit/rollback).

pwrite()

Writes data to a specified offset in the file. Must be reentrant if is_reentrant is true. Returns number of bytes written or -1 on error.

pread

Reads data from a specified offset in the file. Must be reentrant if is_reentrant is true. Returns number of bytes read or -1 on error.

write()

Writes data to the current file position. Returns number of bytes written or -1 on error.

seek()

Sets the current file position for subsequent read/write operations. Returns 0 on success, -1 on error.

truncate()

Truncates the file to the specified size. Returns 0 on success, -1 on error.

read()

Reads data from the current file position. Returns number of bytes read or -1 on error.

rawread()

Reads raw (unprocessed) data from a specified offset—bypassing decryption and decompression. Used only when MCO_CFG_DISK_WARMUP is enabled. Must be reentrant if is_reentrant is true. Returns number of bytes read or -1 on error.

flush()

Flushes OS file buffers to persistent storage. Returns 0 on success, -1 on error.

size()

Retrieves the current file size. Returns 0 on success, -1 on error.

close()

Closes the file. Returns 0 on success, -1 on error.

advise()

Provides a hint to the file system about future access patterns (e.g., whether a region will be needed soon). Returns 0 on success, -1 on error.

punch_hole()

Deallocates (deallocates or “punches a hole” in) a region of the file starting at pos for size bytes. Returns 0 on success, -1 on error.

compact()

Performs file compaction (shrinking) — supported only for transactional files when MCO_DB_INMEM_PAGE_MAP is enabled. Compaction occurs at transaction commit. Returns 0 on success, -1 on error.

commit()

Commits pending changes (transactional file systems only). Returns 0 on success, -1 on error.

rollback()

Rolls back pending changes (transactional file systems only). Returns 0 on success, -1 on error.

mcoxflash (Raw Flash with software FTL),

struct mco_xflash_media

geometry

Flash device geometry (page size, pages per block, total blocks).

is_block_bad()

Checks whether the specified erase block is marked as bad. Returns non-zero if bad.

flag_block_bad()

Marks the specified erase block as bad (e.g., due to a write/erase failure).

block_erase()

Erases the specified block. Returns MCO_S_OK on success or MCO_E_DISK_BAD_BLOCK if the block is faulty.

page_prog()

Programs (writes) data to the specified page. Returns MCO_S_OK on success or MCO_E_DISK_BAD_BLOCK on failure.

is_page_erased()

Checks whether the specified page has been erased (i.e., contains only 0xFF bytes). Returns non-zero if erased.

page_read()

Reads a portion of data from the specified page at a given offset. Returns MCO_S_OK on success or MCO_E_DISK_CRC_MISMATCH if an ECC error is detected.

page_copy()

Copies the contents of one page to another (used during wear leveling or garbage collection). Returns MCO_S_OK on success or an error code on failure.

media_sync()

Flushes any pending data to persistent storage (e.g., forces write-back cache to commit). Returns MCO_S_OK on success.

media_close()

Closes the flash media device and releases associated resources.

struct mco_xflash_geometry_t

page_size

Page size in bytes.

pages_per_block

Number of pages within a single erase block.

n_blocks

Total number of erase blocks in the flash device.

Please refer to the eXtremeDB/rt headers (the definitions of mco_file_t and mco_xflash_media_t in include/mco.h) for verbose declarations.

The xflash approach is illustrated with the flash-crud sample and the chapter "Using Flash Simulator"