C and C++ applications specify storage devices at runtime via the array of
mco_device_tstructures: the argument (nameddevsin most SDK samples) passed tomco_db_open_dev(). Typically, an array of device structures is stack-allocated and initialized prior to callingmco_db_open_dev(). Each memory device is defined by amco_device_tstructure that specifies:typeThe type of memory region; must be one of the following values defined in
mco.h:
MCO_MEMORY_CONV— Conventional (non-shared) memory.MCO_MEMORY_NAMED— Named (shared) memory.MCO_MEMORY_INT_DESC— Some operating environments address shared memory regions through an integer descriptor. This option accommodates these operating systems through the memory device.MCO_MEMORY_FILE— A file device.MCO_MEMORY_MULTIFILE— A multi-file device.MCO_MEMORY_RAID— A RAID device.MCO_MEMORY_CYCLIC_FILE_BUF— A pair of files used to implement a cyclic buffer. This option is intended to overcome the limitation of some file systems that do not providefsync()or some equivalent file system call to make all changes persistent. It is used with theMCO_COMMIT_DELAYEDtransaction commit policy wherelog_params.max_commit_delayis set to the persistent media manufacturer’s specifications. (See Writing Data to Persistent Storage for explanation of the meaning of these transaction policies.)assignmentThe use for this device; must be one of the following values defined in
mco.h:
MCO_MEMORY_ASSIGN_DATABASE— A memory region for meta-data and user-data, indexes and other database structures.MCO_MEMORY_ASSIGN_CACHE— A memory region for the disk manager cache (page pool).MCO_MEMORY_ASSIGN_PERSISTENT— A persistent storage device (can be file, multi-file or RAID). Note that if a persistent device is defined and the application does not explicitly assigndisk_page_size(see below) the runtime will assign it the default value of 4096 bytes. If multiple applications or tasks attempt to assign different disk page sizes for the same database the runtime returns an error code.MCO_MEMORY_ASSIGN_LOG— A device that contains the database log file.MCO_MEMORY_ASSIGN_HA_ASYNC_BUF— An asynchronous buffer device for eXtremeDB High Availability.MCO_MEMORY_ASSIGN_PIPE_BUF— A buffer for an eXtremeDB Transaction Logging pipe.sizeThe size of this device (in bytes).devA union containing a pointer to pre-allocated memory for conventional or shared memory devices or a filename and flags for persistent storage devices. This union is defined as follows in
mco.h:typedef struct mco_device_t_ { unsigned int type; unsigned int assignment; mco_size_t size; union { struct { void * ptr; int flags; } conv; struct { char name[MCO_MAX_MEMORY_NAME]; unsigned int flags; void * hint; } named; struct { int flags; char name[MCO_MAX_FILE_NAME]; } file; struct { int flags; char name[MCO_MAX_MULTIFILE_NAME]; mco_offs_t segment_size; } multifile; struct { int flags; char name[MCO_MAX_MULTIFILE_NAME]; int level; mco_offs_t offset; } raid; struct { unsigned long handle; } idesc; } dev; } mco_device_t, *mco_device_h;(See the descriptions below for
devparameter settings for different types of memory management. Note that theidescportion of the struct is reserved for future use, intended for all other custom embedded memory driver implementations. Currently this capability is not used by the runtime.)Conventional memory parameters
For in-memory (
conv) databases, thestructcontains a pointer to allocated memory and flags. Theptrelement is set to the address of the memory block allocated for the database. Theflagselement is used internally by the runtime and not used by the application; it can safely be set to zero or ignored by the application.Shared memory parameters
For a shared memory (
named) device, it is necessary to specify aname,flagsand ahintaddress where the shared memory block is located in the operating system shared memory pool. Setting thedev.named.hintparameter to zero causes eXtremeDB to determine the actual shared memory segment address. But this could fail when called from a second process attempting to open the shared database. In this case it is the application's responsibility to provide a validhintaddress. (Please see Shared Memory Runtime Options for a detailed explanation of the shared memory hint address implications on Windows and Unix-Linux systems.)On Unix-Linux systems using the POSIX
mcompsxmemory driver, thenamed.flagscan be set toMCO_RT_POSIX_SHM_SHARED(for an “all process accessible” database) orMCO_RT_POSIX_SHM_ANONYMOUS(for a database “private” for the process). For all other memory drivers this element is not used and should be set to zero initially.Persistent storage parameters
For persistent storage devices a
namemust be specified; formultifileoptionallysegment_sizeandflagsmay be specified; and forraiddeviceslevelandoffsetmust be specified and optionallyflags. The file open mode is defined by theflagsparameter which can have values as follows (different combinations of flags can be OR’ed with the pipe ‘|’ operator):MCO_FILE_OPEN_DEFAULTThe standard, unified and fail-safe way to open a file in the manner supported by target platform file system. Depending on the file-system wrapper linked into the target executable it does the following:
mcofcblk.c— File implementation for the KDSA storage system. The code uses acblk_open()call to create the file descriptor for IO operations.mcofkdsa.c— File implementation for the KDSA storage system. The code usesxpd_init_device_name(),xpd_connect_via_link()andxpd_set_handle_flags()calls to initialize the device header for successive IO operations.mcofucos.c— File implementation for the uCOS FAT32 file system; use (FILE_MODE_READ | FILE_MODE_WRITE) for theFILE_OpenShort()call. This opens the file for reading and writing. There is no check for file existence, so if the file exists the content will be lost.Note the following meanings for the two Windows APIs below:
GENERIC_READ: Read access.GENERIC_WRITE: Write access.FILE_SHARE_READ: Share for reading.FILE_SHARE_WRITE: Share for writing.OPEN_ALWAYS: Opens a file, always.mcofw32.c— File implementation for the Win32 (NT family). The wrapper calls theCreateFile()routine passing flag combinations (GENERIC_READ|GENERIC_WRITE), (FILE_SHARE_READ|FILE_SHARE_WRITE) andOPEN_ALWAYS. This creates the specified file unconditionally and opens it for write and read access.mcofwrt.c— File implementation for WinRT. The wrapper calls theCreateFiles2()routine passing flag combinations (GENERIC_READ|GENERIC_WRITE), (FILE_SHARE_READ|FILE_SHARE_WRITE) andOPEN_ALWAYS. This creates the specified file unconditionally and opens it for write and read access.mcofmem.c— “In-memory file”; as this implementation does not open any real files but simulates them with memory there are no file system calls made.The following file system wrappers share the same POSIX interface to open the requested file. They initialize the file handle by a call to
open()with the (O_RDWR|O_CREAT) flags combination. This call creates the file if it does not exist and opens it for reading and writing. Note from POSIX specs:
O_RDWR— Open for reading and writing.O_RDONLY— Open for reading only.O_CREAT— Create file if it does not exist.O_LARGEFILE— Allow files whose sizes cannot be represented in anoff_t(but can be represented in anoff64_t) to be opened.mcofecos.c— File implementation for eCos.mcofintp.c— File implementation for Integrity OS (POSIX).mcofose.c— File implementation for OSE Embedded File System. EFS is a file system for the OSE Real Time Kernel.mcofuni.c— File implementation for Unix withoutpreadandpwritesupport.mcofvx.c— File implementation for VxWorks.- The following file system wrappers do a similar call to
open()as above but in addition they use flagO_LARGEFILEto enable operations on large files (>4G).mcofu98aio.c— Asynchronous file implementation for Unix supportingpreadandpwrite(Unix98 standard).mcofu98.c— File implementation for Unix supportingpreadandpwrite(Unix98 standard).mcofu98zip.c— Compressed file implementation for Unix supportingpreadandpwrite(Unix98 standard).mcofu98ziplog.c— Compressed file implementation for Unix supportingpreadandpwrite(Unix98 standard).MCO_FILE_OPEN_READ_ONLYSets read-only mode for a file; no modifications of the file will be allowed for eXtremeDB. Depending on the target file system it does the following:
mcofcblk.c, mcofkdsa.c, mcofmem.c— Does nothing; no implementation required.mcofecos.c, mcofintp.c, mcofose.c, mcofu98aio.c, mcofu98.c, mcofu98zip.c, mcofu98ziplog.c, mcofuni.c, mcofvx.c— Adds flagO_RDONLY(POSIX: open for reading only) into flags combination foropen()call.mcofucos.c— ExcludesFILE_MODE_WRITEfrom flags combination forFILE_OpenShort()call (leavesFILE_SHARE_READflag only).mcofw32.c, mcofwrt.c— ExcludesGENERIC_WRITEflags from normal (GENERIC_READ | GENERIC_WRITE) flags combination for callingCreateFiles()orCreateFiles2().MCO_FILE_OPEN_TRUNCATEInstructs eXtremeDB to truncate the file being opened to zero length before use. Depending on the implementation of file system wrapper:
mcofcblk.c— Callscblk_file_truncate()on the file.mcofkdsa.c— Callskdsa_file_truncate()on the file.mcofucos.c— CallsFILE_Truncate()on the file.mcofmem.c— No calls required.mcofecos.c, mcofintp.c, mcofose.c, mcofu98aio.c, mcofu98.c, mcofu98zip.c, mcofu98ziplog.c, mcofuni.c, mcofvx.c— Adds theO_TRUNCflag (POSIX: truncate size to 0) to flags combination for theopen()call.mcofw32.c, mcofwrt.c— Sets flagCREATE_ALWAYS(The WinAPI creates a new file, always) instead ofOPEN_ALWAYS(which opens a file, always) for the call of theCreateFiles() / CreateFiles2()routine.MCO_FILE_OPEN_NO_BUFFERINGTells the underlying file system not to cache the file content but run IO operations immediately. Depending on the file system wrapper implementation, it:
mcofcblk.c, mcofkdsa.c, mcofucos.c, mcofmem.c— Does nothing — nothing is required.mcofecos.c, mcofintp.c, mcofose.c, mcofu98aio.c, mcofu98.c, mcofu98zip.c, mcofu98ziplog.c, mcofuni.c, mcofvx.c— Includes theO_DIRECTflag (POSIX: Try to minimize the cache effects of the I/O to and from this file) into the flag combination for theopen()call. For SunOS: calls thedirectio( DIRECTIO_ON )routine (SUNOS: provides advice to the system about the expected behavior of the application when accessing the data in the file).mcofw32.c, mcofwrt.c— Adds flagFILE_FLAG_NO_BUFFERING(WinAPI: The file or device is being opened with no system caching for data reads and writes) into the flags combination for the call of theCreateFiles()/CreateFiles2()routine.MCO_FILE_OPEN_EXISTINGRequires eXtremeDB to open the file only if it exists; eXtremeDB will not create a new file. Depending on the file system wrapper implementation, it:
mcofcblk.c, mcofkdsa.c, mcofmem.c, mcofucos.c— Does nothing — nothing is required.mcofecos.c, mcofintp.c, mcofose.c, mcofu98aio.c, mcofu98.c, mcofu98zip.c, mcofu98ziplog.c, mcofuni.c, mcofvx.c— Excludes theO_CREATflag from the initial flags combination for theopen()call. This means thatopen()will open the file only if it exists. No new file will be created.mcofw32.c, mcofwrt.c— Adds flagOPEN_EXISTING(WinAPI: Opens a file or device, only if it exists) into the initial flags combination for the call of theCreateFiles()/CreateFiles2()routine.MCO_FILE_OPEN_TEMPORARYInstructs the underlying file system to open the file as temporary. Depending on the file system wrapper implementation, it does the following:
mcofcblk.c, mcofkdsa.c, mcofmem.c, mcofucos.c, mcofecos.c, mcofintp.c, mcofose.c, mcofu98aio.c, mcofu98.c, mcofu98zip.c, mcofu98ziplog.c, mcofuni.c, mcofvx.c— Does nothing and works with a file marked asMCO_FILE_OPEN_TEMPORARYas with a typical file.mcofw32.c, mcofwrt.c— Sets flagFILE_ATTRIBUTE_TEMPORARY(WinAPI: The file is being used for temporary storage) to the flag combination for the call of theCreateFiles()/CreateFiles2()routine.MCO_FILE_OPEN_FSYNC_FIXEnables the flush-operation fix for UNIX-98 file system wrappers. The fix confirms that the underlying file system (
ext3file system specifically) really puts the file content modification to the storage. This usage is required forext3only and in some rare conditions when it is confirmed that the underlying file system has flush-related issues.MCO_FILE_OPEN_SUBPARTITIONWorks for the RAID-implementation of database persistent storage. This flag specifies that the file system wrapper must take the value of the offset field of the
mco_device_tstruct into account for all segments of the RAID.MCO_FILE_OPEN_FSYNC_AIO_BARRIERWorks for the AIO (
u98aio) file system wrapper. This flag specifies that the wrapper must execute a barrier for flush operations — all IO operations must be finished before the flush operation.MCO_FILE_OPEN_COMPRESSEDWorks for the compressed (
u98zipandu98ziplog) file system wrappers. This flag specifies that the file must be compressed. Otherwise the file system wrapper will work with the file transparently.MCO_FILE_OPEN_LOCKWorks for the
u98implementation only. This flag instructs the file system wrapper to set up rules for inter-locking the file. The implementation calls theflock()routine (POSIX: A call toflock()may block if an incompatible lock is held by another process.)MCO_FILE_OPEN_NO_READ_BUFFERINGandMCO_FILE_OPEN_NO_WRITE_BUFFERINGWorks for the
u98implementation only. Controls usage of pre-read and pre-write hints for the underlying file system. The implementation usesposix_fadvise()(POSIX: Programs can useposix_fadvise()to announce an intention to access file data in a specific pattern in the future, thus allowing the kernel to perform appropriate optimizations.) Use these flags only if it is advised byMcObject Support.MAP_HUGETLBMost modern Linux systems allow configuring some part of their virtual memory space for use by huge pages. The huge pages feature enables the Linux kernel to manage large pages of memory in addition to the standard 4KB (on x86 and x86_64) or 16KB (on IA64) page size. When the system needs to access a virtual memory location, it uses the page tables to translate the virtual address to a physical address. Using huge pages means that the system needs to load fewer such mappings into the Translation Lookaside Buffer (TLB), which is the cache of page tables on a CPU that speeds up the translation of virtual addresses to physical addresses. Enabling the huge pages feature allows the kernel to use hugetlb entries in the TLB that point to huge pages. The hugetlb entries mean that the TLB entries can cover a larger address space, requiring many fewer entries to map the memory.
On systems with more than 16GB of memory running eXtremeDB databases, enabling the huge pages feature can improve database performance.
Specifying the
MAP_HUGETLBflag (for mmap) makes it possible to use the huge page feature.SHM_HUGETLBThis flag has the same effect as
MAP_HUGETLBbut for eXtremeDB shared memory databases (i.e. for shmget).