Incremental Backup and Restore in C

Incremental online backup can be performed while normal database activity proceeds uninterrupted. While the library level backup is the recommended method of implementation, the eXtremeDB runtime exports low level APIs to allow applications to provide a custom backup implementation.

For alternative data export options see page Data Export and Import in C

Library Level Backup

The library level is a convenient way to integrate a database backup process into C applications. This approach assumes that the backup is done to a file system. The backup and restore functions are provided in the mcobackup library. Please see the Incremental Backup Implementation page for a detailed description of how the backup process is performed. Also, please see the SDK sample 18-backup_online for implementation details.

In order to implement a full or incremental backup, the application must first define filenames for the backup files and labels for backup "tags" then set the appropriate parameters in the mco_db_params_t structure passed to mco_db_open_dev(). For example:

 
    char * filename = "MyDb.bak";
    char * label_full = "Full";
    char * label_inc1 = "Inc1";
    char * label_inc2 = "Inc2";
     
    db_params.backup_map_size = 4 * DATABASE_SEGMENT_SIZE / DISK_PAGE_SIZE / 16;
    db_params.backup_max_passes = 10;
    db_params.backup_min_pages = 10;
    sprintf(db_params.backup_map_filename, "%s","Backup_map");
    db_params.mode_mask = MCO_DB_INCREMENTAL_BACKUP;
     
    rc = mco_db_open_dev(db_name, MyDb_get_dictionary(), &dev, 1, &db_params );
     

The backup map is used to keep track of database pages processed by the backup algorithm. For persistent databases the backup map is written to a file when the mco_db_close() API is called. If no backup_map_filename is specified, the default name of <database_name>.bm is used.

The backup is an iterative process controlled by the two conditions backup_max_passes and backup_min_pages.

backup_map_size	The size of the backup counters array in bytes. It must be power of two. This parameter is ignored and calculated automatically if the parameter disk_max_database_size is defined (non-zero)
backup_min_pages	The minimum number of modified pages required to perform an incremental backup. When the number of remaining pages is less than backup_min_pages the backup process locks the database exclusively and finalizes the backup. If the number of modified pages is small, it takes a very short time to write them out and complete the backup. (Note that a value of zero disables the backup.)
backup_max_passes	Irrespective of backup_min_pages, this parameter represents the maximum number of loops until an incremental backup is performed.

A single backup file can contain several independent backups, possibly of different types (full or incremental) or with different tags or timestamps. Each backup is "framed" within a header and a footer (this frame is called a backup record).

To create a backup file, applications call mco_backup_create() specifying a file_name, a label and the backup type MCO_BACKUP_TYPE_SNAPSHOT. This creates the initial backup record. Then successive incremental backup records can be added by calling mco_backup_create()with backup type MCO_BACKUP_TYPE_INCREMENTAL. Or the backup type MCO_BACKUP_TYPE_AUTO can be specified to cause automatic selection of a snapshot or incremental record depending on the content of the backup file; if there is no snapshot in the file yet, the backup process will create one; otherwise a partial (incremental) backup record is created.

Restoring or Verifying a Backup

Once created, the backup file can be used to restore the database from a specific backup record by calling the mco_backup_restore() API with the appropriate label. To view the records in a backup file call the mco_backup_list() API which displays the name, type, label and other properties of the backup records. It may sometimes be useful to check backup files for consistency without performing the restore. The mco_backup_verify() function is provided for this purpose.

The following C application code snippets demonstrate how these APIs might be used. First, to backup a database:

 
    {
        mco_db_h con;
        MCO_RET rc;
        char * filename = “backup.bak”;
        char * label = “a label”;
 
        /* connect to a database */
        rc = mco_db_connect( db_name, &con );
 
        /* backup the database */
        rc = mco_backup_create( con, filename, label,
                        MCO_BACKUP_TYPE_AUTO, 1, 0, 0);
    }
     

Then to restore a database from the backup file:

 
    {
        mco_db_h con;
        MCO_RET rc;
        char * filename = “backup.bak”;
        char * label = “a label”;
 
        /* connect to a database */
        rc = mco_db_connect( db_name, &con );
 
        /* restore the database */
        rc = mco_backup_restore( con, filename, label, 0, 0);
    }
     

Library Level API Reference:

See page Incremental Backup C API for detailed descriptions and examples of the library level APIs.

Low level C API backup functions

The eXtremeDB backup services can be overridden via custom callback function implementations. Also, services can be turned on/off through the internal runtime configuration file target/mcolib/mcocfg.h define MCO_CFG_BACKUP.

By default, the backup services are enabled and the following backup-related APIs are exported:

 
    typedef MCO_RET (*mco_backup_info_proc_t)( uint4 phase,
        const mco_backup_info_t * info, void * param );
 
    typedef MCO_RET (*mco_backup_chunk_proc_t)( uint4 chunk_id,
        const char * mem, uint4 mem_sz, void * param );
 
    typedef MCO_RET (*mco_restore_chunk_proc_t)( uint4* chunk_id,
        char ** mem, uint4 * mem_sz, void * param );
         

The mco_backup_info_proc_t() and mco_backup_chunk_proc_t() callbacks are called by internal eXtremeDB runtime function mco_backup(). The first is intended to store informational headers that frame a backup record. The second stores a chunk of data. The mco_restore_chunk_proc_t() callback is called by internal eXtremeDB runtime function mco_restore() and is intended to restore a chunk of data from the backup storage. Applications cannot call these callback APIs directly.

This core-level incremental backup/restore API does not limit the application to use a file system or any other specific type of storage. It is up to the application to decide how to store its backups. Internally the runtime functions mco_backup() and mco_restore()assure the integrity of the data chunks. Optionally the application may encrypt or compress the data in the process.

The restore procedure requires a database with the same database properties (such as size, memory page size etc.) and overwrites its content with the data pages read from the backup record. As with the library-level backup, it is necessary to have at least one snapshot (full backup record), and any possible number of incremental records, to restore the content of a database.

The following code snippet demonstrates one possible simple implementation of these callback APIs:

 
    /* backup callbacks */
    MCO_RET mco_backup_info(uint4 phase, const mco_backup_info_t* info, void* param)
    {
        mco_backup_ctx_t *ctx = (mco_backup_ctx_t *)param;
        mco_backup_info_t hdr = *info;
         
        switch (phase) 
        {
            case MCO_BACKUP_PHASE_1:
                …
                /* write backup header */
                if (fwrite(&hdr, sizeof(hdr), 1, ctx->f) != 1) 
                {
                    return MCO_E_DISK_WRITE;
                }
                ...
                break;
 
            case MCO_BACKUP_PHASE_2:
                ...
                /* write backup footer */
                if (fwrite(&hdr, sizeof(hdr), 1, ctx->f) != 1) 
                {
                    return MCO_E_DISK_WRITE;
                }
                ...
                break;
 
            default:
                return MCO_E_ILLEGAL_PARAM;
        }
 
        return MCO_S_OK;
    }
 
    MCO_RET mco_backup_chunk(uint4 chunk_id, const char* chunk, uint4 chunk_size, 
                    void* param )
    {
        mco_backup_ctx_t *ctx = (mco_backup_ctx_t *)param;
        ...
         
        /* write the chunk */
        if (fwrite( chunk, chunk_size, 1, ctx->f) != 1) 
        {
            return MCO_E_DISK_WRITE;
        }
        ...
        return MCO_S_OK;
    }
 
    MCO_RET mco_backup_create(mco_db_h con, char const* file_name, char const* label, 
                    mco_backup_type type, char* err_buf, unsigned int * err_buf_sz)
    {
        MCO_RET rc;
        mco_backup_ctx_t ctx;
        ...
 
        /* open a backup file */
        ctx.f = fopen(file_name, "rb+”);
        ...
 
        /* run backup procedure passing-in two callbacks */
        rc = mco_backup(con, label, type, mco_backup_info, mco_backup_chunk, &ctx);
        ...
 
        /* close backup file */
        fclose(ctx.f);
        ...
 
        return rc;
    }
 
    MCO_RET mco_restore_chunk(uint4* chunk_id, char** pchunk, uint4* pchunk_size, void * param)
    {
        mco_backup_ctx_t *ctx = (mco_backup_ctx_t *) param;
        ...
 
        /* read chunk into a buffer and return read size and the content */
        ...
 
        if ( ((*pchunk_size)=fread( ((*pchunk)=buf), size, 1, ctx->f)) != 1) 
        {
            return MCO_E_DISK_READ;
        }
 
        ...
        return MCO_S_OK;
    }
 
    MCO_RET mco_backup_restore(mco_db_h con, char const* file_name, char const* label, 
                    char const* cipher, char* err_buf, unsigned int * err_buf_sz)
    {
        MCO_RET rc;
        mco_backup_ctx_t ctx;
        mco_backup_info_t info;
        ...
         
        /* open a backup file */
        ctx.f = fopen(file_name, "rb+”);
        ...
         
        /* read backup header and footer*/
        fread( &info, sizeof(info), ctx.f );
        ...
         
        /* apply the backup from the file passing-in chunk read callback and the
            backup info (header & footer) */
        rc = mco_restore(con, &info, mco_restore_chunk, &ctx);
        ...
 
        /* close backup file */
        fclose(ctx.f);
        ...
 
        return rc;
    }