THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
Current state: "Under DiscussionFinished"
Discussion thread: here
JIRA: here (<- link to https://issues.apache.org/jira/browse/SOLR-XXXX)
Released: NA
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast). Confluence supports inline comments that can also be used.
Motivation
Solr's current backup/restore functionality has several frustrating limitations.
| Jira | ||||||
|---|---|---|---|---|---|---|
|
Released: no
Motivation
Solr's current backup/restore functionality has several frustrating limitations.
Current index backups are based on Current index backups are based on full snapshots. Snapshot-based backups are slow and expensive because they copy all the files of the index regardless of how little the index may have changed since the last backup. Some, much, or all of the backup may be spent transferring data already present in the backup repository - a needless inefficiency.
...
- Incremental i.e. only the changed data is saved instead of a full copy. This allows users to save on storage costs as well as significantly speed up the backup process if only small changes have happened since the last backup.
- Cloud friendly i.e. supports one or more blob storage systems available in major public clouds such as Amazon S3, Google Cloud Storage, Azure Blob Storage etc.
- Safe against index corruption i.e. backups should succeed only if the backed up index is not corrupt
- Restorable to existing collections i.e. it should be possible to restore to the source collection (or any existing collection, assuming it is compatible with the source)
Observant readers might recognize (1) and (3) from code that Cao Manh Dat proposed in SOLR-13608. In that sense this SIP is a superset of that ticket, created to
...
cover a broader swath of functionality and generate more discussion/review of the design. Both Dat's ticket and this SIP are informed by code written by Dat, Shalin Mangar, and others, which is available in rough form here
Public Interfaces
The proposed changes involve changes to several different levels of public interfaces.
...
Solr can support restoring to existing collections by making use of the "read only" mode that was introduced in SOLR-1372113271. The restore API can put the target collection in read-only mode, restore a backup for each shard, and then toggle off "read only" mode.
...
Listing files in a directory is a common operation in backup and restore. However, the list of files are usually well known at write time. Therefore, we write a manifest file per backup, per directory (if needed) once all files in the directory have been written. This manifest lists the files that are part of the manifest (or directory). The list operation of the backup repository for blob stores can use the manifest file to return the list of files consistently. This is similar to how Lucene writes segment files at the end.
...
Regardless of the BackupRepository in use, this SIP proposes that backups be taken in an incremental manner, so that only those index files not stored by previous backups will be stored for the given backup. This will result in changes to the format of each backup. The general thrust of these changes is that a given backup "location" can (and should) be used to store multiple backups, and that backup includes a metadata file used to indicate which Lucene index files are a part of the backup and the path to each of these within the umbrella backup "location".
Proposed Changes: Backup File Format
For details on the specific backup file format being proposed and how it enables incremental backups to be done accurately, see the SIP sub-page dedicated to this topic here.
Proposed Changes: HTTP API
...