...
Offline Backup/Restore of Trafodion
Trafodion full offline backup and restore operation are performed using HBase snapshot fetaure. Snapshots are a light weight way to keep the current state of a table without copying the data. After making changes to a table or in case of recovery, restoring the snapshot gives the previous state of the table. More on snapshots can be found at: http://blog.cloudera.com/blog/2013/03/introduction-to-apache-hbase-snapshots/.
Snapshots are taken for all the Trafodion tables including the Trafodion metadata tables and then exported to a HDFS location using the HBase ExportSnapshot Mapreduce job. When we need to restore the backed files from the HDFS location the snapshots are imported to the target system using the ExportSnapshot MapReduce job also and then restored. In the case of Trafodion full backup/restore the The process is as follows:
Full Offline Backup
- Trafodion needs to be completely down before starting offline backup
- Snapshots To do a backup snapshots are taken for all the Trafodion tables using the HBase shell snapshot command.
- Trafodion is restarted. Regular activity can resume on these tables, but they will not be included in the backup.
- Snapshots are exported one Once snapshots are taken, they are exported one by one to HDFS folder using the ExportSnapshot MapReduce job that is part of HBase install base. The MapReduce job can be run in parallel using –m option.TrafExportSnapshot class. For small tables this uses HDFS file copy while for large tables MapReduce is to copy HFiles. The number of mappers can vary by table. It is determined is determined by size of table and number of files to be copy for that table,
- If necessary, If needed the backup files can be archived or moved to a different cluster using hadoop distcp or other methods.
Full Offline Restore
- Trafodion needs to be completely down before doing the starting full offline backup
Full Offline Restore
- restore
- There must be no Trafodion related tables in HBase
- Snapshots To perform the restore the snapshots that were saved in HDFS are imported one by one to HBase using the ExportSnapshot MapReduce job. The MapReduce job can be run in parallel.TrafExportSnapshot class. This is reverse of the copy that was done during backup.
- Once the Once the snapshots are imported, they can be are restored using ‘restore_snapshot‘ HBase shell command.Trafodion needs to be completely down before doing the full offline restore
Trafodion Tables
Trafodion tables are created in HBase with names that have a format like ‘TRAFODION.<schema-name>.<table-name>’. To back up all Trafodion tables, the backup tool selects and backs up all table that have a name pattern like TRAFODION.<any-schema-name>.<any-table-name>. Besides user tables, metadata tables (TRAFODION._MD_.<table-name>) and Repository table repository tables (TRAFODION._REPOS_.<table-name> are ) and transaction related tables (TRAFODION._MD_.<table-name>) are also backed up.
Sudo Access
The backup and restore tools can be used in both the development and clustered environment with either Cloudera or Horton Works distributions. To run the backup and restore scripts in a clustered environment, the user running the scripts needs to have sudo access to be able to run commands as HBase user (user under which HBase server runs), HDFS user and Trafodion user without requiring to enter a password. However sudo access is not needed when the scripts are run in the development environment.
Backup Files Security
The backup and restore operations should be performed using a user with sudo access that can sudo to HBase user. Both backup and restore use the ExportSnapshot MapReduce job which requires read and write access to these locations:
- The HBase root directory which is defined by the hbase.rootdir property. This folder is owned by HBase user
- The backup folder which needs to be either owned by the HBase user or HBase user should have read/write access to it. the Backup script tries to create the destination HDFS folder if it does not exist.
Scripts
Backup script: run_full_trafodion_backup.sh
The ./run_full_trafodion_backup.sh script performs full offline backup of all trafodion tables and copies the backup files to an HDFS location.
The command to use the script is as follows:
...
In a clustered environment one possibility is that root or some admin type id is used to run backup and restore. An alternative is that trafodion user is given access to run any command under the HBase or HDFS user id with a password prompt. If this acceptable from a security perspective it can be achieved by adding this line to /etc/sudoers
trafodion ALL=(hbase) NOPASSWD: ALL, (hdfs) NOPASSWD: ALL, (trafodion) NOPASSWD: ALL
Backup Files Security
Both backup and restore use the TrafExportSnapshot which requires read and write access to these locations:
- The HBase root directory which is defined by the hbase.rootdir property. This folder is owned by HBase user
- The backup folder which needs to be either owned by the HBase user or HBase user should have read/write access to it. the Backup script tries to create the destination HDFS folder if it does not exist.
Scripts
Backup script: run_full_trafodion_backup.sh
The ./run_full_trafodion_backup.sh script performs full offline backup of all trafodion tables and copies the backup files to an HDFS location.
The command to use the script is as follows:
./run_full_trafodion_backup.sh -b backup_ -b backup_folder -u trafodion_user -h HBase_user -d hdfs_user -m mappers -l 10 -n -o Where: -b backup_folder (Optional) HDFS path where all the Trafodion object are exported and saved The HDFS path needs to have a format like hdfs://<host>:<port>/<folder>/... If the path is not provided the script generates a path with a format like hdfs://<host>:<port>/trafodion-backlups/backup_<timestamp> and unless -n is specified the user is asked whether to confirm the use of the generated path. -u trafodion user (Optional) The user under which Trafodion server runs. If not provided and if -n option is not specified the user is asked whether the default trafodion user 'trafodion' can be used or not. If the answer is yes then the default trafodion user is used otherwise the script exits. -h hbase user (Optional) The user under which HBase server runs. If not provided the script tries to compute it and if it does not succeed it considers using the default HBase user 'hbase'. Unless the -n option is specified, the user is asked to confirm the selection afterwards. -d hdfs user (Optional) The user under which HDFS server runs. If not provided the script tries to compute it and if it does not succeed it considers using the default HDFS user 'hdfs'. Unless the -n option is specified, the user is asked to confirm the selection afterwards.''
-m mappers
(Optional) Number of mappers. If unspecified or 0, each snapshot will use a number suitable for its size.
-n
The default value is 2. -n (Optional) Non interactive mode. With this option the script does not prompt
the user to confirm the use of computed or default values when a parameter
like trafodion user, HBasehbase user, hdfs user or backup path is not provided.
Example: ./run_full_trafodion_backup.sh -b hdfs://<host>:<port>/<hdfs-path> -m 4
Checks performed before backing up the Trafodion tables
The backup script performs the following checks before starting the actual backup:
-o
(Optional) offline. With this option trafodion will not be restarted after
snapshots are taken.
-l
(Optional) Snapshot size limit in MB above which map reduce is used for copy. Snapshots with size below this value
will be copied using HDFS FileUtil.copy. Default value is 100 MB. FileUtil.copy is invoked through a class provided
by Trafodion. Use 0 for this option to use HBase' ExportSnaphot class instead.
Example: ./run_full_trafodion_backup.sh -b hdfs://<host>:<port>/<hdfs-path> -n
Checks performed before backing up the Trafodion tables
The backup script performs the following checks before starting the actual backup:
- Make sure Trafodion is completely down.
- Make
- Make sure Trafodion is completely down.
- Make sure the backup folder exists, is empty and HBase user has permissions to write to it. If it does not exists it tries to create it.
- Make sure HBase snapshots are enabled.
- Make sure the user running the scripts have sudo access.
- Create the HBase home directory if it does not exist.
- Try to figure out the HBase and HDFS users if they are not provided.
Restore: run_full_trafodion_backup.sh.
The ./run_full_trafodion_restore.sh script performs full offline restore of all trafodion tables from a HDFS location
The command to use the script is as follows:
- if it does not exist.
- Try to figure out the HBase and HDFS users if they are not provided.
Restore: run_full_trafodion_backup.sh.
The ./run_full_trafodion_restore.sh script performs full offline restore of all trafodion tables from a HDFS location
The command to use the script is as follows:
./run_full_trafodion_restore.sh -b backup_folder -b backup_dir -u trafodion_user -h hbase_user -m mappers -l 10 -n
Where: -b backup_folder (Not Optional) HDFS path where all the Trafodion object are exported and saved The HDFS path needs to have a format like hdfs://<host>:<port>/<folder>/... -u trafodion user (Optional) The user under which Trafodion server runs. If not provided and if -n option is not specified the user is asked whether the default trafodion user 'trafodion' can be used or not. If the answer is yes then the./run_full_trafodion_restore.sh -b backup_folder -b backup_dir -u trafodion_user -h hbase_user -m mappers -n Where: -b backup_folder (Not Optional) HDFS path where all the Trafodion object are exported and saveddefault Thetrafodion HDFSuser pathis needsused tootherwise havethe a format like hdfs://<host>:<port>/<folder>/... -u trafodion user script exits. -h hbase user (Optional) The user under which TrafodionHBase server runs. If not provided the andscript tries to ifcompute -nit optionand isif notit specifieddoes thenot usersucceed isit askedconsiders whetherusing the default trafodionhbase user 'trafodionhbase'. canUnless bethe used-n oroption not.is Ifspecified, the answeruser is yesasked then theto default trafodion user is used otherwiseconfirm the scriptselection exitsafterwards. -hd hbasehdfs user (Optional) The user under which HBaseHDFS server runs. If not provided the script tries to compute it and if it does not succeed it considers using the default hbaseHDFS user 'hbasehdfs'. Unless the -n option is specified, the user is asked to confirm the selection afterwards. -dm mappers
(Optional) Number of mappers. If unspecified or 0, each snapshot will use a number suitable for its size.
-l
(Optional) Snapshot size limit in MB above which map reduce is used for copy. Snapshots with size below this value
will be copied using HDFS FileUtil.copy. Default value is 100 MB. FileUtil.copy is invoked through a class provided
by Trafodion. Use 0 for this option to use HBase' ExportSnaphot class instead.
-n hdfs user (Optional) The user under which HDFS server runs. If not provided the script tries to compute it and if it does not succeed it considers using the default HDFS user 'hdfs'. Unless the -n option is specified, the user is asked to confirm the selection afterwards. -m mappers (Optional) Number of mappers. The default value is 2. -n (Optional) Non interactive mode. With this option the script does not prompt the user to confirm the use of computed or default values when a parameter like trafodion user, HBase user, or hdfs user or backup path is not provided.
Example: ./run_full_trafodion_restore.sh -b hdfs://<host>:<port>/<hdfs-path> -m 4n
Checks performed prior to starting restore
...
./run_full_trafodion_backup.sh -b hdfs://<host>:8020/bulkload/backup -m 16n Where hdfs://<host>:8020/bulkload/backup is the hdfs location where we want the backed up snapshot to reside in this example.
...
./run_full_trafodion_restore.sh -b /bulkload/backup -m 4n
Backup/Restore on development workstation
For the ExportSnapshot TrafExportSnapshot to work in the development environment, we need to copy few jar files under $MY_SQROOT/sql/local_hadoop/hbase/lib directory. Otherwise we get exceptions errors when running the ExportSnapshot MapReduce job. These workaround were figured out while testing on the workstations and there could be other ways to avoid the exceptions.
...
./run_full_trafodion_backup.sh -b hdfs://localhost:<port-number>/bulkload/backup -m 4n
Restore
To run the restore:
run_full_trafodion_restore.sh -b hdfs://localhost:<port-number>/bulkload/backup -m 4n
Testing
The test cases that were performed so far include:
...