Page History

...

Offline Backup/Restore of Trafodion

Trafodion full offline backup and restore operation are performed using HBase snapshot fetaurefeature. Snapshots are a light weight way to keep the current state of a table without copying the data. After making changes to a table or in case of recovery, restoring the snapshot gives the previous state of the table. More on snapshots can be found at: http://blog.cloudera.com/blog/2013/03/introduction-to-apache-hbase-snapshots/.

Snapshots are taken for all the Trafodion tables including the Trafodion metadata tables and then exported to a HDFS location using the HBase ExportSnapshot Mapreduce job. When we need to restore the backed files from the HDFS location the snapshots are imported to the target system using the ExportSnapshot MapReduce job also and then restored. In the case of Trafodion full backup/restore the The process is as follows:

Full Offline Backup

Trafodion needs to be completely down before starting offline backup
Snapshots To do a backup snapshots are taken for all the Trafodion tables using the HBase shell snapshot command.
Trafodion is restarted. Regular activity can resume on these tables, but they will not be included in the backup.
Snapshots are exported Once snapshots are taken, they are exported one by one to HDFS folder using the ExportSnapshot MapReduce job that is part of HBase install base. The MapReduce job can be run in parallel using –m option.TrafExportSnapshot class. For small tables this uses HDFS file copy while for large tables MapReduce is to copy HFiles. The number of mappers can vary by table. It is determined by table size and number of files to be copied for that table,
If necessary, If needed the backup files can be archived or moved to a different cluster using hadoop distcp or other methods.

Full Offline Restore

Trafodion needs to be completely down before doing the starting full offline backup

Full Offline Restore

restore
There must be no Trafodion related tables in HBase
Snapshots To perform the restore the snapshots that were saved in HDFS are imported one by one to HBase using the ExportSnapshot MapReduce job. The MapReduce job can be run in parallel.TrafExportSnapshot class. This is reverse of the copy that was done during backup.
Once the snapshots Once the snapshots are imported, they can be are restored using ‘restore_snapshot‘ HBase shell command.Trafodion needs to be completely down before doing the full offline restore

Trafodion Tables

Trafodion tables are created in HBase with names that have a format like ‘TRAFODION.<schema-name>.<table-name>’. To back up all Trafodion tables, the backup tool selects and backs up all table that have a name pattern like TRAFODION.<any-schema-name>.<any-table-name>. Besides user tables, metadata tables (TRAFODION._MD_.<table-name>) and Repository table repository tables (TRAFODION._REPOS_.<table-name> are ) and transaction related tables (TRAFODION._MD_.<table-name>) are also backed up.

Sudo Access

The backup and restore tools can be used in both the development and clustered environment with either Cloudera or Horton Works distributions. To run the backup and restore scripts in a clustered environment, the user running the scripts needs to have sudo access to be able to run commands as HBase user (user under which HBase server runs), HDFS user and Trafodion user without requiring to enter a password. However sudo access is not needed when the scripts are run in the development environment.

Backup Files Security

The backup and restore operations should be performed using a user with sudo access that can sudo to HBase user. Both backup and restore use the ExportSnapshot MapReduce job which requires read and write access to these locations:

The HBase root directory which is defined by the hbase.rootdir property. This folder is owned by HBase user
The backup folder which needs to be either owned by the HBase user or HBase user should have read/write access to it. the Backup script tries to create the destination HDFS folder if it does not exist.

Scripts

Backup script: run_full_trafodion_backup.sh

The ./run_full_trafodion_backup.sh script performs full offline backup of all trafodion tables and copies the backup files to an HDFS location.

The command to use the script is as follows:

...

In a clustered environment one possibility is that root or some admin type id is used to run backup and restore. An alternative is that trafodion user is given access to run any command under the HBase or HDFS user id with a password prompt. If this acceptable from a security perspective it can be achieved by adding this line to /etc/sudoers

trafodion  ALL=(hbase) NOPASSWD: ALL, (hdfs) NOPASSWD: ALL, (trafodion) NOPASSWD: ALL

Backup Files Security

Both backup and restore use the TrafExportSnapshot which requires read and write access to these locations:

The HBase root directory which is defined by the hbase.rootdir property. This folder is owned by HBase user
The backup folder which needs to be either owned by the HBase user or HBase user should have read/write access to it. the Backup script tries to create the destination HDFS folder if it does not exist.

Scripts

Backup script: run_full_trafodion_backup.sh

The ./run_full_trafodion_backup.sh script performs full offline backup of all trafodion tables and copies the backup files to an HDFS location.

The command to use the script is as follows:

./run_full_trafodion_backup.sh -b backup_folder -u trafodion_ -b backup_folder -u trafodion_user -h HBase_user -d hdfs_user -m mappers -l 10 -n -o

Where:

-b backup_folder

     (Optional) HDFS path where all the Trafodion object are exported and saved

     The HDFS path needs to have a format like hdfs://<host>:<port>/<folder>/...

     If the path is not provided the script generates a path with a format like

     hdfs://<host>:<port>/trafodion-backlups/backup_<timestamp> and unless -n

     is specified the user is asked whether to confirm the use of the generated

     path.

-u trafodion user

     (Optional) The user under which Trafodion server runs. If not provided and

     if -n option is not specified the user is asked whether the default

     trafodion user 'trafodion' can be used or not. If the answer is yes then the

      default trafodion user is used otherwise the script exits.

-h hbase user

     (Optional) The user under which HBase server runs. If not provided the script

     tries to compute it and if it does not succeed it considers using the default

     HBase user 'hbase'. Unless the -n option is specified, the user is asked to

     confirm the selection afterwards.

-d hdfs user

     (Optional) The user under which HDFS server runs. If not provided the script

     tries to compute it and if it does not succeed it considers using the default

     HDFS user 'hdfs'. Unless the -n option is specified, the user is asked to

     confirm the selection afterwards.''

-m mappers


     (Optional) Number of mappers. TheIf defaultunspecified valueor 0, each snapshot will use a number suitable for its size.
-n

     (Optional) Non interactive mode. is 2.

-n

     (Optional) Non interactive mode. With this option the script does not prompt


     the user to confirm the use of computed or default values when a parameter


     like trafodion user, HBasehbase user, hdfs user or backup path is not provided.

-o
Example: ./run_full_trafodion_backup.sh  -b hdfs://<host>:<port>/<hdfs-path>  -m 4

Checks performed before backing up the Trafodion tables

The backup script performs the following checks before starting the actual backup:

 (Optional) offline. With this option trafodion will not be restarted after
     snapshots are taken.
-l
     (Optional) Snapshot size limit in MB above which map reduce is used for copy. Snapshots with size below this value 
      will be copied using HDFS FileUtil.copy. Default value is 100 MB. FileUtil.copy is invoked through a class provided
      by Trafodion. Use 0 for this option to use HBase' ExportSnaphot class instead.

Example: ./run_full_trafodion_backup.sh  -b hdfs://<host>:<port>/<hdfs-path>  -n

Checks performed before backing up the Trafodion tables

The backup script performs the following checks before starting the actual backup:

Make sure Trafodion is completely down.Make sure Trafodion is completely down.
Make sure the backup folder exists, is empty and HBase user has permissions to write to it. If it does not exists it tries to create it.
Make sure HBase snapshots are enabled.
Make sure the user running the scripts have sudo access.
Create the HBase home directory if it does not exist.
Try to figure out the HBase and HDFS users if they are not provided.

Restore: run_full_trafodion_backup.sh.

The ./run_full_trafodion_restore.sh script performs full offline restore of all trafodion tables from a HDFS location

The command to use the script is as follows:

if it does not exist.
Try to figure out the HBase and HDFS users if they are not provided.

Restore: run_full_trafodion_backup.sh.

The ./run_full_trafodion_restore.sh script performs full offline restore of all trafodion tables from a HDFS location

The command to use the script is as follows:

./run_full_trafodion_restore.sh -b backup_folder -b backup_dir -u trafodion_user -h hbase_user  -m mappers -l 10 -n

Where:

-b  backup_folder
     (Not Optional) HDFS path where all the Trafodion object are exported and
      saved The HDFS path needs to have a format like hdfs://<host>:<port>/<folder>/...
-u trafodion user
     (Optional) The user under which Trafodion server runs. If not provided and
     if -n option is not specified the user is asked whether the default
     trafodion user 'trafodion' can be used or not. If the answer is yes then the./run_full_trafodion_restore.sh -b backup_folder -b backup_dir -u trafodion_user -h hbase_user  -m mappers -n

Where:

-b  backup_folder

     (Not Optional) HDFS path where all the Trafodion object are exported and

      saveddefault Thetrafodion HDFSuser pathis needsused tootherwise havethe a format like hdfs://<host>:<port>/<folder>/...

-u trafodion user
script exits.
-h hbase user
     (Optional) The user under which TrafodionHBase server runs. If not provided the andscript

     tries to ifcompute -nit optionand isif notit specifieddoes thenot usersucceed isit askedconsiders whetherusing the default

     trafodionhbase user 'trafodionhbase'. canUnless bethe used-n oroption not.is Ifspecified, the answeruser is yesasked then theto

      default trafodion user is used otherwiseconfirm the scriptselection exitsafterwards.

-hd hbasehdfs user

     (Optional) The user under which HBaseHDFS server runs. If not provided the script

     tries to compute it and if it does not succeed it considers using the default

     hbaseHDFS user 'hbasehdfs'. Unless the -n option is specified, the user is asked to

      confirm the selection afterwards.

-dm mappers
     (Optional) Number of mappers. If unspecified or 0, each snapshot will use a number suitable for its size.

-l
     (Optional) Snapshot size limit in MB above which map reduce is used for copy. Snapshots with size below this value 
      will be copied using HDFS FileUtil.copy. Default value is 100 MB. FileUtil.copy is invoked through a class provided
      by Trafodion. Use 0 for this option to use HBase' ExportSnaphot class instead.

-n hdfs user

     (Optional) The user under which HDFS server runs. If not provided the script

     tries to compute it and if it does not succeed it considers using the default

     HDFS user 'hdfs'. Unless the -n option is specified, the user is asked to

      confirm the selection afterwards.

-m mappers

     (Optional) Number of mappers. The default value is 2.

-n

     (Optional) Non interactive mode. With this option the script does not prompt

     the user to confirm the use of computed or default values when a parameter

      like trafodion user, HBase user, or hdfs user or backup path is not provided.

Example: ./run_full_trafodion_restore.sh  -b hdfs://<host>:<port>/<hdfs-path>  -m 4n

Checks performed prior to starting restore

...

./run_full_trafodion_backup.sh -b hdfs://<host>:8020/bulkload/backup -m 16n

Where hdfs://<host>:8020/bulkload/backup is the hdfs location where we want the backed up snapshot to reside in this example.

...

./run_full_trafodion_restore.sh  -b /bulkload/backup -m 4n

Backup/Restore on development workstation

For the ExportSnapshot TrafExportSnapshot to work in the development environment, we need to copy few jar files under $MY_SQROOT/sql/local_hadoop/hbase/lib directory. Otherwise we get exceptions errors when running the ExportSnapshot MapReduce job. These workaround were figured out while testing on the workstations and there could be other ways to avoid the exceptions.

...

./run_full_trafodion_backup.sh -b hdfs://localhost:<port-number>/bulkload/backup -m 4n

Restore

To run the restore:

run_full_trafodion_restore.sh -b hdfs://localhost:<port-number>/bulkload/backup -m 4n

Testing

The test cases that were performed so far include:

...

Space shortcuts

Page tree

Versions Compared

Old Version 1

New Version Current

Key

Offline Backup/Restore of Trafodion

Full Offline Backup

Full Offline Restore

Full Offline Restore

Trafodion Tables

Sudo Access

Backup Files Security

Scripts

Backup script: run_full_trafodion_backup.sh

Backup Files Security

Scripts

Backup script: run_full_trafodion_backup.sh

Checks performed before backing up the Trafodion tables

Checks performed before backing up the Trafodion tables

Restore: run_full_trafodion_backup.sh.

Restore: run_full_trafodion_backup.sh.

Checks performed prior to starting restore

Backup/Restore on development workstation

Restore

Testing