Introduction

CloudStack DB upgrade takes care of updating the database schema and updating data (if needed) between the releases. The upgrade is initiated by the cloud-management process start - if the version of the software installed is greater than the data base version, the upgrade is being performed.

DB upgrade code structure

  • When management server starts up, we perform number of system integrity checks. All the checker classes inherit from SystemIntegrityChecker interface, and have to be listed in components.xml configuration file (last 2 checks are DB upgrade related checks): <system-integrity-checker>
            <checker name="ManagementServerNode"/>
            <checker name="EncryptionSecretKeyChecker"/>
            <checker name="DatabaseIntegrityChecker"/>
            <checker name="DatabaseUpgradeChecker"/>
        </system-integrity-checker>The checkers are executed in the order they are defined.

 DataBaseIntegrityChecker verifies if the DB ready for the upgrade. When this check fails, user has to perform some manual modifications in order to make it work. As an example, in 2.1.x we had a bug when single Local Storage was associated with more than one Hypervisors in the DB, so user had to remove one   of the references. We couldn't do it on behalf of him, because we had no idea which reference is the correct one.

DataBaseUpgradeChecker does the actual upgrade. In 3.0.x code 2 classes implement this interface -  DataBaseUpgradeChecker  and PremiumDataBaseUpgradeChecker. It's left from the times when we had Premium and Open Source versions of cloudStack software, and DB/DB upgrade procedures varied based on the version. We are planning to merge these 2 checkers in the near future.

  • DataBaseUpgradeChecker determines the current version of the software from the RPM package, and compares it with the latest DB version taken from the last row in cloud.version table. If the DB version is less than the software version, we decide which set of upgrade files should be executed - get the set from _upgradeMap table.
  • For each file defined in upgrade set, DataBaseUpgradeChecker  1) DBUpgrade.getPrepareScripts() and run the scripts with runScript() 2) DBUpgrade.performDataMigration(). Execution order is the same as the files appear in upgrade set.
  • After prepareScripts and dataMigration is finished, DataBaseUpgradeChecker executes the cleanup part for all the files from upgrade set - DBUpgrade.getCleanupScripts() and run them with runScript(). Cleanup scripts are very useful in case when some data has to be dropped, but only at the end of the upgrade as we rely on this data while performing the "Data Migration" part.

Adding new upgrade path

Lets assume we already released 3.0.2 version of the product, and started development for 3.0.3. If anything from below is modified in 3.0.3 release, we need to provide the DB upgrade:

  • new DB table is added
  • field was added/removed from/to existing table
  • indexes were added/removed from/to existing tables
  • changes to the data were made - new config variable is inserted to the cloud.configuration table, the field variable was modified (say, we changed Running state of the vm to say Alive instead)

1) Add Upgrade302to303() implementing DbUpgrade

2) Add prepare and cleanup scripts (if needed) to setup/db/db directory. The naming convention is "schema-<versiontoupgradefrom>to<versinotoupgradeto>.sql" and  "schema-<versiontoupgradefrom>to<versinotoupgradeto>-cleanup.sql":

/setup/db/db/schema-302to303.sql

/setup/db/db/schema-302to303-cleanup.sql

In schema-302to303.sql specify all the upgrade steps that can be made without java code help - dropping the fields, changing the field name, etc.

In schema-302to303-cleanup.sql specify all cleanup steps. It will be executed only after prepared scripts and data migration is executed. We rarely need cleanup scripts, only when some field/value is required during the data migration and has to be dropped later on.

3) Add upgrade path from 3.0.2 to _upgradeMap in DataBaseUpgradeChecker; example:

_upgradeMap.put("3.0.2", new DbUpgrade[]

{ new Upgrade302to303() }
);

4) Append "new Upgrade302to303()" to all entries in _upgradeMap:

_upgradeMap.put("3.0.1", new DbUpgrade[]

{ new Upgrade301to302(), new Upgrade302to303() }
);

5) In Upgrade302to303()

  • Set the version to upgrade from and the version to upgrade to in method:

     

public String[] getUpgradableVersionRange() {

          return new String[]

{ "3.0.2", "3.0.3" }
;

      }
  • Set the version to upgrade to in method:  
 public String getUpgradedVersion()
{         return "3.0.3";     }

  • pecify all the Prepare scripts in: 

    public File[] getPrepareScripts()
{             throw new CloudRuntimeException("Unable to find db/schema-301to302.sql");         }
        return new File[]
{ new File(script) }
;
    }* Specify the cleanup scripts in method:      public File[] getCleanupScripts() {
        String script = Script.findScript("", "db/schema-301to302-cleanup.sql");
        if (script == null)
{             throw new CloudRuntimeException("Unable to find db/schema-301to302-cleanup.sql");         }
        return new File[]
{ new File(script) }
;    " macrohasbody="false" wikihasprecedingnewline="false" wikihastrailingnewline="false" >
{         String script = Script.findScript("", "db/schema-301to302.sql");         if (script == null)
{             throw new CloudRuntimeException("Unable to find db/schema-301to302.sql");         }
        return new File[]
{ new File(script) }
;
    }* Specify the cleanup scripts in method:      public File[] getCleanupScripts() {
        String script = Script.findScript("", "db/schema-301to302-cleanup.sql");
        if (script == null)
{             throw new CloudRuntimeException("Unable to find db/schema-301to302-cleanup.sql");         }
        return new File[]
{ new File(script) }

;     }

    If any data migration is needed, code it as a part of performDataMigration().

6) And the last step - always add the and execute UnitTest for the db upgrade path. You can look atAdvanceZone217To224UpgradeTest to get the idea.

Hints

  • When you want to drop the field/key that existed in N-2 version, but was missing in N-1 version, you have to handle both scenarios in N-1->N upgrade. Do it in the Java code, performDataMigration() method using these calls from DbUpgradeUtils helper class:
         - dropKeysIfExist
         - dropPrimaryKeyIfExists
        - dropTableColumnsIfExist* Always close ResultSet and PreparedStatements in finally() block during the DataMigration. It helps to eliminate memory leaks.

  • If you want to suppress errors in INSERT (be careful with this though), use INSERT IGNORE as opposed to INSERT. We mostly use it when have to insert global config variable missing in just some of the previous releases:       

     INSERT IGNORE INTO `cloud`.`configuration` VALUES ('Advanced', 'DEFAULT', 'management-server', 'direct.agent.load.size', '16', 'The number of direct agents to load each time');
  • Never modify the upgrade path leading to the version that is already released, unless its a blocker bug and the customer needs a patch. Example: 4.1 was released, and there is a bug in 4.0->4.1 upgrade path. The fix should be made in 4.1->4.2 patch, never in 4.0->4.1

Known issues

  • No rollback is supported. We always insist on saving the DB dump before running the upgrade so in case something goes wrong, the Admin can do manual rollback. Steps to do the manual rollback:

  1. Get the patch with the fix for the failed upgrade
  2. Drop the partially updated database
  3. Apply previously saved DB dump
  4. Run the upgrade once again

 

  • No labels