...
The OpenOffice.org Community forums are hosted under the domain http://user.services.openoffice.org/ and comprise 9 National Language (NL) forums: Chinese, Dutch, English, French, Hungarian, Italian, Japanese, Spanish and Vietnamese (“Current Prod”). A total of 15 language packs are supported across the 9 NL forums, (the 9 main language plus 6 other major language variants). Across all forums there are some 75,000 registered members, who have created 300,000 posts on 78,000 topics. A typical usage is ~200,000 page requests/day, with another ~800,000 image, CSS images and other furniture resources needed to load these pages.
The forums are currently implemented in phpBB at version 3.0.8 running on Solaris Coolstack on a 4 core Solaris zone in the Oracle Germany data centre Hamburg machine room (at an average 3-8% us+sy load). PhpBB only requires phpBB works on a standard LAMP stack. Also note the , and the test and development systems, which shadows of the production system (“Test”), are currently implemented on an Ubuntu 10.04 LTS VM running a standard Ubuntu server minimal LAMP configuration. The phpBB application uses both a file system and D/B (currently MySQL) to maintain application data. These are currently some 2.5GiB and 1.5GiB in raw size. The target infrastructure in Apache.org will be an Ubuntu 10.04 LTS VM running on the Apache ESXi production environment (“Target”).
The phpBB application forum software has a number of special-to-OOo modifications, and the process for carrying these forward over phpBB upgrade has now been refined over 5 phpBB application version upgrades. The configuration also include includes a number of custom scripts which will require some rework for an Apache.org environment and to comply with Apache SysOps and Security Standards.
However, since cloning of this system across domains is routinely carried out as part of current support processes, any relocation to Apache Infrastructure is viewed as a low technical risk activity.
The work needed to upgrade and migrate the forums divides into 4 task areas, which are each separately addressed in a subsection below: (1) the set-up of the ooo-forums VM platform; (2) the application work on the customised phpBB / MySQL application; (3) the actual migration of the live service from Oracle Inc to the Apache Foundation infrastructure. (4) Work on the rebranding and other content changes to comply with agreed project requirements
Detailed System Configuration Notes
- The configuration is largely build on a phpBB build. This is currently at version 3.0.8. This one behind current, but upgrade to 3.0.9 was deferred in view of the pending relocation to Apache Infrastructure, and will be included in this migration.
- Due to architectural limitations in phpBB, the 9 10 forums are implemented as 9 10 separate logical systems each with its own directory hierarchy for code and application data and its own schema within MySQL. However, since the 3.0.4 upgrade, all NL-forum-specific mods have been merged into a single common code base, and with each forum is symlinked back to a version specific directory within /var/lib/phpBB. This is mainly for manageability, but this also has a performance dividend as the APC opcode cache resolves all 9 10 copies of each php module back to its real file-name and thus shares the same cache space.
- Within /var/lib/phpBB two directories are maintained for each phpBB version: refXYY and comXYY, where XYY is the phpBB version number. refXXY contains the unpacked "out-of-box" kits as downloaded from the phpBB website, and comXYY the site-specific version which is symlinked into the live hierarchy. The site-specific version is generated by a patch file on the refXYY version. This file is the key customisation configuration item (CI) currently comprising some 990 line (currently ~990 lines) which must be regressed across each phpBB version upgrade. This For each phpBB version upgrade, this patch file is regressed / refined / retest retested on a dev VM off production and , then a new pair of directory hierarchies is installed on prod prior to cut-over of each upgrade.
- The Apache and MySQL configurations are tuned to this application and transaction rate, but this tuning is pretty standard. Nothing esoteric.
- The configuration also contains a management account use to implement a number of batch script to support the release process and routine D/B and file-system housekeeping / monitoring and to implement a mailbox to request forum posting system. The goal here in doing this was to ensure that the application and system could run "zero-admin" between upgrades.
...
- A hot-copy backup of the DB is currently taken nightly to allow the Datacentre backup system to backup a consistent dataset. A typical daily/weekly/monthly copy of these backup sets and Apache log logs are also maintained online within the file-system. A weekly offsite delta/conpressed compressed copy of the backup is also taken for D/R purposes.
Task Breakdown
The work needed to upgrade and migrate the forums divides into 4 task areas, which are each separately addressed in a subsection below: (1) the set-up of the ooo-forums VM platform; (2) the application work on the customised phpBB / MySQL application; (3) the actual migration of the live service from Oracle to Apache infrastructure. (4) Work on the rebranding and other content changes to comply with agreed project requirements
Set-up of the ooo-forums VM platform
As this work is under the aegis of the @infra project, I have raised a JIRA ticket [INFRA-3888] Initial Configuration of the VM ooo-forums.apache.org to cover this work.
- Build Test VM using Ubuntu VM and pull a copy of Current Prod to validate. This work will be done within the current OOo (Oracle) working practices to de-risk the migration ahead of task 3. [completed]
- Set up initial build of Target VM under VMware ESXi. This is a bare Ubuntu 10.04 LTS VM configured as a “standard server” and including the Apache-standard security extension. [completed]
- Set up phpbb role
- Document as per INFRA-3888.
Application work on the customised phpBB / MySQL application
- Obtain project endorsement to transfer the application content. [completed]
- Transfer the Current Prod build, with a snapshot of the form content from Current Prod as at 9th Aug 2011 to form the basis for full scale testing pre-prod. This will content be stale and not synced to Current Prod. All forum content changes with be discarded at task 10. [complete]
- Build and integrate Language Pack extensions [completed]
- Regress forum customisations
- Agree impacts / change with NL forum admins
- Full dress rehearsal of migration
- Integrate agreed backup options on Target
- Integrate agreed zero-admin management scripts.
As with our previous build processes, the details and work-in-progress are documented within my User pages on the OOo community wiki here.
Migration of the live service from Oracle to Apache infrastructure
- Obtain go-validation from project including DNS cut-over.
- (24 hrs before DNS cut-over) Bring Current Prod offline for ~3 hrs whilst the current prod D/B and file system updates post 28th Jul 2011 are backed up, transferred to Apache and loaded onto Target. Oracle will enable DNS redirection for http://user.services.openoffice.org/ to the new external public IP address (140.211.11.67). Bring service back on-line: Target is now the Live Production environment, albeit though DNS redirection from the still Oracle-managed openoffice.org domain. The service on ex-Current Prod is now offline.
- DNS cut-over. This takes up to 24hrs to cascade globally. User access to Live Production continues whether direct or redirected via the Oracle IP addr. The now ex-Current Prod can be decommissioned by Oracle as necessary.
Rebranding and other content changes to comply with agreed project requirements
- Agree branding changes to be applied to forums and implement on Target as a test off-production.
- Develop and maintain list of post-cutover improvements and fixed
Technical Issues
- ACM Cache. ACM caching is required, given the usage volumes on the forum. Because of Solaris/Coolstack issues, we previously could not use any of the standard memory variant ACM modules, so TerryE developed a mysqli-based ACM module. With moving to a standard LAMP stack, this is no longer required. acm_acp will be adopted, retiring acm_mysqli.
- Version 3.0.9 upgrade. This new version of phpBB was released last month. The forums are currently running at the previous version 3.0.8. Upgrade to version 3.0.9 is now viewed as a high priority issue.
- phpBB has dropped support for three NL Language packs, (All Spanish dialects and minority languages). Only 44 users are impacted so their default language will be moved to es as part of the migration.