Overview

The OpenOffice.org Community forums are hosted under the domain http://user.services.openoffice.org/ and comprise 9 National Language (NL) forums: Chinese, Dutch, English, French, Hungarian, Italian, Japanese, Spanish and Vietnamese (“Current Prod”). A total of 15 language packs are supported across the 9 NL forums, (the 9 main language plus 6 other major language variants). Across all forums there are some 75,000 registered members, who have created 300,000 posts on 78,000 topics. A typical usage is ~200,000 page requests/day, with another ~800,000 images and other resources needed to load these pages.

The forums are currently implemented in phpBB at version 3.0.8 running on Solaris Coolstack on a 4 core Solaris zone in the Oracle Hamburg machine room (at an average 3-8% us+sy load). phpBB works on a standard LAMP stack, and the test and development systems, which shadows the production system (“Test”), are currently implemented on an Ubuntu 10.04 LTS VM running a standard Ubuntu server minimal LAMP configuration. The phpBB application uses both a file system and D/B (currently MySQL) to maintain application data. These are currently some 2.5GiB and 1.5GiB in raw size. The target infrastructure in Apache.org will be an Ubuntu 10.04 LTS VM running on the Apache ESXi production environment (“Target”).

The forum software has a number of special-to-OOo modifications, and the process for carrying these forward over phpBB upgrade has now been refined over 5 phpBB version upgrades. The configuration also includes a number of custom scripts which will require some rework for an Apache.org environment and to comply with Apache SysOps and Security Standards.

However, since cloning of this system across domains is routinely carried out as part of current support processes, any relocation to Apache Infrastructure is viewed as a low technical risk activity.

The work needed to upgrade and migrate the forums divides into 4 task areas, which are each separately addressed in a subsection below: (1) the set-up of the ooo-forums VM platform; (2) the application work on the customised phpBB / MySQL application; (3) the actual migration of the live service from Oracle Inc to the Apache Foundation infrastructure. (4) Work on the rebranding and other content changes to comply with agreed project requirements

Detailed System Configuration Notes

The configuration is largely build on a phpBB build. This is currently at version 3.0.8. This one behind current, but upgrade to 3.0.9 was deferred in view of the pending relocation to Apache Infrastructure, and will be included in this migration.

Due to architectural limitations in phpBB, the 10 forums are implemented as 10 separate logical systems each with its own directory hierarchy for code and application data and its own schema within MySQL. However, since the 3.0.4 upgrade, all NL-specific mods have been merged into a single common code base, with each forum symlinked back to a version specific directory within /var/lib/phpBB. This is mainly for manageability, but this also has a performance dividend as the APC opcode cache resolves all 10 copies of each php module back to its real file-name and thus shares the same cache space.

Within /var/lib/phpBB two directories are maintained for each phpBB version: refXYY and comXYY, where XYY is the phpBB version number. refXXY contains the unpacked "out-of-box" kits as downloaded from the phpBB website, and comXYY the site-specific version which is symlinked into the live hierarchy. The site-specific version is generated by a patch file on the refXYY version. This file is the key customisation configuration item (CI) (currently ~990 lines) which must be regressed across each phpBB version upgrade. For each phpBB version upgrade, this patch file is regressed / refined / retested on a dev VM, then a new pair of directory hierarchies is installed on prod prior to cut-over.

The Apache and MySQL configurations are tuned to this application and transaction rate, but this tuning is pretty standard. Nothing esoteric.

The configuration also a management account use to implement a number of batch script to support routine D/B and file-system housekeeping / monitoring and to implement a mailbox to request forum posting system. The goal in doing this was to ensure that the application and system could run "zero-admin" between upgrades.

No end-user access to the system is required as all user, (forum) administration and moderator functions are implement though phpBBs own web interface. Most of the system administration is automated, but the system administrator does require occasional secure interactive access and sudo access to privileged operations. The watchdog scripts also require limited access. (E.g. They do a loop-back to check that the application is up and running and will bounce the AMP as an immediate action if a system stall is detected. This has been an intermittent problem with phpBB and the coolstack in the past).

A hot-copy backup of the DB is currently taken nightly to allow the Datacentre backup system to backup a consistent dataset. A typical daily/weekly/monthly copy of these backup sets and Apache logs are also maintained online within the file-system. A weekly offsite delta/compressed copy of the backup is also taken for D/R purposes.

Task Breakdown

The work needed to upgrade and migrate the forums divides into 4 task areas, which are each separately addressed in a subsection below: (1) the set-up of the ooo-forums VM platform; (2) the application work on the customised phpBB / MySQL application; (3) the actual migration of the live service from Oracle to Apache infrastructure. (4) Work on the rebranding and other content changes to comply with agreed project requirements

Set-up of the ooo-forums VM platform

As this work is under the aegis of the @infra project, I have raised a JIRA ticket [INFRA-3888] Initial Configuration of the VM ooo-forums.apache.org to cover this work.

Build Test VM using Ubuntu VM and pull a copy of Current Prod to validate. This work will be done within the current OOo (Oracle) working practices to de-risk the migration ahead of task 3. [completed]
Set up initial build of Target VM under VMware ESXi. This is a bare Ubuntu 10.04 LTS VM configured as a “standard server” and including the Apache-standard security extension. [completed]
Set up phpbb role
Document as per INFRA-3888.

Application work on the customised phpBB / MySQL application

Obtain project endorsement to transfer the application content. [completed]
Transfer the Current Prod build, with a snapshot of the form content from Current Prod as at 9th Aug 2011 to form the basis for full scale testing pre-prod. This will content be stale and not synced to Current Prod. All forum content changes with be discarded at task 10. [complete]
Build and integrate Language Pack extensions [completed]
Regress forum customisations
Agree impacts / change with NL forum admins
Full dress rehearsal of migration
Integrate agreed backup options on Target
Integrate agreed zero-admin management scripts.

As with our previous build processes, the details and work-in-progress are documented within my User pages on the OOo community wiki here.

Migration of the live service from Oracle to Apache infrastructure

Obtain go-validation from project including DNS cut-over.
(24 hrs before DNS cut-over) Bring Current Prod offline for ~3 hrs whilst the current prod D/B and file system updates post 28th Jul 2011 are backed up, transferred to Apache and loaded onto Target. Oracle will enable DNS redirection for http://user.services.openoffice.org/ to the new external public IP address (140.211.11.67). Bring service back on-line: Target is now the Live Production environment, albeit though DNS redirection from the still Oracle-managed openoffice.org domain. The service on ex-Current Prod is now offline.
DNS cut-over. This takes up to 24hrs to cascade globally. User access to Live Production continues whether direct or redirected via the Oracle IP addr. The now ex-Current Prod can be decommissioned by Oracle as necessary.

Rebranding and other content changes to comply with agreed project requirements

Agree branding changes to be applied to forums and implement on Target as a test off-production.
Develop and maintain list of post-cutover improvements and fixed

Technical Issues

ACM Cache. ACM caching is required, given the usage volumes on the forum. Because of Solaris/Coolstack issues, we previously could not use any of the standard memory variant ACM modules, so TerryE developed a mysqli-based ACM module. With moving to a standard LAMP stack, this is no longer required. acm_acp will be adopted, retiring acm_mysqli.

Version 3.0.9 upgrade. This new version of phpBB was released last month. The forums are currently running at the previous version 3.0.8. Upgrade to version 3.0.9 is now viewed as a high priority issue.

phpBB has dropped support for three NL Language packs, (All Spanish dialects and minority languages). Only 44 users are impacted so their default language will be moved to es as part of the migration.