In the flex-utilities repo, there is an app called MD5Checker.  It is used to periodically verify the MD5 checksums of some of the dependency downloads for Flex such as the AIR SDK, playerglobal.swc, and Google Closure Library.  The providers of these downloads do not appear to have posted MD5s, but we've found that the Installer frequently has problems downloading these artifacts correctly, so we've taken to computing the MD5s ourselves and posting the results in flex.apache.org/installer/sdk-installer-config-4.0.xml.

In hindsight, it probably would have been better for the CI server to compute MD5s and leave them in files on the CI server, but the installer scripts are hardwired to look on flex.apache.org, so we'll probably live with this scheme forever.

There is a CI server job on apacheflexbuilds.cloudapp.net:8080 that computes the MD5s every hour.  Instead of downloading each very large AIR SDK, it checks the eTag http header and only downloads and recomputes the MD5 if the eTag has changed.

If a file's MD5 changes, the CI server send a build failure to the dev@ mailing list.  We originally thought MD5s would not change very often, but Adobe frequently updates beta SDK and for some reason occasionally alters the contents of existing older SDKs.

To update, check the email.  It will tell you which node(s) in sdk-installer-config-4.0 to change, but be careful.  There are a lot of similar entries in the file.

In the flex-utilities repo, there is a build.xml in the MD5Checker folder.  It builds and runs MD5Checker, but also can commit the updated file to the web-site SVN if you supply the path to your SVN working copy for flex.apache.org.

To execute the update, create a local.properties file that specifies svn.site=<path to web-site working copy>.  Then the build script will copy the modified sdk-installer-config-4.0.xml to the working copy and commit it to the web-site SVN.  It will prompt for apache.username which you can also supply in local.properties and it will prompt for password.  The password will not be obfuscated so be careful who is watching your screen.  You will still need to use ASF CMS to push the updated file to the production web server so future MD5Checker runs and our Installer customers will see the changes.

The CI Server has limited memory, and the AIR SDKs have grown in size such that if an AIR SDK changes, the CI Server will hang while computing the new MD5.  You can tell the MD5Checker job is hung because the progress bar in the Jenkins UI will be red.  If you see this, manually cancel the Jenkins job so the other jobs can run, then run MD5Checker locally and update the sdk-installer-config-4.0.xml file as described above.

You may also notice that some times there are several "build failed" emails followed by a "build fixed" then later, more "build failed" emails.  I'm not quite sure why this happens, but my theory is that things like the AIR SDK is published on a CDN and CDNs are essentially mirrors behind the same URL.  It takes time for these files to propagate to all mirrors, and during that time, different runs of MD5Checker will pull the new version and fail and sometimes the old version a succeed.  I've found that after a day or two, we get pretty consistent failures and updating sdk-installer-config-4.0 as described above is warranted.

Another occasional failure scenario is that the "build failed" email shows that the MD5 checksum is the same, but the cacheID (which is the eTag HTTP Header) is different.  I don't know why that sometimes is the case, but if you see more and more "build failed" messages like that, the remedy is to copy the cacheID from the "new version" in the email to the cacheID2 in the sdk-installer-config-4.0.xml file and update the file in SVN and on the web-site as described above.

Very occasionally, you will update SVN with new MD5s and MD5Checker will still fail!  When this happens, manually check http://flex.apache.org/installer/sdk-installer-config-4.0.xml to see if your changes actually got published.  Sometimes you will miss a step in our web-site publishing process, but sometimes, the ASF CMS just doesn't seem to see the change.  If that happens, I would try manually changing something in the file that doesn't matter, like a cacheID2 for the new entry, or adding an XML comment somewhere and publish again until you see the updates.  If you are having trouble after a few attempts, contact Apache Infra. 

In some future, MD5Checker and/or the build.xml could be improved to auto-update the sdk-installer-config-4.0.xml automatically, without human intervention, but that would imply leaving SVN credentials on the CI Server and I am not willing to put my credentials on that machine.  Also, I haven't found a way to "hit the publish button" in ASF CMS through Ant.  I think it is possible if someone wants to try.

Adding New Entries

To add a new entry, copy an existing entry and update the tag name from VersionAA to VersionAB or whatever is next in the sequence.  Update occurrences of the old version number to the new version number so that the right URL is represented.  No need to worry about the cacheIDs or MD5 entry.  Then save it, commit it to SVN, and publish it.  Then go run MD5Checker locally and follow the update steps above.  You could try to get the cachdID and MD5 entries right by computing the MD5 with some utility and sniffing the eTag header with some packet sniffer, but IMO, it isn't worth it and you might as well check that MD5Checker can read your changes to the sdk-installer-config-4.0.xml file.

 

 

  • No labels

6 Comments

  1. Piotr Zarzycki

    Alex Harui,

    I did follow your suggestion and build MD4Checker locally, using build.xml.

    Initial fail was:

    BUILD FAILED
    D:\Work\flex-utilities\MD5Checker\build.xml:66: D:\Work\flex-utilities\MD5Checker\MD5CheckerLog.txt doesn't exist

    I've create that file. What should be the content of that file ? My fail looks like that:

    https://paste.apache.org/1heI

    Piotr

    1. Alex Harui

      When you get a "error while loading initial content" that means there is a mismatch between the -app.xml file and the SWF.  The -app.xml is  set to AIR 3.7, so the SWF has to be built with an older AIR SDK.  Looks like you are using AIR 28.  You might be able to specify swfVersion 20 and still use AIR 28.  The reason we don't update the -app.xml is so we don't have to keep updating AIR on the CI servers. 

      1. Piotr Zarzycki

        I was able to run on MD5Checker with your suggestion. Unfortunately on Windows it will be failing due to large size of AdobeAir - The same fail as it was in Installer.

        You probably using Mac to run MD5Checker ? 

        1. Alex Harui

          Yes I use Mac.  I thought we changed the AntOnAIR tasks that MD5Checker uses to fix the Installer, so I was expecting MD5Checker to no longer have problems.  Also, I think the script runs MD5Checker using ADL so maybe you can use swfVersion=20 run against a 64-bit AIR SDK.

      2. Piotr Zarzycki

        I have downloaded manually all files and generate MD5 checks. Only cache values may not be corrected.

        1. Alex Harui

          If the cache values are not right, the CI Server will run MD5Checker and try to download the AIR SDK so if there are memory issues still, then the CI Server will likely hang on running MD5Checker.