This is the workflow to be used to create a release for either the Daffodil or the Daffodil VS Code Extension project.

...

To improve reproducibility and to minimize the effects and variability of a the users user's environment, release candidates should be created using the "Daffodil release candidate container". Note that although commands to use this container have been tested with and use podman, you should be able to replace podman with docker if if you would rather use it instead. Install the container software of choice using your systems package manager or from the containers container software website. For example:

$ sudo dnf install podman

...

1. Create a [DISCUSS] thread on dev@daffodil.apache.org to make a decision as a community if the timing is correct for a release and what open issues should be resolved for a release. Ensure there is at leas least 72 hours for discussion before moving forward.
     
2. Upon agreement, someone should volunteer to be the "Release Manager" to take the responsibility to prepare the release candidate.
     

3. Prior to creating the release candidate, the version of the project should contain still contains the -SNAPSHOT keyword. Create and merge a pull request to remove this -SNAPSHOT keyword in preparation for a non-snapshot release.
   
   The new version should not contain an -rcX  suffix--the suffix is automatically added to release candidate artifact file names where necessary by the release canidate candidate container.
   

Creating a Release Candidate

...

1. Build the Daffodil release candidate image:

   Code Block
   language bash
   $ podman build -t daffodil-release-candidate /path/to/daffodil.git/containers/release-candidate/

   This may take 10-15 minutes the first time, but should be significantly faster in subsequent runs unless something changes in the image where a full rebuild is needed.
   
   

    This requires substantial free disk space.  If you need to control where podman puts this storage, you can specify the --tmpdir, --root, and --runroot directories as ABSOLUTE paths. (Relative paths are known to cause issues.). Note that if you specify these additional flags on this podman build command you must also specify them for the podman run command below.
     

2. Run the daffodil release candidate container. The command is here, but before running it, prepare the necessary inputs (below) that you will need to provide it when prompted. There are timeouts, and if you wait to dig these things up when prompted they will time out and you'll need to start over.  Also, you must run this command in a standard terminal window that supports cursor-positioning via ANSI escape sequences, which it uses heavily.

   Code Block
   language bash
   $ podman run -it --privileged --rm \ -v ~/.gitconfig:/root/.gitconfig \ -v ~/.gnupg/:/root/.gnupg/ \ -v ~/.ssh/:/root/.ssh/ \ --hostname daffodil.build \ daffodil-release-candidate

   
     Note: This must be run from an ordinary xterm/terminal window, as it uses escape sequences to position the cursor and text. It will not work in an Emacs shell buffer.
   
    Note that the -v option is used to bind mount files/directories from the host system in the container so the container has access to git configurations and gpg/ssh keys.
   
    The --dry-run  option can be provided to the very end of the above podman command (e.g. ... daffodil-release-candidate --dry-run ) to perform a test without publishing any artifacts. An optional parameter can also be provided to perform a dry run with a different GitHub reposotory repository and branch (e.g. ... daffodil-release-candidate --dry-run user/repo@branch).
   
   The --entrypoint /bin/bash option can be provided before the last argument in the above podman command (e.g. ... --entrypoint /bin/bash daffodil-release-candidate ) to interactively view the configuration settings and manually run the daffodil-release-candidate script. This can be useful for debugging the container or testing changes.
      

3. The container will periodically ask for user input (e.g. usernames, passwords) to sign and publish release files. ( Dig out all these before you run the script above so you are ready to provide them when prompted). Note that none of the prompts have a default value - you always must type something before pressing the Enter key. Prompted information includes:
     
   1. The project to release. Options are either "1" for Daffodil or "2" for Daffodil VS Code Extension.
       
   2. Release candidate label. For example: "rc1" if this is the first release candidate. This should not include the release version number (e.g. 2.0.0-rc1  ) –it – it should only contain the rcX part.
        
   3. Long format of your signing key ID. Available keys will be displayed before the script asks for the keys, but can also be found by running gpg --list-secret-keys --keyid-format long  . The key ID should be the 40 character one ID to minimize chance of collisions. Ensure this is the key ID for your apache.org email for your signing key, not some other identity.
       
   4. Git name and email (e.g. "John Doe" and "john.doe@company.com"). This is the name (first name and surname) and email you want to show up as the "Committer" when the release script creates a git tag or commit, which could potentially be different than what is the the bind mounted .gitconfig  file. This is not your GitHub or Apache credentials--simply the name and email address you use for Daffodil commits.
      
        If you have Github's Email Privacy setting on, be sure to use the private email provided in your Github Settings, otherwise the complete-release  script will fail.
       
   5. Apache username and password. This is the username and password credentials used to log in at https://id.apache.org.
       
   6. GitHub SSH key password. The Daffodil repository is cloned using SSH authentication. If your SSH key is password protected, you may be prompted for that password.
        
   7. Private GPG password (also called the 'pass phrase'). The release process signs artifacts with your private GPG key–you will be prompted for a password to enable it to use your private key to sign these artifacts.
        

4. After entering the necessary information the script will run. It may output a few error messages that are not true errors about gpg using your private key as the default key for signing, and a few others also.  The script will perform the actions listed below.
     

   1. Create a zip of the source
      
      
   2. Create helper binaries. For Daffodil this inclues a .tgz, .zip, .msi, and .rpm. For daffodil-vscode this inclues a .vsix).
      
      
   3. Create sha512 checksum and GPG signatures (".asc" files - ASCII armored detached signatures) of the above files.
      
      
   4. Place the above files to in the Apache dist dev directory. This These svn files are not committed until further action is taken.
      
      
   5. Create a signed git tag. This tag is not pushed until further action is taken.
       
   6. For Daffodil only, create javadoc and scaladoc and move to docs in the daffodil site repository docs directory for this release , and create a commit. This commit is not pushed until further action is taken.
      
      

   7. For Daffodil only, stage jars/poms to https://repository.apache.org along with their GPG signatures (".asc"), MD5 checksums (".md5"), and SHA1 checksums (".sha1").
       

     Once the script completes, at that point the terminal window has a shell prompt where commands you type are being given to the running container.

     You must complete the remaining steps without exiting this shell.

    
5. You should verify the existence of the generated files. Note The script will list the files and locations to verify. This includes:
   
   

   1. Verify the checksums and signatures are created in the Apache dist directories and are read ready for commit, for example:

      Code Block
      $ cd /root/daffodil-dist/ $ ls -R $ svn status

        A script that can be adapted to verify the signatures and checksums is in the comments of this page.
       

   2. Verify the git tag is attached to the correct commit in the project repo, for example when releasing the the Daffodil project:

      Code Block
      language bash
      $ git -C /root/daffodil/ log -n 1

          

   3. For Daffodil only, verify the javadoc and scala docs for the version to be released exist in the daffodil site repository.

      This can be done with git log:

      Code Block
      language bash
      $ git -C /root/daffodil-site/ log -n 1 -p

      or via the file system:

      Code Block
      $ cd /root/daffodil-site/ $ ls -R

      
      

   4. For Daffodil only, verify the all the expected jars/poms at https://repository.apache.org/ exist.
       
        A specific release of Daffodil may or may not be cross-compiled for more than one version of Scala. (As of Daffodil 3.23.0, only Scala 2.12 is supported, but future releases may support more than one version of Scala.)
      For all modules, verify the proper scala version(s) exists, and for each scala version, there exists a jar, javadoc jar, sources jar, and pom, and for each of those files there exists the GPG signature (".asc" armored ASCII file), and the associated md5 and sha1 checksums of all of these. To do so, visit https://repository.apache.org/, login in the top right using id.apache.org credentials, select "Staging Repositories" on the left, and find the orgapachedaffodil-XXXX repository (there should be only one). Inspect the "Content" tab to make sure the appropriate jars are uploaded and appear valid.
       
        A script which can be adapted to download and verify the signatures and checksums for the contents of repository.apache.org is in a comment on this page. 
       
   5. If any of the above do not look correct, perform the following steps:
        
      1. For Daffodil only, "Drop" the stages files at https://repository.apache.org--log into that URL with id.apache.org credentials, check the box for the Daffodil staging repository, and click "Drop" at the top.
          
      2. Type exit  to close the container. All files/commits created in the container will be deleted.
          
      3. Fix the issue and repeat the "Create Release Candidate" process from the beginning.
           
6. After verifying all is correct, follow the instructions to complete the release candidate. These steps include:
      

   1. Run the command:

      Code Block
      language bash
      $ /root/complete-release

      All the previous commands have prepared commits and tags in the three repositories (Apache Dist, Daffodil, and Site). Running this command will push those commits and tag to the remote repositories.
        

   2. For Daffodil only, "Close" the staged files at https://repository.apache.org--log into that URL with id.apache.org credentials, check the box for the Daffodil staging repository, and click "Close" at the top.
          
   3. Type exit  to close the container.

...

The previous steps have pushed the new scaladoc and javadoc docs to the daffodil-site Daffodil Site Repository repository. In order for your local sandbox copy of that repository to have this same content, you must fetch those updates, which is typically done via:

...

The most recent commit should then show the addition of new scaladoc and javadoc docs to the site. 

Create a new release file in the site/_releases/ directory in the Daffodil Site Repository directory, updating the page to include a summary of the changes and links/descriptions of the bugs that were fixed in this release. The usual procedure for gathering the materials of interest bugs fixed is as follows:

1. Go to JIRA's Releases page to view only the unreleased versions.
2. If you see only one unreleased version in the list, fill out the Version name text box of the next unreleased version above the list and click Add to add a new unreleased version to the list. 
3. If the release candidate version has a green bar that is not completely filled in, you need to move some issues to the next unreleased version.  Click on the release candidate's version to view its issues.
4. Click on the number of "Issues to do" above the list of issues (the number will be non-zero if the green bar was not completely filled in).
5. Click on "View in Issue Navigator" above the list of issues.
6. Do a Bulk Change to move these issues to the next unreleased version (Top Right: Tools > Bulk Change, select all issues, click Next, select Edit Issues, click Next, select Change Fix Version/s, select Replace all, pick the next unreleased version, and so on).
7. Go back to JIRA's Releases page and click on the release candidate's version again. If the number of "Issues in progress" is not zero, you also need to click on it and move its issues to the next unreleased version too (repeat steps 5-6 and come back to this step.
8. Click on "Release Notes" near the top of the page.
9. Click on "Configure Release Notes", change "Please Select Style" to Text, and press Create.
10. Scroll down to the bottom where you will find a text area with the project release notes ready to be copied to another document.
11. Copy the release notes to the new release file in the site/_releases/ directory, change all "[DAFFODIL-XXX]" to "{% jira XXXX %}", change all indentation and category lines to markdown format, and save your changes.
12. Find all tickets that have the version being released as their Fix Version, that are not resolved or closed (i.e Unresolved), and do a bulk change (Top Right: Tools > Bulk Edit) and move them to the next release (under Unreleased Versions)
       
13. Do a search for all tickets that have the version being released as their Fix Version, and export it as a CSV (Top Right: Export > CSV (All fields)
      
14. In Excel (or equivalent), delete the columns that aren't needed, then do some manipulation to get that list of bugs in the format for the release page(i.e \{% Jira XXXX %} Title)
      
15. For changes that need more description (e.g. big features), create sections on the release page and add those descriptions.
      

16. For the section on Dependency updates, the following command shows the changes or new dependencies that should be mentioned. 

    Code Block
    language bash
    $ git diff v3.13.0 -- project/Dependencies.scala

    You will of course change "v3.13.0" in that command to the tag of the prior release to the one you are preparing. 

...