(Work in progress)

See also basic layering documentation: Proposal: Data Layering for Base64, Line-Folding, Compression, Etc

Extension of it for Checksums/CRC/Parity: Proposal: Checksums, CRC, Parity - Layering Enhancements

A Basic Mechanism Exists

As part of the work for Unable to render Jira issues macro, execution error. , there are new test layers in the daffodil-test module.

These include an IPv4 checksum layer example/test,  a checkDigits example/test, and AIS Payload Armoring example/test. 

These illustrate a way of doing "field extensibility" of layer transforms now, without any particular dynamic layer loading feature.

You pass the name of your layer transform using property

dfdlx:layerTransform="someLayer"

Then somewhere in Scala/Java code before compiling the schema one must call LayerCompilerRegistry.register(SomeLayerCompiler), and the SomeLayerCompiler object has to be defined on the classpath.

In the case of the tests in daffodil-test, the scala test driver code does this call.

This is sufficient for application developers using Daffodil via API from software.

However: It won't let a user run a new layer transform from the Daffodil CLI tool.

Enabling CLI Loading of Layers

To make the CLI work:

• The dfdlx:layerTransform property, if the name does not correspond to a registered LayerCompiler, it could dynamically load a class corresponding to it, using the service loader techniques already in use for UDFs and loadable validators. 

This mechanism would also eliminate the need for API code to explicitly call the LayerCompilerRegistry.register(...) method. However, it is good practice to do that anyway, so as to get a Java linkage error (build/compile time of the software) rather than a dynamic runtime error. 

The 'daffodil-runtime1-layers' Module

The following existing built-in layering transforms are found in the daffodil-runtime1-layers module.

• fourByteSwap
• base64_MIME
• gzip
• lineFolded_IMF
• lineFolded_iCalendar

In daffodil-test there are tests for the above built-ins, but also examples of separately defined layer transforms.

• aisPayloadArmoring 
• IPv4Checksum - stripped down version of IPv4 checksums
• CheckDigits - check digit calculation in the style of the "OTH-Gold" data format. 

Documentation

A web page on the Daffodil user site should contain links to doc about each layer transform that is part of the daffodil-layers jar. 

Scaladoc(?) of each Layer Transform and of the Layering API. 

• These links should be to the scaladoc pages for each of these transforms.
• Each  transform's scaladoc should be where the transforms are thoroughly documented. 
• ??? There are pros/cons here.... needs discussion 
  • scaladoc is created in src/main/scala and it would end up almost repeating the example code which is in the src/test/resources and src/test/scala directories

Test Automation Plan

Testing of User-like Situations - Outside the Daffodil Repository Tree

DFDL Schemas Regression Testing Rig 

There is a general schema regression testing rig that enables developers to easily maintain a large tree of DFDL schemas for regression testing against daffodil. 

This tool/repo should be published. 

TODO: Create Apache Daffodil project repository for this.

This uses git submodules so that it does not copy schemas.

This should then test the "real" schemas mentioned below.

Updating "Real" DFDL Schema Examples that use Layers

The ethernet-ip DFDL schema (on github) can be made into a full exercise of  IPv4 checksums.

• It could further be extended to do ICMP, TCP, UDP checksums also. 
• The PCAP schema which uses this ethernet-ip schema would inherit this also. 

The OpenDFDL (on github - note: should move to DFDLSchemas) example checkDigits schema can be made into a complete example.

The DFDLSchemas (on github) GPS-SPS example computes parity bits. This could also be made into a complete test example - though this is quite large.