...

As part of the work for

You can see this in PR: https://github.com/apache/daffodil/pull/643

There is an IPv4 These include an IPv4 checksum layer example/test, and   a checkDigits example/test, and AIS Payload Armoring example/test. 

This illustrates These illustrate a way of doing "field extensibility" of layer transforms now, without any particular dynamic layer loading feature.

You pass the name of your layer transform using property

dfdlx:layerTransform=

...

"

...

someLayer"

That name must be the name associated with a class derived from LayerCompiler.

LayerCompilers are dynamically loaded service providers. See the META-INF/services/org.apache.daffodil.layers.LayerCompiler files in daffodil-runtime1-layers/src/main/resources, and daffodil-test/src/test/resources

The 'daffodil-runtime1-layers' Module

The following existing built-in layering transforms are found in the daffodil-runtime1-layers module.

• fourByteSwap
• base64_MIME
• gzip
• lineFolded_IMF
• lineFolded_iCalendar

In daffodil-test there are tests for the above built-ins, but also examples of separately defined layer transforms.

• aisPayloadArmoring 
• IPv4Checksum - stripped down version of IPv4 checksums
• CheckDigits - check digit calculation in the style of the "OTH-Gold" data format. 

Documentation

There are two kinds:

• user documentation for a DFDL schema author on how to use one of the built-in layers.
• developer documentation for how to define a new layer transform for use in a data format DFDL schema. 

User documentation should be via a web page on the Daffodil user site. 

Developer documentation should be within the scaladoc of the Layering API classes. 

• TBD: How to capture and exhibit this scaladoc for user-developers. 

Test Automation Plan

Then somewhere in Scala/Java code before compiling the schema one must call LayerTransformFactory.register(MyLayerTransformFactory), and the MyLayerTransformFactory object has to be defined on the classpath.

In the case of my tests in daffodil-test, the scala test driver code does this call.

This is likely sufficient for application developers using Daffodil via API from software.

However: It won't let you run a new layer transform from the Daffodil CLI tool.

Enabling CLI Loading of Layers

To make the CLI work:

• The enum check in dfdlx.xsd should just be removed entirely, and the diagnostic message if the layer transform is not found should list all the currently registered layer transform names.
• Provide a way that the evaluation of dfdlx:layerTransform property, if the name does not correspond to a registered LayerTransformFactory, it could guess the name of the corresponding LayerTransformFactory object, (e.g, by uppercase first letter, and append "LayerTransformFactory" to the string value of the property) and try to dynamically load that class.

Additional Changes

The following existing (as of Daffodil 3.1.0) layering transforms should be removed from daffodil-runtime1 and put in a daffodil-layers module.

• aisPayloadArmoring 
• fourByteSwap

These enums for LayerTransformEnum should be removed as they are not implemented:

• quotedPrintable
• compress
• base64
• base64url

Test Plan

Testing during Daffodil 'sbt test' Build Cycle

Tests in daffodil-test will illustrate how dynamic loading of a layer from a jar works, but the jar in that case isn't a published jar.

This uses stripped down versions of the IPv4 header checksum, and OTH-Gold-style check-digits. 

• Improvement: Be sure to call these FakeIPv4 or similar renaming to avoid confusion with the real ethernet-ip schemas. 

Testing of User-like Situations - Outside the Daffodil Repository Tree

DFDL Schemas Regression Testing RigRig 

There is a general schema regression testing rig that enables developers to easily maintain a large tree of DFDL schemas for regression testing against daffodil. 

This tool/repo should be published. 

TODO: Create Apache Daffodil project repository for this.

This uses git submodules so that it does not copy schemas.

...