Page History

(This page is work-in-progress/WIPAs-is-built 2021-10-06)

This page describes an enhancement to the DFDL language, the Daffodil DFDL Layering extension, enabling DFDL schemas using this extension to describe data formats in such a way as to enable algorithmic computations of CRCs, Checksums, Parity bits, etc. These must be are checked when parsing, to insure data validity and/or well-formedness, and must be computed are recomputed and incorporated into the data infoset when unparsing data. 

Principles of Operation

If Daffodil's layers are able to read and write DFDL variables, then these can be . Variables are used to provide parameters to layer computations, and to receive results back from layer computations. 

...

At unparse time, a variable that is set by a layer computation can also calculate the value of a checksum/crc/parity across the layer data in the same manner as when parsing. The element that will carry carries this value in the infoset would can have a dfdl:outputValueCalc property referencing this variable to insert the new checksum/crc/parity value into the infoset for subsequent unparsing by normal means.

All the normal DFDL behaviors around forward referencing would apply if the infoset element carrying the checksum/crc/parity value appears before the layered sequence itself. 

Some details:

• Layer transformations must be are able to read and write DFDL variables, which means they need access to the parse or unparse state of the computation, not just the I/O layer state. 
• When parsing, the DFDL variables read or written by a layer transformation must be defined or it is a runtime an SDE. 
• When unparsing, DFDL variables read by a layer transformation must either have a value, or the read must cause causes suspension of the layer transformation if the variable is not yet defined. This layer transformation must be is resumed once the variable gets its value. 
• When unparsing, computations of dfdl:outputValueCalc, or dfdl:length or other runtime-valued expressions can reference DFDL variables that will are (to be) set by a layer transform. Such computations must will suspend if the layer transformation has not yet set the variable, and resume when the variable is set. 
• The restriction where a layered sequence may not carry DFDL statements such as setVariable or newVariableInstance is lifted. Such layered sequences are likely to be the locations where one or more newVariableInstance statements should appear to create instances for use by the layer transformation. 

Examples

Worked examples showing layering transformations that implement checksums and parity are:

• IPv4 packets - See the ethernetIP DFDL schema on the DFDL schemas github site. 
  daffodil-test src/test/resources/org/apache/daffodil/layers/IPv4.tdml test, related IPv4.dfdl.xsd schema. This is fake, but very much like IPv4. 
  
  • There is also a PR to update the ethernetIP github DFDL schema project to compute checksums.TBD: currently a pull request:  https://github.com/DFDLSchemas/ethernetIP/pull/1. However, there are checksums in the IPv4Header, UDP, TCP, ICMP, etc. So the work involved here is extensive.
  • This work likely should be done only after

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key DAFFODIL-1927

    is complete so that the pluggable layers that compute these checksums can be created as a plugin. 
• GPS data with parity - See the GPS-SPS DFDL schema on the DFDL schemas github site.
  • TBD: currently a pull request: https://github.com/DFDLSchemas/gps-sps/pull/1
  • This work likely should be done only after

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key DAFFODIL-1927

    is complete so that the pluggable layers for this parity can be done as a plugin. 
• Check-digit computation - See the daffodil-test src/test/resources/org/apache/daffodil/layers/TestCheckDigit.tdml test file and related checkDigit.dfdl.xsd schema.