THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
This page is a location to centralize notes about implementation of advanced DFDL features, and to highlight places where the DFDL specification may need to be clarified, augmented, or corrected.
Infoset can contain elements that have dfdl:outputValueCalc
It is essential that when unparsing it is allowed for the infoset to already contain output-computed elements. The values of these will be discarded and recomputed during unparsing, but they have to be tolerated if present, and created if not present.
Allow IVC/OVC on Global Elements, SimpleTypes
The restriction that IVC/OVC properties are not allowed on global element is arbitrary. It was probably put in place with an eye towards reducing complexity of implementations. It is, however, unnecessary and gets in the way of some useful cases. This restriction should be removed.
Similarly, IVC/OVC are not allowed to be declared on simple types. This restriction is also unnecessary and makes reuse harder.
Allowing some DFDL properties to Coexist with IVC
For elements with dfdl:inputValueCalc, the element is not represented, so format properties don't make sense and should cause SDE if expressed directly on the element. However, some properties are completely compatible with dfdl:inputValueCalc. In particular dfdl:choiceBranchKey does not conflict, because it is not a format property.
This restriction in DFDL is easily worked around by wrapping a sequence around the branch element having IVC property, and putting the dfdl:choiceBranchKey on the sequence.
SDE for element in hidden group that has no default, nor dfdl:outputValueCalc
Within a hidden group, if an element has no fixed/default value, and no OVC, then because it is hidden and so does not appear in the infoset, there's no way for it to get a value at all when unparsing.
...
This SDE is too strong: If the element has IVC, then it doesn't need a value for unparsing unless an expression that is used at unparse time (i.e., not in an assert or discriminator) references the element. In many cases the value of an IVC may be used only in dfdl:occursCount expressions, or in assert/discriminator test expressions. Since those aren't evaluated when unparsing, there is no unparsing situation would never need the value, so the SDE is too strong.
IVC and OVC on the same element
In many situations we have found that we need both IVC and OVC on the same element. This occurs when the element is in a hidden group, and is essentially being used as a variable.
...
Another use of this is that sometimes for unparsing, one needs a location for an intermediate calculation to be saved (to avoid redundantly computing it). Such an element needs to carry an OVC, but also have no representation. Having IVC on it explicitly makes it have no representation, no alignment, and no implications as far as separators showing up for it in separated sequences. There is no way to achieve this other than using IVC.
Array Variables and dfdl:newVariableInstance
In many cases dfdl:newVariableInstance is not sufficient because what are needed are array variables.
...
The headache being the relative path to it., and then the fact that the most natural thing to do is to put both IVC and OVC on it (though the OVC may not be needed in many cases).
...
Variables: The direction property, and forward reference from dfdl:setVariable value and dfdl:newVariableInstance defaultValue Expressions
Consider unparsing with dfdl:setVariable expression referring to OVC forward-referencing element.
| Code Block |
|---|
<xs:element name="len" type="xs:int" dfdl:outputValueCalc="{ dfdl:valueLength(../data) + 10 }"/>
<xs:sequence>
<xs:annotation><xs:appinfo ...>
<dfdl:setVariable name="var">{ ../len }" />
</xs:appinfo></xs:annotation>
<xs:element name="data" dfdl:length="{ $var }">
....
</xs:element>
....
</xs:sequence> |
In the above, when unparsing, the output value calc for len can be evaluated, but we must delay its evaluation and unparsing until the subsequent data element is available.
The next thing the unparse has to do, after delaying the unparsing of 'len' is set the var variable. This requires the value of len, which has been deferred.
We have real schemas (e.g., even PCAP) where this occurs.
It is clear that variables need to be able to be evaluated at unparse time, and the expressions used with them to default or set their values need to be able to forward-reference into the infoset when they are evaluated at unparse time.
We have prototyped a extension direction property for dfdl:defineVariable, which allows one to declare the variable dfdl:direction 'parseOnly', 'unparseOnly', or 'both'.
We believe using dfdl:setVariable is inconsistent with use of dfdl:newVariableInstance having a default value, as there are race conditions between reading a default value and setting the value that are complex. Stylistically, one should declare a variable with no default value nor external value, if one intends to use it with dfdl:newVariableInstance.
The following scenario comes from PCAP and illustrates use of dfdl:newVariableInstance and variables with forward-referencing defaultValue expressions:
| Code Block |
|---|
<!-- Internally used by IPAddressGroup at unparse time
These are IPAddressGroup's local variables. -->
<dfdl:defineVariable name="remainingDottedAddr" type="xs:string" dfdlx:direction="unparseOnly"/>
<dfdl:defineVariable name="priorRemainingDottedAddr" type="xs:string" dfdlx:direction="unparseOnly"/>
<!-- Parameter for IPAddressGroup used at unparse time -->
<dfdl:defineVariable name="ipAddressElement" type="xs:string" dfdlx:direction="unparseOnly"/>
<!--
A PCAP schema has two different IP addresses, IPSrc and IPDest.
They are defined in terms of a common group definition pcap:IPAddressGroup
which works like a common "subroutine" within the schema. The variable pcap:ipAddressElement is
a formal parameter of IPAddressGroup.
-->
<xs:group name="IPSrcGrp">
<xs:sequence>
<xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
<!--
Note how this defaultValue expression forward references to the
Element containing the string (e.g, 1.2.3.4) to be used at unparse time only,
as the variable is itself an 'unparseOnly' variable declaration.
-->
<dfdl:newVariableInstance ref="pcap:ipAddressElement" defaultValue='{ IPSrc }'/><!-- example 1.2.3.4 -->
</xs:appinfo></xs:annotation>
<xs:element name="IPSrcString">
<xs:complexType>
<xs:group ref="pcap:IPAddressGroup"/>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:group>
<xs:group name="IPDestGrp">
<xs:sequence>
<xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:newVariableInstance ref="pcap:ipAddressElement" defaultValue='{ IPDest }'/><!-- example 1.2.3.4 -->
</xs:appinfo></xs:annotation>
<xs:element name="IPDestString">
<xs:complexType>
<xs:group ref="pcap:IPAddressGroup"/>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:group>
<xs:group name="IPAddressGroup">
<xs:annotation><xs:documentation><![CDATA[
This group is a reusable subroutine of DFDL. It parses a string like 1.2.3.4 into 4 integers named Byte1, Byte2, Byte3, Byte4
containing the "." separated integers.
Arguably, this is extreme for DFDL. I mean an infoset with the 4 bytes it should be a good enough parsed representation of the
4 bytes. But this serves as a useful example regardless.
This is used to unparse IP addresses expressed in the dotted notation that is common.
There is one parameter. Users must bind the $pcap:ipAddressElement variable to the string to be so parsed
using dfdl:newVariableInstance.
]]></xs:documentation></xs:annotation>
<xs:sequence>
<xs:sequence>
<xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:newVariableInstance ref="pcap:priorRemainingDottedAddr"
defaultValue='{ $pcap:ipAddressElement }'/><!-- example 1.2.3.4 -->
<dfdl:newVariableInstance ref="pcap:remainingDottedAddr"
defaultValue='{ $pcap:priorRemainingDottedAddr }'/><!-- example 1.2.3.4 -->
</xs:appinfo></xs:annotation>
<xs:element name="Byte1" type="xs:unsignedByte"
dfdl:outputValueCalc="{
xs:unsignedByte(fn:substring-before($pcap:remainingDottedAddr, '.'))
}"/>
<xs:sequence>
<xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:newVariableInstance ref="pcap:priorRemainingDottedAddr"
defaultValue='{ $pcap:remainingDottedAddr }'/><!-- example 1.2.3.4 -->
<dfdl:newVariableInstance ref="pcap:remainingDottedAddr"
defaultValue='{ fn:substring-after($pcap:priorRemainingDottedAddr, ".") }'/><!-- example 2.3.4 -->
</xs:appinfo></xs:annotation>
<xs:element name="Byte2" type="xs:unsignedByte"
dfdl:outputValueCalc="{
xs:unsignedByte(fn:substring-before($pcap:remainingDottedAddr, '.'))
}"/>
<xs:sequence>
<xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:newVariableInstance ref="pcap:priorRemainingDottedAddr"
defaultValue='{ $pcap:remainingDottedAddr }'/><!-- example 2.3.4 -->
<dfdl:newVariableInstance ref="pcap:remainingDottedAddr"
defaultValue='{ fn:substring-after($pcap:priorRemainingDottedAddr, ".") }'/><!-- example 3.4 -->
</xs:appinfo></xs:annotation>
<xs:element name="Byte3" type="xs:unsignedByte"
dfdl:outputValueCalc="{
xs:unsignedByte(fn:substring-before($pcap:remainingDottedAddr, '.'))
}"/>
<xs:sequence>
<xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:newVariableInstance ref="pcap:priorRemainingDottedAddr"
defaultValue='{ $pcap:remainingDottedAddr }'/><!-- example 3.4 -->
<dfdl:newVariableInstance ref="pcap:remainingDottedAddr"
defaultValue='{ fn:substring-after($pcap:priorRemainingDottedAddr, ".") }'/><!-- example 4 -->
</xs:appinfo></xs:annotation>
<xs:element name="Byte4" type="xs:unsignedByte"
dfdl:outputValueCalc="{
xs:unsignedByte($pcap:remainingDottedAddr)
}"/>
</xs:sequence>
</xs:sequence>
</xs:sequence>
</xs:sequence>
</xs:sequence>
</xs:group>
<!-- These groups are then used like so -->
...
<xs:sequence dfdl:hiddenGroupRef="pcap:IPSrcGrp"/>
<!-- IPSrc will be of the usual IP address form: 1.2.3.4 -->
<xs:element name="IPSrc" type="xs:string"
dfdl:inputValueCalc="{
fn:concat(../IPSrcString/Byte1, '.',
../IPSrcString/Byte2, '.',
../IPSrcString/Byte3, '.',
../IPSrcString/Byte4) }"/>
<xs:sequence dfdl:hiddenGroupRef="pcap:IPDestGrp"/>
<xs:element name="IPDest" type="xs:string"
dfdl:inputValueCalc="{
fn:concat(../IPDestString/Byte1, '.',
../IPDestString/Byte2, '.',
../IPDestString/Byte3, '.',
../IPDestString/Byte4) }"/>
.... |
The above DFDL schema enables the 4 bytes of IP Source Address and 4 bytes of IP Destination Address to be parsed into this logical XML infoset:
Data Bytes (hex) 01 02 03 04 05 06 07 08
| Code Block |
|---|
<IPSrc>1.2.3.4</IPSrc>
<IPDest>5.6.7.8</IPDest> |
These will unparse back to the same 8 bytesThis breaks the rule that when variables are evaluated, things they reference must have already been evaluated. Basically, when we delay evaluating the OVC for len we are suspending anything that depends on len having a value as well.
Parse-Time Forward Reference
Several standards in the MIL-STD and NATO STANAG space express formats using a forward reference idiom that requires those forward references to be used at parse time.
...
Note that this issue doesn't actually involve IVC/OVC. It is independent of them. There is a workaround, and so this problem doesn't require a solution as part of DFDL v1.0.
Restricting IVC/OVC in Unordered Sequences
Elements having IVC are not allowed as the root of a choice branch. For the same reasons IVC elements should not be allowed as children of an unordered sequence, or a sequence with floating elements.
Unparsing and Choice Groups, Especially Hidden Choice Groups
There are two problems related to unparsing and xs:choice.
Problem 1: dfdl:outputValueCalc that must refer into a child element inside a xs:choice
Example of this is this fragment of a messaging format where each message has a label and sublabel element that indicate which message it is. Logically, users think of, and expect to find the label and sublabel elements within the message structure, but we must have them before the element in order to use an xs:choice with dfdl:choiceDispatchKey to determine the message.
...
The ESA DFDL4Space project has invented a similar wildcard concept as a DFDL extension. They allow a match not to just an entire name, as a path step, but the allow extensions of a name prefix. E.g, ../pathStep(.*)/ meaning any step the name of which begins with "pathStep". The ".*" notation is motivated by regular expressions, but there is no use of patterns richer than ".*". This generality is not needed as far as we can see because one can always model such data creating a parent element named "pathStep" in this case, and then the children of this have names that would match the "*" part of the pattern. So instead of "pathStep(.*)" matching pathStepData1 and pathStepData2, you would structure the elements so that "pathStep/*" matches /pathStep/Data1 and pathStep/Data2.
Problem 2: Choices inside hidden groups
The DFDL Spec discussion of unparsing choices assumes an element that appears within one of the branches of the choice exists in the infoset. This isn't necessarily the case. If the choice is part of a complex representation that is hidden, then no infoset element will exist corresponding to any branch.
...