Page History

...

This page is a location to centralize notes about implementation of advanced DFDL features, and to highlight places where the DFDL specification may need to be clarified, augmented, or corrected.

Infoset can contain elements that have dfdl:outputValueCalc

It is essential that when unparsing it is allowed for the infoset to already contain output-computed elements. The values of these will be discarded and recomputed during unparsing, but they have to be tolerated if present, and created if not present.

Allow IVC/OVC on Global Elements, SimpleTypes

The restriction that IVC/OVC properties are not allowed on global element is arbitrary. It was probably put in place with an eye towards reducing complexity of implementations. It is, however, unnecessary and gets in the way of some useful cases.  This restriction should be removed.

Similarly, IVC/OVC are not allowed to be declared on simple types. This restriction is also unnecessary and makes reuse harder.

Allowing some DFDL properties to Coexist with IVC

For elements with dfdl:inputValueCalc, the element is not represented, so format properties don't make sense and should cause SDE if expressed directly on the element. However, some properties are completely compatible with dfdl:inputValueCalc. In particular dfdl:choiceBranchKey does not conflict, because it is not a format property.

This restriction in DFDL is easily worked around by wrapping a sequence around the branch element having IVC property, and putting the dfdl:choiceBranchKey on the sequence.

SDE for element in hidden group that has no default, nor dfdl:outputValueCalc

Within a hidden group, if an element has no fixed/default value, and no OVC, then because it is hidden and so does not appear in the infoset, there's no way for it to get a value at all when unparsing. 

...

This SDE is too strong: If the element has IVC, then it doesn't need a value for unparsing unless an expression that is used at unparse time (i.e., not in an assert or discriminator) references the element. In many cases the value of an IVC may be used only in dfdl:occursCount expressions, or in assert/discriminator test expressions. Since those aren't evaluated when unparsing, there is no unparsing situation would never need the value, so the SDE is too strong.

IVC and OVC on the same element

In many situations we have found that we need both IVC and OVC on the same element. This occurs when the element is in a hidden group, and is essentially being used as a variable.

...

Another use of this is that sometimes for unparsing, one needs a location for an intermediate calculation to be saved (to avoid redundantly computing it). Such an element needs to carry an OVC, but also have no representation. Having IVC on it explicitly makes it have no representation, no alignment, and no implications as far as separators showing up for it in separated sequences. There is no way to achieve this other than using IVC.

Array Variables and dfdl:newVariableInstance

In many cases dfdl:newVariableInstance is not sufficient because what are needed are array variables.

...

The headache being the relative path to it., and then the fact that the most natural thing to do is to put both IVC and OVC on it (though the OVC may not be needed in many cases).

Forward reference from dfdl:setVariable and dfdl:newVariableInstance Expressions

Consider unparsing with dfdl:setVariable expression referring to OVC forward-referencing element.

...

This breaks the rule that when variables are evaluated, things they reference must have already been evaluated. Basically, when we delay evaluating the OVC for len we are suspending anything that depends on len having a value as well.

Parse-Time Forward Reference

Several standards in the MIL-STD and NATO STANAG space express formats using a forward reference idiom that requires those forward references to be used at parse time.

...

Note that this issue doesn't actually involve IVC/OVC. It is independent of them. There is a workaround, and so this problem doesn't require a solution as part of DFDL v1.0.

Restricting IVC/OVC in Unordered Sequences

Elements having IVC are not allowed as the root of a choice branch. For the same reasons IVC elements should not be allowed as children of an unordered sequence, or a sequence with floating elements. 

Unparsing and Choice Groups, Especially Hidden Choice Groups

There are two problems related to unparsing and xs:choice.

Problem 1: dfdl:outputValueCalc that must refer into a child element inside a xs:choice

Example of this is this fragment of a messaging format where each message has a label and sublabel element that indicate which message it is. Logically, users think of, and expect to find the label and sublabel elements within the message structure, but we must have them before the element in order to use an xs:choice with dfdl:choiceDispatchKey to determine the message.

...

The ESA DFDL4Space project has invented a similar wildcard concept as a DFDL extension. They allow a match not to just an entire name, as a path step, but the allow extensions of a name prefix. E.g, ../pathStep(.*)/ meaning any step the name of which begins with "pathStep". The ".*" notation is motivated by regular expressions, but there is no use of patterns richer than ".*".  This generality is not needed as far as we can see because one can always model such data creating a parent element named "pathStep" in this case, and then the children of this have names that would match the "*" part of the pattern. So instead of "pathStep(.*)" matching pathStepData1 and pathStepData2, you would structure the elements so that "pathStep/*" matches /pathStep/Data1 and pathStep/Data2.

Problem 2: Choices inside hidden groups

The DFDL Spec discussion of unparsing choices assumes an element that appears within one of the branches of the choice exists in the infoset. This isn't necessarily the case. If the choice is part of a complex representation that is hidden, then no infoset element will exist corresponding to any branch.

...