Introduction
Some functionalities are either not supported or too difficult to express using the DPath expression language. While it is possible to add desired functionality to the codebase, some may be applicable to only a small dataset and won't have widespread use. It also leaves the burden of implementing said functionalities on the Daffodil developers. We would like Daffodil to be able to register and execute external/user defined functions (UDFs) in DPath expressions.
Use Cases/Examples
A trivial use case would be adding a replace function callable from a DPath expression. In the DFDL Schema, we might call something like the below; Where the function will return "Hello_World" if the element resolves to "Hello World".
dfdl:choiceDispatchKey="{ sdf:replace(xs:string(header/MessageIdentifier/MessageTextFormatIdentifier), ' ', '_') }"
Another use case would be implementing normalizing elevation above Mean-Sea-Level (MSL) to Height-Above-Ellipsoid (HAE) for Link16F1 data. In the DFDL schema, we might call something like the below; where the functions will return the result of the conversion.
dfdl:outputValueCalc="{ mhdf:convert_to_hae(xs:int(../land_point_track_initial_word/composite_elevation/raw), xs:int(../land_point_track_continuation_word_3/hae_adjustment/raw), xs:float(../land_point_track_initial_word/composite_elevation/scaling_factor)) }"
Requirements
- The UDF will be defined in a stand-alone library outside of the Daffodil codebase
- The UDF must be accessible to and callable from the Daffodil code
- Daffodil must be able to process and pass the return value from the UDF back to the Schema
- The support of UDFs in the DFDL Schema must be language agnostic and not Java, Scala or Daffodil specific
Proposed Solution
The Daffodil solution will use a combination of JAVA's ServiceLoader and Reflection APIs.
UDF Implementation
The implementer will be expected to implement at least 2 classes. One will be a "provider" class containing a lookup function, that returns an initialized object based on a supplied name and namespace. This "provider" class will act as a traditional service provider as explained in the ServiceLoader API, and should have a src/META-INF/services folder in its JAVA project that contains the fully qualified name of the UDFunctionProvider class that we supply. Through this, this class will be made visible to the ServiceLoader API and the function class object can be returned to Daffodil. This class will also be expected to have access to all the function classes it's responsible for, and be able to initialize them as needed.
public class myUDFunctionProvider extends UDFunctionProvider { public Objext lookupFunctionClass(String namespace, String name) { switch( String.join("_", namespace, name)) { case "ns1_func1": return new func1(); case "ns2_func2": return new func2(); case "ns1_func3": return new func3_randomName(); default: return null; } } }
The rest of the classes will contain the functionality of the UDF embodied in an evaluate method. This class is what will be initialized and returned by the provider class. This UDF class will be expected to implement 3 methods as seen below. Because the parameter types and the return types of the evaluate function are dependent on the functionality, and we really only care about the name, we will not provide an abstract class for it. Each function that the implementer wishes to expose must be defined in a class with an evaluate function, and must be available to the lookup function.
getName: String getNamespace: String evaluate: Any //Return type is function dependent, but must be called evaluate
Daffodil Service Loader
Daffodil will use the ServiceLoader API to poll for UDF Provider classes and return the desired function class on request.
As mentioned above, we will provide an abstract class for UDF "Provider" class. It will contain one abstract lookup function class method . This class will be externally available for the implementer to use. Daffodil will also have an internal class that will implement a function that calls the ServiceLoader iterator, and call the lookupFunctionClass method previously mentioned. If not found, we will return null, else we will return the desired object. This internal class will be implemented in Java as the ServiceLoader proved difficult to use in Scala.
public Object lookupFunctionClass(String namespace, String name) { Object funcObj = null; try { for (UDFunctionProvider fp : loader) { funcObj = fp.lookupFunction(namespace, name); if (funcObj != null) { break; } } } catch (Exception e) { e.printStackTrace(); } return funcObj; }
Daffodil DPath Processing
Acquiring the UDFunctionClass
The Internal class referenced above will be instantiated, and its lookup function called with the namespace and name from functionQName object. If it returns null, we'll throw an error, otherwise we'll use Reflection API to query for the parameter type, return type and to find the evaluate function.
val functionClass = functionClassObject.getClass // infer argument types; this method supports overloading evaluate function val pTypes = argumentsFromDpath.map { o => o.getClass() } val fEvaluate = functionClass.getMethods.find { p => p.getName == "evaluate" && p.getParameterTypes.sameElements(pTypes) } val dResultType = NodeInfo.fromObject(fEvaluate.getReturnType) val dArgTypes = fEvaluate.getParameterType.map { NodeInfo.fromObject }
Calling the UDFunctionClass
Within the DPath processing code, Daffodil will define 2 case classes, a UDFunctionCallExpr and a UDFunctionCall. The dResultType and dArgTypes arguments for UDFArgsListExpr will be inferred from the udFunctionClass as shown above. The Expression functionObject will be set to the below.
UDFunctionCallExpr(functionQNameString, functionQName, args, dResultType, dArgTypes, UDFunctionCall(_, udFunctionClass))
The function prototype for use cases would look something like