A prior version of the proposal can be found here.

Introduction

Some functionalities are either not supported or too difficult to express using the DFDL expression language. While it is possible to add desired functionality to the codebase, some may be applicable to only a small dataset and won't have widespread use. It also leaves the burden of implementing said functionalities on the Daffodil developers. We would like Daffodil to be able to register and execute external/user defined functions (UDFs) in DFDL expressions.

See 

Use Cases/Examples

A trivial use case would be adding a 'replace' function that is callable from a DFDL expression. In the DFDL Schema, we might call  something like the below; where transformedElem will contain "Hello_World", if someElement resolves to "Hello World". 

...

@UserDefinedFunctionIdentification(
	name = "convert_to_hae", 
	namespace = "http://extOther.UDFunction.ElevationConversions.com" 
)
public class MSLConversions {
	public double evaluate(double latitude, double longitude, double msl) {
		//implementation..
	}
}

Requirements

1. The UDF will be defined in a stand-alone library outside of the Daffodil codebase
2. The UDF must be accessible to and callable from the Daffodil code
3. Daffodil must be able to process and pass the return value from the UDF back to the Schema
4. The support of UDFs in the DFDL Schema must be language agnostic and not Java, Scala or Daffodil specific

Proposed Solution

The Daffodil solution will use a combination of JAVA's ServiceLoader and Reflection APIs.

Daffodil Provided Jar

Daffodil will provide a UserDefinedFunction interface, a UserDefinedFunctionProvider abstract class, a  UserDefinedFunctionIdentification annotation class, and two exception classes: UserDefinedFunctionFatalException and UserDefinedFunctionProcessingError.

...

The UserDefinedFunctionProcessingError exception can be thrown when an implementer wishes to throw a recoverable that'll induce backtracking. The UserDefinedFunctionFatalException exception can be thrown to halt processing all together and abort Daffodil.

UDF Implementation

The implementer will be expected to implement at least 2 classes: a provider class and at least one UDF class.

...

The UDF classes will contain the functionality of the UDF embodied in an evaluate method. The UDF class will be expected to implement an evaluate method as well as apply the Daffodil provided UserDefinedFunctionIdentification annotation to the class . Because the parameter types and the return types of the evaluate function are dependent on the functionality, and we really only care about the name, we will not provide an abstract function for it. Each function that the implementer wishes to expose must implement the UserDefinedFunction interface, contain an evaluate function, and have the UserDefinedFunctionIdentification annotation. See 135859458 Proposal: Feature User Defined Functions for a sample UDF class.

Daffodil Service Loader

Daffodil will use the ServiceLoader API to poll for UDF Provider classes and return the desired function class on request.

Daffodil will have an internal object that uses the ServiceLoader iterator to aggregate and validate all the provider classes and the UDF classes they provide. This class will do this aggregation and validation at compile time, and will only initialize a UDF object and look up its method if an attempt is made to call the UDF. And providers or UDFs that don't validate during compile time will be dropped It. Any attempts to call a dropped UDF from the schema will result in an SDE.

Daffodil DFDL Processing

Acquiring the UDF

The Internal object referenced above will be instantiated only if a function call from the schema is not recognized as one of our previously supported functions. We will call this object's lookup function to find the UDF based on the name and namespace. If it finds the UDF, it will return a case class containing the UDF class, the evaluate method, its NodeInfo.Kind param types and return type. The aforementioned are necessary to call the UDF at runtime.  If the UDF is not found, we'll throw an SDE. 

val udfCallingInfo = UserDefinedFunctionService.lookupUserDefinedFunctionCallingInfo(namespace, fName)

val UserDefinedFunctionService.UserDefinedFunctionCallingInfo(udf, ei) = udfCallingInfo.get
val UserDefinedFunctionService.EvaluateMethodInfo(evaluateMethod, evaluateParamTypes, evaluateReturnType) = ei

Calling the

...

UDF

Within the DFDL expression processing code, Daffodil will define 2 case classes, a UserDefinedFunctionCallExpr and a UserDefinedFunctionCall. UserDefinedFunctionCallExpr will extend Daffodil's FunctionCallBase, and override inherentType, targetTypeForSubexpression and compiledDPath.  It will call UserDefinedFunctionCall as follows.

...

case class UserDefinedFunctionCall(
  functionQNameString: String,
  recipes: List[CompiledDPath],
  userDefinedFunction: UserDefinedFunction,
  evaluateFxn: UserDefinedFunctionMethod)
  extends FNArgsList(recipes) {

  override def computeValue(values: List[Any], dstate: DState) = {
    val jValues = values.map { _.asInstanceOf[Object] }
    try {
      val res = evaluateFxn.method.invoke(userDefinedFunction, jValues: _*)
      res
    } catch {
      case e: InvocationTargetException => {
        val targetException = e.getTargetException
        targetException match {
          case te: UserDefinedFunctionProcessingError =>
            throw new UserDefinedFunctionProcessingErrorException(
              s"User Defined Function '$functionQNameString'",
              Maybe(dstate.compileInfo.schemaFileLocation), dstate.contextLocation, Maybe(te), Maybe.Nope)
          case te: Exception =>
            throw new UserDefinedFunctionFatalErrorException(
              s"User Defined Function '$functionQNameString' Error",
              te, userDefinedFunction.getClass.getName)
        }
      }
      case e @ (_: IllegalArgumentException | _: NullPointerException | _: ReflectiveOperationException) =>
        throw new UserDefinedFunctionProcessingErrorException(
          s"User Defined Function '$functionQNameString'",
          Maybe(dstate.compileInfo.schemaFileLocation), dstate.contextLocation, Maybe(e), Maybe.Nope)
      case e: ExceptionInInitializerError =>
        throw new UserDefinedFunctionFatalErrorException(
          s"User Defined Function '$functionQNameString' Error",
          e, userDefinedFunction.getClass.getName)
    }
  }
}

Diagnostics

We intend to supply the user will at least the following errors/warning

• Warning: Any ignored/dropped User Defined Function or User Defined Function Providers
• Error: Errors loading User Defined Function Providers or initializing User Defined Functions
• Info: User Defined Function Loaded
• SDE: No User Defined function class with specified name/namespace found

Testing

Prototype

UDF Jars: HAEMSLConversions.jar and UDFunctionProviderImpl.jar. Both extend UDFunctionProvider.jar.

...