A prior version of the proposal can be found here.

The Java org.xml.sax SAX API is a well known and understood API for handling XML data in an event driven manner. Beause the SAX API is event based, rather than other approaches such as DOM, it allows for efficient processing of XML with reduced memory usage.

This proposes changes Daffodil  to add support parsing and unparsing in conformance with the SAX API.

Note that this proposal does not address issues that are related to reducing Daffodil memory usage (a common benefit of the SAX API), such as creating InfosetOuputter events as early as possible rather than at the end of a parse, or allowing parts of the internal infoset representation to be garbage collected. Those issues will be resolved in separate proposals.

Requirements

• Daffodil shall implement the Java SAX API in accordance with the org.xml.sax SAX API documentation
• Daffodil shall maintain support for existing APIs
• Daffodil shall create SAX events representing a DFDL infoset while parsing data
• Daffodil shall receive SAX events representing a DFDL Infoset to unparse data

SAX Overview

The two main components of a SAX API are the XMLReader and ContentHandler interfaces.

The XMLReader defines an interface for reading/parsing XML documents, including defining the entrypoint to begin the parse as well as functions to configure some aspects of the parse. Although this interface is named "XMLReader" and is traditionally used for parsing XML data, the interface itself does not strictly require that an implementation actually parse XML. In actuality, it only defines that the XMLReader implementation creates XML events. How it determines when and which XML events to create is entirely up to the implementation.

The ContentHandler defines an interface for receiving the XML events created by the XMLReader. What the ContentHandler does with the received events is entirely up to the implementation.

Fortunately, these two SAX API components translate quite nicely to Daffodil's concepts of a Parser and Unparser. Much like a XMLReader, a Parser reads in data and converts it to XML. And much like a ContentHandler, an Unparser reads in XML and takes actions (i.e. unparses data) based on the XML. The following sections describe how Daffodil can implement the XMLReader and ContentHandler to support parse and unparse.

Parse

XMLReader

The XMLReader interface requires various getters and setters for internal mutable state. For example, the ContentHandler is set with the setContentHandler(...) method. Although it might feel natural to modify the DataProccesor to implement the XMLReader interface, the DataProcessor is considered to be immutable, which conflicts with the mutability of the XMLReader. For this reason, a new function is added to the DataProcessor to create an implementation of the XMLReader called a DaffodilParseXMLReader, for example:

val dataProcessor = processorFactory.onPath("/")
val xmlReader = dataProcessor.newXMLReaderInstance()

The newXMLReaderInstance()  method allocates a Daffodil specific implementation of the XMLReader interface that can be used to parse data using the SAX API. Because the DataProcessor has all the actual logic for parsing data, it must be passed into the XMLReader implementation so that it can parse data at the appropriate time. Thus, the implementation for this function looks something along the lines of the following:

class DataProcessor(...) {
  ...
  def newXMLReaderInstance(): DaffodilParseXMLReader = new DaffodilParseXMLReader(this)
}

class DaffodilParseXMLReader(dp: DataProcessor) extends XMLReader {
  ...
}

A list of the XMLReader interface methods and specific details related to the Daffodil implementation are described below. See the XMLReader for more details on how an XMLReader should behave for particular functions. 

In addition the the above functions, the following functions are added to support other input types that Daffodil supports, which may allow for some optimizations.

SAXInfosetOutputter

The SAXInfosetOutputter is an implementation of the Daffodil InfosetOutputter interface responsible for converting InfosetOutputter events to SAX ContentHandler events. According to the SAX API, applications may register a new or different ContentHandler with the XMLReader in the middle of a parse, and the SAX parser must begin using the new handler immediately. Because of this, the SAXInfosetInputter must take the XMLReader as a parameter, and any time a SAX event is generated, it must call getContentHandler()  on that parameter. The definition for this class looks like:

class SAXInfosetOutputter(xmlReader: DaffodilParseXMLReader) extends InfosetOutputter with XMLInfosetOutputter { ... }

Fortunately, the InfosetOutputter events correlate nicely to the ContentHandler events. Below is their mapping. Note that in some cases a single InfosetOutputter event may require calling multiple ContentHandler events.

Other functions in the ContentHandler interface will not be used.

Unparse

ContentHandler

The ContentHandler interface is used to receive and react to SAX XML events. In order to unparse data based on these events, Daffodil must unparse data based on the events that are received. However, the design of the unparser and InfosetInputters behaves opposite to this–rather than receiving events, the unparser and InfosetInputter requests the next event. This is essentially push vs pull, or SAX vs StAX. To support unparsing based on SAX events, we must convert these push-style SAX events into the pull style events that Daffodil requires.

To accomplish this, we need a ContentHandler implementation specific to Daffodil that can receive all the SAX events and unparse to a specified OutputStream. Similar to parsing, this ContentHandler shall be created by means of the DataProcessor:

val dataProcessor = processorFactory.onPath("/")
val contentHandler = dataProcessor.newContentHandlerInstance(output)

The newContentHandlerInstance()  method allocates a Daffodil specific implementation of the ContentHandler interface that can be used to unparse data using the SAX API. Because the DataProcessor has all the actual logic for unparsing data, it must be passed into the ContentHandler implementation so that it can unparse data at the appropriate time. In addition, the ContentHandler must know where to unparse data to, so an OutputStream is also provided as a parameter. The implementation for this function looks something along the lines of the following:

class DataProcessor(...) {
  ...
  def newContentHandlerInstance(output: OutputStream): DaffodilUnparseContentHandler = new DaffodilUnparseContentHandler(this, output)
}

class DaffodilUnparseContentHandler(dp: DataProcessor, output: OutputStream) extends ContentHandler {
  ...
}

SAXInfosetInputter

The SAXInfosetInputter is a new InfosetInputter that supports the SAX style events. This InfosetInputter has a mutable SAXInfosetEvent with information on each piece of state that the InfosetInputter could request (e.g. event type, localname, namespace, text content). This is the currentEvent SAXInfosetEvent, so when the InfosetInputter getter functions are called, this SAXInfosetInputter simply returns the respective values from the currentEvent.

How and when this event is mutated requires coordination between the push-style ContentHandler and the pull-style SAXInfosetInputter. This coordination is handled using coroutines and is describe in the following section.

Coroutine Coordination

The legacy Daffodil Coroutine library allows for pausing the execution of a subroutine to temporarily yield to the caller, and allow the caller to resume the coroutine back to where it paused earlier. This library was revived to coordinate the interactions between the ContentHandler and the SAXInfosetInputter.

The dataProcessor.unparse(...) method is run within the SAXInfosetInputter coroutine, so its calls are also part of the coroutine execution stack. The ContentHandler is run in the main execution stack, with control passed back and forth between it and the SAXInfosetInputter coroutine, until the unparse completes and the SAXInfosetInputter yields control for the last time.

The coordination between the ContentHandler and the SAXInfosetInputter in the coroutine behaves with the following rules:

• The ContentHandler maintains the private state of the next SAXInfosetEvent to provide to the SAXInfosetInputter, which is copied and stored by the SAXInfosetInputter, in the nextEvent member, when it calls hasNext
• The ContentHandler triggers the SAXInfosetInputter coroutine, and thus the unparse, when the startDocument SAX event is received
• When the unparser calls the hasNext method in the SAXInfosetInputter (inside the coroutine), it either returns true if there is a stored nextEvent, false if we've already received an EventDocument, or it calls resume passing in any status information from the currentEvent and yielding control to the ContentHandler to create a new event. When control resumes back to hasNext, after the ContentHandler calls resume (with the new event) on its end, it will store the new event in nextEvent and return true.
• When the unparser calls the next method in the SAXInfosetInputter (inside the coroutine), it calls hasNext first to see if there is a nextEvent queued to copy, if there is, it copies nextEvent into currentEvent. A false hasNext is an error state, as next should not be called if hasNext is false.
• This continues, until hasNext in SAXInfosetInputter returns false, and the unparse call completes. The unparseResult is queried for errors and any errors are set via SAXInfosetEvent.causeError The unparseResult is set via SAXInfosetEvent.unparseResult. 

An outline for what the above rules and coroutine interactions look like is below:

class DaffodilUnparseContentHandler(dp: DataProcessor, output: OutputStream) extends ContentHandler with coroutine[SAXInfosetEvent] {
  
  private val infoseEvent = new SAXInfosetEvent()
  private val inputter = new SAXInfosetInputter(this, dp, output)


  private def sendtoInputter(....) {
    // queueing the infosetEvent for SAXInfosetInputter
	val infosetEventWithResponse = this.resume(inputter, Try(infosetEvent))
    infosetEvent.clear()
    // if event is wrapped in a Try failure, we will not have an unparseResult, so we only set
    // unparseResults for events wrapped in Try Success, including those events that have expected
    // errors
    if (infosetEventWithResponse.isSuccess && infosetEventWithResponse.get.unparseResult.isDefined) {
      unparseResult = infosetEventWithResponse.get.unparseResult.get
    }
    // the exception from events wrapped in Try failures and events wrapped in Try Successes
    // (with an unparse error state i.e unparseResult.isError) are collected and thrown to stop
    // the execution of the xmlReader
    if (infosetEventWithResponse.isFailure || infosetEventWithResponse.get.isError) {
      val causeError = if(infosetEventWithResponse.isFailure) {
        infosetEventWithResponse.failed.get
      } else {
        infosetEventWithResponse.get.causeError.get
      }
      causeError match {
        case unparseError: DaffodilUnparseErrorSAXException =>
          // although this is an expected error, we need to throw it so we can stop the xmlReader
          // parse and this thread
          throw unparseError
        case unhandled: DaffodilUnhandledSAXException => throw unhandled
        case unknown => throw new DaffodilUnhandledSAXException("Unknown exception: ", new Exception(unknown))
      }
    }
  }

  def startDocument() {
    // Start the coroutine
    infosetEvent.eventType = One(StartDocument)
    sendToInputter()
  }

  def endDocument() {
    infosetEvent.eventType = One(EndDocument)
    sendToInputter()
  }
  
  ...
}

class SAXInfosetInputter(unparseContentHandler: DaffodilUnparseContentHandler, dp: DataProcessor, output: OutputStream) extends InfosetInputter with coroutine[SAXInfosetEvent] {

  val currentEvent: SAXInfosetEvent = new SAXInfosetEvent
  val nextEvent: SAXInfosetEvent = new SAXInfosetEvent  

  def getEventType: InfosetInputterEventType = currentEvent.eventType.orNull
  def getLocalName: String = currentEvent.localName.orNull
  def getNamespaceURI: String = currentEvent.namespaceURI.orNull
  ...

  def hasNext: Boolean = {
    if (endDocumentReceived) false
    else if (!nextEvent.isEmpty) true
    else {
      val event = this.resume(unparseContentHandler, Try(currentEvent))
      copyEvent(source = event, dest = nextEvent)
      true
    }
  }

  def next(): Unit = {
    if (hasNext()) {
      copyEvent(source = Try(nextEvent), dest = currentEvent)
      nextEvent.clear()
      if (currentEvent.eventType.contains(EndDocument)) endDocumentReceived = true
    } else {
      // we should never call next() if hasNext() is false
      Assert.abort()
    }
  }
  ...
}

class SAXInfosetEvent() {

  var localName: Maybe[String] = Nope
  var simpleText: Maybe[String] = Nope
  var namespaceURI: Maybe[String] = Nope
  var eventType: Maybe[InfosetInputterEventType] = Nope
  var nilValue: Maybe[String] = Nope
  var causeError: Maybe[SAXException] = Nope
  var unparseResult: Maybe[UnparseResult] = Nope

  def isError: Boolean
  def clear: Unit
  def isEmpty: Boolean
}