The Java org.xml.sax SAX API is a well known and understood API for handling XML data in an event driven manner. Beause the SAX API is event based, rather than other approaches such as DOM, it allows for efficient processing of XML with reduced memory usage.

This proposes changes Daffodil add support parsing and unparsing in conformance with the SAX API.

Note that this proposal does not address issues that are related to reducing Daffodil memory usage (a common benefit of the SAX API), such as creating InfosetOuputter events as early as possible rather than at the end of a parse, or allowing parts of the internal infoset representation to be garbage collected. Those issues will be resolved in separate proposals.

Requirements

• Daffodil shall implement the Java SAX API in accordance with the org.xml.sax SAX API documentation
• Daffodil shall maintain support for existing APIs
• Daffodil shall create SAX events representing a DFDL infoset while parsing data
• Daffodil shall receive SAX events representing a DFDL Infoset to unparse data

SAX Overview

The two main components of a SAX API are the XMLReader and ContentHandler interfaces.

The XMLReader defines an interface for reading/parsing XML documents, including defining the entrypoint to begin the parse as well as funtions to configure some aspects of the parse. Although this interface is named "XMLReader" and is traditionally used for parsing XML data, the interface itself does not strictly require that an implementation actually parse XML. In actuality, it only defines that the XMLReader implementation creates XML events. How it determines when and which XML events to create is entirely up to the implementation.

The ContentHandler defines an interface for receiving the XML events created by the XMLReader. What the ContentHandler does with the received events is entirely up to the implementation.

Fortunately, these two SAX API components translate quite nicely to Daffodil's concepts of a Parser and Unparser. Much like a XMLReader, a Parser reads in data and converts it to XML. And much like a ContentHandler, an Unparser reads in XML and takes actions (i.e. unparses data) based on the XML. The following sections describe how Daffodil can implement the XMLReader and ContentHandler to support parse and unparse.

Parse

XMLReader

The XMLReader interace requires various getters and setters for internal mutable state. For example, the ContentHandler is set with the setContentHandler(...) method. Although it might feel natural to modify the DataProccesor to implement the XMLReader interface, the DataProcessor is considered to be immutable, which conflicts with the mutability of the XMLReader. For this reason, a new function is added to the DataProcessor to create an implementation of the XMLReader called a DaffodilXMLReader, for example:

val dataProcessor = processorFactory.onPath("/")
val xmlReader = dataProcessor.newXMLReaderInstance()

The newXMLReaderInstance()  method allocates a Daffodil specific implementation of the XMLReader interface that can be used to parse data using the SAX API. Because the DataProcessor has all the actual logic for parsing data, it must be passed into the XMLReader implementation so that it can parse data at the appropriate time. Thus, the implementation for this function looks something along the lines of the following:

class DataProcessor(...) {
  ...
  def newXMLReaderInstance(): DaffodilXMLReader = new DaffodilXMLReader(this)
}

class DaffodilXMLReader(dp: DataProcessor) extends XMLReader {
  ...
}

A list of the XMLReader interface methods and specific details related to the Daffodil implementation are described below. See the XMLReader for more details on how an XMLReader should behave for particular functions. 

In addition the the above funtions, the following functinons are added to support other input types that Daffodil supports, which may allow for some optimizations.

SAXInfosetOutputter

The SAXInfosetOutputter is an implementation of the Daffodil InfosetOutputter interface responsible for converting InfosetOutputter events to SAX ContentHandler events. According to the SAX API, applications may register a new or different ContentHandler with the XMLReader in the middle of a parse, and the SAX parser must begin using the new handler immediately. Because of this, the SAXInfosetInputter must take the XMLReader as a parameter, and any time a SAX event is generated, it must call getContentHandler()  on that paremater. The definition for this class looks like:

class SAXInfosetOutputter(rdr: DaffodilXMLReader) extends InfosetInputter { ... }

Fortunately, the InfosetOutputter events correlate nicely to the InfosetOutputter events. Below is their mapping. Note that in some cases a single InfosetOutputter event may require calling multiple ContentHandler events.

Other functions in the ContentHandler interface will not be used.

Unparse

ContentHandler

The ContentHandler interface is used to receive and react to SAX XML events. In order to unparse data based on these events, Daffodil must unparse data based on the events that are received. However, the design of the unparser and InfosetInputter's behaves opposite to this–rather than receiveing events the unparser and InfosetInputter requests the next event. This is essentially push vs pull, or SAX vs StAX. To support unparsing based on SAX events, we must convert these push-style SAX events inout the pull style events that Daffodil requires.

To accomplsih this, we need an ContentHandler implementation specific to Daffodil that can receive all the SAX events and unparse to a specified OutputStream. Similar to parsing, this ContentHandler shall be created by means of the DataProcessor:

val dataProcessor = processorFactory.onPath("/")
val contentHandler = dataProcessor.newContentHandlerInstance(output)

The newContentHandlerInstance()  method allocates a Daffodil specific implementation of the ContentHandler interface that can be used to unparse data using the SAX API. Because the DataProcessor has all the actual logic for unparsing data, it must be passed into the ContentHandler implementation so that it can unparse data at the appropriate time. In addtion, the ContentHandler must know where to unparse data to, so an OutputStream is also provided as a parameter. The implementation for this function looks something along the lines of the following:

class DataProcessor(...) {
  ...
  def newContentHandlerInstance(output: OutputStream): DaffodilContentHandler = new DaffodilContentHandler(this, output)
}

class DaffodilContentHandler(dp: DataProcessor, output: OutputStream) extends ContentHandler {
  ...
}

SAXInfosetInputter

The SAXInfosetInputter is a new InfosetInputter that supports the SAX style events. This InfosetInputter has mutable state for each piece of state that the InfosetInputter could request (e.g. event type, localname, namespace, text content). When the InfosetInputter getter functions are called, this SAXInfosetInputter simply returns the values of the mutable state.

How this state is mutated and when requires coordination between the push-style ContentHandler and the pull-style SAXInfosetInputter. This coordination is handled using coroutines and is describe in the following section.

Coroutine Coordination

The Scala Coroutines library allows for pausing the execution of a subroutine to temporarily yeild to the caller, and allow the caller to resume the coroutine back to where it paused earlier. This library will be used to coordinate the interactions between the ContentHandler and the SAXInfosetInputter.

The dataProcessor.unparse(...) method is run as a coroutine. Because the SAXInfosetInputter methos are called from the unparse() method, it's calls are also part of the coroutine execute stack. The ContentHandler is run in the main execution stack, with control yielded to it from the SAXInfosetInputter coroutine.

The coordination between the ContentHandler and the SAXInfosetInputter in the coroutine behaves with the following rules:

• The ContentHandler maintains private state of the next event information to provide to the SAXInfosetInputter
• The ContentHandler starts the coroutine, and thus the unparse, when the startDocument SAX method is received
• When the unparser calls the hasNext method in the SAXInfosetInputter (inside the coroutine), it calls yieldval passing the constant value HAS_NEXT. When control resumes back to hasNext after the yield, it can assume the hasNext state has been set appropriately and return it.
• When the unparser calls the next method in the SAXInfosetInputter (inside the coroutine), it calls yieldval passing the constant value NEXT. When control resumes back to next after the yield, it can assume the mutable event state has been set appropriately
• When the ContentHandler is yielded back control, if the yielded value is HAS_NEXT, then:
  • If the ContentHandler does not have event information or it is incomplete, then it continues execution and recieves SAX events until complete event information is received (or there are no more events, such as endDocument was received). It then mutates the hasNext state in the SAXInfosetInputter accordingly and resumes the coroutine.
  • If the ContentHandler already has complete event information, then it simply resumes the coroutine–the hasNext state has already been set from the previous rule.
• When the ContentHandler is yielded back control, if the yielded value is NEXT, then:
  • The ContentHandler mutates the state of the SAXInfosetInputter based on its private event state, resets its private event state, and resumes the coroutine.

An outline for what the above rules and coroutine interactions look like is below:

class DaffodilContentHandler(dp: DataProcessor, output: OutputStream) extends ContentHandler {
  
  private val nextEvent = new EventState()

  private val inputter = new SAXInfosetInputter()

  private val unparseCoroutine = coroutine { (inputter, output) =>
    dp.unparse(inputter, output)
  }

  private def handleUpdatedEvent(....) {
    // currently trying to answer hasNext, trying to find a
    // complete InfosetInputter event before we can asnwer
    Assert(unparseCoroutine.value == HAS_NEXT)
    if (nextEvent.isComplete) {
      inputter.hasNext = true
      unparseCoroutine.resume
      while (unparseCoroutine.value == HAS_NEXT) {
        // keep resuming if unparse keeps asking hasNext, answer won't change
        unparseCoroutine.resume
      }
      Assert(unparseCoroutine.value == NEXT)
      nextEvent.copyToInputter(inputter)
      nextEvent.reset
      unparseCoroutine.resume
      Assert(unparseCoroutine.value == HAS_NEXT)
      // now looking for next complete event, keep handling SAX events
      // until we gather a complete event
    } else {
      // do not have a complete event yet, keep handling SAX events
      // until we gather a complete event
    }
  }

  def startDocument() {
    // Start the coroutine
    call(unparseCoroutine(inputter, output))
    unparseCoroutine.resume

    nextEvent.updateEvent(START_DOCUMENT)
    handleUpdatedEvent()
  }

  def endDocument() {
    nextEvent.updateEvent(END_DOCUMENT)
    handleUpdatedEvent()
    inputter.hasNext = false
    unparseCoroutine.resume
    val res = unaprseCoroutine.result
    ...
  }

  def someSaxEvent(...) {
    nextEvent.updateEvent(...)
    handleUpdatedEvent()
  }

}

class SAXInfosetInputter(event: Event) {

  // mutable infoset inputters state
  var hasNext: Boolean = false
  var eventType: InfosetInputterEventType = _
  var localName: String = _
  var simpleText: String = _
  ...

  def hasNext: Boolean = {
    yeildval(HAS_NEXT)
    hasNext
  }

  def next(): Unit = {
    yeildval(NEXT)
  }

  def getEventType: InfosetInputterEventType = eventType
  def getLocalName: String = localName
  ...
}