Distributed Runtime

A space to capture thoughts and notes about how we bring the Tuscany SCA distributed runtime back to life. While the nature of a distributed runtime implies that more than one runtime of more than one type will be involved in a running SCA application I've put this page under the Java SCA subroject as it seems sensible to work with distributing one type of runtime (java) before branching out.

So from the mail thread 1 on this subject here are some initial thoughts and questions...

Distributing SCA Artifacts

The assembly model specification 2 deals briefly with distributed runtimes in its discussion of SCA Domains

"An SCA Domain represents a complete runtime configuration, potentially distributed over a series of interconnected runtime nodes."

The assembly spec, however, is not prescriptive about how an SCA Domain should be mapped and supported across multiple runtime nodes. Here I believe the term runtime node (or just node) is used to describe a process running an SCA runtime in which components can be run, e.g. the Java or C++ runtimes that Tuscany is developing.

Terminology

SCADomain, Composite, Component, Service, Reference - as used in SCA

Runtime - Provides the runtime environment for SCA component instances. May be as simple as a single Java VM or may be provided by a more scalable and reliable solution such as a compute cluster. Must be able to expose the endpoints required by the services of associated components. Must be able to support the client technology for the references of associated components.

Component Instance - The running component that services requests. A single component definition in a SCDL file may give rise to one or more component instances depending on how the component is scoped (using the @Scope annotation).

Scenarios

Start a runtime

Starting a runtime will be a manual task initially. This task could be automated with some scheduling/provisioning system that is able to select appropriate processors for runtime execution.

• a file describing the configuration
• a reference to a registry/manager (or a mechanism for finding this reference) where it can
  • find the configuration information
  • register for configuration notifications or configuration update notifications

Stop a runtime

An manual task in the first instance. In the future would be done through notification.

• runtimes that reference components in this runtime must be notified so that they can stop or reconfigure these components.

Configure a distributed SCA Domain

In the user providing the initial configuration of the distributed domain must

• annotate the SCDL to assign components to runtimes
• create separate configuration that describes this relationship
  • in a file
  • in a registry

Update the configuration of a distributed SCA Domain

Configuration of the domain can be changed in various ways

• Install Contribution
• Remove Contrbution
• Update Contribution
• Add Deployment Composite (not clear to me why this is different from updating contribution)
• Update Deployment Composite (not clear to me why this is different from updating contribution)
• Add component to runtime

Changes in configuration affect the distributed domain in the same way as a stand alone domain, e.g.

• New/Updated components must be activated
  • new services started
  • new references wired
• Removed components must be stopped
  • services stopped
  • references unwired
  • And components that depend on the removed component must also be stopped etc.

Also, with the distributed domain,

• new local references to services now on other runtimes must be resolved depending on SCA binding
  • by refering to local configuration
  • by refering to a registry of endpoints
• new enpoints for local wires must be exposed using the SCABinding
  • in some cases this will require registration of the new endpoint (either centrally or at the runtime holding the source of the wire)

Runtime failure

In the case that a runtime fails for whatever reason a new runtime is required.

• do nothing and raise an error
• restart the failed rutime
  • taking all the appropriate actions to resolve local wires
  • notify all dependent runtimes of restart

To restart a runtime a model of the configuration of the failed runtime is required based on initial configuration and subsequent configuration updates.

Scoping The Disitribution Problem

There are many exisiting technologies that deal with managing compute nodes and job scheduling. So it's probably safe to start by ignoring the issue of how the system picks processors on which runtime nodes will run (1).

There are also many technologies that providing scalable, robust and/or high performance service hosting solutions. So we can probably also ignore the isssue of how component instances are actually constructed as the runtime representation of components deployed to a runtime (3).

So that leaves us to consider how the components of a domain are associated with the runtimes of a domain (2).

Runtimes and Components

In the non-distributed case a single runtime loads all contributions and runs all components. In the distributed case many runtimes will be active in a domain. Each component deployed in the domain will be associated with a single runtime.

There is no restriction on the number of component instances a runtime can create, or indeed how these component instances are run. For example, all components instances could run in a single VM or be distributed across a number of VM's to provide improved performance or failover for example.

Is is assumed that a component cannot be associated with more than one runtime. The distribution of component (implementations) for reliability or performance reasons is a product of the implementation of a runtime and not part of the configuration of the SCADomain.

SCA Binding

The SCABinding is the default binding used within an SCA assembly. In the runtime in a single VM case it implies local connections. In the distributed runtime case it hides all of the complexity of ensuring that nodes wired between runtimes are able to communicate.

Meta Data

Using information from the SCA Assembly specification and the implied requirements of a distribute runtime we can determine what data is required to configure and control the distributed SCADomain.

SCADomain
  Name (DomainA)
  BaseURI
  Domain Level Composite
    Component (ComponentA)
      implementation
        composite
      Service
      Reference
  Installed Contributions
    Initial Package
      Contribution (file system, jar, zip etc)
        URI (ContributionA)
        /META-INF/
          sca-contribution.xml
            deployable (composite QName)
            import (namespace, location)
            export (namespace)
          sca-contribution-generated.xml
            deployable (composite QName)
            import (namespace, location)
            export (namespace)
          deployables
            *.composite
        *.composite
          URI
          Component (ComponentA)
            Service
            Reference
        Other Resources
          URI
    Dependent Contributions
      Contribution snapshot
    Deployment-time Composites
      *.composite

  Runtimes
   runtime
    implementation 
    name (runtimeA)
    hostname/ip
    binding.ws:
      scheme http://localhost:8080/acbd
      scheme https://localhost:442/abcd
    binding.sca:
      scheme http://localhost:1234
    binding.jsonrpc:
      scheme http://localhost:8085/jsonxyz
    topology and resolution (tell me what to run ,  find me a remote artifact in the domain etc)
       binding.file topology.xml
       binding.ws http://my.registry.com
       binding.jxta
    management (start, stop, reconfigure etc.)
       binding.jmx 

  Topology 
    domainA:
      runtimeA:
       components:
          DomainA/ComponentA
          DomainA/ComponentB

Runtime Design Thoughts

Connectivity

Runtime Interfaces

Incremental Steps

In the first instance we should set the bar fairly low. I.e have the target be running a sample application across two SCA runtimes supporting java component implementations. This pretty much picks up where we were with the distribution support before the core modularization effort and so allows us to leverage the work already done where appropriate.
We can achieve this quite easily with the following mapping

Events - only provided at runtime startup
Domain Configuration - a text editor
Configuration - a file system
Messages - JMS

Meta Data Reading

The runtime and tolpology information will be read separately from the> First the runtime configuration provides the based configuration for the runtime and tells the runtime how to get topology information.

Topology information tells the runtime which components to run.

Contributions provide all of the resources required to run components.

Allocating Components To Nodes

Somehow we need to tell each runtime which parts of the SCA model to run. So, if CA is to run on N1 we have some options...

1. Annotate the existing metadata (the SCDL) with the information
2. Create separate metadata that maps N1 to CA
3. Assume that all nodes run all components and use message delivery as the distinguishing factor

The first option was chosen for the existing distributed runtime implementation. There would likely have to be a hierarchical nature to these annotations where you might mark a composite as belonging to a node or the individual components of a composite. Services and references can be assumed to belong to nodes running the related components.

Notfifying Nodes of Allocations

You can imagine, in the longer term, a scheme where running nodes are notified what components they should be running. This implies a number of service interfaces and a set of interacting services to maintain this information. In the first instace we could take the simpler approach of using a (shared) file system to share the message about what node is running what. In fact we could have each node read all of the model information. In that way each node is able to read the allocation annotation and determine what artefacts it's interested in based on its allocated name. Not ncessarily very service oriented but gets us going.

Each node will also be able tell which nodes are running the other artefacts in the domain. This is importation as each node has to invent remote wires to replace the local wiring between components being distributed. CA and CB in our case.

As we may want to swap out this approach in the future we should consider the mechanism which configures a distributed nodes as replaceable. The default would be for a node read all of the contributions from an SCA domain on a file system and consume the resulting set of contribution requests taking note of which ones it has to run itself and, based on this information, which local wires need replacing with remote wires.

SCA Binding

Where two components that are connected locally in the SCDL are run on different node we would expect the runtime to be smart enough to invent a remote connection between the two. For the time being we can make some rules about what type of connection is constructued in these circumstances. For example, we could assume that the protocol is going to be WebServices and that each node will be configured with the information required to derive the required host name, port and path required to create an endpoint for the automatically created bindings. We don't have to use web services. Anything that works now is an option. We should just pick the one we think will be simplest to use.

References

1 http://www.mail-archive.com/tuscany-dev%40ws.apache.org/msg16971.html
2 http://www.osoa.org/display/Main/Service+Component+Architecture+Specifications