Distributed Runtime

A space to capture thoughts and notes about how we bring the Tuscany SCA distributed runtime back to life. While the nature of a distributed runtime implies that more than one runtime of more than one type will be involved in a running SCA application I've put this page under the Java SCA subroject as it seems sensible to work with distributing one type of runtime (java) before branching out.

So from the mail thread 1 on this subject here are some initial thoughts and questions...

Distributing SCA Artifacts

The assembly model specification 2 deals briefly with distributed runtimes in its discussion of SCA Domains

"An SCA Domain represents a complete runtime configuration, potentially distributed over a series of interconnected runtime nodes."

The assembly spec, however, is not prescriptive about how an SCA Domain should be mapped and supported across multiple runtime nodes. Here I believe the term runtime node (or just node) is used to describe a process running an SCA runtime in which components can be run, e.g. the Java or C++ runtimes that Tuscany is developing.

Terminology

SCADomain, Composite, Component, Service, Reference - as used in SCA

Runtime - Provides the runtime environment for SCA component instances. May be as simple as a single Java VM or may be provided by a more scalable and reliable solution such as a compute cluster. Must be able to expose the endpoints required by the services of associated components. Must be able to support the client technology for the references of associated components.

Component Instance - The running component that services requests. A single component definition in a SCDL file may give rise to one or more component instances depending on how the component is scoped (using the @Scope annotation).

Scoping The Disitribution Problem

There are many exisiting technologies that deal with managing compute nodes and job scheduling. So it's probably safe to start by ignoring the issue of how the system picks processors on which runtime nodes will run (1).

There are also many technologies that providing scalable, robust and/or high performance service hosting solutions. So we can probably also ignore the isssue of how component instances are actually constructed as the runtime representation of components deployed to a runtime (3).

So that leaves us to consider how the components of a domain are associated with the runtimes of a domain (2).

Runtimes and Components

In the non-distributed case a single runtime loads all contributions and runs all components. In the distributed case many runtimes will be active in a domain. Each component deployed in the domain will be associated with a single runtime.

There is no restriction on the number of component instances a runtime can create, or indeed how these component instances are run. For example, all components instances could run in a single VM or be distributed across a number of VM's to provide improved performance or failover for example.

Is is assumed that a component cannot be associated with more than one runtime. The distribution of component (implementations) for reliability or performance reasons is a product of the implementation of a runtime and not part of the configuration of the SCADomain.

SCA Binding

The SCABinding is the default binding used within an SCA assembly. In the runtime in a single VM case it implies local connections. In the distributed runtime case it hides all of the complexity of ensuring that nodes wired between runtimes are able to communicate.

Meta Data

Using information from the SCA Assembly specification and the implied requirements of a distribute runtime we can determine what data is required to configure and control the distributed SCADomain.

SCADomain
  Name (DomainA)
  BaseURI
  Domain Level Composite
    Component (ComponentA)
      implementation
        composite
      Service
      Reference
  Installed Contributions
    Initial Package
      Contribution (file system, jar, zip etc)
        URI (ContributionA)
        /META-INF/
          sca-contribution.xml
            deployable (composite QName)
            import (namespace, location)
            export (namespace)
          sca-contribution-generated.xml
            deployable (composite QName)
            import (namespace, location)
            export (namespace)
          deployables
            *.composite
        *.composite
          URI
          Component (ComponentA)
            Service
            Reference
        Other Resources
          URI
    Dependent Contributions
      Contribution snapshot
    Deployment-time Composites
      composite
*.
  Runtimes
   runtime
    implementation 
    name (runtimeA)
    hostname/ip
    binding.ws:
      scheme http://localhost:8080/acbd
      scheme https://localhost:442/abcd
    binding.sca:
      scheme http://localhost:1234
    binding.jsonrpc:
      scheme http://localhost:8085/jsonxyz
    topology and resolution (tell me what to run ,  find me a remote artifact in the domain etc)
       binding.file topology.xml
       binding.ws http://my.registry.com
       binding.jxta
    management (start, stop, reconfigure etc.)
       binding.jmx 

  Topology 
    domainA:
      runtimeA:
       components:
          DomainA/ComponentA
          DomainA/ComponentB

Scenarios

Define an SCA Domain

Can be as simple as a file structure (based on hierarchy from assembly spec) in a shared file system

Start a runtime manually

Start the runtime on the appropriate hardware giving it the initial domain contributions

Stop a runtime manually

Stop a running runtime. Any runtimes that host components that reference components in this runtime must be notified so that they can stop these components.

A runtime can load all of the contributions and, using the meta data, determine which components it is responsible for. Alternatively a runtime can wait to be notified of the domain artifacts it is responsible for. Various domain management implementations could be provided.

Add a contribution

• Install Contribution

Each runtime that reference components in the contribution loads the new contribution.

Remove a contribution

• Remove Contrbution

Each runtime that reference components in the contribution removes the contribution. What happens to anything that depends on this contribution.

Contribution modified at runtime

• Update Contribution

Add a composite

• Add Deployment Composite

Update a composite

• Update Deployment Composite

Runtime failure

In the case that a runtime fails for whatever reason a new runtime must be started and configured in the same way as the old one. To do this a model of the state of the old runtime is required. Also the creation of a new runtime, possibly on new hardware, implies that any components that reference endpoints on the failed runtime must be updated to reference the new runtime.

Component assignment modified at runtime

• Assign Component
• Unassign Component

When initial contributions are loaded the topology information is used to determine which components run on which nodes. If at some point components are moved from one runtime to another (Why?) then the component must be unassigned from the current runtime and assigned to the new one.

The implication of this is that any components that reference this moved component must be notified that reference component has a new endpoint.

• Update Component Reference

Start the domain automatically

Requires some provisioning/management solution.

Runtime Design

Overview

Incremental Steps

In the first instance we should set the bar fairly low. I.e have the target be running a sample application across two SCA runtimes supporting java component implementations. This pretty much picks up where we were with the distribution support before the core modularization effort and so allows us to leverage the work already done where appropriate.
We can achieve this quite easily with the following mapping

Events - only provided at runtime startup
Domain Configuration - a text editor
Configuration - a file system
Messages - JMS

Allocating Components To Nodes

Somehow we need to tell each runtime which parts of the SCA model to run. So, if CA is to run on N1 we have some options...

1. Annotate the existing metadata (the SCDL) with the information
2. Create separate metadata that maps N1 to CA
3. Assume that all nodes run all components and use message delivery as the distinguishing factor

The first option was chosen for the existing distributed runtime implementation. There would likely have to be a hierarchical nature to these annotations where you might mark a composite as belonging to a node or the individual components of a composite. Services and references can be assumed to belong to nodes running the related components.

Notfifying Nodes of Allocations

You can imagine, in the longer term, a scheme where running nodes are notified what components they should be running. This implies a number of service interfaces and a set of interacting services to maintain this information. In the first instace we could take the simpler approach of using a (shared) file system to share the message about what node is running what. In fact we could have each node read all of the model information. In that way each node is able to read the allocation annotation and determine what artefacts it's interested in based on its allocated name. Not ncessarily very service oriented but gets us going.

Each node will also be able tell which nodes are running the other artefacts in the domain. This is importation as each node has to invent remote wires to replace the local wiring between components being distributed. CA and CB in our case.

As we may want to swap out this approach in the future we should consider the mechanism which configures a distributed nodes as replaceable. The default would be for a node read all of the contributions from an SCA domain on a file system and consume the resulting set of contribution requests taking note of which ones it has to run itself and, based on this information, which local wires need replacing with remote wires.

SCA Binding

Where two components that are connected locally in the SCDL are run on different node we would expect the runtime to be smart enough to invent a remote connection between the two. For the time being we can make some rules about what type of connection is constructued in these circumstances. For example, we could assume that the protocol is going to be WebServices and that each node will be configured with the information required to derive the required host name, port and path required to create an endpoint for the automatically created bindings. We don't have to use web services. Anything that works now is an option. We should just pick the one we think will be simplest to use.

References

1 http://www.mail-archive.com/tuscany-dev%40ws.apache.org/msg16971.html
2 http://www.osoa.org/display/Main/Service+Component+Architecture+Specifications