Distributed Runtime

A space to capture thoughts and notes about how we bring the Tuscany SCA distributed runtime back to life. While the nature of a distributed runtime implies that more than one runtime of more than one type will be involved in a running SCA application I've put this page under the Java SCA subroject as it seems sensible to work with distributing one type of runtime (java) before branching out.

So from the mail thread 1 on this subject here are some initial thoughts and questions...

Distributing SCA Artifacts

The assembly model specification 2 deals briefly with distributed runtimes in its discussion of SCA Domains

"An SCA Domain represents a complete runtime configuration, potentially distributed over a series of interconnected runtime nodes."

The assembly spec, however, is not prescriptive about how an SCA Domain should be mapped and supported across multiple runtime nodes. Here I believe the term runtime node (or just node) is used to describe a process running an SCA runtime in which components can be run, e.g. the Java or C++ runtimes that Tuscany is developing.

Terminology

SCADomain, Composite, Component, Service, Reference - as used in SCA

Runtime - Provides the runtime environment for SCA component instances. May be as simple as a single Java VM or may be provided by a more scalable and reliable solution such as a compute cluster. Must be able to expose the endpoints required by the services of associated components. Must be able to support the client technology for the references of associated components.

Component Instance - The running component that services requests. A single component definition in a SCDL file may give rise to one or more component instances depending on how the component is scoped (using the @Scope annotation).

Scoping The Disitribution Problem

There are many exisiting technologies that deal with managing compute nodes and job scheduling. So it's probably safe to start by ignoring the issue of how the system picks processors on which runtime nodes will run (1).

There are also many technologies that providing scalable, robust and/or high performance service hosting solutions. So we can probably also ignore the isssue of how component instances are actually constructed as the runtime representation of components deployed to a runtime (3).

So that leaves us to consider how the components of a domain are associated with the runtimes of a domain (2).

Runtimes and Components

In the non-distributed case a single runtime loads all contributions and runs all components. In the distributed case many runtimes will be active in a domain. Each component deployed in the domain will be associated with a single runtime.

There is no restriction on the number of component instances a runtime can create, or indeed how these component instances are run. For example, all components instances could run in a single VM or be distributed across a number of VM's to provide improved performance or failover for example.

Is is assumed that a component cannot be associated with more than one runtime. The distribution of component (implementations) for reliability or performance reasons is a product of the implementation of a runtime and not part of the configuration of the SCADomain.

Scenarios

Domain loaded at runtime start

domain contribution modified at runtime

runtime failure runtime brought up in same places

hardware failure runtime brought up somewhere else

Meta Data

SCADomain
  Name (DomainA)
  BaseURI
  Domain Level Composite
    Component (ComponentA)
      implementation
        composite
      Service
      Reference
  Installed Contributions
    Contribution (file system, jar, zip etc)
      URI (ContributionA)
      /META-INF/
        sca-contribution.xml
          deployable (composite QName)
          import (namespace, location)
          export (namespace)
        sca-contribution-generated.xml
          deployable (composite QName)
          import (namespace, location)
          export (namespace)
        deployables
          *.composite
      *.composite
        Component (ComponentA)
          Service
          Reference

  Runtimes
   runtime
    implementation 
    name (runtimeA)
    hostname/ip
    binding.ws:
      scheme http://localhost:8080/acbd
      scheme https://localhost:442/abcd
    binding.sca:
      scheme http://localhost:1234
    binding.jsonrpc:
      scheme http://localhost:8085/jsonxyz
    topology and resolution (tell me what to run ,  find me a remote artifact in the domain etc)
       binding.file topology.xml
       binding.ws http://my.registry.com
       binding.jxta
    management (start, stop, reconfigure etc.)
       binding.jmx 

  Topology 
    domainA:
      runtimeA:
       components:
          DomainA/ComponentA
          DomainA/ComponentB

Incremental Steps

In the first instance we should set the bar fairly low. I.e have the target be running a sample application across two SCA runtimes supporting java component implementations. This pretty much picks up where we were with the distribution support before the core modularization effort and so allows us to leverage the work already done where appropriate. In true don't run before you can walk style we can add more complex features once we can satisfy the simple scenarios. As we pull these ideas together we should be prepared to decide whether an idea is destined for the first pass or whether we should park it for the future.

Distributing SCA Artefacts

Allocating Components To Nodes

Somehow we need to tell each runtime which parts of the SCA model to run. So, if CA is to run on N1 we have some options...

1. Annotate the existing metadata (the SCDL) with the information
2. Create separate metadata that maps N1 to CA
3. Assume that all nodes run all components and use message delivery as the distinguishing factor

The first option was chosen for the existing distributed runtime implementation. There would likely have to be a hierarchical nature to these annotations where you might mark a composite as belonging to a node or the individual components of a composite. Services and references can be assumed to belong to nodes running the related components.

Notfifying Nodes of Allocations

You can imagine, in the longer term, a scheme where running nodes are notified what components they should be running. This implies a number of service interfaces and a set of interacting services to maintain this information. In the first instace we could take the simpler approach of using a (shared) file system to share the message about what node is running what. In fact we could have each node read all of the model information. In that way each node is able to read the allocation annotation and determine what artefacts it's interested in based on its allocated name. Not ncessarily very service oriented but gets us going.

Each node will also be able tell which nodes are running the other artefacts in the domain. This is importation as each node has to invent remote wires to replace the local wiring between components being distributed. CA and CB in our case.

As we may want to swap out this approach in the future we should consider the mechanism which configures a distributed nodes as replaceable. The default would be for a node read all of the contributions from an SCA domain on a file system and consume the resulting set of contribution requests taking note of which ones it has to run itself and, based on this information, which local wires need replacing with remote wires.

Default Binding

Where two components that are connected locally in the SCDL are run on different node we would expect the runtime to be smart enough to invent a remote connection between the two. For the time being we can make some rules about what type of connection is constructued in these circumstances. For example, we could assume that the protocol is going to be WebServices and that each node will be configured with the information required to derive the required host name, port and path required to create an endpoint for the automatically created bindings. We don't have to use web services. Anything that works now is an option. We should just pick the one we think will be simplest to use.

User Experience

So I would expect a manager of a distributed SCA runtime to go through a
number of stages in getting the system up and running.

Define an SCA Domain (Looking at the mailing list Luciano is thinking these thoughts also)

• domain definition
• as simple as a file structure (based on hierarchy from assembly spec) in a shared file system.
• could implement more complex registry based system
• allocate nodes to the domain
• As simple as running up SCA runtime on each node in the domain.
• For more complex scenarios might want to use a scheduling/management system

Add contributions to the domain

• Identify contributions required to form running service network (developers will build/define contributions)
• Contribution these contributions to the domain
• in the simple shared file system scenario I would image they just end up in on this file system available to all nodes in the domain.

Add contributions to the Virtual Domain Level Composite

• At this point it think we have to know where artifacts are physically going to run
• It could be that all runtimes load all contributions and only expose those destined for them, I.e. each node has the full model loaded but knows which bits it's running.
• Alternatively we could target each node specifically and ask it to load a particular installed contribution and define a distributed model.

Manage the Domain

• Need to be able to target the logical service provided by the domain at the appropriate runtime node

Terminology

node -

References

1 http://www.mail-archive.com/tuscany-dev%40ws.apache.org/msg16971.html
2 http://www.osoa.org/display/Main/Service+Component+Architecture+Specifications