You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 14 Next »

A page to capture thoughts and notes about the Tuscany SCA distributed runtime. Feel free to help in developing the ideas here. I've put the source for the diagrams on this page in my sandbox here

While the nature of a distributed runtime implies that more than one runtime of more than one type will be involved in a running SCA application, it seems sensible to work with distributing one type of runtime (java) before branching out.

Distributed SCA Runtimes - Notes

The assembly model specification 2 deals briefly with distributed runtimes in its discussion of SCA Domains
"An SCA Domain represents a complete runtime configuration, potentially distributed over a series of interconnected runtime nodes."

The assembly spec, however, is not prescriptive about how an SCA Domain should be mapped and supported across multiple runtime nodes. Here I believe the term runtime node (or just node) is used to describe a process running an SCA runtime in which components can be run, e.g. the Java or C++ runtimes that Tuscany is developing.

Furthermore, the SCA specifications do not talk about the form of repository in which the metadata of the SCA Domain is held - although clearly the metadata must live somewhere. The Domain repository could take a variety of forms, from a simple shared set of directories to a sophisticated distributed database system. SCA runtimes should be able to interact with a variety of Domain repositories.

Motivation

Some use cases that provide motivation for distributing a domain

  1. Represent the widely distributed nature of a typical SOA so that SCA presents a cross enterprise description of assembled components
  2. Policy matching where components require particular resources and hence particular, and separate, nodes
  3. HA/Load balancing/Performance scenarios where a single component appears on multiple nodes
  4. Load balancing/Performance scenarios where domain is spread across multiple nodes (same as 1 & 2 I believe)
  5. Dynamic wiring/Registry based service location, i.e. the SCA binding is called upon to automatically locate services based on registry entries.(overlaps with all of the above)

Terminology

SCADomain, Composite, Component, Service, Reference - as described in the SCA specifications. Note that a Domain may span multiple runtime nodes.  A Composite may also span multiple runtime nodes.

Runtime

The logical container for all one or more SCA Domains containing components. A runtime comprises one or more (distributed) runtime nodes.

Node

Provides an environment inside which SCA component instances execute. It's an operating system process, separate from other Nodes. Its form may be as simple as a single Java VM or it may take the more scalable and reliable form, such as a compute cluster.

Each node must be capable of supporting at least:

  • one implementation type
  • one binding type (which may be restricted to binding.sca)

A runtime node must be able to expose the service endpoints required by the components it runs. It must be able to support the client technology for the references of associated components.

Component Instance

The running component that services requests. A single component definition in a SCDL file may give rise to one or more component instances depending on how the component is scoped (using the @Scope annotation).

Cardinality


In the non-distributed case a single runtime node loads all contributions and runs all components.
In the distributed case, A Domain may span many nodes.

Each component must be associated (through specific configuration or some matching algorithm) with one or more nodes. If the same component appears in more than one node, the runtime is responsible for deciding how messages are routed to the correct node.

There is no restriction on the number of component instances a node can create, or indeed how these component instances are run. For example, all component instances could run in a single VM or be distributed across a number of VM's to provide improved performance or failover for example.

Some questions have been raised about cardinality

(question) Should load balancing or HA type scenarios be able to be described in the topology by allowing components to be assinged to more than one node?

Answer: Yes. However, this may well be best handled by a "layered runtime" approach, where a whole cluster of nodes is presented to the rest of the distributed runtime as a single node.

(question) Should a runtime be able to run more than one domain?

Answer: Yes. Multiple Domains can run on the same runtime. It is up to the runtime implementation to ensure that appropriate partitioning is achieved, since SCA Domains are intended to be isolated (for example, a reference in one domain cannot directly reference a service in another domain through it SCA component & service names).

Scoping The Distribution Problem


There are many existing technologies that deal with managing compute nodes and job scheduling. So it's probably safe to start by ignoring the issue of how the system picks processors on which runtime nodes will run (1).
There are also many technologies that providing scalable, robust and/or high performance service hosting solutions. So we can probably also ignore the issue of how component instances are actually constructed as the runtime representation of components deployed to a runtime (3).
So that leaves us to consider how the components of a domain are associated with runtime nodes (2).

Scenarios

starting the distributed domain

  • start all of the nodes that the domain will use
  • associate each node with the domain
  • load required contributions
  • add composites from the contributions to the domain level composite
  • assign components to nodes and start the components

choosing a component instance

Assigning components to nodes defins the endpoint for a components services. The distributed domain uses this information to create default bindings for the cross node wires. If a component is assigned to multiple nodes then the runtime is responsible for selecting the appropriate node based on, scope, conversational status of target component and also non specified goals such as load balancing

management

The features of the node and the domain as a whole are subject to management operations. Thus the various components will expose service interfaces through which the status and configuration of nodes and the domains are managed.

failure

Unexpect failure is enevitable. If a node fails a new node should be started and configured to match the failed node. This implies having the configuration avaialble external to the failed node and a mechanism whereby other nodes can be notified of the new endpoints that result.

Supporting The Scenarios

starting a node

Run a node exe giving it a a runtime name and a node name. The node exe will embed a Tuscany runtime.

stopping a node

Stop a node exe

starting a runtime

All nodes for a runtime are started

stopping a runtime

All nodes for a runtime are stopped

starting a domain

Domain is established in all nodes of a runtime and configuration is provided regarding, for example, scheme base URLS

stopping a domain

All domain components are stopped

install a contribution

A domain (each node in the domain) loads all of the resources associated with the contribution. Each node in the domain is given access to the resources that it requires to run the components in the contribution.

remove a contribution

All of the contribution's resources are removed from the domain

add a composite

A composite from the contribution is added to the domain level composite. The components from the composite are therefore available for running on nodes.

remove a composite

The composites components are removed from the domain level composite and will no longer run in the domain.

add a component to a node

The component from the domain level composite is assigned to a node and is run there.

remove a component from a node

The specified component is stopped at the specified node.

adding a contribution

The domain (each node in the domain) reads the contribution and makes assigned components ready for use

removing a contribution

All components assoicated with the contribution are stopped and removed

Managing The Distributed Domain

SCA Binding

The SCABinding is the default binding used within an SCA assembly. In the runtime in a single VM case it implies local connections. In the distributed runtime case it hides all of the complexity of ensuring that nodes wired between runtimes are able to communicate.

When a message oriented binding is used here we benefit from the abstract nature of the endpoints, I.e queues can be created given runtimeId/ServiceID and messages can be targetted at these queues without knowledge of where the message consumers are physically.

Whene a point to point protocol is used a physical endpoint is required. So a registry of endpoints to SCA bound service is required to allow the SCA binding to find the appropriate target. This registry can either be static, i.e. derived from the base urls given in the domain topology configuration, or dynamic in nature, i.e. set up at runtime.

Within the same domain/runtime multiple technologies may be required to implement the SCA binding as messages pass between different runtime node implementations.

Modelling The Distributed Domain

Using information from the SCA Assembly specification and the implied requirements of a distribute runtime we can determine what data is required to configure and control the distributed SCADomain.

SCADomain
  Name (DomainA)
  BaseURI
  Domain Level Composite
    Component (ComponentA)
      implementation
        composite
      Service
      Reference
    Installed Contributions
    Initial Package
    Contribution (file system, jar, zip etc)
      URI (ContributionA)
      /META-INF/
        sca-contribution.xml
          deployable (composite QName)
          import (namespace, location)
          export (namespace)
        sca-contribution-generated.xml
          deployable (composite QName)
          import (namespace, location)
          export (namespace)
        deployables
          *.composite
      *.composite
        URI
        Component (ComponentA)
          Service
          Reference
      Other Resources
        URI
      Dependent Contributions
        Contribution snapshot
      Deployment-time Composites
        *.composite

Over and above the contributed information we need to associate components with runtime nodes.

Runtime
  name (runtimeA)
  Node
    name (nodeeA)
    DomainA
      scheme http://localhost:8080/acbd
      scheme https://localhost:442/abcd
      ComponentA

We know how SCDL is used to represent the application composites. We can view the runtime node configuration as a new set of components, interfaces, services and references. In SCA terms we can consider that each node implements a system composite that provides the service interfaces required to manage the node, for example.

<composite xmlns="http://www.osoa.org/xmlns/sca/1.0"
           name="nodeA">
    <component name="ComponentRegistry">
        <implementation.java class="org.apache.tuscany.sca.distributed.node.impl.DefaultComponentRegistry"/>
    </component>
</composite>

Having this meand that we can expose out local component registry using any bindings that Tuscany supports. Imagine that out component registry has an interface that allows out to

getComponentNode
setComponentNode
etc.

Then we might choose to initialise the registry with the follwoing type of information.

<runtime>
    <node name="nodeA">
        <schema name="http" baseURL="http://localhost:80" />
        <schema name="https" baseURL="https://localhost:443" />
        <component name="CalculatorServiceComponent" />
    </node>
    <node name="nodeB">
        <schema name="http" baseURL="http://localhost:81"/>
        <schema name="https" baseURL="https://localhost:444" />
        <component name="AddServiceComponent"/>
    </node>
    <node name="nodeC">
        <schema name="http" baseURL="http://localhost:81"/>
        <schema name="https" baseURL="https://localhost:444" />
        <component name="SubtractServiceComponent"/>
    </node>
</runtime>

Of course we can read this configuration locally form a file, have it delivered via a service interface or even retrieve it via a reference.

Message Based Configuration

If we wanted to make the component assignement message driven then of course we could define a service interface for the topology service. With suitable methods:

  • assignComponent()
  • removeComponent()

Other Services

The objective with this kind of approach is to allow the flexible definition of other useful services for the runtime, for example,

  • management
  • notification.
  • sca binding control
  • runtime control

We do need to work out how these services are managed by the runtime. How much special treatment do they need.

Incremental Steps

Stage 1 - 1st pass simple scenario (done)

  • Domain Configuration - a text editor
  • Events - runtime startup, i.e. just starting the runtime executable
  • Configuration - a file system
  • Messages - JMS

Stage 2 - More efficient SCABinding implementation and endpoint resolution

  • Domain Configuration - a centralised model parallels the model in each node and
    allows a restarted node to determine, based on node name,
    started domains and loaded contributions
  • Events - doman start
    add contribution
    remove contribution
    domain stop
  • Configuration - a file system
  • Messages - a point protocol, e.g. JSONRPC supported by and endpoint registry

stage 3 - More dynamic configuration of distributed domain

  • ???

References

1 http://www.mail-archive.com/tuscany-dev%40ws.apache.org/msg16971.html
2 http://www.osoa.org/display/Main/Service+Component+Architecture+Specifications
3 http://www.mail-archive.com/tuscany-dev@ws.apache.org/msg18613.html

  • No labels