You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

A page to capture thoughts and notes about the Tuscany SCA distributed runtime. Feel free to help in developing the ideas here. I've put the source for the diagrams on this page in my sandbox here

While the nature of a distributed runtime implies that more than one runtime of more than one type will be involved in a running SCA application I've it seems sensible to work with distributing one type of runtime (java) before branching out.

Distributed SCA Runtimes - Notes

The assembly model specification 2 deals briefly with distributed runtimes in its discussion of SCA Domains

"An SCA Domain represents a complete runtime configuration, potentially distributed over a series of interconnected runtime nodes."

The assembly spec, however, is not prescriptive about how an SCA Domain should be mapped and supported across multiple runtime nodes. Here I believe the term runtime node (or just node) is used to describe a process running an SCA runtime in which components can be run, e.g. the Java or C++ runtimes that Tuscany is developing.

Terminology

SCADomain, Composite, Component, Service, Reference - as used in SCA

Runtime - The logical container for all the the domains running components. Physically a runitme comprises 1 or more (distributed) runtime nodes.

Runtime Node - Provides the runtime environment for SCA component instances. May be as simple as a single Java VM or may be provided by a more scalable and reliable solution such as a compute cluster. Must be able to expose the endpoints required by the services of associated components. Must be able to support the client technology for the references of associated components.

Component Instance - The running component that services requests. A single component definition in a SCDL file may give rise to one or more component instances depending on how the component is scoped (using the @Scope annotation).

Scenarios

1. Starting a runtime manually

Starting a runtime (collection of nodes) will be a manual task. This task could be automated in the future with some scheduling/provisioning system that is able to select appropriate processors for runtime execution.

Each node in the runtime will be started and will read its node configuration from a file. From this it is able to determine which domains and components it is responsible for and what the scheme edpoints are.

The contributions relating to the domains associated with the node are read from the file system.

Remote references are configured according to the default schemes and and SCDL based configuration.

Any cross node SCA bindings are converted into remote bindings using a point to point protocol. In the process
of doing this the component endpoints that are generated are registered in a central registry (do we need to do this? we could potentially derive them form the node all of the node configurations)

2. Stopping a runtime manually

Stopping a runtime is s manual task. Each node is killed independently.

3. Runtime failure

If a runtime fails then a new runtime is required to replace it. It is assumed that provisioning/deployment of the new runtime is handled by some other tool. When the new runtime startes it has to be able to take the place of the old one by

  • loading the appropriate components for the node
  • notify all remote refrencing components that the runtime is available
  • resolving all reference remote components to other nodes in the runtime.

The configuration of the failed runtime comprises the initial configuration and subsequent configuration updates that the failed node was configured with. Implies that this configuration must be available somewhere.

4 Adding a contribution

Changes in configuration affect the distributed domain in the same way as a stand alone domain, e.g.

  • New/Updated components must be activated
    • new services started
    • new references wired
  • Removed components must be stopped
    • services stopped
    • references unwired
    • And components that depend on the removed component must also be stopped etc.

Also, with the distributed domain,

  • new local references to services now on other runtimes must be resolved depending on SCA binding
    • by refering to local configuration
    • by refering to a registry of endpoints
  • new enpoints for local wires must be exposed using the SCABinding
    • in some cases this will require registration of the new endpoint (either centrally or at the runtime holding the source of the wire)

5 Updating a contribution

6 Adding a component to a node

7 Managing service endpoints

7 Choosing a component instance

Scoping The Disitribution Problem

There are many exisiting technologies that deal with managing compute nodes and job scheduling. So it's probably safe to start by ignoring the issue of how the system picks processors on which runtime nodes will run (1).

There are also many technologies that providing scalable, robust and/or high performance service hosting solutions. So we can probably also ignore the isssue of how component instances are actually constructed as the runtime representation of components deployed to a runtime (3).

So that leaves us to consider how the components of a domain are associated with the runtimes of a domain (2).

Cardinality

domain 1 - 1 runtime 1 - n runtime node

In the non-distributed case a single runtime node loads all contributions and runs all components. In the distributed case many runtime nodes will be active in a domain. Each component deployed in the domain will be associated with a single runtime.

There is no restriction on the number of component instances a runtime can create, or indeed how these component instances are run. For example, all components instances could run in a single VM or be distributed across a number of VM's to provide improved performance or failover for example.

Is is assumed that a component cannot be associated with more than one runtime node. The distribution of component (implementations) for reliability or performance reasons is a product of the implementation of a runtime node and not part of the configuration of the SCADomain.

Some questions have been raised about cardinality

Should load balancing or HA type scenarios be able to be described in the topology by allowing components to be assinged to more than one node
Should a runtime be able to run more than one domain.

SCA Binding

The SCABinding is the default binding used within an SCA assembly. In the runtime in a single VM case it implies local connections. In the distributed runtime case it hides all of the complexity of ensuring that nodes wired between runtimes are able to communicate.

When a message oriented binding is uses here we benefit from the abstract nature of the endpoints, I.e queues can be created given runtimeId/ServiceID and messages can be targetted at these queues without knowledge of where the message consumers are physically.

Whene a point to point protocol is used a physical endpoint is required. So a registry of endpoints to SCA bound service is required to allow the SCA binding to find the appropriate target. This registry can either be static, i.e. derived from the base urls given in the domain topology configuration, or dynamic in nature, i.e. set up at runtime.

Within the same domain/runtime multiple technologies may be required to implement the SCA binding as messages pass between different runtime node implementations.

Modelling The Distributed Domain

Using information from the SCA Assembly specification and the implied requirements of a distribute runtime we can determine what data is required to configure and control the distributed SCADomain.

SCADomain
  Name (DomainA)
  BaseURI
  Domain Level Composite
    Component (ComponentA)
      implementation
        composite
      Service
      Reference
    Installed Contributions
    Initial Package
    Contribution (file system, jar, zip etc)
      URI (ContributionA)
      /META-INF/
        sca-contribution.xml
          deployable (composite QName)
          import (namespace, location)
          export (namespace)
        sca-contribution-generated.xml
          deployable (composite QName)
          import (namespace, location)
          export (namespace)
        deployables
          *.composite
      *.composite 
        URI
        Component (ComponentA)
          Service
          Reference
      Other Resources
        URI
      Dependent Contributions
        Contribution snapshot
      Deployment-time Composites
        *.composite

Over and above the contributed information we need to associate components with runtime nodes. 

Runtime
  Node
    name (runtimeA)
    DomainA
      scheme http://localhost:8080/acbd   
      scheme https://localhost:442/abcd   
      ComponentA

We know how SCDL is used to represent the application composites. We can view the runtime node configuration as a new set of components, interfaces, services and references. In SCA terms we can consider that each node implements a system composite that provides the service interfaces required to manage the node, for example. 

<composite xmlns="http://www.osoa.org/xmlns/sca/1.0"
           name="nodeA">
    <component name="ComponentRegistry">
        <implementation.java class="org.apache.tuscany.sca.distributed.node.impl.DefaultComponentRegistry"/>
    </component>  
</composite>

Having this meand that we can expose out local component registry using any bindings that Tuscany supports. Imagine that out component registry has an interface that allows out to

getComponentNode
setComponentNode
etc.

Then we might choose to initialise the registry with the follwoing type of information. 

<runtime>
    <node name="nodeA">
        <schema name="http" baseURL="http://localhost:80" /> 
        <schema name="https" baseURL="https://localhost:443" />
        <component name="CalculatorServiceComponent" />
    </node>
    <node name="nodeB">
        <schema name="http" baseURL="http://localhost:81"/> 
        <schema name="https" baseURL="https://localhost:444" />
        <component name="AddServiceComponent"/>
    </node>   
    <node name="nodeC">
        <schema name="http" baseURL="http://localhost:81"/> 
        <schema name="https" baseURL="https://localhost:444" />
        <component name="SubtractServiceComponent"/>
    </node>       
</runtime>

Of course we can read this configuration locally form a file, have it delivered via a service interface or even retrieve it via a reference.

Message Based Configuration

If we wanted to make the component assignement message driven then of course we could define a service interface for the topology service. With suitable methods:

  • assignComponent()
  • removeComponent()

Other Services

The objective with this kind of approach is to allow the flexible definition of other useful services for the runtime, for example,

  • management
  • notification.
  • sca binding control
  • runtime control

We do need to work out how these services are managed by the runtime. How much special treatment do they need.

Network based configuration

Managing The Distributed Domain

Incremental Steps

Stage 1 - 1st pass simple scenario

  • Domain Configuration - a text editor
  • Events - runtime startup, i.e. just starting the runtime executable
  • Configuration - a file system
  • Messages - JMS

Stage 2 - More efficient SCABinding implementation

  • Domain Configuration - a text editor
  • Events - only provided at runtime startup
  • Configuration - a file system
  • Messages - a faster point to point protocol, e.g. JSONRPC, required some endpoint registry

Stage 3 - Integration with management/provisioning
?

Stage 4 - Dynamic configuration

  • Domain Configuration - A tool for creating changes and events
    • What sort of events should be supported here
  • Events - initial startup and changes
  • Configuration - repository
  • Messages - a faster point to point protocol, e.g. JSONRPC, required some endpoint registry

Stage 1 Code Changes

Currently the runtime has the following structure

SCADomain
ReallySmallRuntime
Contribution
Assembly Model
DomainComposite (from domain)
Deployed composites refering to activated assembly model

I have added the following packages to the codebase

modules/
topology/topology-xml - describe and read the node to component mapping (not acutally used at the moment)
binding.sca - provides and abstraction of a automatically configured remote runtime
disitributed - the various other infrastructure changes I needed - worth a closer look
samples/
calculator-distributed

References

1 http://www.mail-archive.com/tuscany-dev%40ws.apache.org/msg16971.html
2 http://www.osoa.org/display/Main/Service+Component+Architecture+Specifications
3 http://www.mail-archive.com/tuscany-dev@ws.apache.org/msg18613.html

  • No labels