Page History

...

While the nature of a distributed runtime implies that more than one runtime of more than one type will be involved in a running SCA application I've , it seems sensible to work with distributing one type of runtime (java) before branching out.

...

The assembly spec, however, is not prescriptive about how an SCA Domain should be mapped and supported across multiple runtime nodes. Here I believe the term runtime node (or just node) is used to describe a process running an SCA runtime in which components can be run, e.g. the Java or C++ runtimes that Tuscany is developing.

Motivation

Furthermore, the SCA specifications do not talk about the form of repository in which the metadata of the SCA Domain is held - although clearly the metadata must live somewhere. The Domain repository could take a variety of forms, from a simple shared set of directories to a sophisticated distributed database system. SCA runtimes should be able to interact with a variety of Domain repositories.

Motivation

Some use cases that provide motivation Some reasons for distributing a domain

1. Represent the widely distributed nature of a typical SOA so that SCA presents a cross enterprise description of assembled components
2. Policy Support policy matching where components require particular resources and hence particular, and separate, nodes
3. HA/Load balancing/Performance scenarios where a single component appears on multiple nodes
4. Load balancing/Performance scenarios where domian si domain is spread across multiple nodes (same as 1 & 2 I believe)

Terminology

SCADomain, Composite, Component, Service, Reference - as used in SCA

Runtime - The logical container for all the the domains running components. Physically a runitme comprises 1 or more (distributed) runtime nodes.

Runtime Node - Provides the runtime environment for SCA component instances. May be as simple as a single Java VM or may be provided by a more scalable and reliable solution such as a compute cluster. Must be able to expose the endpoints required by the services of associated components. Must be able to support the client technology for the references of associated components.

1. Dynamic wiring/Registry based service location, i.e. the SCA binding is called upon to automatically locate services based on registry entries.(overlaps with all of the above)

Terminology

See http://cwiki.apache.org/confluence/display/TUSCANYWIKI/Terminology

Scoping The Distribution Problem

There are many existing technologies that deal with managing compute nodes and job scheduling. So it's probably safe to start by ignoring the issue of how the system picks processors on which runtime nodes will run. So the runtime management of the nodes themselves is out of scope.

There are also many technologies that provide scalable, robust and/or high performance service hosting solutions. So we can also ignore the issue of how component instances are actually constructed as the runtime representation of components deployed to a runtime. For example if a JVM clustering solution is chosen to implement a node then we assume that local method calls within that cluster will be handled by the clustering technology and no special action is required. If higher level clustering technology is in operation then intergration with the runtime is required. In this case, where each node in the cluster runs part of the domain a component can be mapped to multiple nodes, the most natural integration point is the SCA binding which must interact with the clustering technology in order locate target component services.

So the initial area of consideration is how the components of a domain are associated with runtime nodes (2).

Image AddedComponent Instance - The running component that services requests. A single component definition in a SCDL file may give rise to one or more component instances depending on how the component is scoped (using the @Scope annotation).

Cardinality

Image Modified
In the non-distributed case a single runtime node loads all contributions and runs all components.
In the distributed case, A Domain may span many runtime nodes will be active in a domain.

Each component must be associated (through specific configuration or some matching algorithm) with one or more nodes. Where a If the same component appears in more than one node, the runtime is responsible for deciding how messages are routed to the correct node.

There is no restriction on the number of component instances a node can create, or indeed how these component instances are run. For example, all component instances could run in a single VM or be distributed across a number of VM's to provide improved performance or failover for example.

...

Should load balancing or HA type scenarios be able to be described in the topology by allowing components to be assinged to more than one node?

Answer: Yes. However, this may well be best handled by a "layered runtime" approach, where a whole cluster of nodes is presented to the rest of the distributed runtime as a single node.

Should a runtime be able to run more than one domain?

Scoping The Disitribution Problem

Image Removed

There are many exisiting technologies that deal with managing compute nodes and job scheduling. So it's probably safe to start by ignoring the issue of how the system picks processors on which runtime nodes will run (1).

There are also many technologies that providing scalable, robust and/or high performance service hosting solutions. So we can probably also ignore the isssue of how component instances are actually constructed as the runtime representation of components deployed to a runtime (3).

So that leaves us to consider how the components of a domain are associated with the runtimes of a domain (2).

Scenarios

Image Removed

Scenarios

starting a node

Run a node exe giving it a node name

stopping a node

Stop a node exe

starting a runtime

All nodes for a runtime are started

stopping a runtime

All nodes for a runtime are stopped

starting a domain

Domain is established in all nodes of a runtime and configuration is provided regarding, for example, scheme base URLS

stopping a domain

All domain compnents are stopped

adding a contribution

The domain (each node in the domain) reads the contribution and makes assigned components ready for use

removing a contribution

All components assoicated with the contribution are stopped and removed

assignComponent

node failure

A new node is started and is configured to match the failed node

configuring endpoints

Service endpoints are configured based on the specified algorithm

choosing a component instance

If a component is assigned to multiple nodes then the runtime is responsible for selecting the appropriate node
based on, scope, conversational status of target component and also non specified goals such as load balancing
Likely to be implemented as a cluster hosting solution

SCA Binding

The SCABinding is the default binding used within an SCA assembly. In the runtime in a single VM case it implies local connections. In the distributed runtime case it hides all of the complexity of ensuring that nodes wired between runtimes are able to communicate.

When a message oriented binding is uses here we benefit from the abstract nature of the endpoints, I.e queues can be created given runtimeId/ServiceID and messages can be targetted at these queues without knowledge of where the message consumers are physically.

Whene a point to point protocol is used a physical endpoint is required. So a registry of endpoints to SCA bound service is required to allow the SCA binding to find the appropriate target. This registry can either be static, i.e. derived from the base urls given in the domain topology configuration, or dynamic in nature, i.e. set up at runtime.

Within the same domain/runtime multiple technologies may be required to implement the SCA binding as messages pass between different runtime node implementations.

Modelling The Distributed Domain

Using information from the SCA Assembly specification and the implied requirements of a distribute runtime we can determine what data is required to configure and control the distributed SCADomain.

SCADomain
  Name (DomainA)
  BaseURI
  Domain Level Composite
    Component (ComponentA)
      implementation
        composite
      Service
      Reference
    Installed Contributions
    Initial Package
    Contribution (file system, jar, zip etc)
      URI (ContributionA)
      /META-INF/
        sca-contribution.xml
          deployable (composite QName)
          import (namespace, location)
          export (namespace)
        sca-contribution-generated.xml
          deployable (composite QName)
          import (namespace, location)
          export (namespace)
        deployables
          *.composite
      *.composite 
        URI
        Component (ComponentA)
          Service
          Reference
      Other Resources
        URI
      Dependent Contributions
        Contribution snapshot
      Deployment-time Composites
        *.composite

Over and above the contributed information we need to associate components with runtime nodes. 

Runtime
  Node
    name (runtimeA)
    DomainA
      scheme http://localhost:8080/acbd   
      scheme https://localhost:442/abcd   
      ComponentA       

We know how SCDL is used to represent the application composites. We can view the runtime node configuration as a new set of components, interfaces, services and references. In SCA terms we can consider that each node implements a system composite that provides the service interfaces required to manage the node, for example.

<composite xmlns="http://www.osoa.org/xmlns/sca/1.0"
           name="nodeA">
    <component name="ComponentRegistry">
        <implementation.java class="org.apache.tuscany.sca.distributed.node.impl.DefaultComponentRegistry"/>
    </component>  
</composite>

Having this meand that we can expose out local component registry using any bindings that Tuscany supports. Imagine that out component registry has an interface that allows out to

getComponentNode
setComponentNode
etc.

Then we might choose to initialise the registry with the follwoing type of information.

<runtime>
    <node name="nodeA">
        <schema name="http" baseURL="http://localhost:80" /> 
        <schema name="https" baseURL="https://localhost:443" />
        <component name="CalculatorServiceComponent" />
    </node>
    <node name="nodeB">
        <schema name="http" baseURL="http://localhost:81"/> 
        <schema name="https" baseURL="https://localhost:444" />
        <component name="AddServiceComponent"/>
    </node>   
    <node name="nodeC">
        <schema name="http" baseURL="http://localhost:81"/> 
        <schema name="https" baseURL="https://localhost:444" />
        <component name="SubtractServiceComponent"/>
    </node>       
</runtime>

Of course we can read this configuration locally form a file, have it delivered via a service interface or even retrieve it via a reference.

Message Based Configuration

If we wanted to make the component assignement message driven then of course we could define a service interface for the topology service. With suitable methods:

• assignComponent()
• removeComponent()

Other Services

The objective with this kind of approach is to allow the flexible definition of other useful services for the runtime, for example,

• management
• notification.
• sca binding control
• runtime control

We do need to work out how these services are managed by the runtime. How much special treatment do they need.

Network based configuration

Managing The Distributed Domain

Image Removed

Incremental Steps

Stage 1 - 1st pass simple scenario

• Domain Configuration - a text editor
• Events - runtime startup, i.e. just starting the runtime executable
• Configuration - a file system
• Messages - JMS

Stage 2 - More efficient SCABinding implementation

• Domain Configuration - a text editor
• Events - only provided at runtime startup
• Configuration - a file system
• Messages - a faster point to point protocol, e.g. JSONRPC, required some endpoint registry

Stage 3 - Integration with management/provisioning
?

Stage 4 - Dynamic configuration

• Domain Configuration - A tool for creating changes and events
  • What sort of events should be supported here
• Events - initial startup and changes
• Configuration - repository
• Messages - a faster point to point protocol, e.g. JSONRPC, required some endpoint registry

Stage 1 Code Changes

Currently the runtime has the following structure

SCADomain
ReallySmallRuntime
Contribution
Assembly Model
DomainComposite (from domain)
Deployed composites refering to activated assembly model

I have added the following packages to the codebase

modules/
topology/topology-xml - describe and read the node to component mapping (not acutally used at the moment)
binding.sca - provides and abstraction of a automatically configured remote runtime
disitributed - the various other infrastructure changes I needed - worth a closer look
samples/
calculator-distributed

References

...

Answer: Yes. Multiple Domains can run on the same runtime. It is up to the runtime implementation to ensure that appropriate partitioning is achieved, since SCA Domains are intended to be isolated (for example, a reference in one domain cannot directly reference a service in another domain through it SCA component & service names).

Scenario - Simple Distributed Components

Image Added

In this basic scenario a number of composites are started across nodes from the command line and once they are all started messages are send through the application

Demonstrates: the sca binding and service resolution within the domain.

Scenario - Standalone Node

Composites are added to the node through the node API and the node is started.

Demonstrates: Resolution of wires across composites with a single node

Scenario - Nodes Connected To A domain

Composites are added through the node API and each node is started

Demonstrates: Compilation of a domain view of the application as composites are started on nodes

Scenario - Nodes Running in a Web App

Nodes started in web apps run the composites from those applications and registers them with the domain

Demonstrated: Compilation of a domain view of the application a web apps are run

Scenario - Virtual Node

A node is associated with a domain that doesn't have a tuscany runtime.

Demonstrates: Ability of Tuscany domain to include components/services that are not running on an SCA runtime.

Scenario - Domain Adding Nodes

A node is started and it becomes part of the domain ready to run composites

Managing The Distributed Domain

Th logical view of how the different parts of the solution communicate is.

Image Added

Messages - the application messages that flow between configured components. Messages will flow over bindings described excplicitly in the assembly model or across the default binding used when no explicit binding is specified.

Configuration - In the disitrubted domain configuration is shared across the nodes with which the domain is associated. This includes information about, contributed resources, running components and their endpoints and domain configuration items such as base URLs.

Events - as the domain runs interesting events will occur, for example, a node fails and is restarted meaning that a set of endpoints change.

Components Of The Solution

Based on the calculator scenario we can take a general view of how the domain organizes running application

Image Added

However there are a number of specific configurations to consider which affect the way that configuration and events are distributed.

Domain Driven

Image Added

Node Driven

Image Added

Stand Alone Node

Image Added

Remote Domain Control

Image Added

APIs

SCADomainFactory

SCADomain

• public void start() throws DomainException;
• public void stop() throws DomainException;
• public String getURI();
• public void addContribution(String uri, URL url) throws DomainException;
• public void removeContribution(String uri) throws DomainException;
• public void addToDomainLevelComposite(QName compositeQName) throws DomainException;
• public void removeFromDomainLevelComposite(QName compositeQName) throws DomainException;
• public void addDeploymentComposite(ContributionURI, CompositeXML) throws DomainException;
• public void startComposite(QName qname) throws DomainException;
• public void stopComposite(QName qname) throws DomainException;
• public <B, R extends CallableReference<B>> R cast(B target) throws IllegalArgumentException;
• public <B> B getService(Class<B> businessInterface, String serviceName);
• public <B> ServiceReference<B> getServiceReference(Class<B> businessInterface, String referenceName);

SCANodeFactory

SCANode

• public String getURI();
• public SCADomain getDomain();
• public void addContribution(String uri, URL url) throws DomainException;
• public void removeContribution(String uri) throws DomainException;
• public void addToDomainLevelComposite(QName compositeQName) throws DomainException;
• public void removeFromDomainLevelComposite(QName compositeQName) throws DomainException;
• public void startComposite(QName composite) throws NodeException;
• public void stopComposite(QName composite) throws NodeException;
• public void start() throws NodeException;
• public void stop() throws NodeException;
• public void destroy() throws NodeException;

SPIs

NodeEvents (node to domain)

• public String registerNode(String nodeURI, String nodeURL);
• public String removeNode(String nodeURI);
• public void registerContribution(String nodeURI, String contributionURI, String contributionURL);
• public void unregisterContribution(String contributionURI);
• public String registerServiceEndpoint(String domainUri, String nodeUri, String serviceName, String bindingName, String URL);
• public String removeServiceEndpoint(String domainUri, String nodeUri, String serviceName, String bindingName);
• public String findServiceEndpoint(String domainUri, String serviceName, String bindingName);

NodeManagement (domain to node)

• public String getURI();
• public void addContribution(String contributionURI, String contributionURL);
• public void deployComposite(String compositeName);
• public void start();
• public void stop();

Interactions

Load Balancing

Reliability and Failover