Versions Compared

Old Version 21

changes.mady.by.user Hao Chen

Saved on

compared with

New Version Current

changes.mady.by.user qingwzhao

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Status

Status
titleIMPROVING

Principle

  • Minimize user operations for managing and monitoring topologies
  • REST Service centralized metadata management
  • Consistent UI experience

Objective

  1. Make sure consistent semantics between the term application (user space) and topology (physical space only)
  2. Topology configuration can be edited on UI
  3. Use REST Service as only entry to manage application (START/STOP) and ApplicationManager to schedule the task execution and update the execution information 
  4. Eagle UI should support to manage/monitor the application on site/application page
  5. Single site/application may have more than one topologies (for example hdfsAuditLogMonitoring and userProfileMonitoring should have topologies running separately)

Architecture

To have a better user experience with EAGLE ui, we could provide an interface to manage a certain topology via the ui page, such as submit a topology

Gliffy Diagram
nametopologyManage

Schema

Application Definition 

ApplicationTopologyArch

Gliffy Diagram
nametopologyManagerArch

Schema

Topology Description Service

  • Service name: TopologyDescriptionService
  • Entity: TopologyDescriptionEntity
  • Table name: eagle_metadata
  • Prefix: topologyDescription
 AttributeTypeDescription
tagetopologyStringtopology name
fieldsexeClassStringtopology entry class
 typeStringtopology type: DSL or CLASS
 descriptionStringdescription on this topology
 versionStringtopology version

Topology Execution Service 

  • Service name: TopologyExecutionService
  • Entity: TopologyExecutionEntityService name: ApplicationManagementService
  • Table name: eagle_metadata
  • Prefix: application_managementtopologyExecution
  • Description: define the relation between an application and an topology, and maintain the execution status as well
jarFile
 AttributeTypeDescription
tagssiteStringtopology site name
 applicationString 
 topologyString 
fieldsfullNameStringtopology execution name: eagle_

application

physical topology = ${site}_${application}_${topology}
 urlStringtopology tracking url
 descriptionStringtopology running status descriptionfieldsmainClassStringapplication main class
 statusStringjar file pathapplication running status {NEW, STARTING, STOPPING, STARTED, STOPPED}
 environmentmodeStringcluster nametopology running mode: cluster or local
 statusenvironmentString{SUCCESS, FAILURE}

...

topology execution environment, e.g., storm
 lastModifiedDatelonglast status update time

Topology Operation Service

  • Service name: ApplicationEnvironmentServiceTopologyOperationService
  • Entity: TopologyOperationEntity
  • Table name:  eagleeagle_metadata
  • Prefix: application_environmenttopologyOperation
typeconfig
 AttributeTypeDescription
tagsnameoperationStringenvironment name{START, STOP, STATUS}
 siteStringe.g., storm, sparkfields 
 applicationStringcluster config

Use cases

 
 operationIDString 
 topologyStringtopology name: ${topology}
fieldsstatusString{INITIALIZED, PENDING, FAILED, SUCCESS}
 messageStringexception message
 lastModifiedDatelonglast status update time

Customized Restful Apis

HTTP MethodURLPayloadDescription
POST/app/operationTopologyOperationEntitycreate an topology operation
DELETE/app/topology/{topology} delete a topology description



 

...

  1. create a topology definition entity

...

  1. click START button
  2. check the topology's current status on the nimbus host
  3. a new thread is created to execute the submitting topology task
  4. update status

...