Versions Compared

Old Version 22

changes.mady.by.user Hao Chen

Saved on

compared with

New Version Current

changes.mady.by.user qingwzhao

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Status

Status
titleIMPROVING

Principle

  • Minimize user operations for managing and monitoring topologies
  • REST Service centralized metadata management
  • Consistent UI experience

Objective

  1. Make sure consistent semantics between the term application (user space) and topology (physical space only)
  2. Topology configuration can be edited on UI
  3. Use REST Service as only entry to manage application (START/STOP) and ApplicationManager to schedule the task execution and update the execution information 
  4. Eagle UI should support to manage/monitor the application on site/application page
  5. Single site/application may have more than one topologies (for example hdfsAuditLogMonitoring and userProfileMonitoring should have topologies running separately)

Architecture

To have a better user experience with EAGLE ui, we could provide an interface to manage a certain topology via the ui page, such as submit a topology

Gliffy Diagram
nameApplicationTopologyArch

Gliffy Diagram
nametopologyManage

Schema

Application Definition 

topologyManagerArch

Schema

Topology Description Service

  • Service name: TopologyDescriptionService
  • Entity: TopologyDescriptionEntity
  • Table name: eagle_metadata
  • Prefix: topologyDescription
 AttributeTypeDescription
tagetopologyStringtopology name
fieldsexeClassStringtopology entry class
 typeStringtopology type: DSL or CLASS
 descriptionStringdescription on this topology
 versionStringtopology version

Topology Execution Service 

  • Service name: TopologyExecutionService
  • Entity: TopologyExecutionEntityService name: ApplicationManagementService
  • Table name: eagle_metadata
  • Prefix: application_managementtopologyExecution
  • Description: define the relation between an application and an topology, and maintain the execution status as well
fieldsmainClass
 AttributeTypeDescription
tagssiteStringtopology site name
 applicationString 
 topologyString 
fieldsfullNameStringtopology execution name: eagle_

application

physical topology = ${site}_${application}_${topology}
 urlStringapplication main classtopology tracking url
 jarFiledescriptionStringjar file pathtopology running status description
 environmentstatusStringenvironment nameapplication running status {NEW, STARTING, STOPPING, STARTED, STOPPED}
 urlmodeStringapplication tracking urltopology running mode: cluster or local
 statusenvironmentString{SUCCESS, FAILURE}topology execution environment, e.g., storm
 descriptionlastModifiedDateStringapplication description

...

longlast status update time

Topology Operation Service

  • Service name: ApplicationEnvironmentServiceTopologyOperationService
  • Entity: TopologyOperationEntity
  • Table name:  eagleeagle_metadata
  • Prefix: application_environmenttopologyOperation
environment typee.g., storm, sparkconfig
 AttributeTypeDescription
tagsnameoperationString{START, STOP, STATUS}
 siteString 
 applicationString 
 operationIDString 
 topologyStringtopology name: ${topology}
fieldsstatusString{INITIALIZED, PENDING, FAILED, SUCCESS}
 messageStringcluster config

Use cases

...

  1. create a topology definition entity

...

  1. click START button
  2. check the topology's current status on the nimbus host
  3. a new thread is created to execute the submitting topology task
  4. update status
exception message
 lastModifiedDatelonglast status update time

Customized Restful Apis

HTTP MethodURLPayloadDescription
POST/app/operationTopologyOperationEntitycreate an topology operation
DELETE/app/topology/{topology} delete a topology description



 

...