Versions Compared

Old Version 12

changes.mady.by.user Sanil Jain

Saved on

compared with

New Version 13

changes.mady.by.user Sanil Jain

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Pros

Cons

  • Simple to extend the existing REST endpoint
  • Need to build authentication
  • If the AM dies all the outstanding requests are discarded (no additional handling needed)
  • Need to build Authorization layer around these rest endpoints 
  • Loading the already Heavy loaded Job coordinator with another service might cause an increase in memory used
  • Need to build a service for discovery or rely on Yarn embedded Servlet
Implementation Details:
  1. ContainerPlacementHandler is a stateless handler dispatching ContainerPlacementRequestMessages from Metastore to Container Placement Service & ContainerPlacementResponseMessages from Container Placement Service to metastore for external controls to query the status of an action. (PR). 
  2. Metastore used today by in Samza by default is Kafka (coordinator stream) which is used to store configs & container mappings & is log compacted
  3. ContainerPlacementRequestMessage & ContainerPlacementResponseMessage are maintained in individual namespaces using NamespaceAwareMetaStoreKey in namespace using NamespaceAwareMetaStore ("samza-place-container-v1")

Key-Value Format

Key for storing the ContainerPlacementRequestMessage & ContainerPlacementResponseMessage in Metastore is chosen to be

...

UUID + "." + messageType(ContainerPlacementResponseMessage or ContainerPlacementRequestMessage). Value will be payload container ContainerPlacementRequestMessage & ContainerPlacementResponseMessage. Messages are written and read to the Metastore through the MetadataStore abstraction. 

ContainerPlacementResponseMessage:

KeyValueField DescriptionField Type
"UUID.subType"uuidUnique identifier of a response messageRequired

processorId 

Logical processor id 0,1,2 of the containerRequired

deploymentIdUnique identifier for a deploymentRequired

subTypeType of message here: ContainerPlacementResponseMessageRequired

destinationHostDestination host where the container is desired to be movedRequired

statusCodeStatus of the current actionRequired

responseMessageResponse message in conjunction to statusRequired

timestampThe timestamp of the response messageRequired

requestExpiryEequest expiry which acts as a timeout for any resource request to cluster resource managerOptional

Sample KV

KeyValue

[1,"samza-place-container-v1","88b0d30c-d518-4307-9e8e-c8529eb30f04.ContainerPlacementResponseMessage"]

{"processorId":"1","deploymentId":"app-atttempt-001","subType":"ContainerPlacementResponseMessage","responseMessage":"Request is accepted","uuid":"88b0d30c-d518-4307-9e8e-c8529eb30f04","destinationHost":"ANY_HOST","statusCode":"ACCEPTED","timestamp":1578694070875}

GC policy for stale messages in metastore

  1. One way to delete stale ContainerPlacementMessages is to delete request / responses from the previous incarnation of the job in the metastore on job restarts
  2. Once the request is complete, ContainerPlacementService can issue an async delete to the metastore

...

KV for ContainerPlacementRequestMessages & ContainerPlacementResponseMessage

...

Key

...

Value

...

processorId

uuid: unique identifier a request, populated by client

applicationId: unique identifier of the deployed app for which the action is taken

destination-host: valid hostname / “ANY_HOST” / “STANDBY”

...

Part 2. Container Placement Service

Container Placement service is a set of APIs built around AM to move/restart containers. The solution proposes to refactor & simplify the current AM code & introduce a ContainerManager which is a single entity managing container actions like start, stop for both active and standby containers. Enlisted are functions of ContainerManager & proposed refactoring around the AM code 

...