Versions Compared

Old Version 10

changes.mady.by.user Kevin Minder

Saved on

compared with

New Version Current

changes.mady.by.user Kevin Minder

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

...

The sequence diagrams below (after the very long Legend) are intended to be a very fairly detailed description of the interactions that occur during the process of defining, submitting and executing a map reduce job on a secure Hadoop 2.x cluster. Different phases of the overall process are covered in each diagram. The are intended to be taken as one continuous flow with the exception of the last diagram which illustrates parallel steps that would occur during the flow.

  1. Bootstrap
  2. Job Definition
  3. Job Submission
  4. Job Initiation
  5. Map Task Execution
  6. Reduce Task Execution
  7. Job Completion
  8. Client Monitoring

Legend

The descriptions of the interactions in the sequence diagrams below take this form.

Code Block
message [Protocol] ( input ) : output

...

The  {{\[Protocol\]}}  portion   describes   the   protocol,   authentication   mechanism   and   identities   exchanged.
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="f2530e20-8dea-4bfc-87d6-04ff0a4fe955"><ac:plain-text-body><![CDATA[
 Abbreviation 
 Description 
 [KRB] 
 Kerberos Protocol 
]]></ac:plain-text-body></ac:structured-macro>
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="0157a5d4-22ed-4210-801c-5985ca5fe7bb"><ac:plain-text-body><![CDATA[
 [RSK:{[RSKT:{kerberos-service-ticket}] 
 RPC protocol with SASL mutual authentication using Kerberos tickets. ]]></ac:plain-text-body></ac:structured-macro>
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="a9479ed7-bb17-456c-9ed5-7e968026d04a"><ac:plain-text-body><![CDATA[
 [RST:{ticket[RSAT:{access-token}] 
 RPC protocol with SASL mutual client authentication using delegation tokens. access tokens (e.g. YARN Node Manager Token). 
[RSDT
]]></ac:plain-text-body></ac:structured-macro>
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="b9431650-48ac-493b-87d1-8c5fc48f5752"><ac:plain-text-body><![CDATA[
[RSD:{delegation-token}]
 RPC protocol with SASL mutual client authentication using delegation tokens (e.g. 
]]></ac:plain-text-body></ac:structured-macro>
HDFS Name Node Delegation Token). 
 [STP] 
 Shuffle data transfer protocol between ShuffleService and ReduceTask. HTTP protocol with TODO. <ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="22130d3c-ca7f-4241-a06b-034a4b554871"><ac:plain-text-body><![CDATA[
 [DTP] 
 Data Block data transfer protocol between the DataNode and a client. HTTP protocol with block tokens plus SHA1 hash exchange. 
]]></ac:plain-text-body></ac:structured-macro>
Suffixes are used in many cases to denote type.
 Abbreviation 
 Description 
 tgt 
 Kerberos Ticket Granting Ticket 
 kst 
 Kerberos Service Ticket: u-jt-kt = A Kerberos Ticket for User u to access the JobTracker jt 
 kp 
 Kerberos Principal: nn-kp = The Kerberos principal for the NameNode nn 
 kt dt 
 Kerberos Ticket Delegation Token: uc-jtnn-kt dt = A Kerberos Ticket for User u to access the JobTracker jt 
 dt 
 Delegation Token 
 tkn 
 Token 
delegation token for identity of the Client that can be presented to the NameNode. 
 tkn 
 Access Token: am-tkn = An access token that can be presented to the ApplicationMaster for access. 
 tkn-sk 
Access Token Secret Key 
 id 
Identifier: job-id = Job Identifier 
Kerberos principals use the principal abbreviation and the kp suffix.
...
 Abbreviation 
 Description 
 u-nn-kt 
 Kerberos service ticket for User u to access NameNode nn 
 u-jt-kt 
 Kerberos service ticket for User u to access JobTracker jt 
 dn-nn-kt 
 Kerberos service ticket for DataNode dn to access NameNode nn 
 jt-nn-kt 
 Kerberos service ticket for JobTracker dn to access NameNode nn 
 tt-jt-kt 
 Kerberos service ticket for TaskTracker tt to access JobTracker jt 
Secure MapReduce2 - Bootstrap
Image Removed
Secure MapReduce2 - Job Definition
Image Removed
Secure MapReduce2 - Job Submission
Image Removed
Secure MapReduce2 - Job Initiation
Image Removed
Secure MapReduce2 - Map Task Execution
Image Removed
Secure MapReduce2 - Reduce Task Execution
Image Removed
Secure MapReduce2 - Job Completion
Image Removed
Secure MapReduce2 - Client Monitoring
...
Bootstrap
This diagram illustrates the interactions that occur when a Hadoop system is starting up and stabilizing. It involves various master components generating secret keys and slave components registering with the masters to receive these secret keys.
Image Added
  1. createBlockAccessTokenSecretKey -
  2. kinit/AS_REQ -
  3. TGS_REQ -
  4. register/heartbeat -
  5. createNodeManagerTokenSecretKey -
  6. createAppContainerTokenSecretKey -
  7. kinit/AS_REQ -
  8. TGS_REQ -
  9. register/heartbeat -
...
Job Definition
This diagram illustrates the steps taken by a client to define a MapReduce job that will later be submitted.
Image Added
  1. TODO
  2. TODO
  3. TODO
...
Job Submission
This diagram illustrates the steps taken during the submission of a MapReduce job.
Image Added
  1. TODO
  2. TODO
  3. TODO
...
Job Initiation
This diagram illustrates the steps taken when a MapReduce job is scheduled for execution.
Image Added
  1. TODO
  2. TODO
  3. TODO
...
Map Task Execution
This diagram illustrates the steps taken when the Map portion of a MapReduce job is executed.
Image Added
  1. TODO
  2. TODO
  3. TODO
...
Reduce Task Execution
This diagram illustrates the steps taken when the Reduce portion of a MapReduce job is executed.
Image Added
  1. TODO
  2. TODO
  3. TODO
...
Job Completion
This diagram illustrates the steps taken a MapReduce job has completed.
Image Added
  1. TODO
  2. TODO
  3. TODO
...
Client Monitoring
This diagram illustrates the steps taken by a Client to monitor the status of a Job throughout the Job's life-cycle. The timeframe for this diagram span several of the diagrams above starting from Job Submission all the way through Job Completion.
Image Added
  1. TODO
  2. TODO
  3. TODO
...
NodeManager Token Flow
This diagram illustrates the flow of NodeManager Tokens throughout a MapReduce Job's life-cycle.
Image Added
  1. TODO
  2. TODO
  3. TODO
...