Versions Compared

Old Version 19

changes.mady.by.user Kevin Minder

Saved on

compared with

New Version 20

changes.mady.by.user Kevin Minder

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The sequence diagrams below are intended to be a very detailed description of the interactions that occur during the process of defining, submitting and executing a map reduce job on a secure Hadoop 2.x cluster. Different phases of the overall process are covered in each diagram. The are intended to be taken as one continuous flow with the exception of the last diagram which illustrates parallel steps that would occur during the flow.

  1. Bootstrap
  2. Job Definition
  3. Job Submission
  4. Job Initiation
  5. Map Task Execution
  6. Reduce Task Execution
  7. Job Completion
  8. Client Monitoring

Legend

The descriptions of the interactions in the sequence diagrams below take this form.

...

Abbreviation

Description

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="27d8287f4cb89deb-d6fd0b18-4a484e54-9dfc985a-01e7de5ffbc97882a423c751"><ac:plain-text-body><![CDATA[

[KRB]

Kerberos Protocol

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="3495e7df15e263fc-4899c2cc-4adc45c6-855384c1-b2c3bb8fe2e9ca2e20281409"><ac:plain-text-body><![CDATA[

[RSKT:{kerberos-service-ticket}]

RPC protocol with SASL mutual authentication using Kerberos tickets.

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="c9c286e621c89e71-e71bd81f-4f374975-b5a1bacb-6e3e792a0933487727f3f3be"><ac:plain-text-body><![CDATA[

[RSAT:{access-token}]

RPC protocol with SASL mutual authentication using access tokens (e.g. YARN Node Manager Token).

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="9671cb54b96bf6fa-1beb6f92-46e54a1e-ada292a4-085ce11ac29e4599120f4d90"><ac:plain-text-body><![CDATA[

[RSDT:{delegation-token}]

RPC protocol with SASL mutual authentication using delegation tokens (e.g. HDFS Name Node Delegation Token).

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="f7c185d12dd3f6c3-6e5d929a-4ea94688-8129be7c-503fd7666151a133ff911e9a"><ac:plain-text-body><![CDATA[

[STP]

Shuffle data transfer protocol between ShuffleService and ReduceTask. HTTP protocol with TODO.

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="ce541ce7c6741e6f-59acb1b3-448c4a6b-a015bae2-b6c4d77f167ca77570fbd6e3"><ac:plain-text-body><![CDATA[

[DTP]

Block data transfer protocol between the DataNode and a client. HTTP protocol with block tokens plus SHA1 hash exchange.

]]></ac:plain-text-body></ac:structured-macro>

...

This diagram illustrates the interactions that occur when a Hadoop system is starting up and stabilizing. It involves various master components generating secret keys and slave components registering with the masters to receive these secret keys.

  1. createBlockAccessTokenSecretKey -
  2. kinit/AS_REQ -
  3. TGS_REQ -
  4. register/heartbeat -
  5. createNodeManagerTokenSecretKey -
  6. createAppContainerTokenSecretKey -
  7. kinit/AS_REQ -
  8. TGS_REQ -
  9. register/heartbeat -

...

Secure MapReduce2 - Job Definition

This diagram illustrates the steps taken by a client to define a MapReduce job that will later be submitted.

  1. TODO
  2. TODO
  3. TODO

...

Secure MapReduce2 - Job Submission

This diagram illustrates the steps taken during the submission of a MapReduce job.

  1. TODO
  2. TODO
  3. TODO

...

Secure MapReduce2 - Job Initiation

This diagram illustrates the steps taken when a MapReduce job is scheduled for execution.

  1. TODO
  2. TODO
  3. TODO

...

Secure MapReduce2 - Map Task Execution

This diagram illustrates the steps taken when the Map portion of a MapReduce job is executed.

  1. TODO
  2. TODO
  3. TODO

...

Secure MapReduce2 - Reduce Task Execution

This diagram illustrates the steps taken when the Reduce portion of a MapReduce job is executed.

  1. TODO
  2. TODO
  3. TODO

...

Secure MapReduce2 - Job Completion

This diagram illustrates the steps taken a MapReduce job has completed.

  1. TODO
  2. TODO
  3. TODO

...

Secure MapReduce2 - Client Monitoring

This diagram illustrates the steps taken by a Client to monitor the status of a Job throughout the Job's life-cycle. The timeframe for this diagram span several of the diagrams above starting from Job Submission all the way through Job Completion.

  1. TODO
  2. TODO
  3. TODO