Versions Compared

Old Version 22

changes.mady.by.user Kevin Minder

Saved on

compared with

New Version Current

changes.mady.by.user Kevin Minder

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

...

Code Block
message [Protocol] ( input ) : output

Wiki MarkupThe {{\[Protocol\]}} portion describes the protocol, authentication mechanism and identities exchanged.

Abbreviation

Description

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="49fbaf65-814b-4e6d-8ec2-f6367009dc82"><ac:plain-text-body><![CDATA[

[KRB]

Kerberos Protocol

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="a008b904-b13f-4fed-beb6-5e63e730113f"><ac:plain-text-body><![CDATA[

[RSKT:{kerberos-service-ticket}]

RPC protocol with SASL mutual authentication using Kerberos tickets.

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="fbff41bd-e7d2-46f6-846d-55d61e3e256d"><ac:plain-text-body><![CDATA[

[RSAT:{access-token}]

RPC protocol with SASL mutual client authentication using access tokens (e.g. YARN Node Manager Token).

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="32a30a62-40a1-4efa-a9f1-cc9bf9293ff9"><ac:plain-text-body><![CDATA[

[RSDT:{delegation-token}]

RPC protocol with SASL mutual client authentication using delegation tokens (e.g. HDFS Name Node Delegation Token). ]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="8828105a-88ff-4042-b65c-e9962ecd72cf"><ac:plain-text-body><![CDATA[

[STP]

Shuffle data transfer protocol between ShuffleService and ReduceTask. HTTP protocol with TODO.

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="854478af-dd69-4d33-8e1f-efbf06dbd968"><ac:plain-text-body><![CDATA[

[DTP]

Block data transfer protocol between the DataNode and a client. HTTP protocol with block tokens plus SHA1 hash exchange.

]]></ac:plain-text-body></ac:structured-macro>

Suffixes are used in many cases to denote type.

Abbreviation

Description

tgt

Kerberos Ticket Granting Ticket

kt kst

Kerberos Service Ticket: u-jt-kt = A Kerberos Ticket for User u to access the JobTracker jt

kp

Kerberos Principal: nn-kp = The Kerberos principal for the NameNode nn

dt

Delegation Token: c-nn-dt = A delegation token for identity of the Client that can be presented to the NameNode.

tkn

Access Token: am-tkn = An access token that can be presented to the ApplicationMaster for access.

tkn-sk

Access Token Secret Key

id

Identifier: job-id = Job Identifier

...

Abbreviation

Description

u-nn-kt

Kerberos service ticket for User u to access NameNode nn

u-jt-kt

Kerberos service ticket for User u to access JobTracker jt

dn-nn-kt

Kerberos service ticket for DataNode dn to access NameNode nn

jt-nn-kt

Kerberos service ticket for JobTracker dn to access NameNode nn

tt-jt-kt

Kerberos service ticket for TaskTracker tt to access JobTracker jt

...

Bootstrap

This diagram illustrates the interactions that occur when a Hadoop system is starting up and stabilizing. It involves various master components generating secret keys and slave components registering with the masters to receive these secret keys.

...

  1. createBlockAccessTokenSecretKey -
  2. kinit/AS_REQ -
  3. TGS_REQ -
  4. register/heartbeat -
  5. createNodeManagerTokenSecretKey -
  6. createAppContainerTokenSecretKey -
  7. kinit/AS_REQ -
  8. TGS_REQ -
  9. register/heartbeat -

...

Job Definition

This diagram illustrates the steps taken by a client to define a MapReduce job that will later be submitted.

  1. TODO
  2. TODO
  3. TODO

...

Job Submission

This diagram illustrates the steps taken during the submission of a MapReduce job.

  1. TODO
  2. TODO
  3. TODO

...

Job Initiation

This diagram illustrates the steps taken when a MapReduce job is scheduled for execution.

  1. TODO
  2. TODO
  3. TODO

...

Map Task Execution

This diagram illustrates the steps taken when the Map portion of a MapReduce job is executed.

  1. TODO
  2. TODO
  3. TODO

...

Reduce Task Execution

This diagram illustrates the steps taken when the Reduce portion of a MapReduce job is executed.

  1. TODO
  2. TODO
  3. TODO

...

Job Completion

This diagram illustrates the steps taken a MapReduce job has completed.

  1. TODO
  2. TODO
  3. TODO

...

Client Monitoring

This diagram illustrates the steps taken by a Client to monitor the status of a Job throughout the Job's life-cycle. The timeframe for this diagram span several of the diagrams above starting from Job Submission all the way through Job Completion.

  1. TODO
  2. TODO
  3. TODO

...

NodeManager Token Flow

This diagram illustrates the flow of NodeManager Tokens throughout a MapReduce Job's life-cycle.

Image Added

  1. TODO
  2. TODO
  3. TODO