THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
If there is a failure, batches can be in in any of the queues. Recovery (or imports) takes all the batches in any of the states and puts them into the loggedQ in LOGGED state. This allows the recovered batches to reregister their interest in their ack tags when they are sent downstream.
Interaction with Retry.
As groups of events are written they have an associated tag and timestamp. If flume.agent.logdir.retransmit millis is exceeded, a retry call will be made on the event group tag. This will trigger a state transition of the data.
| Note |
|---|
This is updated as of FLUME-746 – should be in 0.9.5 |
State when attempting to retry | State after retry | Consequence |
|---|---|---|
IMPORT | IMPORT | Unexpected condition |
WRITING | WRITING | Unexepcted condition |
LOGGED | LOGGED | Do nothing. Already queued for (re-)sending |
SENDING | SENDING | Do nothing. Sending already in progress |
SENT | LOGGED | Demote sent data to LOGGED so it will eventually be resent |
E2EACKED | E2EACKED | Do nothing. Data s ack no need to resent |
ERROR | ERROR | Do nothing. Already in bad state, keep it there |
The collector side.
| No Format |
|---|
ackChecker -> roller -> dfsClosedNotifier /-> dfswriter -> hdfs. \-> rolltag acktag map --------------\-> ackDone -> Master's Ack Manager |
...