Page History

...

Extends the Reducer API and at this point only runs the progressService. It is invoked only when the numLoaders driver config is > 0. See above. VB: It is still unclear to me, how this would support throttling as indicated in this ticket, Looking for some details on how invoking a reducer helps.

public class SqoopReducer extends Reducer<SqoopWritable, NullWritable, SqoopWritable, NullWritable> {
..
      progressService.scheduleAtFixedRate(new SqoopProgressRunnable(context), 0, 2, TimeUnit.MINUTES);
} 

SqoopOutputFormatLoadExecutor and SqoopOutputFormatDataReader

...

 

1. The LoaderContext is set up in the ConsumerThread.run(..) method. 
2. Loader's load method is invoked passing the SqoopOutputFormatDataReader and the LoaderContext
3. The load method invokes the SqoopOutputFormatDataReader to read to records from the SqoopRecordWriter associated with the SqoopNullOutputFormat

...