org.apache.giraph.master.input

Class MasterInputSplitsHandler

java.lang.Object

org.apache.giraph.master.input.MasterInputSplitsHandler

public class MasterInputSplitsHandler
extends Object

Handler for input splits on master Since currently Giraph fails if worker fails while reading input, we didn't complicate this part with retries yet, later it could be added by keeping track of which worker got which split and then if worker dies put these splits back to queues.

Field Summary

Fields

Modifier and Type	Field and Description
static StrConfOption	DONE_FRACTIONS_TO_STORE_IN_COUNTERS Store in counters timestamps when we finished reading these fractions of input

Constructor Summary

Constructors

Constructor and Description
MasterInputSplitsHandler(boolean useLocality, org.apache.hadoop.mapreduce.Mapper.Context context) Constructor

Method Summary

Modifier and Type	Method and Description
void	addSplits(InputType splitsType, List<org.apache.hadoop.mapreduce.InputSplit> inputSplits, GiraphInputFormat inputFormat) Add splits
static Map<String,Set<String>>	getCounterGroupAndNames()
static org.apache.hadoop.mapreduce.Counter	getSplitFractionDoneTimestampCounter(InputType inputType, double fraction, org.apache.hadoop.mapreduce.Mapper.Context context) Get counter
void	initialize(MasterClient masterClient, List<WorkerInfo> workers) Initialize
void	sendSplitTo(InputType splitType, int workerTaskId, boolean isFirstSplit) Called after we receive a split request from some worker, should send split back to it if available, or send it information that there is no more available

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DONE_FRACTIONS_TO_STORE_IN_COUNTERS

public static final StrConfOption DONE_FRACTIONS_TO_STORE_IN_COUNTERS

Store in counters timestamps when we finished reading these fractions of input

Constructor Detail

MasterInputSplitsHandler

public MasterInputSplitsHandler(boolean useLocality,
                                org.apache.hadoop.mapreduce.Mapper.Context context)

Constructor

Parameters:

useLocality - Whether to use locality information or not

context - Context for accessing counters

Method Detail

initialize

public void initialize(MasterClient masterClient,
                       List<WorkerInfo> workers)

Initialize

Parameters:

masterClient - Master client

workers - List of workers

addSplits

public void addSplits(InputType splitsType,
                      List<org.apache.hadoop.mapreduce.InputSplit> inputSplits,
                      GiraphInputFormat inputFormat)

Add splits

Parameters:

splitsType - Type of splits

inputSplits - Splits

inputFormat - Format

sendSplitTo

public void sendSplitTo(InputType splitType,
                        int workerTaskId,
                        boolean isFirstSplit)

Called after we receive a split request from some worker, should send split back to it if available, or send it information that there is no more available

Parameters:

splitType - Type of split requested

workerTaskId - Id of worker who requested split

isFirstSplit - Whether this is the first split a thread is requesting, or this request indicates that previously requested input split was done

getSplitFractionDoneTimestampCounter

public static org.apache.hadoop.mapreduce.Counter getSplitFractionDoneTimestampCounter(InputType inputType,
                                                                                       double fraction,
                                                                                       org.apache.hadoop.mapreduce.Mapper.Context context)

Get counter

Parameters:

inputType - Type of input for counter

fraction - Fraction for counter

context - Context to get counter from

Returns:

Counter

getCounterGroupAndNames

public static Map<String,Set<String>> getCounterGroupAndNames()