org.apache.giraph.worker

Class WorkerInputSplitsHandler

java.lang.Object

org.apache.giraph.worker.WorkerInputSplitsHandler

public class WorkerInputSplitsHandler
extends Object

Requests splits from master and keeps track of them

Constructor Summary

Constructors

Constructor and Description
WorkerInputSplitsHandler(WorkerInfo workerInfo, int masterTaskId, WorkerClient workerClient) Constructor

Method Summary

Modifier and Type	Method and Description
void	receivedInputSplit(InputType splitType, byte[] serializedInputSplit) Called when an input split has been received from master, adding it to the map
byte[]	reserveInputSplit(InputType splitType, boolean isFirstSplit) Try to reserve an InputSplit for loading.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WorkerInputSplitsHandler

public WorkerInputSplitsHandler(WorkerInfo workerInfo,
                                int masterTaskId,
                                WorkerClient workerClient)

Constructor

Parameters:

workerInfo - Worker info of this worker

masterTaskId - Task id of master

workerClient - Worker client, used for communication

Method Detail

receivedInputSplit

public void receivedInputSplit(InputType splitType,
                               byte[] serializedInputSplit)

Called when an input split has been received from master, adding it to the map

Parameters:

splitType - Type of split

serializedInputSplit - Split

reserveInputSplit

public byte[] reserveInputSplit(InputType splitType,
                                boolean isFirstSplit)

Try to reserve an InputSplit for loading. While InputSplits exists that are not finished, wait until they are. NOTE: iterations on the InputSplit list only halt for each worker when it has scanned the entire list once and found every split marked RESERVED. When a worker fails, its Ephemeral RESERVED znodes will disappear, allowing other iterating workers to claim it's previously read splits. Only when the last worker left iterating on the list fails can a danger of data loss occur. Since worker failure in INPUT_SUPERSTEP currently causes job failure, this is OK. As the failure model evolves, this behavior might need to change. We could add watches on inputSplitFinishedNodes and stop iterating only when all these nodes have been created.

Parameters:

splitType - Type of split

isFirstSplit - Whether this is the first split input thread reads

Returns:

reserved InputSplit or null if no unfinished InputSplits exist