org.apache.giraph.block_app.framework.piece

Class DefaultParentPiece<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable,WV,WM extends org.apache.hadoop.io.Writable,S>

java.lang.Object

org.apache.giraph.block_app.framework.piece.AbstractPiece<I,V,E,M,WV,WM,S>

org.apache.giraph.block_app.framework.piece.DefaultParentPiece<I,V,E,M,WV,WM,S>

Type Parameters:

I - Vertex id type

V - Vertex value type

E - Edge value type

M - Message type

WV - Worker value type

WM - Worker message type

S - Execution stage type

All Implemented Interfaces:

Iterable<AbstractPiece>, Block

Direct Known Subclasses:

Piece, PieceWithWorkerContext

public abstract class DefaultParentPiece<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable,WV,WM extends org.apache.hadoop.io.Writable,S>
extends AbstractPiece<I,V,E,M,WV,WM,S>

Additional abstract implementations for all pieces to be used. Code here is not in AbstractPiece only to allow for non-standard non-user-defined pieces.
Only logic used by the underlying framework directly is in AbstractPiece itself.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.giraph.block_app.framework.piece.AbstractPiece

AbstractPiece.InnerVertexReceiver, AbstractPiece.InnerVertexSender

Field Summary

Fields

Modifier and Type	Field and Description
static EnumConfOption<MessageEncodeAndStoreType>	MESSAGE_ENCODE_AND_STORE_TYPE_MIN_FORCE This option will tell which message encode & store enum to force, when combining is not enabled.

Constructor Summary

Constructors

Constructor and Description
DefaultParentPiece()

Method Summary

Modifier and Type	Method and Description
protected boolean	allowOneMessageToManyIdsEncoding() Override to specify that this Piece allows one to many ids encoding to be used for messages.
protected Class<M>	getMessageClass() Override to specify type of the message this Piece sends, if it does send messages.
MessageClasses<I,M>	getMessageClasses(ImmutableClassesGiraphConfiguration conf) Returns MessageClasses definition for messages being sent by this Piece.
protected MessageCombiner<? super I,M>	getMessageCombiner(ImmutableClassesGiraphConfiguration conf) Override to specify message combiner to be used, if any.
protected MessageValueFactory<M>	getMessageFactory(ImmutableClassesGiraphConfiguration conf) Override to specify message value factory to be used, which creates objects into which messages will be deserialized.
VertexSender<I,V,E>	getVertexSender(BlockWorkerSendApi<I,V,E,M> workerApi, S executionStage) Override to do vertex send processing.
AbstractPiece.InnerVertexSender	getWrappedVertexSender(BlockWorkerSendApi<I,V,E,M> workerApi, S executionStage) Add automatic handling of reducers to getVertexSender.
protected boolean	receiveIgnoreExistingVertices() Override to specify that receive of this Piece (and send of next Piece) ignore existing vertices, and just process received messages.
protected void	reduceDouble(ReducerHandle<org.apache.hadoop.io.DoubleWritable,?> reduceHandle, double value)
protected void	reduceFloat(ReducerHandle<org.apache.hadoop.io.FloatWritable,?> reduceHandle, float value)
protected void	reduceInt(ReducerHandle<org.apache.hadoop.io.IntWritable,?> reduceHandle, int value)
protected void	reduceLong(ReducerHandle<org.apache.hadoop.io.LongWritable,?> reduceHandle, long value)
void	registerReducers(CreateReducersApi reduceApi, S executionStage) Override to register any potential reducers used by this piece, through calls to reduceApi, which will return reducer handles for simple.
void	wrappedRegisterReducers(BlockMasterApi masterApi, S executionStage) Add automatic handling of reducers to registerReducers.

Methods inherited from class org.apache.giraph.block_app.framework.piece.AbstractPiece

equals, forAllPossiblePieces, getPieceCount, getVertexReceiver, hashCode, iterator, masterCompute, nextExecutionStage, registerAggregators, toString, workerContextReceive, workerContextSend

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

MESSAGE_ENCODE_AND_STORE_TYPE_MIN_FORCE

public static final EnumConfOption<MessageEncodeAndStoreType> MESSAGE_ENCODE_AND_STORE_TYPE_MIN_FORCE

This option will tell which message encode & store enum to force, when combining is not enabled. MESSAGE_ENCODE_AND_STORE_TYPE and this property are basically upper and lower bound on message store type, when looking them in order from not doing anything special, to most advanced type: BYTEARRAY_PER_PARTITION, EXTRACT_BYTEARRAY_PER_PARTITION, POINTER_LIST_PER_VERTEX resulting encode type is going to be: pieceEncodingType = piece.allowOneMessageToManyIdsEncoding() ? POINTER_LIST_PER_VERTEX : BYTEARRAY_PER_PARTITION) Math.max(index(minForce), Math.min(index(maxAllowed), index(pieceType); This is useful to force all pieces onto particular message store, even if they do not overrideallowOneMessageToManyIdsEncoding, though that might be rarely needed. This option might be more useful for fully local computation, where overall job behavior is quite different.

Constructor Detail

DefaultParentPiece

public DefaultParentPiece()

Method Detail

registerReducers

public void registerReducers(CreateReducersApi reduceApi,
                             S executionStage)

Override to register any potential reducers used by this piece, through calls to reduceApi, which will return reducer handles for simple. Tip: Without defining a field, first write here name of the field and what you want to reduce, like: totalSum = reduceApi.createLocalReducer(SumReduce.DOUBLE); and then use tools your IDE provides to generate field signature itself, which might be slightly complex: ReducerHandle<DoubleWritable, DoubleWritable> totalSum;

getVertexSender

public VertexSender<I,V,E> getVertexSender(BlockWorkerSendApi<I,V,E,M> workerApi,
                                           S executionStage)

Override to do vertex send processing. Creates handler that defines what should be executed on worker during send phase. This logic gets executed first. This function is called once on each worker on each thread, in parallel, on their copy of Piece object to create functions handler. If returned object implements Postprocessor interface, then corresponding postprocess() function is going to be called once, after all vertices corresponding thread needed to process are done.

getMessageClass

protected Class<M> getMessageClass()

Override to specify type of the message this Piece sends, if it does send messages. If not overwritten, no messages can be sent.

getMessageFactory

protected MessageValueFactory<M> getMessageFactory(ImmutableClassesGiraphConfiguration conf)

Override to specify message value factory to be used, which creates objects into which messages will be deserialized. If not overwritten, or null is returned, DefaultMessageValueFactory will be used.

getMessageCombiner

protected MessageCombiner<? super I,M> getMessageCombiner(ImmutableClassesGiraphConfiguration conf)

Override to specify message combiner to be used, if any. Message combiner itself should be immutable (i.e. it will be call simultanously from multiple threads)

allowOneMessageToManyIdsEncoding

protected boolean allowOneMessageToManyIdsEncoding()

Override to specify that this Piece allows one to many ids encoding to be used for messages. You should override this function, if you are sending identical message to all targets, and message itself is not extremely small.

receiveIgnoreExistingVertices

protected boolean receiveIgnoreExistingVertices()

Override to specify that receive of this Piece (and send of next Piece) ignore existing vertices, and just process received messages. Useful when distributed processing on groups that are not vertices is needed. This flag allows you not to worry whether a destination vertex exist, and removes need to clean it up when finished. One example is if each vertex is in a cluster, and we need to process something per cluster. Alternative are reducers, which have distributed reduction, but mostly master still does the processing afterwards, and amount of data needs to fit single machine (master).

getMessageClasses

public MessageClasses<I,M> getMessageClasses(ImmutableClassesGiraphConfiguration conf)

Description copied from class: AbstractPiece

Returns MessageClasses definition for messages being sent by this Piece.

Specified by:

getMessageClasses in class AbstractPiece<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable,WV,WM extends org.apache.hadoop.io.Writable,S>

getWrappedVertexSender

public final AbstractPiece.InnerVertexSender getWrappedVertexSender(BlockWorkerSendApi<I,V,E,M> workerApi,
                                                                    S executionStage)

Description copied from class: AbstractPiece

Add automatic handling of reducers to getVertexSender. Only for Framework internal use.

Specified by:

getWrappedVertexSender in class AbstractPiece<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable,WV,WM extends org.apache.hadoop.io.Writable,S>

wrappedRegisterReducers

public final void wrappedRegisterReducers(BlockMasterApi masterApi,
                                          S executionStage)

Description copied from class: AbstractPiece

Add automatic handling of reducers to registerReducers. Only for internal use.

Specified by:

wrappedRegisterReducers in class AbstractPiece<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable,WV,WM extends org.apache.hadoop.io.Writable,S>

reduceDouble

protected final void reduceDouble(ReducerHandle<org.apache.hadoop.io.DoubleWritable,?> reduceHandle,
                                  double value)

reduceFloat

protected final void reduceFloat(ReducerHandle<org.apache.hadoop.io.FloatWritable,?> reduceHandle,
                                 float value)

reduceLong

protected final void reduceLong(ReducerHandle<org.apache.hadoop.io.LongWritable,?> reduceHandle,
                                long value)

reduceInt

protected final void reduceInt(ReducerHandle<org.apache.hadoop.io.IntWritable,?> reduceHandle,
                               int value)