org.apache.giraph.block_app.library

Class Pieces

java.lang.Object

org.apache.giraph.block_app.library.Pieces

public class Pieces
extends Object

Utility class for creating common Pieces and computations for processing graphs.

Method Summary

Modifier and Type	Method and Description
static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object>	forAllVertices(String pieceName, Consumer<Vertex<I,V,E>> process) For each vertex execute given process function.
static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object>	forAllVerticesOnReceive(String pieceName, Consumer<Vertex<I,V,E>> process) For each vertex execute given process function.
static Piece<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable,NoMessage,Object>	masterCompute(String pieceName, Consumer<BlockMasterApi> process) Execute given function on master.
static <S,R extends org.apache.hadoop.io.Writable,I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object>	reduce(String name, ReduceOperation<S,R> reduceOp, SupplierFromVertex<I,V,E,S> valueSupplier, Consumer<R> reducedValueConsumer) Creates single reducer piece - given reduce class, supplier of values on worker, reduces and passes the result to given consumer on master.
static <S,R extends org.apache.hadoop.io.Writable,I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object>	reduceAndBroadcast(String name, ReduceOperation<S,R> reduceOp, SupplierFromVertex<I,V,E,S> valueSupplier, ConsumerWithVertex<I,V,E,R> reducedValueConsumer) Creates single reducer and broadcast piece - given reduce class, supplier of values on worker, reduces and broadcasts the value, passing it to the consumer on worker for each vertex.
static <S,R extends org.apache.hadoop.io.Writable,I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object>	reduceAndBroadcastWithArrayOfHandles(String name, int numHandles, Supplier<ReduceOperation<S,R>> reduceOp, SupplierFromVertex<I,V,E,Long> handleHashSupplier, SupplierFromVertex<I,V,E,S> valueSupplier, ConsumerWithVertex<I,V,E,R> reducedValueConsumer) Like reduceAndBroadcast, but uses array of handles for reducers and broadcasts, to make it feasible and performant when values are large.
static <S,R extends org.apache.hadoop.io.Writable,I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object>	reduceWithMaster(String name, ReduceOperation<S,R> reduceOp, SupplierFromVertex<I,V,E,S> valueSupplier, PairConsumer<R,BlockMasterApi> reducedValueConsumer) Creates single reducer piece - given reduce class, supplier of values on worker, reduces and passes the result to given consumer on master.
static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object>	removeVertices(String pieceName, SupplierFromVertex<I,V,E,Boolean> shouldRemoveVertex) Creates Piece which removes vertices for which supplier returns true.
static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable> SendMessagePiece<I,V,E,M>	sendMessage(String name, Class<M> messageClass, SupplierFromVertex<I,V,E,M> messageSupplier, SupplierFromVertex<I,V,E,Iterator<I>> targetsSupplier, ConsumerWithVertex<I,V,E,Iterable<M>> messagesConsumer) Creates Piece that for each vertex, sends message provided by messageSupplier to all targets provided by targetsSupplier.
static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable> SendMessageWithCombinerPiece<I,V,E,M>	sendMessage(String name, MessageCombiner<? super I,M> messageCombiner, SupplierFromVertex<I,V,E,M> messageSupplier, SupplierFromVertex<I,V,E,Iterator<I>> targetsSupplier, ConsumerWithVertex<I,V,E,M> messagesConsumer) Creates Piece that for each vertex, sends message provided by messageSupplier to all targets provided by targetsSupplier, and uses given messageCombiner to combine messages together.
static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable> SendMessagePiece<I,V,E,M>	sendMessageToNeighbors(String name, Class<M> messageClass, SupplierFromVertex<I,V,E,M> messageSupplier, ConsumerWithVertex<I,V,E,Iterable<M>> messagesConsumer) Creates Piece that for each vertex, sends message provided by messageSupplier to all neighbors of current vertex.
static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable> SendMessageWithCombinerPiece<I,V,E,M>	sendMessageToNeighbors(String name, MessageCombiner<? super I,M> messageCombiner, SupplierFromVertex<I,V,E,M> messageSupplier, ConsumerWithVertex<I,V,E,M> messagesConsumer) Creates Piece that for each vertex, sends message provided by messageSupplier to all neighbors of current vertex, and uses given messageCombiner to combine messages together.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

forAllVertices

public static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object> forAllVertices(String pieceName,
                                                                                                                                                                                               Consumer<Vertex<I,V,E>> process)

For each vertex execute given process function. Computation is happening in send phase of the returned Piece.

masterCompute

public static Piece<org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable,org.apache.hadoop.io.Writable,NoMessage,Object> masterCompute(String pieceName,
                                                                                                                                                        Consumer<BlockMasterApi> process)

Execute given function on master.

forAllVerticesOnReceive

public static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object> forAllVerticesOnReceive(String pieceName,
                                                                                                                                                                                                        Consumer<Vertex<I,V,E>> process)

For each vertex execute given process function. Computation is happening in the receive phase of the returned Piece. This function should be used if you need returned Piece to interact with subsequent Piece, as that requires passed function to be executed during receive phase,

removeVertices

public static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object> removeVertices(String pieceName,
                                                                                                                                                                                               SupplierFromVertex<I,V,E,Boolean> shouldRemoveVertex)

Creates Piece which removes vertices for which supplier returns true.

reduce

public static <S,R extends org.apache.hadoop.io.Writable,I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object> reduce(String name,
                                                                                                                                                                                                                                 ReduceOperation<S,R> reduceOp,
                                                                                                                                                                                                                                 SupplierFromVertex<I,V,E,S> valueSupplier,
                                                                                                                                                                                                                                 Consumer<R> reducedValueConsumer)

Creates single reducer piece - given reduce class, supplier of values on worker, reduces and passes the result to given consumer on master.

Type Parameters:

S - Single value type, objects passed on workers

R - Reduced value type

I - Vertex id type

V - Vertex value type

E - Edge value type

reduceWithMaster

public static <S,R extends org.apache.hadoop.io.Writable,I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object> reduceWithMaster(String name,
                                                                                                                                                                                                                                           ReduceOperation<S,R> reduceOp,
                                                                                                                                                                                                                                           SupplierFromVertex<I,V,E,S> valueSupplier,
                                                                                                                                                                                                                                           PairConsumer<R,BlockMasterApi> reducedValueConsumer)

Creates single reducer piece - given reduce class, supplier of values on worker, reduces and passes the result to given consumer on master.

Type Parameters:

S - Single value type, objects passed on workers

R - Reduced value type

I - Vertex id type

V - Vertex value type

E - Edge value type

reduceAndBroadcast

public static <S,R extends org.apache.hadoop.io.Writable,I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object> reduceAndBroadcast(String name,
                                                                                                                                                                                                                                             ReduceOperation<S,R> reduceOp,
                                                                                                                                                                                                                                             SupplierFromVertex<I,V,E,S> valueSupplier,
                                                                                                                                                                                                                                             ConsumerWithVertex<I,V,E,R> reducedValueConsumer)

Creates single reducer and broadcast piece - given reduce class, supplier of values on worker, reduces and broadcasts the value, passing it to the consumer on worker for each vertex.

Type Parameters:

S - Single value type, objects passed on workers

R - Reduced value type

I - Vertex id type

V - Vertex value type

E - Edge value type

reduceAndBroadcastWithArrayOfHandles

public static <S,R extends org.apache.hadoop.io.Writable,I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Piece<I,V,E,NoMessage,Object> reduceAndBroadcastWithArrayOfHandles(String name,
                                                                                                                                                                                                                                                               int numHandles,
                                                                                                                                                                                                                                                               Supplier<ReduceOperation<S,R>> reduceOp,
                                                                                                                                                                                                                                                               SupplierFromVertex<I,V,E,Long> handleHashSupplier,
                                                                                                                                                                                                                                                               SupplierFromVertex<I,V,E,S> valueSupplier,
                                                                                                                                                                                                                                                               ConsumerWithVertex<I,V,E,R> reducedValueConsumer)

Like reduceAndBroadcast, but uses array of handles for reducers and broadcasts, to make it feasible and performant when values are large. Each supplied value to reduce will be reduced in the handle defined by handleHashSupplier%numHandles

Type Parameters:

S - Single value type, objects passed on workers

R - Reduced value type

I - Vertex id type

V - Vertex value type

E - Edge value type

sendMessage

public static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable> SendMessagePiece<I,V,E,M> sendMessage(String name,
                                                                                                                                                                                                                                Class<M> messageClass,
                                                                                                                                                                                                                                SupplierFromVertex<I,V,E,M> messageSupplier,
                                                                                                                                                                                                                                SupplierFromVertex<I,V,E,Iterator<I>> targetsSupplier,
                                                                                                                                                                                                                                ConsumerWithVertex<I,V,E,Iterable<M>> messagesConsumer)

Creates Piece that for each vertex, sends message provided by messageSupplier to all targets provided by targetsSupplier. Received messages are then passed to and processed by provided messagesConsumer. If messageSupplier or targetsSupplier returns null, current vertex is not going to send any messages.

sendMessageToNeighbors

public static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable> SendMessagePiece<I,V,E,M> sendMessageToNeighbors(String name,
                                                                                                                                                                                                                                           Class<M> messageClass,
                                                                                                                                                                                                                                           SupplierFromVertex<I,V,E,M> messageSupplier,
                                                                                                                                                                                                                                           ConsumerWithVertex<I,V,E,Iterable<M>> messagesConsumer)

Creates Piece that for each vertex, sends message provided by messageSupplier to all neighbors of current vertex. Received messages are then passed to and processed by provided messagesConsumer. If messageSupplier returns null, current vertex is not going to send any messages.

sendMessage

public static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable> SendMessageWithCombinerPiece<I,V,E,M> sendMessage(String name,
                                                                                                                                                                                                                                            MessageCombiner<? super I,M> messageCombiner,
                                                                                                                                                                                                                                            SupplierFromVertex<I,V,E,M> messageSupplier,
                                                                                                                                                                                                                                            SupplierFromVertex<I,V,E,Iterator<I>> targetsSupplier,
                                                                                                                                                                                                                                            ConsumerWithVertex<I,V,E,M> messagesConsumer)

Creates Piece that for each vertex, sends message provided by messageSupplier to all targets provided by targetsSupplier, and uses given messageCombiner to combine messages together. Received combined message is then passed to and processed by provided messageConsumer. (null is passed to it, if vertex received no messages) If messageSupplier or targetsSupplier returns null, current vertex is not going to send any messages.

sendMessageToNeighbors

public static <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable,M extends org.apache.hadoop.io.Writable> SendMessageWithCombinerPiece<I,V,E,M> sendMessageToNeighbors(String name,
                                                                                                                                                                                                                                                       MessageCombiner<? super I,M> messageCombiner,
                                                                                                                                                                                                                                                       SupplierFromVertex<I,V,E,M> messageSupplier,
                                                                                                                                                                                                                                                       ConsumerWithVertex<I,V,E,M> messagesConsumer)

Creates Piece that for each vertex, sends message provided by messageSupplier to all neighbors of current vertex, and uses given messageCombiner to combine messages together. Received combined message is then passed to and processed by provided messageConsumer. (null is passed to it, if vertex received no messages) If messageSupplier returns null, current vertex is not going to send any messages.