org.apache.giraph.block_app.reducers.array

Class HugeArrayUtils

java.lang.Object

org.apache.giraph.block_app.reducers.array.HugeArrayUtils

public class HugeArrayUtils
extends Object

Utility class when we are dealing with huge arrays (i.e. large number of elements) within reducing/broadcasting. In Giraph, for each reducer there is a worker machine which is it's owner, which does partial aggregation for it. So if we have only single huge reducer - other workers will have to wait, while that single worker is doing huge reducing operation. Additionally single reducer should be smaller then max netty message, which is 1MB. On the other hand, each reducer has a meaningful overhead, so we should try to keep number of reducers as low as possible. By default we are being conservative, to keep individual reducers small, with striping into 500k reducers by default. If you know exact sizes of your objects you can specify exact number you want. So when we have huge array, we don't want one reducer/broadcast for each element, but we also don't want one reducer/broadcast for the whole array. This class allows transparent split into reasonable number of reducers (~500k), which solves both of the above issues.

Method Summary

Modifier and Type	Method and Description
static <V extends org.apache.hadoop.io.Writable> BroadcastArrayHandle<V>	broadcast(int count, Int2ObjFunction<V> valueSupplier, BlockMasterApi master) Broadcast a huge array, by splitting into NUM_STRIPES number of parts.
static <V extends org.apache.hadoop.io.Writable> BroadcastArrayHandle<V>	broadcast(int count, Int2ObjFunction<V> valueSupplier, PrimitiveTypeOps<V> typeOps, BlockMasterApi master) Broadcast a huge array, by splitting into NUM_STRIPES number of parts.
static <S,R extends org.apache.hadoop.io.Writable> ReducerArrayHandle<S,R>	createGlobalReducerArrayHandle(int fixedSize, ReduceOperation<S,R> elementReduceOp, CreateReducersApi reduceApi) Create global array of reducers, by splitting the huge array into NUM_STRIPES number of parts.
static <S,R extends org.apache.hadoop.io.Writable> ReducerArrayHandle<S,R>	createGlobalReducerArrayHandle(int fixedSize, ReduceOperation<S,R> elementReduceOp, CreateReducersApi reduceApi, int maxNumStripes) Create global array of reducers, by splitting the huge array into maxNumStripes number of parts.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

createGlobalReducerArrayHandle

public static <S,R extends org.apache.hadoop.io.Writable> ReducerArrayHandle<S,R> createGlobalReducerArrayHandle(int fixedSize,
                                                                                                                 ReduceOperation<S,R> elementReduceOp,
                                                                                                                 CreateReducersApi reduceApi)

Create global array of reducers, by splitting the huge array into NUM_STRIPES number of parts.

Parameters:

fixedSize - Number of elements

elementReduceOp - ReduceOperation for individual elements

reduceApi - Api for creating reducers

Returns:

Created ReducerArrayHandle

createGlobalReducerArrayHandle

public static <S,R extends org.apache.hadoop.io.Writable> ReducerArrayHandle<S,R> createGlobalReducerArrayHandle(int fixedSize,
                                                                                                                 ReduceOperation<S,R> elementReduceOp,
                                                                                                                 CreateReducersApi reduceApi,
                                                                                                                 int maxNumStripes)

Create global array of reducers, by splitting the huge array into maxNumStripes number of parts.

Parameters:

fixedSize - Number of elements

elementReduceOp - ReduceOperation for individual elements

reduceApi - Api for creating reducers

maxNumStripes - Maximal number of reducers to create.

Returns:

Created ReducerArrayHandle

broadcast

public static <V extends org.apache.hadoop.io.Writable> BroadcastArrayHandle<V> broadcast(int count,
                                                                                          Int2ObjFunction<V> valueSupplier,
                                                                                          BlockMasterApi master)

Broadcast a huge array, by splitting into NUM_STRIPES number of parts.

Parameters:

count - Number of elements

valueSupplier - Supplier of value to be broadcasted for a given index

master - Master API

Returns:

Created BroadcastArrayHandle

broadcast

public static <V extends org.apache.hadoop.io.Writable> BroadcastArrayHandle<V> broadcast(int count,
                                                                                          Int2ObjFunction<V> valueSupplier,
                                                                                          PrimitiveTypeOps<V> typeOps,
                                                                                          BlockMasterApi master)

Broadcast a huge array, by splitting into NUM_STRIPES number of parts. Efficient for primitive types, using BasicArray underneath.

Parameters:

count - Number of elements

valueSupplier - Supplier of value to be broadcasted for a given index

typeOps - Element TypeOps

master - Master API

Returns:

Created BroadcastArrayHandle