org.apache.giraph.graph

Class GiraphTransferRegulator

java.lang.Object

org.apache.giraph.graph.GiraphTransferRegulator

public class GiraphTransferRegulator
extends Object

Utility class to manage data transfers from a local worker reading InputSplits. Currently, this class measures # of vertices and edges per outgoing Collection of graph data (destined for a particular Partition and remote worker node, preselected by the master.) TODO: implement defaults and configurable options for measuring the size of input <V> or <E> data per read vertex, and setting limits on totals per outgoing graph data Collection etc. (See GIRAPH-260)

Field Summary

Fields

Modifier and Type	Field and Description
static String	MAX_EDGES_PER_TRANSFER Maximum edges to read from an InputSplit locally that are to be routed to a remote worker, before sending them.
static int	MAX_EDGES_PER_TRANSFER_DEFAULT Default maximum number of vertices per temp partition before sending.
static String	MAX_VERTICES_PER_TRANSFER Maximum vertices to read from an InputSplit locally that are to be routed to a remote worker, before sending them.
static int	MAX_VERTICES_PER_TRANSFER_DEFAULT Default maximum number of vertices per temp partition before sending.

Constructor Summary

Constructors

Constructor and Description
GiraphTransferRegulator(org.apache.hadoop.conf.Configuration conf) Default constructor

Method Summary

Modifier and Type	Method and Description
void	clearCounters() Clear storage to reset for reading new InputSplit
long	getMaxEdgesPerTransfer() Getter for MAX edge count to initiate a transfer
long	getMaxVerticesPerTransfer() Getter for MAX vertex count to initiate a transfer
long	getTotalEdges() Getter for total edge count for the current InputSplit
long	getTotalVertices() Getter for total vetex count for the current InputSplit
<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> void	incrementCounters(PartitionOwner partitionOwner, Vertex<I,V,E> vertex) Increment V & E counts for new vertex read, store values for that outgoing _temporary_ Partition, which shares the Partition ID for the actual remote Partition the collection will eventually be processed in.
boolean	transferThisPartition(PartitionOwner owner) Is this outbound data Collection full, and ready to transfer?

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_VERTICES_PER_TRANSFER

public static final String MAX_VERTICES_PER_TRANSFER

Maximum vertices to read from an InputSplit locally that are to be routed to a remote worker, before sending them.

See Also:

Constant Field Values

MAX_VERTICES_PER_TRANSFER_DEFAULT

public static final int MAX_VERTICES_PER_TRANSFER_DEFAULT

Default maximum number of vertices per temp partition before sending.

See Also:

Constant Field Values

MAX_EDGES_PER_TRANSFER

public static final String MAX_EDGES_PER_TRANSFER

Maximum edges to read from an InputSplit locally that are to be routed to a remote worker, before sending them.

See Also:

Constant Field Values

MAX_EDGES_PER_TRANSFER_DEFAULT

public static final int MAX_EDGES_PER_TRANSFER_DEFAULT

Default maximum number of vertices per temp partition before sending.

See Also:

Constant Field Values

Constructor Detail

GiraphTransferRegulator

public GiraphTransferRegulator(org.apache.hadoop.conf.Configuration conf)

Default constructor

Parameters:

conf - the Configuration for this job

Method Detail

transferThisPartition

public boolean transferThisPartition(PartitionOwner owner)

Is this outbound data Collection full, and ready to transfer?

Parameters:

owner - the partition owner for the outbound data

Returns:

'true' if the temp partition data is ready to transfer

clearCounters

public void clearCounters()

Clear storage to reset for reading new InputSplit

incrementCounters

public <I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> void incrementCounters(PartitionOwner partitionOwner,
                                                                                                                                                                  Vertex<I,V,E> vertex)

Increment V & E counts for new vertex read, store values for that outgoing _temporary_ Partition, which shares the Partition ID for the actual remote Partition the collection will eventually be processed in.

Type Parameters:

I - the vertex id type.

V - the vertex value type.

E - the edge value type.

Parameters:

partitionOwner - the owner of the Partition this data will eventually belong to.

vertex - the vertex to extract counts from.

getMaxEdgesPerTransfer

public long getMaxEdgesPerTransfer()

Getter for MAX edge count to initiate a transfer

Returns:

max edge count per transfer

getMaxVerticesPerTransfer

public long getMaxVerticesPerTransfer()

Getter for MAX vertex count to initiate a transfer

Returns:

max edge count per transfer

getTotalEdges

public long getTotalEdges()

Getter for total edge count for the current InputSplit

Returns:

the # of total edges counted in this InputSplit

getTotalVertices

public long getTotalVertices()

Getter for total vetex count for the current InputSplit

Returns:

the total # of vertices counted in this InputSplit