org.apache.giraph.ooc.policy

Class MemoryEstimatorOracle

java.lang.Object

org.apache.giraph.ooc.policy.MemoryEstimatorOracle

All Implemented Interfaces:

OutOfCoreOracle

public class MemoryEstimatorOracle
extends Object
implements OutOfCoreOracle

Implementation of OutOfCoreOracle that uses a linear regression model to estimate actual memory usage based on the current state of computation. The model takes into consideration 5 parameters: y = c0 + c1*x1 + c2*x2 + c3*x3 + c4*x4 + c5*x5 y: memory usage x1: edges loaded x2: vertices loaded x3: vertices processed x4: bytes received due to messages x5: bytes loaded/stored from/to disk due to OOC.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.giraph.ooc.policy.OutOfCoreOracle

OutOfCoreOracle.IOAction

Field Summary

Fields

Modifier and Type	Field and Description
static FloatConfOption	AM_HIGH_THRESHOLD If mem-usage is above this threshold, active threads are set to 0
static FloatConfOption	AM_LOW_THRESHOLD If mem-usage is below this threshold, active threads are set to max
static LongConfOption	CHECK_MEMORY_INTERVAL Memory check interval in msec
static FloatConfOption	CREDIT_HIGH_THRESHOLD If mem-usage is above this threshold, credit is set to 0
static FloatConfOption	CREDIT_LOW_THRESHOLD If mem-usage is below this threshold, credit is set to max
static FloatConfOption	GC_MINIMUM_RECLAIM_FRACTION Used to detect a high memory pressure situation
static FloatConfOption	MANUAL_GC_MEMORY_PRESSURE If mem-usage is above this threshold and no Full GC has been called, we call it manually
static FloatConfOption	OOC_THRESHOLD OOC starts if mem-usage is above this threshold

Constructor Summary

Constructors

Constructor and Description
MemoryEstimatorOracle(ImmutableClassesGiraphConfiguration conf, OutOfCoreEngine oocEngine) Constructor.

Method Summary

Modifier and Type	Method and Description
boolean	approve(IOCommand command) Whether a command is appropriate to bring the memory to a more desired state.
void	commandCompleted(IOCommand command) Notification of command completion.
void	gcCompleted(com.sun.management.GarbageCollectionNotificationInfo gcInfo) When a new GC has completed, we can get an accurate measurement of the memory usage.
OutOfCoreOracle.IOAction[]	getNextIOActions() Get the next set of viable IO actions to help bring memory to a more desired state.
void	startIteration() Resets all the counters used in the memory estimation.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CHECK_MEMORY_INTERVAL

public static final LongConfOption CHECK_MEMORY_INTERVAL

Memory check interval in msec

MANUAL_GC_MEMORY_PRESSURE

public static final FloatConfOption MANUAL_GC_MEMORY_PRESSURE

If mem-usage is above this threshold and no Full GC has been called, we call it manually

GC_MINIMUM_RECLAIM_FRACTION

public static final FloatConfOption GC_MINIMUM_RECLAIM_FRACTION

Used to detect a high memory pressure situation

AM_HIGH_THRESHOLD

public static final FloatConfOption AM_HIGH_THRESHOLD

If mem-usage is above this threshold, active threads are set to 0

AM_LOW_THRESHOLD

public static final FloatConfOption AM_LOW_THRESHOLD

If mem-usage is below this threshold, active threads are set to max

CREDIT_HIGH_THRESHOLD

public static final FloatConfOption CREDIT_HIGH_THRESHOLD

If mem-usage is above this threshold, credit is set to 0

CREDIT_LOW_THRESHOLD

public static final FloatConfOption CREDIT_LOW_THRESHOLD

If mem-usage is below this threshold, credit is set to max

OOC_THRESHOLD

public static final FloatConfOption OOC_THRESHOLD

OOC starts if mem-usage is above this threshold

Constructor Detail

MemoryEstimatorOracle

public MemoryEstimatorOracle(ImmutableClassesGiraphConfiguration conf,
                             OutOfCoreEngine oocEngine)

Constructor.

Parameters:

conf - Configuration

oocEngine - OOC engine.:w

Method Detail

startIteration

public void startIteration()

Resets all the counters used in the memory estimation. This is called at the beginning of a new superstep.

The number of vertices to compute in the next superstep gets reset in GraphTaskManager.processGraphPartitions(org.apache.hadoop.mapreduce.Mapper<?, ?, ?, ?>.Context, java.util.List<org.apache.giraph.partition.PartitionStats>, org.apache.giraph.graph.GraphState, org.apache.giraph.comm.messages.MessageStore<I, org.apache.hadoop.io.Writable>, int) right before PartitionStore.startIteration() gets called.

Specified by:

startIteration in interface OutOfCoreOracle

getNextIOActions

public OutOfCoreOracle.IOAction[] getNextIOActions()

Description copied from interface: OutOfCoreOracle

Get the next set of viable IO actions to help bring memory to a more desired state.

Specified by:

getNextIOActions in interface OutOfCoreOracle

Returns:

an array of viable IO actions, sorted from highest priority to lowest priority

approve

public boolean approve(IOCommand command)

Description copied from interface: OutOfCoreOracle

Whether a command is appropriate to bring the memory to a more desired state. A command is not executed unless it is approved by the oracle. This method is specially important where there are multiple IO threads performing IO operations for the out-of-core mechanism. The approval becomes significantly important to prevent all IO threads from performing identical command type, if that is a necessity. For instance, execution of a particular command type by only one thread may bring the memory to a desired state, and the rest of IO threads may perform other types of commands.

Specified by:

approve in interface OutOfCoreOracle

Parameters:

command - the IO command that is about to execute

Returns:

'true' if the command is approved for execution. 'false' if the command should not be executed

commandCompleted

public void commandCompleted(IOCommand command)

Description copied from interface: OutOfCoreOracle

Notification of command completion. Oracle may update its status and commit the changes a command may cause.

Specified by:

commandCompleted in interface OutOfCoreOracle

Parameters:

command - the IO command that is completed

gcCompleted

public void gcCompleted(com.sun.management.GarbageCollectionNotificationInfo gcInfo)

When a new GC has completed, we can get an accurate measurement of the memory usage. We use this to update the linear regression model.

Specified by:

gcCompleted in interface OutOfCoreOracle

Parameters:

gcInfo - GC information