org.apache.giraph.ooc.policy

Class SimpleGCMonitoringOracle

java.lang.Object

org.apache.giraph.ooc.policy.SimpleGCMonitoringOracle

All Implemented Interfaces:

OutOfCoreOracle

public class SimpleGCMonitoringOracle
extends Object
implements OutOfCoreOracle

Out-of-core oracle to adaptively control data kept in memory, with the goal of keeping the memory state constantly at a desired state. This oracle monitors GC behavior to keep track of memory pressure. After each GC is done, this oracle retrieve statistics about the memory pressure (memory used, max memory, and how far away memory is compared to a max optimal pressure). Based on the the past 2 recent memory statistics, the oracle predicts the status of the memory, and sets the rate of load/store of data from/to disk. If the rate of loading data from disk is 'l', and the rate of storing data to disk is 's', the rate of data injection to memory from disk can be denoted as 'l-s'. This oracle determines what 'l-s' should be based on the prediction of memory status. Assume that based on the previous GC call the memory usage at time t_0 is m_0, and based on the most recent GC call the memory usage at time t_1 is m_1. So, the rate of memory increase is alpha = (m_1 - m_0) / (t_1 - t_0). Assume that the ideal memory pressure happens when the memory usage is m_ideal. So, at time 't_2 = t_1 + (t_1 - t_0)', we want m_ideal. That means the ideal rate would be beta = (m_ideal - m_1) / (t_2 - t_1). If the date injection rate to memory so far was i, the new injection rate should be: i_new = i - (alpha - beta)

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.giraph.ooc.policy.OutOfCoreOracle

OutOfCoreOracle.IOAction

Field Summary

Fields

Modifier and Type	Field and Description
static FloatConfOption	OPTIMAL_MEMORY_PRESSURE The optimal memory pressure at which GC behavior is close to ideal.

Constructor Summary

Constructors

Constructor and Description
SimpleGCMonitoringOracle(ImmutableClassesGiraphConfiguration conf, OutOfCoreEngine oocEngine) Constructor

Method Summary

Modifier and Type	Method and Description
boolean	approve(IOCommand command) Whether a command is appropriate to bring the memory to a more desired state.
void	commandCompleted(IOCommand command) Notification of command completion.
void	gcCompleted(com.sun.management.GarbageCollectionNotificationInfo gcInfo) Notification of GC completion.
OutOfCoreOracle.IOAction[]	getNextIOActions() Get the next set of viable IO actions to help bring memory to a more desired state.
void	startIteration() Called at the beginning of a superstep.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OPTIMAL_MEMORY_PRESSURE

public static final FloatConfOption OPTIMAL_MEMORY_PRESSURE

The optimal memory pressure at which GC behavior is close to ideal. This fraction may be dependant on the GC strategy used for running a job, but generally should not be dependent on the graph processing application.

Constructor Detail

SimpleGCMonitoringOracle

public SimpleGCMonitoringOracle(ImmutableClassesGiraphConfiguration conf,
                                OutOfCoreEngine oocEngine)

Constructor

Parameters:

conf - configuration

oocEngine - out-of-core engine

Method Detail

gcCompleted

public void gcCompleted(com.sun.management.GarbageCollectionNotificationInfo gcInfo)

Description copied from interface: OutOfCoreOracle

Notification of GC completion. Oracle may take certain decisions based on GC information (such as amount of time it took, memory it reclaimed, etc.)

Specified by:

gcCompleted in interface OutOfCoreOracle

Parameters:

gcInfo - GC information

startIteration

public void startIteration()

Description copied from interface: OutOfCoreOracle

Called at the beginning of a superstep.

Specified by:

startIteration in interface OutOfCoreOracle

getNextIOActions

public OutOfCoreOracle.IOAction[] getNextIOActions()

Description copied from interface: OutOfCoreOracle

Get the next set of viable IO actions to help bring memory to a more desired state.

Specified by:

getNextIOActions in interface OutOfCoreOracle

Returns:

an array of viable IO actions, sorted from highest priority to lowest priority

approve

public boolean approve(IOCommand command)

Description copied from interface: OutOfCoreOracle

Whether a command is appropriate to bring the memory to a more desired state. A command is not executed unless it is approved by the oracle. This method is specially important where there are multiple IO threads performing IO operations for the out-of-core mechanism. The approval becomes significantly important to prevent all IO threads from performing identical command type, if that is a necessity. For instance, execution of a particular command type by only one thread may bring the memory to a desired state, and the rest of IO threads may perform other types of commands.

Specified by:

approve in interface OutOfCoreOracle

Parameters:

command - the IO command that is about to execute

Returns:

'true' if the command is approved for execution. 'false' if the command should not be executed

commandCompleted

public void commandCompleted(IOCommand command)

Description copied from interface: OutOfCoreOracle

Notification of command completion. Oracle may update its status and commit the changes a command may cause.

Specified by:

commandCompleted in interface OutOfCoreOracle

Parameters:

command - the IO command that is completed