Out-of-core oracle to adaptively control data kept in memory, with the goal
of keeping the memory state constantly at a desired state. This oracle
monitors GC behavior to keep track of memory pressure.
After each GC is done, this oracle retrieve statistics about the memory
pressure (memory used, max memory, and how far away memory is compared to a
max optimal pressure). Based on the the past 2 recent memory statistics,
the oracle predicts the status of the memory, and sets the rate of load/store
of data from/to disk. If the rate of loading data from disk is 'l', and the
rate of storing data to disk is 's', the rate of data injection to memory
from disk can be denoted as 'l-s'. This oracle determines what 'l-s' should
be based on the prediction of memory status.
Assume that based on the previous GC call the memory usage at time t_0 is
m_0, and based on the most recent GC call the memory usage at time t_1 is
m_1. So, the rate of memory increase is alpha = (m_1 - m_0) / (t_1 - t_0).
Assume that the ideal memory pressure happens when the memory usage is
m_ideal. So, at time 't_2 = t_1 + (t_1 - t_0)', we want m_ideal. That means
the ideal rate would be beta = (m_ideal - m_1) / (t_2 - t_1). If the date
injection rate to memory so far was i, the new injection rate should be:
i_new = i - (alpha - beta)
Nested Class Summary
Nested classes/interfaces inherited from interface org.apache.giraph.ooc.policy.OutOfCoreOracle
The optimal memory pressure at which GC behavior is close to ideal. This
fraction may be dependant on the GC strategy used for running a job, but
generally should not be dependent on the graph processing application.
Whether a command is appropriate to bring the memory to a more desired
state. A command is not executed unless it is approved by the oracle. This
method is specially important where there are multiple IO threads
performing IO operations for the out-of-core mechanism. The approval
becomes significantly important to prevent all IO threads from performing
identical command type, if that is a necessity. For instance, execution of
a particular command type by only one thread may bring the memory to a
desired state, and the rest of IO threads may perform other types of