DiskBackedEdgeStore (Apache Giraph Parent 1.4.0-SNAPSHOT API)

java.lang.Object
- org.apache.giraph.ooc.data.DiskBackedDataStore<VertexIdEdges<I,E>>
- - org.apache.giraph.ooc.data.DiskBackedEdgeStore<I,V,E>

Type Parameters:: I - Vertex id; V - Vertex data; E - Edge data

All Implemented Interfaces:: EdgeStore<I,V,E>

public class DiskBackedEdgeStore<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>
extends DiskBackedDataStore<VertexIdEdges<I,E>>
implements EdgeStore<I,V,E>

Implementation of an edge-store used for out-of-core mechanism.

Field Summary
- Fields inherited from class org.apache.giraph.ooc.data.DiskBackedDataStore
  hasPartitionDataOnFile, MINIMUM_BUFFER_SIZE_TO_FLUSH, oocEngine

Constructor Summary

Constructors
Constructor and Description
`DiskBackedEdgeStore(EdgeStore<I,V,E> edgeStore, ImmutableClassesGiraphConfiguration<I,V,E> conf, OutOfCoreEngine oocEngine)` Constructor

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`addEntryToInMemoryPartitionData(int partitionId, VertexIdEdges<I,E> edges)` Adds a single entry for a given partition to the in-memory data store.
`void`	`addPartitionEdges(int partitionId, VertexIdEdges<I,E> edges)` Add edges belonging to a given partition on this worker.
`protected int`	`entrySerializedSize(VertexIdEdges<I,E> edges)` Gets the size of a given entry in bytes.
`boolean`	`hasEdgesForPartition(int partitionId)` Check if edge store has edge for a given partition
`protected long`	`loadInMemoryPartitionData(int partitionId, int ioThreadId, DataIndex index)` Loads data of a partition into data store.
`long`	`loadPartitionData(int partitionId)` Loads and assembles all data for a given partition, and put it into the data store.
`void`	`moveEdgesToVertices()` Move all edges from temporary storage to their source vertices.
`long`	`offloadBuffers(int partitionId)` Offloads raw data buffers of a given partition to disk, and returns the number of bytes offloaded from memory to disk.
`protected long`	`offloadInMemoryPartitionData(int partitionId, int ioThreadId, DataIndex index)` Offloads data of a partition in data store to disk.
`long`	`offloadPartitionData(int partitionId)` Offloads partition data of a given partition in the data store to disk, and returns the number of bytes offloaded from memory to disk.
`protected VertexIdEdges<I,E>`	`readNextEntry(DataInput in)` Reads the next available raw entry from a given input stream.
`void`	`readPartitionEdgeStore(int partitionId, DataInput input)` Serialize the edges of a given partition, and adds it to the partition store (assumes that the edge store does not have any edge from the partition already).
`protected void`	`writeEntry(VertexIdEdges<I,E> edges, DataOutput out)` Writes a single raw entry to a given output stream.
`void`	`writePartitionEdgeStore(int partitionId, DataOutput output)` Deserialize the edges of a given partition, and removes the associated data from the store.

Methods inherited from class org.apache.giraph.ooc.data.DiskBackedDataStore
addEntry, getCandidateBuffersToOffload, loadPartitionDataProxy, offloadBuffersProxy, offloadPartitionDataProxy

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Constructor Detail
 - DiskBackedEdgeStore
```
public DiskBackedEdgeStore(EdgeStore<I,V,E> edgeStore,
 ImmutableClassesGiraphConfiguration<I,V,E> conf,
 OutOfCoreEngine oocEngine)
```
 Constructor
 
 Parameters:
 
 edgeStore - In-memory edge store for which out-of-core edge store would be a wrapper
 
 conf - Configuration
 
 oocEngine - Out-of-core engine
- Method Detail
 - addPartitionEdges
```
public void addPartitionEdges(int partitionId,
 VertexIdEdges<I,E> edges)
```
 Description copied from interface: EdgeStore
 
 Add edges belonging to a given partition on this worker. Note: This method is thread-safe.
 
 Specified by:
 
 addPartitionEdges in interface EdgeStore
 
 Parameters:
 
 partitionId - Partition id for the incoming edges.
 
 edges - Incoming edges
 - moveEdgesToVertices
```
public void moveEdgesToVertices()
```
 Description copied from interface: EdgeStore
 
 Move all edges from temporary storage to their source vertices. Note: this method is not thread-safe and is called once all vertices and edges are read in INPUT_SUPERSTEP.
 
 Specified by:
 
 moveEdgesToVertices in interface EdgeStore
 - writePartitionEdgeStore
```
public void writePartitionEdgeStore(int partitionId,
 DataOutput output)
 throws IOException
```
 Description copied from interface: EdgeStore
 
 Deserialize the edges of a given partition, and removes the associated data from the store. Note: This method is not thread-safe (i.e. should not be called for the same partition at the same time).
 
 Specified by:
 
 writePartitionEdgeStore in interface EdgeStore
 
 Parameters:
 
 partitionId - Id of partition to deserialize
 
 output - Output to write the edge store to
 
 Throws:
 
 IOException
 - readPartitionEdgeStore
```
public void readPartitionEdgeStore(int partitionId,
 DataInput input)
 throws IOException
```
 Description copied from interface: EdgeStore
 
 Serialize the edges of a given partition, and adds it to the partition store (assumes that the edge store does not have any edge from the partition already). Note: This method is not thread-safe (i.e. should not be called for the same partition at the same time).
 
 Specified by:
 
 readPartitionEdgeStore in interface EdgeStore
 
 Parameters:
 
 partitionId - Id of partition to serialize
 
 input - Input to read the partition from
 
 Throws:
 
 IOException
 - hasEdgesForPartition
```
public boolean hasEdgesForPartition(int partitionId)
```
 Description copied from interface: EdgeStore
 
 Check if edge store has edge for a given partition
 
 Specified by:
 
 hasEdgesForPartition in interface EdgeStore
 
 Parameters:
 
 partitionId - Id of partition
 
 Returns:
 
 True iff edge store have messages for the given partition
 - loadPartitionData
```
public long loadPartitionData(int partitionId)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Loads and assembles all data for a given partition, and put it into the data store. Returns the number of bytes transferred from disk to memory in the loading process.
 
 Specified by:
 
 loadPartitionData in class DiskBackedDataStore<VertexIdEdges>
 
 Parameters:
 
 partitionId - id of the partition to load and assemble all data for
 
 Returns:
 
 number of bytes loaded from disk to memory
 
 Throws:
 
 IOException
 - offloadPartitionData
```
public long offloadPartitionData(int partitionId)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Offloads partition data of a given partition in the data store to disk, and returns the number of bytes offloaded from memory to disk.
 
 Specified by:
 
 offloadPartitionData in class DiskBackedDataStore<VertexIdEdges>
 
 Parameters:
 
 partitionId - id of the partition to offload its data
 
 Returns:
 
 number of bytes offloaded from memory to disk
 
 Throws:
 
 IOException
 - offloadBuffers
```
public long offloadBuffers(int partitionId)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Offloads raw data buffers of a given partition to disk, and returns the number of bytes offloaded from memory to disk.
 
 Specified by:
 
 offloadBuffers in class DiskBackedDataStore<VertexIdEdges>
 
 Parameters:
 
 partitionId - id of the partition to offload its raw data buffers
 
 Returns:
 
 number of bytes offloaded from memory to disk
 
 Throws:
 
 IOException
 - writeEntry
```
protected void writeEntry(VertexIdEdges<I,E> edges,
 DataOutput out)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Writes a single raw entry to a given output stream.
 
 Specified by:
 
 writeEntry in class DiskBackedDataStore<VertexIdEdges>
 
 Parameters:
 
 edges - entry to write to output
 
 out - output stream to write the entry to
 
 Throws:
 
 IOException
 - readNextEntry
```
protected VertexIdEdges<I,E> readNextEntry(DataInput in)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Reads the next available raw entry from a given input stream.
 
 Specified by:
 
 readNextEntry in class DiskBackedDataStore<VertexIdEdges>
 
 Parameters:
 
 in - input stream to read the entry from
 
 Returns:
 
 entry read from an input stream
 
 Throws:
 
 IOException
 - loadInMemoryPartitionData
```
protected long loadInMemoryPartitionData(int partitionId,
 int ioThreadId,
 DataIndex index)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Loads data of a partition into data store. Returns number of bytes loaded.
 
 Specified by:
 
 loadInMemoryPartitionData in class DiskBackedDataStore<VertexIdEdges>
 
 Parameters:
 
 partitionId - id of the partition to load its data
 
 ioThreadId - id of the IO thread performing the load
 
 index - data index chain for the data to load
 
 Returns:
 
 number of bytes loaded from disk to memory
 
 Throws:
 
 IOException
 - offloadInMemoryPartitionData
```
protected long offloadInMemoryPartitionData(int partitionId,
 int ioThreadId,
 DataIndex index)
 throws IOException
```
 Description copied from class: DiskBackedDataStore
 
 Offloads data of a partition in data store to disk. Returns the number of bytes offloaded to disk
 
 Specified by:
 
 offloadInMemoryPartitionData in class DiskBackedDataStore<VertexIdEdges>
 
 Parameters:
 
 partitionId - id of the partition to offload to disk
 
 ioThreadId - id of the IO thread performing the offload
 
 index - data index chain for the data to offload
 
 Returns:
 
 number of bytes offloaded from memory to disk
 
 Throws:
 
 IOException
 - entrySerializedSize
```
protected int entrySerializedSize(VertexIdEdges<I,E> edges)
```
 Description copied from class: DiskBackedDataStore
 
 Gets the size of a given entry in bytes.
 
 Specified by:
 
 entrySerializedSize in class DiskBackedDataStore<VertexIdEdges>
 
 Parameters:
 
 edges - input entry to find its size
 
 Returns:
 
 size of given input entry in bytes
 - addEntryToInMemoryPartitionData
```
protected void addEntryToInMemoryPartitionData(int partitionId,
 VertexIdEdges<I,E> edges)
```
 Description copied from class: DiskBackedDataStore
 
 Adds a single entry for a given partition to the in-memory data store.
 
 Specified by:
 
 addEntryToInMemoryPartitionData in class DiskBackedDataStore<VertexIdEdges>
 
 Parameters:
 
 partitionId - id of the partition to add the data to
 
 edges - input entry to add to the data store

Class DiskBackedEdgeStore<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

Field Summary

Fields inherited from class org.apache.giraph.ooc.data.DiskBackedDataStore

Constructor Summary

Method Summary

Methods inherited from class org.apache.giraph.ooc.data.DiskBackedDataStore

Methods inherited from class java.lang.Object

Constructor Detail

DiskBackedEdgeStore

Method Detail

addPartitionEdges

moveEdgesToVertices

writePartitionEdgeStore

readPartitionEdgeStore

hasEdgesForPartition

loadPartitionData

offloadPartitionData

offloadBuffers

writeEntry

readNextEntry

loadInMemoryPartitionData

offloadInMemoryPartitionData

entrySerializedSize

addEntryToInMemoryPartitionData