org.apache.giraph.io.formats

Class GiraphFileInputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>

org.apache.giraph.io.formats.GiraphFileInputFormat<K,V>

Type Parameters:

K - Key

V - Value

Direct Known Subclasses:

GiraphTextInputFormat

public abstract class GiraphFileInputFormat<K,V>
extends org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>

Provides functionality similar to FileInputFormat, but allows for different data sources (vertex and edge data).

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

org.apache.hadoop.mapreduce.lib.input.FileInputFormat.Counter

Field Summary

Fields

Modifier and Type	Field and Description
static String	EDGE_INPUT_DIR Edge input file paths.
static String	NUM_EDGE_INPUT_FILES Number of edge input files.
static String	NUM_VERTEX_INPUT_FILES Number of vertex input files.
static String	VERTEX_INPUT_DIR Vertex input file paths.

Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

DEFAULT_LIST_STATUS_NUM_THREADS, INPUT_DIR, INPUT_DIR_RECURSIVE, LIST_STATUS_NUM_THREADS, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE

Constructor Summary

Constructors

Constructor and Description
GiraphFileInputFormat()

Method Summary

Modifier and Type	Method and Description
static void	addEdgeInputPath(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path) Add a Path to the list of edge inputs.
static void	addVertexInputPath(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path) Add a Path to the list of vertex inputs.
static org.apache.hadoop.fs.Path[]	getEdgeInputPaths(org.apache.hadoop.mapreduce.JobContext context) Get the list of edge input Paths.
List<org.apache.hadoop.mapreduce.InputSplit>	getEdgeSplits(org.apache.hadoop.mapreduce.JobContext job) Generate the list of edge input splits.
static org.apache.hadoop.fs.Path[]	getVertexInputPaths(org.apache.hadoop.mapreduce.JobContext context) Get the list of vertex input Paths.
List<org.apache.hadoop.mapreduce.InputSplit>	getVertexSplits(org.apache.hadoop.mapreduce.JobContext job) Generate the list of vertex input splits.
protected List<org.apache.hadoop.fs.FileStatus>	listEdgeStatus(org.apache.hadoop.mapreduce.JobContext job) List edge input directories.
protected List<org.apache.hadoop.fs.FileStatus>	listVertexStatus(org.apache.hadoop.mapreduce.JobContext job) List vertex input directories.
static void	setEdgeInputPath(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path) Set the Path for edge input.
static void	setVertexInputPath(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path path) Set the Path for vertex input.

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, getSplits, isSplitable, listStatus, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

Methods inherited from class org.apache.hadoop.mapreduce.InputFormat

createRecordReader

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

VERTEX_INPUT_DIR

public static final String VERTEX_INPUT_DIR

Vertex input file paths.

See Also:

Constant Field Values

EDGE_INPUT_DIR

public static final String EDGE_INPUT_DIR

Edge input file paths.

See Also:

Constant Field Values

NUM_VERTEX_INPUT_FILES

public static final String NUM_VERTEX_INPUT_FILES

Number of vertex input files.

See Also:

Constant Field Values

NUM_EDGE_INPUT_FILES

public static final String NUM_EDGE_INPUT_FILES

Number of edge input files.

See Also:

Constant Field Values

Constructor Detail

GiraphFileInputFormat

public GiraphFileInputFormat()

Method Detail

addVertexInputPath

public static void addVertexInputPath(org.apache.hadoop.conf.Configuration conf,
                                      org.apache.hadoop.fs.Path path)
                               throws IOException

Add a Path to the list of vertex inputs.

Parameters:

conf - the Configuration to store the input paths

path - Path to be added to the list of vertex inputs

Throws:

IOException

setVertexInputPath

public static void setVertexInputPath(org.apache.hadoop.conf.Configuration conf,
                                      org.apache.hadoop.fs.Path path)
                               throws IOException

Set the Path for vertex input.

Parameters:

conf - Configuration to store in

path - Path to set

Throws:

IOException - on I/O errors

addEdgeInputPath

public static void addEdgeInputPath(org.apache.hadoop.conf.Configuration conf,
                                    org.apache.hadoop.fs.Path path)
                             throws IOException

Add a Path to the list of edge inputs.

Parameters:

conf - the Configuration to store the input paths

path - Path to be added to the list of edge inputs

Throws:

IOException

setEdgeInputPath

public static void setEdgeInputPath(org.apache.hadoop.conf.Configuration conf,
                                    org.apache.hadoop.fs.Path path)
                             throws IOException

Set the Path for edge input.

Parameters:

conf - Configuration to store in

path - Path to set

Throws:

IOException - on I/O errors

getVertexInputPaths

public static org.apache.hadoop.fs.Path[] getVertexInputPaths(org.apache.hadoop.mapreduce.JobContext context)

Get the list of vertex input Paths.

Parameters:

context - The job

Returns:

The list of input Paths

getEdgeInputPaths

public static org.apache.hadoop.fs.Path[] getEdgeInputPaths(org.apache.hadoop.mapreduce.JobContext context)

Get the list of edge input Paths.

Parameters:

context - The job

Returns:

The list of input Paths

listVertexStatus

protected List<org.apache.hadoop.fs.FileStatus> listVertexStatus(org.apache.hadoop.mapreduce.JobContext job)
                                                          throws IOException

List vertex input directories.

Parameters:

job - the job to list vertex input paths for

Returns:

array of FileStatus objects

Throws:

IOException - if zero items.

listEdgeStatus

protected List<org.apache.hadoop.fs.FileStatus> listEdgeStatus(org.apache.hadoop.mapreduce.JobContext job)
                                                        throws IOException

List edge input directories.

Parameters:

job - the job to list edge input paths for

Returns:

array of FileStatus objects

Throws:

IOException - if zero items.

getVertexSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getVertexSplits(org.apache.hadoop.mapreduce.JobContext job)
                                                             throws IOException

Generate the list of vertex input splits.

Parameters:

job - The job

Returns:

The list of vertex input splits

Throws:

IOException

getEdgeSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getEdgeSplits(org.apache.hadoop.mapreduce.JobContext job)
                                                           throws IOException

Generate the list of edge input splits.

Parameters:

job - The job

Returns:

The list of edge input splits

Throws:

IOException