org.apache.giraph.io.hcatalog

Class HCatalogEdgeInputFormat<I extends org.apache.hadoop.io.WritableComparable,E extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.giraph.conf.DefaultImmutableClassesGiraphConfigurable<I,V,E>

org.apache.giraph.io.GiraphInputFormat<I,org.apache.hadoop.io.Writable,E>

org.apache.giraph.io.EdgeInputFormat<I,E>

org.apache.giraph.io.hcatalog.HCatalogEdgeInputFormat<I,E>

Type Parameters:

I - Vertex id

E - Edge value

All Implemented Interfaces:

GiraphConfigurationSettable<I,org.apache.hadoop.io.Writable,E>, ImmutableClassesGiraphConfigurable<I,org.apache.hadoop.io.Writable,E>

public abstract class HCatalogEdgeInputFormat<I extends org.apache.hadoop.io.WritableComparable,E extends org.apache.hadoop.io.Writable>
extends EdgeInputFormat<I,E>

HCatalog EdgeInputFormat for reading edges from Hive/Pig.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	HCatalogEdgeInputFormat.HCatalogEdgeReader<I extends org.apache.hadoop.io.WritableComparable,E extends org.apache.hadoop.io.Writable> EdgeReader for HCatalogEdgeInputFormat.
protected static class	HCatalogEdgeInputFormat.SingleRowHCatalogEdgeNoValueReader<I extends org.apache.hadoop.io.WritableComparable> HCatalogEdgeInputFormat.HCatalogEdgeReader for tables holding a complete edge in each row where the edges contain no data other than IDs they point to.
protected static class	HCatalogEdgeInputFormat.SingleRowHCatalogEdgeReader<I extends org.apache.hadoop.io.WritableComparable,E extends org.apache.hadoop.io.Writable> HCatalogEdgeInputFormat.HCatalogEdgeReader for tables holding a complete edge in each row.

Constructor Summary

Constructors

Constructor and Description
HCatalogEdgeInputFormat()

Method Summary

Modifier and Type	Method and Description
protected abstract HCatalogEdgeInputFormat.HCatalogEdgeReader<I,E>	createEdgeReader() Create EdgeReader.
EdgeReader<I,E>	createEdgeReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context) Create an edge reader for a given split.
protected GiraphHCatInputFormat	getHCatInputFormat() Get underlying HCatalog input format.
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context, int minSplitCountHint) Get the list of input splits for the format.

Methods inherited from class org.apache.giraph.io.GiraphInputFormat

checkInputSpecs, readInputSplit, writeInputSplit

Methods inherited from class org.apache.giraph.conf.DefaultImmutableClassesGiraphConfigurable

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HCatalogEdgeInputFormat

public HCatalogEdgeInputFormat()

Method Detail

getSplits

public final List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context,
                                                                    int minSplitCountHint)
                                                             throws IOException,
                                                                    InterruptedException

Description copied from class: GiraphInputFormat

Get the list of input splits for the format.

Specified by:

getSplits in class GiraphInputFormat<I extends org.apache.hadoop.io.WritableComparable,org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

Parameters:

context - The job context

minSplitCountHint - Minimum number of splits to create (hint)

Returns:

The list of input splits

Throws:

IOException

InterruptedException

getHCatInputFormat

protected GiraphHCatInputFormat getHCatInputFormat()

Get underlying HCatalog input format. Used for creating readers.

Returns:

GiraphHCatInputFormat stored.

createEdgeReader

protected abstract HCatalogEdgeInputFormat.HCatalogEdgeReader<I,E> createEdgeReader()

Create EdgeReader.

Returns:

HCatalogEdgeInputFormat.HCatalogEdgeReader instance.

createEdgeReader

public EdgeReader<I,E> createEdgeReader(org.apache.hadoop.mapreduce.InputSplit split,
                                        org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                 throws IOException

Description copied from class: EdgeInputFormat

Create an edge reader for a given split. The framework will call EdgeReader.initialize(InputSplit, TaskAttemptContext) before the split is used.

Specified by:

createEdgeReader in class EdgeInputFormat<I extends org.apache.hadoop.io.WritableComparable,E extends org.apache.hadoop.io.Writable>

Parameters:

split - the split to be read

context - the information about the task

Returns:

a new record reader

Throws:

IOException