org.apache.giraph.io.hcatalog

Class HCatalogVertexInputFormat<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.giraph.conf.DefaultImmutableClassesGiraphConfigurable<I,V,E>

org.apache.giraph.io.GiraphInputFormat<I,V,E>

org.apache.giraph.io.VertexInputFormat<I,V,E>

org.apache.giraph.io.hcatalog.HCatalogVertexInputFormat<I,V,E>

Type Parameters:

I - Vertex id

V - Vertex value

E - Edge value

All Implemented Interfaces:

GiraphConfigurationSettable<I,V,E>, ImmutableClassesGiraphConfigurable<I,V,E>

public abstract class HCatalogVertexInputFormat<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>
extends VertexInputFormat<I,V,E>

Abstract class that users should subclass to load data from a Hive or Pig table. You can easily implement a HCatalogVertexInputFormat.HCatalogVertexReader by extending either HCatalogVertexInputFormat.SingleRowHCatalogVertexReader or HCatalogVertexInputFormat.MultiRowHCatalogVertexReader depending on how data for each vertex is stored in the input table.

The desired database and table name to load from can be specified via GiraphHCatInputFormat.setVertexInput(org.apache.hadoop.mapreduce.Job, org.apache.hcatalog.mapreduce.InputJobInfo) as you setup your vertex input format with GiraphConfiguration.setVertexInputFormatClass(Class).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected class	HCatalogVertexInputFormat.HCatalogVertexReader Abstract class that users should subclass based on their specific vertex input.
protected class	HCatalogVertexInputFormat.MultiRowHCatalogVertexReader HCatalogVertexReader for tables holding vertex info across multiple rows sorted by vertex id column, so that they appear consecutively to the RecordReader.
protected class	HCatalogVertexInputFormat.SingleRowHCatalogVertexReader HCatalogVertexReader for tables holding complete vertex info within each row.

Constructor Summary

Constructors

Constructor and Description
HCatalogVertexInputFormat()

Method Summary

Modifier and Type	Method and Description
protected abstract HCatalogVertexInputFormat.HCatalogVertexReader	createVertexReader() create vertex reader instance.
VertexReader<I,V,E>	createVertexReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context) Create a vertex reader for a given split.
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context, int minSplitCountHint) Get the list of input splits for the format.

Methods inherited from class org.apache.giraph.io.GiraphInputFormat

checkInputSpecs, readInputSplit, writeInputSplit

Methods inherited from class org.apache.giraph.conf.DefaultImmutableClassesGiraphConfigurable

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HCatalogVertexInputFormat

public HCatalogVertexInputFormat()

Method Detail

getSplits

public final List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context,
                                                                    int minSplitCountHint)
                                                             throws IOException,
                                                                    InterruptedException

Description copied from class: GiraphInputFormat

Get the list of input splits for the format.

Specified by:

getSplits in class GiraphInputFormat<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

Parameters:

context - The job context

minSplitCountHint - Minimum number of splits to create (hint)

Returns:

The list of input splits

Throws:

IOException

InterruptedException

createVertexReader

protected abstract HCatalogVertexInputFormat.HCatalogVertexReader createVertexReader()

create vertex reader instance.

Returns:

HCatalogVertexReader

createVertexReader

public final VertexReader<I,V,E> createVertexReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                    org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                             throws IOException

Description copied from class: VertexInputFormat

Create a vertex reader for a given split. Guaranteed to have been configured with setConf() prior to use. The framework will also call VertexReader.initialize(InputSplit, TaskAttemptContext) before the split is used.

Specified by:

createVertexReader in class VertexInputFormat<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

Parameters:

split - the split to be read

context - the information about the task

Returns:

a new record reader

Throws:

IOException