org.apache.giraph.io.hcatalog

Class HCatalogVertexOutputFormat<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

java.lang.Object

org.apache.giraph.conf.DefaultImmutableClassesGiraphConfigurable<I,V,E>

org.apache.giraph.io.VertexOutputFormat<I,V,E>

org.apache.giraph.io.hcatalog.HCatalogVertexOutputFormat<I,V,E>

Type Parameters:

I - Vertex id

V - Vertex value

E - Edge value

All Implemented Interfaces:

GiraphConfigurationSettable<I,V,E>, ImmutableClassesGiraphConfigurable<I,V,E>

public abstract class HCatalogVertexOutputFormat<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>
extends VertexOutputFormat<I,V,E>

Abstract class that users should subclass to store data to Hive or Pig table. You can easily implement a HCatalogVertexOutputFormat.HCatalogVertexWriter by extending HCatalogVertexOutputFormat.SingleRowHCatalogVertexWriter or HCatalogVertexOutputFormat.MultiRowHCatalogVertexWriter depending on how you want to fit your vertices into the output table.

The desired database and table name to store to can be specified via HCatOutputFormat.setOutput(org.apache.hadoop.mapreduce.Job, org.apache.hcatalog.mapreduce.OutputJobInfo) as you setup your vertex output format with GiraphConfiguration setVertexOutputFormatClass(Class)}. You must create the output table.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	HCatalogVertexOutputFormat.HCatalogVertexWriter<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> Abstract class that users should subclass based on their specific vertex output.
static class	HCatalogVertexOutputFormat.MultiRowHCatalogVertexWriter<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> HCatalogVertexWriter to write each vertex in multiple rows.
protected static class	HCatalogVertexOutputFormat.SingleRowHCatalogVertexWriter<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable> HCatalogVertexWriter to write each vertex in each row.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hcatalog.mapreduce.HCatOutputFormat	hCatOutputFormat hcat output format

Constructor Summary

Constructors

Constructor and Description
HCatalogVertexOutputFormat()

Method Summary

Modifier and Type	Method and Description
void	checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext context) Check for validity of the output-specification for the job.
protected abstract HCatalogVertexOutputFormat.HCatalogVertexWriter<I,V,E>	createVertexWriter() create vertex writer.
VertexWriter<I,V,E>	createVertexWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context) Create a vertex writer for a given split.
org.apache.hadoop.mapreduce.OutputCommitter	getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext context) Get the output committer for this output format.

Methods inherited from class org.apache.giraph.conf.DefaultImmutableClassesGiraphConfigurable

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

hCatOutputFormat

protected org.apache.hcatalog.mapreduce.HCatOutputFormat hCatOutputFormat

hcat output format

Constructor Detail

HCatalogVertexOutputFormat

public HCatalogVertexOutputFormat()

Method Detail

checkOutputSpecs

public final void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext context)
                            throws IOException,
                                   InterruptedException

Description copied from class: VertexOutputFormat

Check for validity of the output-specification for the job. (Copied from Hadoop OutputFormat)

This is to validate the output specification for the job when it is a job is submitted. Typically checks that it does not already exist, throwing an exception when it already exists, so that output is not overwritten.

Specified by:

checkOutputSpecs in class VertexOutputFormat<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

Parameters:

context - information about the job

Throws:

IOException - when output should not be attempted

InterruptedException

getOutputCommitter

public final org.apache.hadoop.mapreduce.OutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                     throws IOException,
                                                                            InterruptedException

Description copied from class: VertexOutputFormat

Get the output committer for this output format. This is responsible for ensuring the output is committed correctly. (Copied from Hadoop OutputFormat)

Specified by:

getOutputCommitter in class VertexOutputFormat<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

Parameters:

context - the task context

Returns:

an output committer

Throws:

IOException

InterruptedException

createVertexWriter

protected abstract HCatalogVertexOutputFormat.HCatalogVertexWriter<I,V,E> createVertexWriter()

create vertex writer.

Returns:

HCatalogVertexWriter

createVertexWriter

public final VertexWriter<I,V,E> createVertexWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                             throws IOException,
                                                    InterruptedException

Description copied from class: VertexOutputFormat

Create a vertex writer for a given split. The framework will call VertexWriter.initialize(TaskAttemptContext) before the split is used.

Specified by:

createVertexWriter in class VertexOutputFormat<I extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable,E extends org.apache.hadoop.io.Writable>

Parameters:

context - the information about the task

Returns:

a new vertex writer

Throws:

IOException

InterruptedException