org.apache.giraph.yarn

Class YarnUtils

java.lang.Object

org.apache.giraph.yarn.YarnUtils

public class YarnUtils
extends Object

Utilities that can only compile with versions of Hadoop that support YARN, so they live here instead of o.a.g.utils package.

Method Summary

Modifier and Type	Method and Description
static void	addFileToResourceMap(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path target) Boilerplate to add a file to the local resources..
static void	addFsResourcesToMap(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> map, GiraphConfiguration giraphConf, org.apache.hadoop.yarn.api.records.ApplicationId appId) Populates the LocalResources list with the HDFS paths listed in the conf under GiraphConstants.GIRAPH_YARN_LIBJARS, and the GiraphConfiguration for this job.
static void	addGiraphConfToLocalResourceMap(GiraphConfiguration giraphConf, org.apache.hadoop.yarn.api.records.ApplicationId appId, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResourceMap) Populate the LocalResources list with the GiraphConf XML file's HDFS path.
static void	addLocalClasspathToEnv(Map<String,String> env, GiraphConfiguration giraphConf) Popuate the environment string map to be added to the environment vars in a remote execution container.
static void	exportGiraphConfiguration(GiraphConfiguration giraphConf, org.apache.hadoop.yarn.api.records.ApplicationId appId) Export our populated GiraphConfiguration as an XML file to be used by the ApplicationMaster's exec container, and register it with LocalResources.
static org.apache.hadoop.fs.Path	getFsCachePath(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.yarn.api.records.ApplicationId appId) Get the base HDFS dir we will be storing our LocalResources in.
static Set<org.apache.hadoop.fs.Path>	getLocalFiles(Set<String> fileNames) Utility function to locate local JAR files and other resources recursively in the dirs on the local CLASSPATH.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

addFsResourcesToMap

public static void addFsResourcesToMap(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> map,
                                       GiraphConfiguration giraphConf,
                                       org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                throws IOException

Populates the LocalResources list with the HDFS paths listed in the conf under GiraphConstants.GIRAPH_YARN_LIBJARS, and the GiraphConfiguration for this job. Also adds the Giraph default application jar as determined by GiraphYarnClient.GIRAPH_CLIENT_JAR constant.

Parameters:

map - the LocalResources list to populate.

giraphConf - the configuration to use to select jars to include.

appId - the ApplicationId, naming the the HDFS base dir for job jars.

Throws:

IOException

getLocalFiles

public static Set<org.apache.hadoop.fs.Path> getLocalFiles(Set<String> fileNames)

Utility function to locate local JAR files and other resources recursively in the dirs on the local CLASSPATH. Once all the files named in fileNames are found, we stop and return the results.

Parameters:

fileNames - the file name of the jars, without path information.

Returns:

a set of Paths to the jar files requested in fileNames.

addFileToResourceMap

public static void addFileToResourceMap(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources,
                                        org.apache.hadoop.fs.FileSystem fs,
                                        org.apache.hadoop.fs.Path target)
                                 throws IOException

Boilerplate to add a file to the local resources..

Parameters:

localResources - the LocalResources map to populate.

fs - handle to the HDFS file system.

target - the file to send to the remote container.

Throws:

IOException

getFsCachePath

public static org.apache.hadoop.fs.Path getFsCachePath(org.apache.hadoop.fs.FileSystem fs,
                                                       org.apache.hadoop.yarn.api.records.ApplicationId appId)

Get the base HDFS dir we will be storing our LocalResources in.

Parameters:

fs - the file system.

appId - the ApplicationId under which our resources will be stored.

Returns:

the path

addLocalClasspathToEnv

public static void addLocalClasspathToEnv(Map<String,String> env,
                                          GiraphConfiguration giraphConf)

Popuate the environment string map to be added to the environment vars in a remote execution container. Adds the local classpath to pick up "yarn-site.xml" and "mapred-site.xml" stuff.

Parameters:

env - the map of env var values.

giraphConf - the GiraphConfiguration to pull values from.

addGiraphConfToLocalResourceMap

public static void addGiraphConfToLocalResourceMap(GiraphConfiguration giraphConf,
                                                   org.apache.hadoop.yarn.api.records.ApplicationId appId,
                                                   Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResourceMap)
                                            throws IOException

Populate the LocalResources list with the GiraphConf XML file's HDFS path.

Parameters:

giraphConf - the GiraphConfifuration to export for worker tasks.

appId - the ApplicationId for this YARN app.

localResourceMap - the LocalResource map of files to export to tasks.

Throws:

IOException

exportGiraphConfiguration

public static void exportGiraphConfiguration(GiraphConfiguration giraphConf,
                                             org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                      throws IOException

Export our populated GiraphConfiguration as an XML file to be used by the ApplicationMaster's exec container, and register it with LocalResources.

Parameters:

giraphConf - the current Configuration object to be published.

appId - the ApplicationId to stamp this app's base HDFS resources dir.

Throws:

IOException