org.apache.giraph.comm.netty

Class NettyClient

java.lang.Object

org.apache.giraph.comm.netty.NettyClient

public class NettyClient
extends Object

Netty client for sending requests. Thread-safe.

Field Summary

Fields

Modifier and Type	Field and Description
static BooleanConfOption	LIMIT_NUMBER_OF_OPEN_REQUESTS Do we have a limit on number of open requests we can have
static BooleanConfOption	LIMIT_OPEN_REQUESTS_PER_WORKER Do we have a limit on number of open requests we can have for each worker.
static int	MAX_CONNECTION_MILLISECONDS_DEFAULT 30 seconds to connect by default
static int	MAX_DESTINATION_TASK_IDS_TO_LIST Maximum number of destination task ids with open requests to list (for debugging)
static int	MAX_REQUESTS_TO_LIST Maximum number of requests to list (for debugging)
static String	NETTY_COUNTERS_GROUP Group name for netty counters
static String	NETWORK_REQUESTS_RESENT_FOR_CHANNEL_FAILURE_NAME How many network requests were resent because channel failed
static String	NETWORK_REQUESTS_RESENT_FOR_CONNECTION_FAILURE_NAME How many network requests were resent because connection failed
static String	NETWORK_REQUESTS_RESENT_FOR_TIMEOUT_NAME How many network requests were resent because they took too long
static io.netty.util.AttributeKey<SaslNettyClient>	SASL Used to authenticate with other workers acting as servers

Constructor Summary

Constructors

Constructor and Description
NettyClient(org.apache.hadoop.mapreduce.Mapper.Context context, ImmutableClassesGiraphConfiguration conf, TaskInfo myTaskInfo, Thread.UncaughtExceptionHandler exceptionHandler) Only constructor

Method Summary

Modifier and Type	Method and Description
void	authenticate() Authenticate all servers in addressChannelMap.
void	connectAllAddresses(Collection<? extends TaskInfo> tasks) Connect to a collection of tasks servers
Long	doSend(int destTaskId, WritableRequest request) Actual send of a request.
static Map<String,Set<String>>	getCounterGroupsAndNames()
FlowControl	getFlowControl()
Long	getNextRequestId(int taskId) Generate and get the next request id to be used for a given worker
int	getNumberOfOpenRequests()
void	logAndSanityCheck() Log information about the requests and check for problems in requests
boolean	masterInvolved(int clientId) Whether master task is involved in the communication with a given client
void	messageReceived(int senderId, long requestId, int response, boolean shouldDrop) Handle receipt of a message.
void	sendWritableRequest(int destTaskId, WritableRequest request) Send a request to a remote server honoring the flow control mechanism (should be already connected)
void	stop() Stop the client.
void	waitAllRequests() Ensure all the request sent so far are complete.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LIMIT_NUMBER_OF_OPEN_REQUESTS

public static final BooleanConfOption LIMIT_NUMBER_OF_OPEN_REQUESTS

Do we have a limit on number of open requests we can have

LIMIT_OPEN_REQUESTS_PER_WORKER

public static final BooleanConfOption LIMIT_OPEN_REQUESTS_PER_WORKER

Do we have a limit on number of open requests we can have for each worker. Note that if this option is enabled, Netty will not keep more than a certain number of requests open for each other worker in the job. If there are more requests generated for a worker, Netty will not actually send the surplus requests, instead, it caches the requests in a local buffer. The maximum number of these unsent requests in the cache is another user-defined parameter (MAX_NUM_OF_OPEN_REQUESTS_PER_WORKER).

MAX_REQUESTS_TO_LIST

public static final int MAX_REQUESTS_TO_LIST

Maximum number of requests to list (for debugging)

See Also:

Constant Field Values

MAX_DESTINATION_TASK_IDS_TO_LIST

public static final int MAX_DESTINATION_TASK_IDS_TO_LIST

Maximum number of destination task ids with open requests to list (for debugging)

See Also:

Constant Field Values

MAX_CONNECTION_MILLISECONDS_DEFAULT

public static final int MAX_CONNECTION_MILLISECONDS_DEFAULT

30 seconds to connect by default

See Also:

Constant Field Values

SASL

public static final io.netty.util.AttributeKey<SaslNettyClient> SASL

Used to authenticate with other workers acting as servers

NETTY_COUNTERS_GROUP

public static final String NETTY_COUNTERS_GROUP

Group name for netty counters

See Also:

Constant Field Values

NETWORK_REQUESTS_RESENT_FOR_TIMEOUT_NAME

public static final String NETWORK_REQUESTS_RESENT_FOR_TIMEOUT_NAME

How many network requests were resent because they took too long

See Also:

Constant Field Values

NETWORK_REQUESTS_RESENT_FOR_CHANNEL_FAILURE_NAME

public static final String NETWORK_REQUESTS_RESENT_FOR_CHANNEL_FAILURE_NAME

How many network requests were resent because channel failed

See Also:

Constant Field Values

NETWORK_REQUESTS_RESENT_FOR_CONNECTION_FAILURE_NAME

public static final String NETWORK_REQUESTS_RESENT_FOR_CONNECTION_FAILURE_NAME

How many network requests were resent because connection failed

See Also:

Constant Field Values

Constructor Detail

NettyClient

public NettyClient(org.apache.hadoop.mapreduce.Mapper.Context context,
                   ImmutableClassesGiraphConfiguration conf,
                   TaskInfo myTaskInfo,
                   Thread.UncaughtExceptionHandler exceptionHandler)

Only constructor

Parameters:

context - Context for progress

conf - Configuration

myTaskInfo - Current task info

exceptionHandler - handler for uncaught exception. Will terminate job.

Method Detail

getCounterGroupsAndNames

public static Map<String,Set<String>> getCounterGroupsAndNames()

masterInvolved

public boolean masterInvolved(int clientId)

Whether master task is involved in the communication with a given client

Parameters:

clientId - id of the communication (on the end of the communication)

Returns:

true if master is on one end of the communication

connectAllAddresses

public void connectAllAddresses(Collection<? extends TaskInfo> tasks)

Connect to a collection of tasks servers

Parameters:

tasks - Tasks to connect to (if haven't already connected)

authenticate

public void authenticate()

Authenticate all servers in addressChannelMap.

stop

public void stop()

Stop the client.

sendWritableRequest

public void sendWritableRequest(int destTaskId,
                                WritableRequest request)

Send a request to a remote server honoring the flow control mechanism (should be already connected)

Parameters:

destTaskId - Destination task id

request - Request to send

doSend

public Long doSend(int destTaskId,
                   WritableRequest request)

Actual send of a request.

Parameters:

destTaskId - destination to send the request to

request - request itself

Returns:

request id generated for sending the request

messageReceived

public void messageReceived(int senderId,
                            long requestId,
                            int response,
                            boolean shouldDrop)

Handle receipt of a message. Called by response handler.

Parameters:

senderId - Id of sender of the message

requestId - Id of the request

response - Actual response

shouldDrop - Drop the message?

waitAllRequests

public void waitAllRequests()

Ensure all the request sent so far are complete. Periodically check the state of current open requests. If there is an issue in any of them, re-send the request.

logAndSanityCheck

public void logAndSanityCheck()

Log information about the requests and check for problems in requests

getFlowControl

public FlowControl getFlowControl()

getNextRequestId

public Long getNextRequestId(int taskId)

Generate and get the next request id to be used for a given worker

Parameters:

taskId - id of the worker to generate the next request id

Returns:

request id

getNumberOfOpenRequests

public int getNumberOfOpenRequests()

Returns:

number of open requests