GRID ASCII HELPER PROTOCOL for Elastic Compute Cloud (EC2) VERSION 1.0 Todd L Miller [tlmiller@cs.wisc.edu] Febuary 2014 HTCondor Project [http://research.cs.wisc.edu/htcondor] Department of Computer Sciences University of Wisconsin-Madison 1210 W. Dayton Street Madison, WI 53706 [http://www.cs.wisc.edu] 1. INTRODUCTION The object of the Grid ASCII Helper Protocol (GAHP) is to allow the use of the client library or package of a grid or cloud service via a simple ASCII-based protocol. A process which implements GAHP is referred to as a GAHP server. GAHP is designed to handle both synchronous (blocking) and asynchronous (non-blocking) calls. The first GAHP specification focused on the GRAM and GASS grid services of the Globus Toolkit. This GAHP specification focuses on the Google Compute Engine (GCE) cloud service. 1.1 WHY A GAHP? Although most grid and cloud services provide client libraries or modules in various languages that can be incorporated directly into applications that wish to use those services, there are several distinct advantages to using an independent GAHP server process instead. For instance, parts of the native API may provide only synchronous/blocking calls. Users who require a non-blocking/asynchronous interface must make use of multiple threads. Even if the native module is thread-safe, thread safety requires that _all_ modules linked into the process be both re-entrant and in agreement upon a threading model. This may be a significant barrier when trying to integrate the service client module into legacy non-thread-safe code, or when attempting to link with commercial libraries which either have no support for threads or define their own threading model. But because the GAHP server runs as a separate process, it can be easily implemented as a multi-threaded server, and still present an asynchronous non-blocking protocol. Worse yet, there may not be a client module available in the language of the application wishing to use the service. With a GAHP server, langauage choice is immaterial. GAHP facilitates the construction of multi-tier systems. A first tier client can easily send ASCII commands via a socket (perhaps securely via an SSH or SSL tunnel) to a second tier running a GAHP server, allowing grid or cloud services to be consolidated at the second or third tier with minimal effort. Furthermore, GAHP, like many other simple ASCII-based protocols, supports the concept of component-based development independent of the software language used with minimal complexity. When a grid service has client modules available in multiple languages, those interfaces can look very different from each other. By using GAHP, a body of software could be written once with one interface and then subsequently utilize a GAHP server written in C, Java, or Perl -- and said GAHP server could be running locally or as a daemon on a remote host. 1.2 AVAILABLE EC2 GAHP SERVERS The HTCondor Project at the University of Wisconsin-Madison has developed an EC2 GAHP server, written in C++, which uses pthreads and libcurl to support the EC2 protocol. Many Unix platforms are supported. This GAHP is available as part of the HTCondor software distribution, version 7.7.0 or later. 2.0 GAHP SERVER IMPLEMENTATION GAHP itself, as a protocol, is independent of the underlying transport protocol and requires only a reliable ordered bi-directional data stream channel. A GAHP server, however, is assumed to read and respond to GAHP commands solely via stdin and stdout. Should stdin to a GAHP server be closed, the GAHP server should immediately shutdown in a manner similar to the receipt of a QUIT command. Therefore, a GAHP server can be easily invoked and managed via SSHD, inted, or rshd. Software can spawn a local GAHP server via an interface such as POSIX.2 popen(). Under no circumstances should a GAHP server block when issued any command. All commands require a response nearly instantaneously. Therefore, most GAHP servers will be implemented as a multi-threaded process. Use of child processes or avoidance of blocking calls within the GAHP server are also options. 3.0 GAHP COMMANDS The following commands must be implemented by all GAHP servers: COMMANDS QUIT RESULTS VERSION The following commands may be implemented by any GAHP server: ASYNC_MODE_ON ASYNC_MODE_OFF RESPONSE_PREFIX The following commands are specific to EC2 and must be implemented by an EC2 GAHP server: EC2_VM_START EC2_VM_STOP EC2_VM_STATUS_ALL EC2_VM_CREATE_KEYPAIR EC2_VM_DESTROY_KEYPAIR EC2_VM_ASSOCIATE_ADDRESS EC2_VM_ATTACH_VOLUME EC2_VM_CREATE_TAGS EC2_VM_SERVER_TYPE EC2_VM_START_SPOT EC2_VM_STOP_SPOT EC2_VM_STATUS_SPOT EC2_VM_STATUS_ALL_SPOT 3.1 CONVENTIONS AND TERMS USED IN SECTION 3.2 The following definitions are common to all GAHP protocol documents. The characters carriage return and line feed (in that order), _or_ solely the line feed character. The space character. line A sequence of ASCII characters ending with a . Request Line A request for action on the part of the GAHP server. Return Line A line immediately returned by the GAHP server upon receiving a Request Line. Result Line A line sent by the GAHP server in response to a RESULTS request, which communicates the results of a previous asynchronous command Request. S: and R: In the Example sections for the commands below, the prefix "S: " is used to signify what the client sends to the GAHP server. The prefix "R: " is used to signify what the client receives from the GAHP server. Note that the "S: " or "R: " should not actually be sent or received. 3.2 GAHP COMMAND STRUCTURE GAHP commands consist of three parts: * Request Line * Return Line * Result Line Each of these "Lines" consists of a variable length character string ending with the character sequence . A Request Line is a request from the client for action on the part of the GAHP server. Each Request Line consists of a command code followed by argument field(s). Command codes are a string of alphanumeric and underscore characters. Upper and lower case alphabetic characters are to be treated identically with respect to command codes. Thus, any of the following may represent the ASYNC_MODE_ON command: async_mode_on Async_Mode_On asYNc_MOde_oN ASYNC_MODE_ON In contrast, the argument fields of a Request Line are _case sensitive_. The Return Line is always generated by the server as an immediate response to a Request Line. The first character of a Return Line will contain one of the following characters: S - for Success F - for Failure E - for a syntax or parse Error Any Request Line which contains an unrecognized or unsupported command, or a command with an insufficient number of arguments, will generate an "E" response. The Result Line is used to support commands that would otherwise block. Any GAHP command which may require the implementation to block on network communication require a "request id" as part of the Request Line. For such commands, the Result Line just communicates if the request has been successfully parsed and queued for service by the GAHP server. At this point, the GAHP server would typically dispatch a new thread to actually service the request. Once the request has completed, the dispatched thread should create a Result Line and enqueue it until the client issues a RESULT command. 3.3 TRANSPARENCY Arguments on a particular Line (be it Request, Return, or Result) are typically separated by a . In the event that a string argument needs to contain a within the string itself, it must be escaped by placing a backslash ("\") in front of the character. Thus, the character sequence "\ " (no quotes) must not be treated as a separator between arguments, but instead as a space character within a string argument. If a string argument contains a backslash character, it must be escaped by preceding it with another backslash character. 3.4 SEQUENCE OF EVENTS Upon startup, the GAHP server should output to stdout a banner string which is identical to the output from the VERSION command without the beginning "S " sequence (see example below). Next, the GAHP server should wait for a complete Request Line from the client (from stdin). The server is to take no action until a Request Line sequence is received. There are no sequencing restrictions on the ordering of Requests in the GAHP API, even though some sequences may be semantically invalid for the underlying service. For example, attempting to stop a VM before starting it. The server shall not produce an "E" or "F" Return Line in the event of such sequences, but may produce a Result Line reflecting an error from the underlying service. 3.5 COMMAND SYNTAX This section contains the syntax for the Request, Return, and Result line for each command. The commands common to all GAHPs are defined first, followed by commands specific to the EC2 GAHP procotol. 3.5.1 COMMON GAHP COMMANDS These commands are common to all GAHP types. ----------------------------------------------- COMMANDS List all the commands from this protocol specification which are implemented by this GAHP server. + Request Line: COMMANDS + Return Line: S ... + Result Line: None. ----------------------------------------------- VERSION Return the version string for this GAHP. The version string follows a specified format (see below). Ideally, the version entire version string, including the starting and ending dollar sign ($) delimiters, should be a literal string in the text of the GAHP server executable. This way, the Unix/RCS "ident" command can produce the version string. The version returned should correspond to the version of the protocol supported. + Request Line: VERSION + Return Line: S $GahpVesion: .. $ * major.minor.subminor = for this version of the protocol, use version x.y.z. * build-month = string with the month abbreviation when this GAHP server was built or released. Permitted values are: "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", and "Dec". * build-day-of-month = day of the month when GAHP server was built or released; an integer between 1 and 31 inclusive. * build-year = four digit integer specifying the year in which the GAHP server was built or released. * general-descrip = a string identifying a particular GAHP server implementation. + Result Line: None. + Example: S: VERSION R: S $GahpVersion: 1.0.0 Nov 26 2001 NCSA\ CoG\ Gahpd $ ----------------------------------------------- QUIT Free any/all system resources (close all sockets, etc) and terminate as quickly as possible. + Request Line: QUIT + Return Line: S Immediately afterwards, the command pipe should be closed and the GAHP server should terminate. + Result Line: None. ----------------------------------------------- RESULTS Display all of the Result Lines which have been queued since the last RESULTS command was issued. Upon success, the first return line specifies the number of subsequent Result Lines which will be displayed. Then each result line appears (one per line) -- each starts with the request ID which corresponds to the request ID supplied when the corresponding command was submitted. The exact format of the Result Line varies based upon which corresponding Request command was issued. IMPORTANT: Result Lines must be displayed in the _exact order_ in which they were queued!!! In other words, the Result Lines displayed must be sorted in the order by which they were placed into the GAHP's result line queue, from earliest to most recent. + Request Line: RESULTS + Return Line(s): S ... ... ... * reqid = integer Request ID, set to the value specified in the corresponding Request Line. + Result Line: None. + Example: S: RESULTS R: S 1 R: 100 0 ----------------------------------------------- ASYNC_MODE_ON Enable Asynchronous notification when the GAHP server has results pending for a client. This is most useful for clients that do not want to periodically poll the GAHP server with a RESULTS command. When asynchronous notification mode is active, the GAHP server will print out an 'R' (without the quotes) on column one when the 'RESULTS' command would return one or more lines. The 'R' is printed only once between successive 'RESULTS' commands. The 'R' is also guaranteed to only appear in between atomic return lines; the 'R" will not interrupt another command's output. If there are already pending results when the asynchronous results available mode is activated, no indication of the presence of those results will be given. A GAHP server is permitted to only consider changes to it's result queue for additions after the ASYNC_MODE_ON command has successfully completed. GAHP clients should issue a 'RESULTS' command immediately after enabling asynchronous notification, to ensure that any results that may have been added to the queue during the processing of the ASYNC_MODE_ON command are accounted for. + Request Line: ASYNC_MODE_ON + Return Line: S Immediately afterwards, the client should be prepared to handle an R appearing in the output of the GAHP server. + Result Line: None. + Example: S: ASYNC_MODE_ON R: S S: GRAM_PING 00001 beak.cs.wisc.edu/jobmanager R: S S: GRAM_PING 00002 nostos.cs.wisc.edu/jobmanager R: S R: R S: RESULTS R: S 2 R: 00001 0 R: 00002 0 Note that you are NOT guaranteed that the 'R' will not appear between the dispatching of a command and the return line(s) of that command; the GAHP server only guarantees that the 'R' will not interrupt an in-progress return. The following is also a legal example: S: ASYNC_MODE_ON R: S S: GRAM_PING 00001 beak.cs.wisc.edu/jobmanager R: S S: GRAM_PING 00002 nostos.cs.wisc.edu/jobmanager R: R R: S S: RESULTS R: S 2 R: 00001 0 R: 00002 0 (Note the reversal of the R and the S after GRAM_PING 00002) ----------------------------------------------- ASYNC_MODE_OFF Disable asynchronous results-available notification. In this mode, the only way to discover available results is to poll with the RESULTS command. This mode is the default. Asynchronous mode can be enable with the ASYNC_MODE_ON command. + Request Line: ASYNC_MODE_OFF + Return Line: S + Result Line: None + Example: S: ASYNC_MODE_OFF R: S ----------------------------------------------- RESPONSE_PREFIX Specify a prefix that the GAHP server should use to prepend every subsequent line of output. This may simplify parsing the output of the GAHP server by the client program, especially in cases where the responses of more one GAHP server are "collated" together. This affects the output of both return lines and result lines for all subsequent commands (NOT including this one). + Request Line: RESPONSE_PREFIX = an arbitrary string of characters which you want to prefix every subsequent line printed by the GAHP server. + Return Line: S + Result Line: None. + Example: S: RESPONSE_PREFIX GAHP: R: S S: RESULTS R: GAHP:S 0 S: RESPONSE_PREFIX NEW_PREFIX_ R: GAHP:S S: RESULTS R: NEW_PREFIX_S 0 ----------------------------------------------- 3.5.2 GAHP COMMANDS SPECIFIC TO EC2 The following commands are specific to the EC2 GAHP. All commands will include the following arguments. * request-ID = a non-zero Request ID * service-URL = the URL used to access the EC2 service's RESTful API * accessKeyFile = the full path to a file containing nothing but the EC2 access key (and an optional trailing newline) * secretKeyFile = the full path to a file containing nothing but the EC2 secret key (and an optional trailing newline) We do not directly specify keys to minimize the chance of interception. Required values must not be the literal string "NULL" (without the quotes), as this denotes that the corresponding attribute was not set. If an optional value is not set, it _must_ be represented in this way. The following is the Return Line for all EC2 GAHP commands unless otherwise specified. + Return Line: * result = the character "S" (without the quotes) for the successful submission of the request (meaning the request is now pending), or an "E" (without the quotes) for an error parsing the request or its arguments (e.g. an unrecognized or unsupported command, or missing or malformed arguments) The following is the Result Line for all EC2 GAHP commands unless otherwise specified. + Result Line: One of the following, where 0 is success and 1 is failure. 1 0 * error-code = a fixed string denoting the type of error * error-string = a variable string with more detail The definitions for error-code and error-string are identical for every Result Line unless otherwise stated. ----------------------------------------------- EC2_VM_START Start an instance of the specified VM. + Request Line: EC2_VM_START [ ]+ The following attributes are required. * image-ID = an existing AMI ID (or equivalent) The following are optional. * ssh-keypair-name = an existing SSH keypair name * user-data-string = a string supplied by the user, in which spaces must be escaped (see above) * user-data-file = the full path to a file containing nothing but the (unencoded) user data. Appended to user-data-string, if both are set. * instance-type = the instance type * availability-zone = the availability zone * subnet-ID = the subnet ID * private-ip-address = the private IP address of the instance * client-token = the client token The following attribute(s) may be omitted. * security-group = a security group. Security groups are passed to the service unmodified. The client may specify an arbitrary number of security group strings, including zero. (Services may limit the number of security groups for an instance, but the EC2 GAHP does not.) + Result Line: Either of the following, where 0 is success and 1 is failure. 0 1 * request-ID = from the corresponding Request Line * instance-ID = the ID of the started instance ----------------------------------------------- EC2_VM_STOP Stop ("terminate") the specified VM instance. + Request Line: EC2_VM_STOP The following attributes are required. * instance-ID = an instance ID (such as may have been returned by EC2_VM_START, although this is not required) ----------------------------------------------- EC2_VM_STATUS_ALL Discover the status of all instances the given credentials can observe. This may include instances not started by the GAHP; the client must maintain its own list of instances if it cares. This does _not_ include spot instances; see EC2_VM_STATUS_SPOT and EC2_VM_STATUS_ALL_SPOT. + Request Line: EC2_VM_STATUS_ALL + Result Line: One of the following, where 0 is success and 1 is failure. 1 0 [ [] [] [] [] ]* * request-ID = from the corresponding Request Line A success returns 0 or more 6-tuples, the last four values of which may be the literal string "NULL" (without the quotes) to denote that the corresponding attribute was not set. The six attributes are: * instance-ID = an instance ID * status = the status of the preceding instance * client-token = the client token, if any * ssh-keypair-name = the SSH keypair name, if any * state-reason-code = the state reason code, if any * public-dns-name = the public DNS name, if any ----------------------------------------------- EC2_VM_CREATE_KEYPAIR Generate and register an SSH keypair on the service. + Request Line: EC2_VM_CREATE_KEYPAIR * ssh-keypair-name = register the generated keypair with this name * private-key-file = the full path to the file to which to write the private key half of the generated keypair ----------------------------------------------- EC2_VM_DESTROY_KEYPAIR Remove the named SSH keypair from the service. + Request Line: EC2_VM_CREATE_KEYPAIR * ssh-keypair-name = delete the keypair named this ----------------------------------------------- EC2_VM_ASSOCIATE_ADDRESS Associate the given Elastic IP with the given instance. Accepts both EC2 and VPC -style Elastic IP specifiers. + Request Line: EC2_VM_CREATE_KEYPAIR * instance-ID = an instance ID * elastic-IP = an Elastic IP specifier ----------------------------------------------- EC2_VM_ATTACH_VOLUME Attach an EBS volume to an instance. + Request Line: EC2_VM_CREATE_KEYPAIR * volume-ID = an EBS volume ID * instance-ID = an instance ID * device = which device to attach the volume as ----------------------------------------------- EC2_VM_CREATE_TAGS Add the specified tags to the given resource. + Request Line: EC2_VM_CREATE_KEYPAIR [ = ]+ CRLF * resource-ID = a resource ID (usually an instance ID) The following two attributes must be specified as a pair separated by a literal "=" (no quotes) but no spaces; escaped spaces will be treated as part of the tag-name or tag-value, depending on which side of the "=" they're on. At least one pair must be present. The EC2 GAHP does not specify a maximum number of pairs. * tag-name = the name of the tag to add * tag-value = the value of the tag to add ----------------------------------------------- EC2_VM_SERVER_TYPE Determine the type of the EC2 service offered at the given URL. + Request Line: EC2_VM_CREATE_KEYPAIR + Result Line: One of the following, where 0 is success and 1 is failure. 1 0 * server-type = one of "Unknown", "Amazon", "OpenStack", "Nimbus", or "Eucalyptus" (without the quotes) ----------------------------------------------- EC2_VM_START_SPOT Request a Spot instance. + Request Line: EC2_VM_START_SPOT [ The following attributes are required. * image-ID = an existing AMI ID (or equivalent) * spot-price = a string specifying the bid ("0.0022") The remainder of this command's specification is identical to EC2_VM_START. + Result Line: Either of the following, where 0 is success and 1 is failure. 0 1 * request-ID = from the corresponding Request Line * spot-instance-ID = the ID of the Spot instance request ----------------------------------------------- EC2_VM_STOP_SPOT Stop ("cancel") a spot request. + Request Line: EC2_VM_STOP_SPOT ----------------------------------------------- EC2_VM_STATUS_SPOT Determine the status of the given Spot instance request. This function is _deprecated_, and should not be used; EC2_VM_STATUS_SPOT_ALL returns the same information, but more efficiently. + Request Line: EC2_VM_STATUS_SPOT + Result Line: Any of the following, where 0 is success and 1 is failure. 0 0 [] [] 1 The shorter succes line indicates that no spot instance request with the given ID was found (but the search completed succesfully). * request-ID = from the corresponding Request Line * spot-instance-ID = the ID of the Spot instance request * state = the state of the Spot instance request * client-token = the client token supplied in the request The following two attributes may be the literal string "NULL" (without the quotes) to denote that the corresponding attribute was not set: * instance-ID = ID of the instance spawned by this request * status-code = the status code of the request ----------------------------------------------- EC2_VM_STATUS_ALL_SPOT Obtain status information about all the spot instance requests visible to the given credentials. + Request Line: EC2_VM_STATUS_ALL_SPOT + Result Line: Either of the following, where 0 is success and 1 is failure. 0 [ [] []]* 1 * request-ID = from the corresponding Request Line A success returns 0 or more 5-tuples, the last two values of which may be the literal string "NULL" (without the quotes) to denote that the corresponding attribute was not set. The five attributes are: * spot-instance-ID = the ID of the Spot instance request * state = the state of the Spot instance request * client-token = the client token supplied in the request * instance-ID = ID of the instance spawned by this request * status-code = the status code of the request