example/service/s3: Add presign URL example (aws#1260)

Creates an example for using S3 presign URL with a service vending presigned URLs to a client and the client using those URLs to build requests for the S3 content.

example/service/s3/presignURL/README.md

@@ -0,0 +1,124 @@
+# Presigned Amazon S3 API Operation Example
+
+This example demonstrates how you can build a client application to retrieve and
+upload object data from Amazon S3 without needing to know anything about Amazon
+S3 or have access to any AWS credentials. Only the service would have knowledge
+of how and where the objects are stored in Amazon S3.
+
+The example is split into two parts `server.go` and `client.go`. These two parts
+simulate the client/server architecture. In this example the client will represent
+a third part user that will request resource URLs from the service. The service
+will generate presigned S3 URLs which the client can use to download and
+upload S3 object content.
+
+The service supports generating presigned URLs for two S3 APIs; `GetObject` and
+`PutObject`. The client will request a presigned URL from the service with an
+object Key. In this example the value is the S3 object's `key`. Alternatively,
+you could use your own pattern with no visible relation to the S3 object's key.
+The server would then perform a cross reference with client provided value to
+one that maps to the S3 object's key.
+
+Before using the client to upload and download S3 objects you'll need to start the
+service. The service will use the SDK's default credential chain to source your
+AWS credentials. See the [`Configuring Credentials`](http://docs.aws.amazon.com/sdk-for-go/api/)
+section of the SDK's API Reference guide on how the SDK loads your AWS credentials.
+
+The server requires the S3 `-b bucket` the presigned URLs will be generated for. A
+`-r region` is only needed if the bucket is in AWS China or AWS Gov Cloud. For 
+buckets in AWS the server will use the [`s3manager.GetBucketRegion`](http://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#GetBucketRegion) utility to lookup the bucket's region.
+
+You should run the service in the background or in a separate terminal tab before
+moving onto the client. 
+
+
+```sh
+go run -tags example server/server.go -b mybucket
+> Starting Server On: 127.0.0.1:8080
+```
+
+Use the `--help` flag to see a list of additional configuration flags, and their
+defaults.
+
+## Downloading an Amazon S3 Object
+
+Use the client application to request a presigned URL from the server and use
+that presigned URL to download the object from S3. Calling the client with the
+`-get key` flag will do this. An optional `-f filename` flag can be provided as 
+well to write the object to. If no flag is provided the object will be written
+to `stdout`
+
+```sh
+go run -tags example client/client.go -get "my-object/key" -f outputfilename
+```
+
+Use the `--help` flag to see a list of additional configuration flags, and their
+defaults.
+
+The following curl request demonstrates the request the client makes to the server
+for the presigned URL for the `my-object/key` S3 object. The `method` query
+parameter lets the server know that we are requesting the `GetObject`'s presigned
+URL. The `method` value can be `GET` or `PUT` for the `GetObject` or `PutObject` APIs.
+
+```sh
+curl -v "http://127.0.0.1:8080/presign/my-object/key?method=GET"
+```
+
+The server will respond with a JSON value. The value contains three pieces of 
+information that the client will need to correctly make the request. First is
+the presigned URL. This is the URL the client will make the request to. Second
+is the HTTP method the request should be sent as. This is included to simplify
+the client's request building. Finally the response will include a list of
+additional headers that the client should include that the presigned request
+was signed with.
+
+```json
+{
+	"URL": "https://mybucket.s3-us-west-2.amazonaws.com/my-object/key?<signature>",
+	"Method": "GET",
+	"Header": {
+		"x-amz-content-sha256":["UNSIGNED-PAYLOAD"]
+	}
+}
+```
+
+With this URL our client will build a HTTP request for the S3 object's data. The
+`client.go` will then write the object's data to the `filename` if one is provided,
+or to `stdout` of a filename is not set in the command line arguments.
+
+## Uploading a File to Amazon S3
+
+Just like the download, uploading a file to S3 will use a presigned URL requested
+from the server. The resigned URL will be built into an HTTP request using the
+URL, Method, and Headers. The `-put key` flag will upload the content of `-f filename`
+or stdin if no filename is provided to S3 using a presigned URL provided by the
+service
+
+```sh
+go run -tags example client/client.go -put "my-object/key" -f filename
+```
+
+Like the download case this will make a HTTP request to the server for the 
+presigned URL. The Server will respond with a presigned URL for S3's `PutObject`
+API operation. In addition the `method` query parameter the client will also
+include a `contentLength` this value instructs the server to generate the presigned
+PutObject request with a `Content-Length` header value included in the signature.
+This is done so the content that is uploaded by the client can only be the size
+the presigned request was generated for.
+
+```sh
+curl -v "http://127.0.0.1:8080/presign/my-object/key?method=PUT&contentLength=1024"
+```
+
+## Expanding the Example
+
+This example provides a spring board you can use to vend presigned URLs to your
+clients instead of streaming the object's content through your service. This
+client and server example can be expanded and customized. Adding new functionality
+such as additional constraints the server puts on the presigned URLs like
+`Content-Type`.
+
+In addition to adding constraints to the presigned URLs the service could be
+updated to obfuscate S3 object's key. Instead of the client knowing the object's 
+key, a lookup system could be used instead. This could be substitution based,
+or lookup into an external data store such as DynamoDB.
+

example/service/s3/presignURL/client/client.go

@@ -0,0 +1,266 @@
+// +build example
+
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"net/http"
+	"os"
+	"strconv"
+	"strings"
+)
+
+// client.go is an example of a client that will request URLs from a service that
+// the client will use to upload and download content with.
+//
+// The server must be started before the client is run.
+//
+// Use "--help" command line argument flag to see all options and defaults. If
+// filename is not provided the client will read from stdin for uploads and
+// write to stdout for downloads.
+//
+// Usage:
+//    go run -tags example client.go -get myObjectKey -f filename
+func main() {
+	method, filename, key, serverURL := loadConfig()
+
+	var err error
+
+	switch method {
+	case GetMethod:
+		// Requests the URL from the server that the client will use to download
+		// the content from. The content will be written to the file pointed to
+		// by filename. Creating it if the file does not exist. If filename is
+		// not set the contents will be written to stdout.
+		err = downloadFile(serverURL, key, filename)
+	case PutMethod:
+		// Requests the URL from the service that the client will use to upload
+		// content to. The content will be read from the file pointed to by the
+		// filename. If the filename is not set, content will be read from stdin.
+		err = uploadFile(serverURL, key, filename)
+	}
+
+	if err != nil {
+		exitError(err)
+	}
+}
+
+// loadConfig configures the client based on the command line arguments used.
+func loadConfig() (method Method, serverURL, key, filename string) {
+	var getKey, putKey string
+	flag.StringVar(&getKey, "get", "",
+		"Downloads the object from S3 by the `key`. Writes the object to a file the filename is provided, otherwise writes to stdout.")
+	flag.StringVar(&putKey, "put", "",
+		"Uploads data to S3 at the `key` provided. Uploads the file if filename is provided, otherwise reads from stdin.")
+	flag.StringVar(&serverURL, "s", "http://127.0.0.1:8080", "Required `URL` the client will request presigned S3 operation from.")
+	flag.StringVar(&filename, "f", "", "The `filename` of the file to upload and get from S3.")
+	flag.Parse()
+
+	var errs Errors
+
+	if len(serverURL) == 0 {
+		errs = append(errs, fmt.Errorf("server URL required"))
+	}
+
+	if !((len(getKey) != 0) != (len(putKey) != 0)) {
+		errs = append(errs, fmt.Errorf("either `get` or `put` can be provided, and one of the two is required."))
+	}
+
+	if len(getKey) > 0 {
+		method = GetMethod
+		key = getKey
+	} else {
+		method = PutMethod
+		key = putKey
+	}
+
+	if len(errs) > 0 {
+		fmt.Fprintf(os.Stderr, "Failed to load configuration:%v\n", errs)
+		flag.PrintDefaults()
+		os.Exit(1)
+	}
+
+	return method, filename, key, serverURL
+}
+
+// downloadFile will request a URL from the server that the client can download
+// the content pointed to by "key". The content will be written to the file
+// pointed to by filename, creating the file if it doesn't exist. If filename
+// is not set the content will be written to stdout.
+func downloadFile(serverURL, key, filename string) error {
+	var w *os.File
+	if len(filename) > 0 {
+		f, err := os.Create(filename)
+		if err != nil {
+			return fmt.Errorf("failed to create download file %s, %v", filename, err)
+		}
+		w = f
+	} else {
+		w = os.Stdout
+	}
+	defer w.Close()
+
+	// Get the presigned URL from the remote service.
+	req, err := getPresignedRequest(serverURL, "GET", key, 0)
+	if err != nil {
+		return fmt.Errorf("failed to get get presigned request, %v", err)
+	}
+
+	// Gets the file contents with the URL provided by the service.
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return fmt.Errorf("failed to do GET request, %v", err)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("failed to get S3 object, %d:%s",
+			resp.StatusCode, resp.Status)
+	}
+
+	if _, err = io.Copy(w, resp.Body); err != nil {
+		return fmt.Errorf("failed to write S3 object, %v", err)
+	}
+
+	return nil
+}
+
+// uploadFile will request a URL from the service that the client can use to
+// upload content to. The content will be read from the file pointed to by filename.
+// If filename is not set the content will be read from stdin.
+func uploadFile(serverURL, key, filename string) error {
+	var r io.ReadCloser
+	var size int64
+	if len(filename) > 0 {
+		f, err := os.Open(filename)
+		if err != nil {
+			return fmt.Errorf("failed to open upload file %s, %v", filename, err)
+		}
+
+		// Get the size of the file so that the constraint of Content-Length
+		// can be included with the presigned URL. This can be used by the
+		// server or client to ensure the content uploaded is of a certain size.
+		//
+		// These constraints can further be expanded to include things like
+		// Content-Type. Additionally constraints such as X-Amz-Content-Sha256
+		// header set restricting the content of the file to only the content
+		// the client initially made the request with. This prevents the object
+		// from being overwritten or used to upload other unintended content.
+		stat, err := f.Stat()
+		if err != nil {
+			return fmt.Errorf("failed to stat file, %s, %v", filename, err)
+		}
+
+		size = stat.Size()
+		r = f
+	} else {
+		buf := &bytes.Buffer{}
+		io.Copy(buf, os.Stdin)
+		size = int64(buf.Len())
+
+		r = ioutil.NopCloser(buf)
+	}
+	defer r.Close()
+
+	// Get the Presigned URL from the remote service. Pass in the file's
+	// size if it is known so that the presigned URL returned will be required
+	// to be used with the size of content requested.
+	req, err := getPresignedRequest(serverURL, "PUT", key, size)
+	if err != nil {
+		return fmt.Errorf("failed to get get presigned request, %v", err)
+	}
+	req.Body = r
+
+	// Upload the file contents to S3.
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return fmt.Errorf("failed to do GET request, %v", err)
+	}
+
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		return fmt.Errorf("failed to put S3 object, %d:%s",
+			resp.StatusCode, resp.Status)
+	}
+
+	return nil
+}
+
+// getPresignRequest will request a URL from the service for the content specified
+// by the key and method. Returns a constructed Request that can be used to
+// upload or download content with based on the method used.
+//
+// If the PUT method is used the request's Body will need to be set on the returned
+// request value.
+func getPresignedRequest(serverURL, method, key string, contentLen int64) (*http.Request, error) {
+	u := fmt.Sprintf("%s/presign/%s?method=%s&contentLength=%d",
+		serverURL, key, method, contentLen,
+	)
+
+	resp, err := http.Get(u)
+	if err != nil {
+		return nil, fmt.Errorf("failed to make request for presigned URL, %v", err)
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("failed to get valid presign response, %s", resp.Status)
+	}
+
+	p := PresignResp{}
+	if err := json.NewDecoder(resp.Body).Decode(&p); err != nil {
+		return nil, fmt.Errorf("failed to decode response body, %v", err)
+	}
+
+	req, err := http.NewRequest(p.Method, p.URL, nil)
+	if err != nil {
+		return nil, fmt.Errorf("failed to build presigned request, %v", err)
+	}
+
+	for k, vs := range p.Header {
+		for _, v := range vs {
+			req.Header.Add(k, v)
+		}
+	}
+	// Need to ensure that the content length member is set of the HTTP Request
+	// or the request will not be transmitted correctly with a content length
+	// value across the wire.
+	if contLen := req.Header.Get("Content-Length"); len(contLen) > 0 {
+		req.ContentLength, _ = strconv.ParseInt(contLen, 10, 64)
+	}
+
+	return req, nil
+}
+
+type Method int
+
+const (
+	PutMethod Method = iota
+	GetMethod
+)
+
+type Errors []error
+
+func (es Errors) Error() string {
+	out := make([]string, len(es))
+	for _, e := range es {
+		out = append(out, e.Error())
+	}
+	return strings.Join(out, "\n")
+}
+
+type PresignResp struct {
+	Method, URL string
+	Header      http.Header
+}
+
+func exitError(err error) {
+	fmt.Fprintln(os.Stderr, err.Error())
+	os.Exit(1)
+}