S3 Direct Form Posting and Go

Introduction

I recently ran into a situation that required files to be directly posted to S3 without using an intermediate machine as a middleman.  This brought about some obvious challenges including but not limited to showing a progress bar or spinner during upload, the fact that the file needed to be posted directly to S3 also initially meant that any errors that were going to be displayed would be ugly S3 provided (and rather abrupt) XML messages.

So is there a better way?  Of course there is, and here’s how I did it.

Getting Started

First and foremost before anything gets posted to S3 you need to ensure that you can create an S3 signature properly.  Now I found this to be the most tiresome and time consuming part of the process.  Reading through outdated docs and non-working samples are enough to make any seasoned programmer want to rage quit after a time.  However, through perseverance and a lot of trial and error, I figured it out.  If you’re using Go, then the following examples are going to be perfect for you, if you’re using a different language, then there’s probably already another working sample out there for you to use.  Just in case however, I will precede each with pseudo code so that it may be portable to other languages where needed.

Encryption

The first thing we are going to want to do is create a function for handling HMAC SHA 256 encryption.  We are going to be using this function over and over again throughout this post.

Pseudo Code

function:
    encrypt(key, value)
        return hmacSha256(key, value)

In Go, I’ve defined the function for the sake of this post as the following.

Go Code

import (
    "crypto/hmac"
    "crypto/sha256"
)

func AwsEncode(key, value []byte) []byte {
    mac := hmac.New(sha256.New, key)
    mac.Write(value)
    return mac.Sum(nil)
}

In understanding the above, let’s walk through it line by line. First we define the function as accepting key and value as a byte slice.  The value is what will be encrypted, the key is the salt used to encrypt the value. The function then returns the encrypted string as a byte slice. First we create a new hmac specifying ssh256 as our encryption scheme, and initialize it with the key. Next, we encrypt the value and return it.

Signing

Next we’re going to want to begin the process of signing the payload which is required for S3 form posting to work.  Once again, let’s start by pseudo coding then showing the actual implementation.

For this we’re going to want to create a series of encryption calls, each based on the previous value and feeding into the next, sounds confusing at first but it’s not that bad, here’s how it works.

Basically what we’re going to want is get the current date in YYYYMMDD format, which will get HMAC SHA 256 encoded using your AWS secret.  For the sake of AWS V4 authentication we’ll need to add a prefix to your AWS secret.  Do not make the mistake of hex or base64 encoding this, leave it returned in its raw state.

Pseudo Code

yyyymmdd = Now.format("yyyymmdd")
encrypt("AWS4" + your_aws_secret, yyyymmdd)

Go Code

import (
    "fmt"
    "crypto/hmac"
    "crypto/sha256"
    "time"
)

now := time.Now()
yyyymmdd := fmt.Sprintf("%d02d%02d", now.Year(), now.Month(), now.Day())
dateKey := AwsEncode([]byte("AWS4" + your_aws_secret), []byte(yyyymmdd))

In true fashion, let’s go line by line here.  First, we create a yyyymmdd value by ensuring the current date is 0 left padded for values mm and dd.  Once the yyyymmdd string is created we move on to creating the encrypted dateKey which is comprised of encrypting your AWS secret along with the string prefix “AWS4”.  Lastly we ensure these values are passed in as byte slices and store the returned dateKey value.

We will use this pattern to create the entire signature.  The only variation will be at the very end when we return the final singed value as hex encoded. Also note, value cited as your_aws_region should contain the string name of the region such as “us-east-1” for example.

Pseudo Code

function:
    sign (string_to_sign)
        dateKey = encrypt("AWS4" + your_aws_secret, yyyymmdd)
        dateRegionKey = encrypt(dateKey, your_aws_region)
        dateRegionServiceKey = encrypt(dateRegionKey, []byte("s3"))
        signingKey = encrypt(dateRegionServiceKey, "aws4_request")
        signature = encrypt(signingKey, string_to_sign)
        return hex(signature)

Go Code

import (
    "fmt"
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "time"
)

func AwsSign(stringToSign string) string {
    your_aws_secret := "SOMEAWSSECRETKEYSTOREDINYOURAIMSETTINGS"
    your_aws_region := "your-region-1"

    now := time.Now()
    yyyymmdd := fmt.Sprintf("%d%02d%02d", now.Year(), now.Month(), now.Day())

    dateKey := AwsEncode([]byte("AWS4" +your_aws_secret), []byte(yyyymmdd))
    dateRegionKey := AwsEncode(dateKey, []byte(your_aws_region))
    dateRegionServiceKey := AwsEncode(dateRegionKey, []byte("s3"))
    signingKey := AwsEncode(dateRegionServiceKey, []byte("aws4_request"))
    signature := AwsEncode(signingKey, []byte(stringToSign))
    return hex.EncodeToString(signature)
}

Now is probably a good time to note that the above code is really laid out for learning purposes, the variable notation used such as your_aws_secret and so forth is really there solely for the purpose of drawing attention.  Your final implementation of the above should either pull the secret and region from a configuration file or be implemented as function parameters depending on your requirements.  That being said, let’s go line by line.  First we’ve setup a function called AwsSign which takes a stringToSign variable.  This variable will be the final content to be signed.  In the case of direct form posting this will variable will contain the JSON policy definition which we will describe later. The AwsSign function returns a string which will be the final signature value.  The next two lines in true implementation will contain your actual AWS secret and AWS region name respectively. We then generate the yyyymmdd string as previously described and generate the dateKey variable based on the “AWS4” prefix along with your AWS secret and yyyymmdd variable.  We then take that variable and feed it back into the AwsEncode function along with the AWS region.  We then take the dateRegionKey and feed it back into the AwsEncode function again along with the service we which to use, in this case “s3”. We then take that value and feed it back into the AwsEncode yet again along with the “aws4_request” string telling AWS that we are using a version 4 request schema. We then take that value and pass it yet again back into AwsEncode along with the stringToSign value finally completing the change of encryption.  The last step is the hex encode the byte slice into a string and return it as the final signature to be used along with direct form posting to S3.

I know that was a bit of gallop and hopefully it made sense.  If now, please feel free to contact me or leave a comment below and I’ll do my best to describe what’s going on here.  For reference the following might provide a more visual flow to the above.

https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html (scroll to the bottom of the page)

Policy

Now we start getting into the guts of the operation, the policy.  The way the form post works is that you define a policy, that policy gets base 64 encoded and added as a form variable named “policy” (go figure), then you pass the same un-encoded policy to be signed using the AWS signature function we defined above and use the returned signature value as a form field called, you guessed it, “signature”.  One of the caveats to understand with policy is that everything defined in the policy has to match in the form, we’ll discuss this a little more in detail later, but for now, let’s define a simple working policy.

Example

{
    expiration: "2019-02-13T12:00:00.000Z",
    conditions: [
        {"bucket", "your_bucket_name"},
        {"success_action_redirect", "https://yourserver.com/your_landing_page"},
        ["starts-with", "$key", "your_root_folder/perhaps_another_folder/"],
        {"acl": "public-read"},
        ["starts-with", "$content-type", ""],
        {"x-amz-meta-uuid": "14365123651274"},
        {"x-amz-server-side-encryption": "AES256"},
        {"x-amz-credential": "your_aws_id/your_aws_region/s3/aws4_request"},
        {"x-amz-algorithm": "ASW4-HMAC-SHA256"},
        {"x-amz-date": "20190212T000000Z"}
    ]
}

So now the fun, let’s walk through exactly what in tarhooty is going on above.

expiration: The first thing you’ll probably notice is the expiration field.  This field describes how long the transaction should live for.  Any easy way to handle this for example would be to take to current date time and add one day to it, allowing for a period of twenty four hours to upload the file.  For a more tightened approach depending on your needs you could set it to seconds, minutes or hours.

conditions: Next, let’s look at the conditions field.  This is an array which describes the conditions that must be met in order for the post upload to succeed.  Anything not met will automatically cause a failure by the AWS S3 API.

bucket: The bucket field is the top most name of your S3 folder structure.  This field is required to ensure the upload goes to the correct bucket.

success_action_redirect: This success action redirect field specifies where you would like S3 to redirect after a successful upload.

starts-with: The starts-with condition allows one to define a restriction for a field in the post upload form.  For example, the “starts-with” “key” requires that the html form containing the “key” value must in fact start with whatever is specified in the last parameter (ex. “uploads/public/”).  When the last parameter in this condition is an empty string “” that means anything goes and this condition will be satisfied as long as there is a key field.

acl: The acl (Access Control List) field determines the visibility of the uploaded object.

x-amz-meta-uuid: This field is a bit of a bizarre one, as far as I can tell this field is required and it’s required to have the following value of “14365123651274”. No clue as to why, wouldn’t work without it…when in Rome.

x-amz-server-side-encryption: This field specifies the encryption scheme being used to sign the policy.

x-amz-credential: The credential field is composed of several values and usually takes the form of {your_aws_id}/{your_aws_region}/s3/aws4_request. A working example if we were to use the values from our AwsSign function would take on the following form “AKIAYOURAWSID/us-east-1/s3/aws4_request”.

x-amz-algorithm: The algorithm field tells the S3 api which algorithm was used for signing.

x-amz-date: The date field would be today’s date.  For simplicity’s sake, I zeroed out the hours minutes and seconds above however, you could and should use a complete date taking the form of YYYYMMDDThhmmssZ.

Constructing The Form

Now that we have a general understanding of all the moving parts involved in making the post request work, let’s bring them all together.  First a walk through of what we’re going to do.

We’re going to need to build a policy with dynamic values populated at request time.

When building the form, we’re going to need to create our policy and base64 encode it and pass it into the form as “policy”.

We’ll also need to AwsSign our policy and pass it into the form as the “signature” field.

Finally, we’ll need to ensure all the values we’ve specified in our policy are met by our form.

Here’s an example HTML form for use with our policy described above.  Please note, when developing your form the order of the elements matters.  It must match the order described in the policy otherwise the post will fail

<form method="post" action="https://your_bucket_name.s3.amazonaws.com/" enctype="multpart/form-data">
    <input type="hidden" name="key" value="your_root_folder/perhaps_another_folder/your_file_to_upload.txt" />
    <input type="hidden" name="acl" value="public-read" />
    <input type="hidden" name="success_action_redirect" value="https://yourserver.com/your_landing_page" />
    <input type="hidden" name="content-type" "" />
    <input type="hidden" name="x-amz-meta-uuid" value="14365123651274" />
    <input type="hidden" name="x-amz-server-side-encryption" value="AES256" />
    <input type="hidden" name="x-amz-credential" value="your_aws_id/your_aws_region/s3/aws4_request" />
    <input type="hidden" name="x-amz-algorithm" value="AWS4-HMAC-SHA256" />
    <input type="hidden" name="x-amz-date" value="20190212T000000Z" />
    <input type="hidden" name="policy" value="{replace-with-actual-base64-encoded-policy}" />
    <input type="hidden" name="x-amz-signature" value="{replace-with-actual-aws-sign}" />
    <input type="file" name="file" />
    <input type="submit" value="Upload" />
</form>

As you can see above all of the fields specified in the policy are present in the form.  The “policy” field will contain the direct result of taking the policy JSON as a string and running through a base64 encode.  The “x-amz-signature” field will need to contain the base64 encoded policy which has been run through the AwsSign function.  Finally we added a file and submit button to select the file (which in this case must be named “your_file_to_upload.txt”) and to submit the form.

Putting It All Together

Server Side

import ( 
    "fmt" 
    "crypto/hmac" 
    "crypto/sha256" 
    "encoding/hex"
    "encoding/base64"
    "time" 
)

func AwsEncode(key, value []byte) []byte {     
    mac := hmac.New(sha256.New, key)     
    mac.Write(value)     return mac.Sum(nil) 
}

func AwsSign(stringToSign string) string {     
    your_aws_secret := "SOMEAWSSECRETKEYSTOREDINYOURAIMSETTINGS"     
    your_aws_region := "your-region-1" 

    now := time.Now() 
    yyyymmdd := fmt.Sprintf("%d%02d%02d", now.Year(), now.Month(), now.Day()) 

    dateKey := AwsEncode([]byte("AWS4" +your_aws_secret), []byte(yyyymmdd)) 
    dateRegionKey := AwsEncode(dateKey, []byte(your_aws_region)) 
    dateRegionServiceKey := AwsEncode(dateRegionKey, []byte("s3")) 
    signingKey := AwsEncode(dateRegionServiceKey, []byte("aws4_request")) 
    signature := AwsEncode(signingKey, []byte(stringToSign)) 
    return hex.EncodeToString(signature) 
}

//define a policy
policy := []byte(`{ 
    expiration: "2019-02-13T12:00:00.000Z", 
    conditions: [ 
        {"bucket", "your_bucket_name"},
        {"success_action_redirect", "https://yourserver.com/your_landing_page"}, 
        ["starts-with", "$key", "your_root_folder/perhaps_another_folder/"], 
        {"acl": "public-read"}, 
        ["starts-with", "$content-type", ""], 
        {"x-amz-meta-uuid": "14365123651274"}, 
        {"x-amz-server-side-encryption": "AES256"}, 
        {"x-amz-credential": "your_aws_id/your_aws_region/s3/aws4_request"}, 
        {"x-amz-algorithm": "ASW4-HMAC-SHA256"}, 
        {"x-amz-date": "20190212T000000Z"} 
    ] 
}`)

//base4 encode policy
hexPolicy := base64.StdEncoding.EncodeToString(policy)
signature := AwsSign(hexPolicy)

Client Side

<form method="post" action="https://your_bucket_name.s3.amazonaws.com/" enctype="multpart/form-data">
    <input type="hidden" name="key" value="your_root_folder/perhaps_another_folder/your_file_to_upload.txt" />
    <input type="hidden" name="acl" value="public-read" />
    <input type="hidden" name="success_action_redirect" value="https://yourserver.com/your_landing_page" />
    <input type="hidden" name="content-type" "" />
    <input type="hidden" name="x-amz-meta-uuid" value="14365123651274" />
    <input type="hidden" name="x-amz-server-side-encryption" value="AES256" />
    <input type="hidden" name="x-amz-credential" value="your_aws_id/your_aws_region/s3/aws4_request" />
    <input type="hidden" name="x-amz-algorithm" value="AWS4-HMAC-SHA256" />
    <input type="hidden" name="x-amz-date" value="20190212T000000Z" />
    <input type="hidden" name="policy" value="{{.hexPolicy}}" />
    <input type="hidden" name="x-amz-signature" value="{{.signature}}" />
    <input type="file" name="file" />
    <input type="submit" value="Upload" />
</form>

The above client side and server side code examples are obviously not meant to be implemented verbatim however, they should be enough to get you started and heading in the right direction.

Moving Beyond Post – Ajax

A few things may have jumped out a you with the approach above.  The values required will need to either be populated dynamically or be populated from the server before the form is rendered.  Both could work.  The main issue I had with both of these approaches is that it required the key name to be known before the form was constructed which could be problematic depending on your requirements, the other was that there’s no spinner or loading for the user.  The user is instead just required to wait and hope all went well.  For me, that’s just not good enough so I devised a way to use AJAX (or XHTTP) to upload the file without refreshing the page. If you’re interested then read on.

So how could we get this solution to more in a more elegant manner.  How about this workflow?

Use sees a file button, clicks the file button and selects the file to be used.

The onchange event is triggered for the file button and in turn sends an ajax (XHTTP) request to the server to generate all the fields required to populate the client form.  The values could contain the full key using the selected filename, signature, everything.

When the ajax (XHTTP) request returns with all the required form values, the client constructs a FormData object and appends the data as form attributes.

//assuming json contains server side AWS values and
//using jQuery to communicate with the file input element

var formData = new FormData();
formData.append("key", json.AwsKey);
formData.append("acl", json.AwsAcl);
...
formData.append("policy", json.AwsPolicy);
formData.append("signature", json.AwsSignature);
formData.append("file", fileInput.prop("files")[0], input.attr("name));
formData.append("upload_file", true);

Finally, once the formData object has been fully populated and in the correct order with all of the required AWS field values, that form could then be sent to the AWS end point specified earlier in the form.

//assuming jQuery being used to post XHTTP request to AWS

//show spinner

$.ajax({
    url: "https://your_bucket_name.s3.amazonaws.com",
    type: "post",
    data: formData,
    contentType: false,
    processData: false,
    success: function(json){
        //do something
    },
    error: function(xhr, status, message){
        //show error
    },
    complete: function(){
        //hide spinner
    }
});

This approach would allow you to upload asynchronously, show a spinner, handle errors and overall be more elegant.

Two important things of note here however.

First – We don’t want to use success_action_redirect in our policy anymore.  If we’re using AJAX then we want to change our policy to use “success_action_status” with the value of “200”

{ 
    expiration: "2019-02-13T12:00:00.000Z", 
    conditions: [ 
        {"bucket", "your_bucket_name"},
        {"success_action_status", "200"}, 
        ["starts-with", "$key", "your_root_folder/perhaps_another_folder/"], 
        {"acl": "public-read"}, 
        ["starts-with", "$content-type", ""], 
        {"x-amz-meta-uuid": "14365123651274"}, 
    "x-amz-server-side-encryption": "AES256"}, 
        {"x-amz-credential": "your_aws_id/your_aws_region/s3/aws4_request"}, 
        {"x-amz-algorithm": "ASW4-HMAC-SHA256"}, 
        {"x-amz-date": "20190212T000000Z"} 
    ] 
}

Second – We’re going to run into a CORS pre-flight limitation that will need to be address.  The fix is to go into your S3 settings on AWS, click on the bucket name, then click on the “Permissions” tab and finally click the “CORS configuration button.

Once there add the following xml and save.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>POST</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Much better, now instead of being redirected we have full control over how we show error messages be alerts, growls, or some other.  Furthermore, we’ve added the correct CORS check and should be good moving forward.

Conclusion

Hopefully this helped some of you out there trying to get AWS direct post working with Go. If I’ve left something out, made a mistake or if you’d just like a bit more information or clarification on something, please leave me a comment and I’ll do my best to answer.  Thanks for stopping by.