Using Amazon S3 and Cloudfront to distribute your private files

Simple 10 minute guide that will direct you on how to create a link on amazon cloudfront that expires after 30 seconds (or whatever time limit you set) to stream or deliver your private downloads or videos.

1. Setup account

First step is to setup an Amazon AWS account here: https://aws-portal.amazon.com/gp/aws/developer/registration/index.html Amazon have recently introduced a free tier to all its AWS services with very generous data allowances, you can read all about it here: http://aws.amazon.com/free/ In addition to these services, the AWS Management Console is available at no charge to help you build and manage your application on AWS.

2. Create an S3 bucket

Now that you have created your amazon S3 account, login to the Management Console and create a new bucket. NOTE: the name needs to be unique... AND PLEASE AVOID USING FULLSTOPS. If you use fullstops, you will start getting “The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. The remote certificate is invalid according to the validation procedure” when accessing the buckets using the AWS library. This has been an issue for years and still has not been fixed, something to do with the different between the EU/US storage locations. There are workarounds, but for simplicity just avoid using them. Click the upload button in your bucket you just created and choose some files. On the window, dont click “start upload” yet, click on “set permissions”. Click “Add More Permissions” and grant “Authorised Users” access to download the file by checking the “Open/Download” checkbox. Start the upload. I personally named my filenames as "SoftwareName_Version.exe" because the AWS library allows simple queries on request to get all items that begin with a certain phrase etc, but whatever naming scheme works for you.

3. Create a new key pair

Next step, Click Here and Create a new cloudfront KeyPair, and make sure you save the .pem file that you are returned when you create it. There is no way to re-download this file, so if you miss saving that file, remove the key and start again. Save this file somewhere safe as you will need it soon to sign the URL with. Also take note of your “Key Pair ID” as you will need this to sign the URL also.

4. Setup a Cloudfront distribution of files in the bucket

Here is the annoying part: Amazon hasn’t implemented the feature to allow you to create a private distribution from inside the AWS management console. You can do it through web service calls... but the much much easier option is to download bucket explorer (free 30 day trial... and you only need to setup the distribution once so it should be more than enough time.) Follow this guide to setup your own private distribution, as they explain it better than I ever could: http://www.bucketexplorer.com/documentation/cloudfront--how-to-manage-private-content-for-amazon-cloudfront-distribution.html

5. Signing the URL

I could not for the life of me work out how to create a signed URL using the AWS library provided by amazon... but I did come across this very useful class that can do it all for me real easily. I can’t remember where it came from but I came across it on the internet somewhere a few months ago. If anyone knows where it’s from, please let me know and I will credit them with the initial code. It also uses the openSSL library. I have hacked it up a bit and added support for relative path load in of the PEM file.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Security.Cryptography;
using System.IO;
namespace PMI.AWS.CloudFront
{
    public class SignedURL
    {
        /// <summary>
        /// This Generates a signed http url using a canned policy.
        /// To create the PEM file and KeyPairID please visit https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key
        /// </summary>
        /// <param name="resourceURL">The URL of the distribution item you are signing.</param>
        /// <param name="expiryTime">UTC time to expire the signed URL</param>
        /// <param name="pemFileLocation">The path and name to the PEM file. Can be either Relative or Absolute.</param>
        /// <param name="keypairId">The ID of the private key used to sign the request</param>
        /// <param name="urlEncode">Whether to URL encode the result</param>
        /// <returns>A String that is the signed http request.</returns>
        public static string GetPreSignedURLWithPEMFile(string resourceURL, DateTime expiryTime, string pemFileLocation, string keypairId, bool urlEncode)
        {
            if (pemFileLocation.StartsWith("~"))
            {
                var baseDirectory = System.AppDomain.CurrentDomain.BaseDirectory;
                pemFileLocation = Path.GetFullPath(baseDirectory + pemFileLocation.Replace("~", string.Empty));
            }
            System.IO.StreamReader myStreamReader = new System.IO.StreamReader(pemFileLocation);
            string pemKey = myStreamReader.ReadToEnd();
            pemKey = pemKey.Replace("\n", "");
            return GetPreSignedURLWithPEMKey(resourceURL, expiryTime, pemKey, keypairId, urlEncode);
        }
        /// <summary>
        /// This Generates a signed http url using a canned policy.
        /// To create the PEM file and KeyPairID please visit https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key
        /// </summary>
        /// <param name="resourceURL">The URL of the distribution item you are signing.</param>
        /// <param name="expiryTime">UTC time to expire the signed URL</param>
        /// <param name="pemFileLocation">The actual pem file</param>
        /// <param name="keypairId">The ID of the private key used to sign the request</param>
        /// <param name="urlEncode">Whether to URL encode the result</param>
        /// <returns>A String that is the signed http request.</returns>
        public static string GetPreSignedURLWithPEMKey(string resourceURL, DateTime expiryTime, string keyPEM, string keypairId, bool urlEncode)
        {
            string xmlKey = JavaScience.opensslkey.DecodePEMKey(keyPEM);
            return GetPreSignedURLWithXMLKey(resourceURL, expiryTime, xmlKey, keypairId, urlEncode);
        }
        /// <summary>
        /// This Generates a signed http url using a canned policy.
        /// To create the PEM file and KeyPairID please visit https://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key
        /// </summary>
        /// <param name="resourceURL">The URL of the distribution item you are signing.</param>
        /// <param name="expiryTime">UTC time to expire the signed URL</param>
        /// <param name="pemFileLocation">The actual pem file</param>
        /// <param name="keypairId">The ID of the private key used to sign the request</param>
        /// <param name="urlEncode">Whether to URL encode the result</param>
        /// <returns>A String that is the signed http request.</returns>
        public static string GetPreSignedURLWithXMLKey(string resourceURL, DateTime expiryTime, string keyXML, string keypairId, bool urlEncode)
        {
            long expiry = (long)(expiryTime.ToUniversalTime() - new DateTime(1970, 1, 1)).TotalSeconds;
            string policy = String.Format(
                @"{{""Statement"":[{{""Resource"":""{0}"",""Condition"":{{""DateLessThan"":{{""AWS:EpochTime"":{1}}}}}}}]}}",
                resourceURL, expiry);
            RSACryptoServiceProvider rsa = new RSACryptoServiceProvider();
            RSACryptoServiceProvider.UseMachineKeyStore = false;
            rsa.FromXmlString(keyXML);
            string signature = UrlSafe(rsa.SignData(Encoding.UTF8.GetBytes(policy), new SHA1CryptoServiceProvider()));
            string formatStr = urlEncode ?
            "{0}%3fExpires%3d{1}%26Signature%3d{2}%26Key-Pair-Id%3d{3}" :
            "{0}?Expires={1}&Signature={2}&Key-Pair-Id={3}";
            return string.Format(formatStr, resourceURL, expiry, signature, keypairId);
        }
        private static string UrlSafe(byte[] data)
        {
            return Convert.ToBase64String(data)
            .Replace('+', '-').Replace('=', '_').Replace('/', '~');
        }
    }
}

and you also need the openSSL library from here: http://www.jensign.com/opensslkey/opensslkey.cs

6. Add your PEM file to your project

Create a new folder, just create a new guid and name your folder that. For eg, “9467a84d-39f8-464e-a284-8535bd0f8c44” or whatever you want. Just make sure its unique and cant be guessed!!! In that file, place your PEM file that you downloaded in step 3. of course if your site is hosted on VPS and you have access to the local filesystem, store your PEM file there and use that path (yep, the helper class below is smart enough to pick up whether its a relative or absolute path)

7. Creating the Signed url

DateTime expires = DateTime.UtcNow.AddSeconds(30);
string signedURL = PMI.AWS.CloudFront.SignedURL.GetPreSignedURLWithPEMFile("[Cloudfront Distribution URL]/" + [Filename Of Item In Bucket], expires, "~/[GUID Folder Created in Step 6]/[PEM Key Filename inside folder]", "[Key Pair ID]", false);

And thats it! "signedURL" is the URL of the file that will expire in 30 seconds and return an "unauthorized" message. All you need to do now to add more items to your distribution is add items to your bucket and make sure the files have “Authenticated users” checked to view and download. this can be used to secure streaming videos, only allow users to have access to a file download for 2 hours etc etc. plus with cloudfronts speed... you know that whoever is looking at your content is going to be getting it the fastest way possible.