Upload File to Amazon S3 With Go

When your application handles a large number of files, you need to store the files in a certain storage system outside your application server. One of the best storage services is Amazon S3. This article will show you how to create an Amazon S3 Bucket, create access key id and secret, and upload files to Amazon S3 with Go.

What is Amazon S3

Amazon S3 or Simple Storage Service is a storage service provided by Amazon AWS. We can use it to store and protect any amount of data for a range of use cases, such as data lakes, websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics.
To store data to S3 with a Go application, first, we need to create the bucket and access credentials.
If you already have your bucket and credentials, you can skip to this section.

Create an S3 bucket

To create your S3 bucket, first, go to the AWS console page. Then search S3 in the search bar. You will be directed to the S3 dashboard page.
Click Create Bucket button.

Input your Bucket name and select your prefered AWS Region. Leave other setting as is for now.

Then click Create Bucket button at the bottom of the page. The bucket will be created.

Create IAM to access the bucket

After the bucket is created, we need to create an access key id and secret so our Go application can access the bucket. To create the credential, go to the AWS console page, then search IAM in the search bar. Then click IAM to go to the IAM dashboard page.
On the IAM dashboard page, click Users on the left sidebar. Then click Add Users.

Input the User name and check the Access key - Programmatic access checkbox. Then click the Next button.

We need to attach S3 access policy to the user.

Click Attach existing policies directly.
Search S3 in the filter.
Then tick AmazonS3FullAccess and click Next button.

You can add tags if you want. It is optional.

Review the user creation to make sure the parameters are correct. Then click Create User.

The user is successfully created.

Save the Access Key ID and Secret access key. It is needed to access the bucket from Go application.

How to upload file to S3 with Go

Once we have the bucket and the access key, our Go app can upload files to the S3. AWS provided official Go SDK to manage S3 (https://github.com/aws/aws-sdk-go/).
To upload a file to S3 we need to create an S3 uploader and call the Upload method of the uploader.
Use the following code sample to create the uploader.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import (
    // ...
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
)

    // ...

    s3Config := &aws.Config{
        Region: aws.String("ap-southeast-1"),
        Credentials: credentials.NewStaticCredentials("KeyID", "SecretKey", ""),
    }
    s3Session := session.New(s3Config)

    uploader := s3manager.NewUploader(s3Session)

On line 11, we create AWS config with our bucket’s region and access key ID and secret key of the IAM. This config is needed to create a session for the uploader. Make sure to import all the necessary libraries.
Once the uploader is created, we can call the UploadWithContext method of the uploader to upload a file to the S3. Let’s see the sample code below.

1
2
3
4
5
6
7
    input := &s3manager.UploadInput{
        Bucket:      aws.String("jajaldoang-test-s3-upload"), // bucket's name
        Key:         aws.String("myfiles/my_cat.jpg"),        // files destination location
        Body:        bytes.NewReader(file),                   // content of the file
        ContentType: aws.String("image/jpg"),                 // content type
    }
    output, err := uploader.UploadWithContext(context.Background(), input)

To upload the file, we need parameter UploadInput which contains the bucket’s name and attributes of the file. The context can be used to limit upload time and cancelation.

Conclusion

To upload files to Amazon S3, we can use the official Go SDK provided by AWS. We need the bucket and access key ID and secret key to upload from the Go app. Once we have those, the Go code to upload the file is simple.

The complete code

The complete code is here (click to expand)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
package main

import (
	"bytes"
	"context"
	"io/ioutil"
	"log"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3/s3manager"
)

var uploader *s3manager.Uploader

func main() {
	uploader = NewUploader()

	upload()
}

func NewUploader() *s3manager.Uploader {
	s3Config := &aws.Config{
		Region:      aws.String("ap-southeast-1"),
		Credentials: credentials.NewStaticCredentials("KeyID", "SecretKey", ""),
	}

	s3Session := session.New(s3Config)

	uploader := s3manager.NewUploader(s3Session)
	return uploader
}

func upload() {
	log.Println("uploading")

	file, err := ioutil.ReadFile("./my_cat.jpg")
	if err != nil {
		log.Fatal(err)
	}

	upInput := &s3manager.UploadInput{
		Bucket:      aws.String("jajaldoang-test-s3-upload"), // bucket's name
		Key:         aws.String("myfiles/my_cat.jpg"),        // files destination location
		Body:        bytes.NewReader(file),                   // content of the file
		ContentType: aws.String("image/jpg"),                 // content type
	}
	res, err := uploader.UploadWithContext(context.Background(), upInput)
	log.Printf("res %+v\n", res)
	log.Printf("err %+v\n", err)
}

S3 aws go