SECRET OF CSS

Managing Private Files With AWS S3 | by Hernan Reyes | Jul, 2022


Learn two methods to handle private files in your application with AWS S3

image by author

Back in 2020, at the company I worked for, I was told that we were moving our images from Cloudinary to AWS S3 and that I would be in charge of performing the migration. Long story short, I did the migration, but in the process, I faced a problem — we have images that must be private and are only meant to be seen by admins. Two years since then, I’m faced with a similar problem, but with a different use case.

In this article, I’ll show you two ways you can manage private files with AWS S3: the one I did in 2020 and the one I’m planning to do now.

This solution is what I used when I migrated images. It basically consists in creating a backend endpoint that will validate if the user requesting the image has the necessary permissions to see the file.

1*WdxMZALw7Qc0

My use case

Give access to admins, so they can see private images that are rarely to be requested (three times as much).

A disadvantage of this approach is that the private file first goes from S3 to our backend, and from there, it goes to the client, which increases transfer costs. But it was OK for our use case because these images are barely requested once or twice and then forget it. After all, they are no longer needed unless we need to validate historical data.

This solution consists in using what is called presign URLs. The idea behind these URLs is to give access to private files for a period of time, and we do it by signing the object key of our S3 with private credentials in the backend, then setting an expiration time in which the presign URL will be valid to be requested.

0*t1 ofCLBZJSPyY7

As you see, for this method, we also use a backend endpoint to validate if the user has permission to see the private file, so we can return the signed URL.

My use case

Render a list of employees with a name and picture (to keep the example simple) that will be frequently visited in the application.

With this, we solve the problem of transfer costs of the first method here instead of transferring the file from S3 to our backend and then to the client. We just send a signed URL to the client that can be used to obtain the file directly from the S3.

A problem that we’ll face here is that every time we sign a URL, a new signature is generated. So the browser won’t be able to cache the image because the URL will always change, so we’ll have something like this every time:

five columns: name, method, status, type, size. five rows with same information: cat.jpg?AWSAccessKeyId…, GET, 200, jpeg, 22.6kb

In every request, the user will download the file instead of getting it from the cache.

0*dWMrTuhpnUOmns6Y

This problem is explained in detail in this article, but basically, we’ll need a way to get the same signature for the presigned URL, so the browser can cache the file.

The other thing is that I need to render a list of employees in the frontend where everyone has a private image, but having an endpoint for signing just one URL at a time will be an N+1 query problem for our server. So when we return the list of employees, we’ll also have to return the pre-signed URLs to avoid this problem.

While I was investigating this, I saw that notion.so and raindrop.io use these presigned URLs to handle private files inside your account.

Something we’ll have to do sooner or later is configuring our S3 to give access to our backend when using the AWS SDK, so we can upload or get files from our S3 bucket.

AWS offers us different methods to access our S3 bucket, but because we are working within the same AWS account, we’ll use IAM policies to provide access to our bucket.

IAM policies

These policies control the access to resources on AWS (S3 in our case) and are attached to entities like users, roles, or groups in your AWS account to define what these entities can do in the different services provided by AWS.

Now we’ll proceed with the configuration:

Here I assume you already have an AWS account

Create the bucket to store our objects

0*QWIDWbZziNP0eTKN

If you have public files, too, you must deactivate the Block Public Access option as I did here.

Create a policy to define the permissions of our user

For my example, I just need three permissions:

  1. PutObject to upload files
  2. PutObjectAcl to define if the file will be public or private
  3. GetObject to get files

Also, I specified that this policy is only valid for the bucket we just created with the field: "Resource": "arn:aws:s3:::hrmtest/*"

Create a user to obtain the credentials needed for our backend

Here, we create the user with the policy we just created in the previous step, so we can get the credentials to use them in our application.

After this, we are ready to use our credentials in our application.

I am preparing a small project where we’ll be able to upload and get files from AWS S3. We’ll use the credentials we downloaded from AWS and will put them inside the .env file that will be read when starting the service, so we can do requests to AWS. Here’s the code:

# AWS S3
S3_ACCESS_KEY_ID=AKIAZCF3JGYORXJMZPMY
S3_SECRET_ACCESS_KEY=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
S3_BUCKET_NAME=hrm-article-demo
S3_BUCKET_REGION=us-east-1

By the time this article is published, the credentials will no longer exist, so don’t bother trying to do something with them.

After we read the credentials, we’ll use them to get a session to AWS S3. Here’s the code:

Upload method

With this, we’ll be able to upload public or private files.

GetFile method

To get the private files, we’ll use this method. For public files, AWS gives us a URL with the following format: https://{bucket-name}.s3.amazonaws.com/{object-key.png}https://hrm-article-demo.s3.amazonaws.com/employees/1399d998-b505-4ce2-b49b-bd8073de9b9e.jpg.

Presign method

With this, we’ll be able to sign our private file’s key, so we can see the file for a period of time.

Another way of doing this without getting the same signature every time is shown below:

It’s a cleaner solution that you can use if you don’t mind getting the same signature every time.

Here, I just show the code that’s of interest to the article. If you want to see all the project’s code, just go to: https://github.com/hernanhrm/blog-examples/tree/main/private-files.

Also, I mentioned that at the backend, we need to verify if the user has the needed permissions to see the requested file, but I didn’t include that code in this article because I’ll make an article on how to handle authentication and authorization in your application.

Now that we have the code, we can play with it.

Uploading an image

We’ll upload a private image, so the only way to see it will be through our endpoint to get files or to sign the URL.

0*Y HVNmYDTU8 tFbU

Once we send the request, we’ll get a response like this:

0*fKYS3pYuS816xm l

As I just told you, the only way to see the image is through our endpoint to get files or sign the URL, so if we try to use the URL that AWS gives us, it will show us something like this:

0*2OIj1w6lO4KLophN

But if we use the endpoint we made to get files from AWS, we’ll get the file:

0*M6DcVzR rsf5Rtjj

Yeah, that’s me. Now let’s try signing the URL:

0*4zwbjdLDJjqcwn49

Now, if we use the presigned URL, we’ll also get the file while it is still valid. Here’s what it looks lke:

0*89cSp44ItKHj20DI

GetEmployees method

And for my use case, I told you that I need to generate a signed URL for the employees’ pictures but that calling an endpoint to sign a URL one by one will be an N+1 query problem. So for that, I’ll use a method that gets the resigned URL when requesting employees. Here’s the code:

So if we upload some images for our employees, when we do a request to: GET api/v1/employees, we’ll get a signed URL with the same signature during the current day every time we execute a request, and as we can see, the browser cached the image.

0*aUnKGRwIHzj9Kk4s

Okay, this got longer than I was expecting, but I showed you how to configure your S3 and two methods to get private files from AWS S3 with use cases that I’ve come across at work. So I hope you can decide what method to use depending on your case.

  1. For the illustrations, Excalidraw
  2. Learn in depth how Pre-sign URLs work
  3. If you speak Spanish, here is a good course about AWS S3 https://ed.team/cursos/s3
  4. Securing AWS S3 uploads using presigned URLs
  5. Project Repository



News Credit

%d bloggers like this: