Automate your File Storage: a guide to setup Buckeeter for your Rails application

Bucketeer is a Heroku add-on that facilitates the integration and handling of S3. I discovered this add-on while working on a project when my team included Review apps in our deployment workflow to test features. We had to configure each environment as close as possible to production. Bucketeer became key to include Amazon Web Services for our file storage. I highly recommend it due to its ease of managing and automating the creation of buckets. 

We used an app.json manifest to configure add-ons, add build packs, and execute scripts. This file is run once when creating the review apps, so the configuration can be automated. However, we were required to change our S3 integration approach to allow that automation.

Previously, we were managing the buckets on our side. That means we were creating, configuring them, and setting their credentials manually. Doing it for every review app would be highly inefficient. Luckily, Bucketeer takes care of it. 

Before we dive into the setup, let's review the advantages and disadvantages of using it. That way you can decide whether it’s a worthwhile alternative for you.

‍

Advantages

1. Simplified Bucket Creation‍

Creating buckets is pretty simple with the Heroku CLI. You can specify the storage size, region, and bucket name with a single command. 

2. Automatized setup

‍The credentials are generated once the bucket is created. They are saved into the Heroku's variables, ready to be accessed in your app. You can retrieve them as follows: 

• ENV["BUCKETEER_AWS_ACCESS_KEY_ID"]
• ENV["BUCKETEER_AWS_SECRET_ACCESS_KEY"]
• ENV["BUCKETEER_BUCKET_NAME"]
• BUCKETEER_AWS_REGION

3. Hands-off S3 Management‍

One of its most appealing aspects is that it relieves us of the management of the S3 account, such as the billing. Heroku takes care of the intricate details of S3, allowing us to focus only on bucket management.

‍

Disadvantages

1. Poor documentation‍

Most likely, everything you need to know can be found within AWS's huge documentation. But there isn’t much support on the web regarding Bucketeer specifically. 

2. Limited access

‍The IAM credentials are all the access we have to the buckets. We don’t have access to the console, which can come in handy when debugging, or the account ID, which is often required to apply some bucket policies. Spoiler: we faced challenges when migrating existing objects to Bucketeer, but that’s for another post!

3. Dependence on Heroku

‍The bucket is attached to the add-on, therefore, to Heroku itself. If any of them get deleted, you will lose the data.

‍

Set up

As stated above, we use an app.json manifest to configure our review apps. I will only show you the intricacies of adding Bucketeer and configuring the bucket. However, it can be customized to match your needs.

We must include the add-on in our Heroku application. We can establish the plan we want to use (otherwise, it'll choose the default one). Bucketeer provides several plans with different storage capacities to match your needs. This step is run before deploying, under the add-ons section:

Next, we’ll create an initializer to hold our OpenAI API key. It is recommended that you store it securely in config/credentials.yml.enc, or as an ENV variable using dotenv.

"addons": [
  {
    "plan": "bucketeer:plan"
  }
]

Remember, this step also takes charge of setting the credentials.  You can call them using the ENV variables described above. 

For some, this may be it. If you need to perform a different action, like setting a bucket policy, you can use the following example:

# app.json
"scripts": {
  "postdeploy": {
    "command": "sh archive.sh"
    }
 }

# archive.sh
bundle exec action:execute

# action, like a rake
namespace :action do
  desc "Set CORS policy"
  task execute: :environment do
    require "aws-sdk-s3"

    set_cors_policy
  end

  def set_cors_policy
    S3.put_bucket_cors({
      bucket:             ENV["BUCKETEER_BUCKET_NAME"],
      cors_configuration: {
        cors_rules: [
          {
            allowed_headers: [
              "*"
            ],
            allowed_methods: [
              "PUT"
            ],
            allowed_origins: [
              "https://#{ENV["HEROKU_APP_NAME"]}.herokuapp.com"
            ],
            expose_headers:  [],
            max_age_seconds: 3000
          }
        ]
      }
    })
 end
end

On a final note, when deleting the app, the buckets are discarded. You don’t have to worry about useless objects being stored unnecessarily.
‍

Conclusion

Doing proper setups may take some time, but it can save a lot of effort and time, especially when it is meant to be run multiple times. In some cases, like this one, you are saved from manual configuration if you have a setup ready to go.

In a nutshell, adding S3 and creating a bucket is straightforward using Bucketeer. It also reduces friction while using AWS, since it manages your account. However, convenience comes at a cost: limited access, dependency on the add-on, and limited documentation. Despite these drawbacks, it’s a powerful tool for managing S3. Its advantages often outweigh the disadvantages.

‍