Coraline Ada Ehmke

Processing Email Attachments with MailGun and ActiveStorage

Coraline Ada Ehmke | April 20, 2019

I implemented an “open code of conduct issue via email” feature on Beacon recently, and today I set out to allow reporters to use file attachments to add screenshots to the issues they report. This meant that I would have to figure out a way to process and store email attachments with ActiveStorage, which turned out to be An Adventure.

I’m using MailGun to allow Beacon to receive emails, and the ActiveStorage feature in Rails 5 to handle file uploads to a Google Cloud bucket. All of the documentation that I could find for ActiveStorage assumed an upload-via-form use case, which wasn’t applicable for me.

I already had a service class called EmailProcessorService to handle processing incoming email and creating an Issue object, so now I needed to add a method to process attachments. The service initializes with an Email instance, provided by MailGun. So I added the new method:

# app/services/email_processor_service.rb
class EmailProcessorService

  attr_reader :email

  def initialize(email)
    @email = email
  end

  ...

  def process_attachments
    email.attachments.each do |attachment|
      # do something
    end
  end

end

I actually did TDD for this feature, so the next thing I did was to extend my Email factory (using FactoryBot of course) to include an attachment:

# spec/factories/email.rb
FactoryBot.define do
  factory :email, class: OpenStruct do
    to {
      [
        {
          full: 'sample_project@issues.coc-beacon.com',
          email: 'sample_project@issues.coc-beacon.com',
          token: 'sample_project', host: 'coc-beacon.com',
          name: nil
        }
      ]
    }
    from { {
      token: 'reporter',
      host: 'foo.com',
      email: 'reporter@foo.com',
      full: 'Reporter <reporter@foo.com>', name: 'Reporter'
    } }
    subject { 'CoC Issue' }
    body { 'Something bad happened.' }
    attachments { [] }

  trait :with_valid_attachment do
    attachments {
      [
        ActionDispatch::Http::UploadedFile.new(
          filename: 'img_1.png',
          type: 'image/png',
          tempfile: File.new(
            "#{File.expand_path(File.dirname(__FILE__))}/fixtures/img_1.png"
          )
        )
      ]
    }
  end

I needed to add an image to spec/fixtures/, so I took a screenshot and saved it there as img_1.png.

Now, in my spec, I could write an example for an email with an attachment:

# spec/services/email_processor_service_spec.rb
require 'rails_helper'

describe EmailProcessorService do

  let!(:project) {
    FactoryBot.create(
      :project,
      account_id: FactoryBot.create(:account).id,
      name: "Sample Project",
      accept_issues_by_email: true
    )
  }
  let!(:issue) {
    FactoryBot.create(:issue, project_id: project.id, issue_number: 101)
  }

  context "with attachment" do

    context "valid image attachment" do
      let(:email_issue) { FactoryBot.build(:email, :with_valid_attachment) }
      let(:service) { EmailProcessorService.new(email_issue) }

      it "creates file attachments" do
        expect{ service.process }.to_not raise_error
        issue = project.issues.last
        expect(issue.uploads.size).to eq(1)
      end
    end

  end

After some iterating, I got the spec to green. Yay!

As I process the email attachments, I want to attach image-type attachments as uploads on an Issue. Here is where that is defined:

# app/models/issue.rb
class Issue < ApplicationRecord
  has_many_attached :uploads, dependent: :destroy
end

(This is straight from the ActiveStorage docs.)

To process the attachments and create the associations, I used the attach method:

# services/email_processor_service.rb
class EmailProcessorService

  ...

  def process_attachments
    email.attachments.each do |attachment|
      issue.uploads.attach(
        io: attachment,
        filename: attachment.original_filename,
        content_type: attachment.content_type
      )
    end
  end

end

Next, I wanted to make sure that the file attachments were valid images. I couldn’t rely on the validations that I had in the Issue model, since there would be no way to alert the user (in this case, the email sender) of a validation error. So I decided to do inline validation in the service, and only process attachments from an allowed set of MIME types. So in the service class:

# services/email_processor_service.rb
class EmailProcessorService

  VALID_MIME_TYPES = [
    "image/gif",
    "image/jpeg",
    "image/png"
  ].freeze

  ...

  def process_attachments
    email.attachments.each do |attachment|
      next unless VALID_MIME_TYPES.include?(attachment.content_type)
      ...
    end
  end

end

Nice! Now to create an example to prove that an unsupported MIME type wouldn’t be attached to an Issue.

First the factory:

# spec/factories/email.rb
...
trait :with_invalid_attachment do
  attachments {
    [
      ActionDispatch::Http::UploadedFile.new(
        filename: 'random.csv',
        type: 'text/csv',
        tempfile: File.new(
          "#{File.expand_path(File.dirname(__FILE__))}/fixtures/random.csv"
        )
      )
    ]
  }
end
...

Again, the random.csv file must exist in spec/fixtures.

Then, in the service spec:

# spec/services/email_processor_service_spec.rb
...
  context "invalid attachment" do
    let(:email_issue) { FactoryBot.build(:email, :with_invalid_attachment) }
    let(:service) { EmailProcessorService.new(email_issue) }
    it "does not file attachments" do
      expect{ service.process }.to_not raise_error
      issue = project.issues.last
      expect(issue.uploads.size).to eq(0)
    end
  end
...

And the test was green! The final case I wanted to test was an email with a mixture of valid (image) attachments and invalid (non-image) attachments:

In the factory:

# spec/factories/email.rb
...
trait :with_valid_and_invalid_attachments do
  attachments {
    [
      ActionDispatch::Http::UploadedFile.new(
        filename: 'img_1.png',
        type: 'image/png',
        tempfile: File.new(
          "#{File.expand_path(File.dirname(__FILE__))}/fixtures/img_1.png"
        )
      ),
      ActionDispatch::Http::UploadedFile.new(
        filename: 'random.csv',
        type: 'text/csv',
        tempfile: File.new(
          "#{File.expand_path(File.dirname(__FILE__))}/fixtures/random.csv"
        )
      )
    ]
  }
end
...

And then the example:

# spec/services/email_processor_service_spec.rb
...
context "valid and invalid attachment" do
  let(:email_issue) { FactoryBot.build(
    :email, :with_valid_and_invalid_attachments
  ) }

  let(:service) { EmailProcessorService.new(email_issue) }

  it "creates only valid file attachments" do
    expect{ service.process }.to_not raise_error
    issue = project.issues.last
    expect(issue.uploads.size).to eq(1)
  end
end
...

Another green spec! I was pretty happy with myself. So I pushed the branch up to GitHub, opened the PR, waited for CI to turn green, and deployed. Time to test in production!

I sent an email to Beacon with file attachments. The issue got created and was visible in the UI, but the attachments weren’t there. And there was an error in the logs: Google::Apis::ClientError: Invalid upload source

I looked at the source for the Google API client, and saw that it was expecting an IO object. But the file attachments from a real email were an ActionDispatch::Http::UploadedFile, which is not an IO object. That’s why my tests passed, but the feature failed in production.

I thought the issue might have to do with temporary file storage locally vs on Heroku, so I went down that rabbit hole for about an hour, setting up a :mirror strategy in my ActiveStorage configuration file to simultaneously write to Google Cloud and a local tempfile. No luck.

After a lot of Googling, I finally ended up in the source code for Http::Upload and saw this method:

def to_io
  @tempfile.to_io
end

That smelled right, so I updated the method in the service class:

# services/email_processor_service.rb
class EmailProcessorService
  ...

  def process_attachments
    email.attachments.each do |attachment|
      next unless VALID_MIME_TYPES.include?(attachment.content_type)
      issue.uploads.attach(
        io: attachment.to_io,
        filename: attachment.original_filename,
        content_type: attachment.content_type
      )
    end
  end

end

So now instead of io: attachment, it’s io: attachment.to_io. Specs were green locally, so I (gasp) pushed to the release branch and sent it up to Heroku. Time to test in production again! And this time, it worked!

Feature complete.

I decided to blog about this because figuring out the right incantation to make this work was a real journey, and I wanted to save others in this situation from having to work it out on their own. I hope someone finds this useful!