I implemented an "open code of conduct issue via email" feature on Beacon recently, and today I set out to allow reporters to use file attachments to add screenshots to the issues they report. This meant that I would have to figure out a way to process and store email attachments with ActiveStorage, which turned out to be An Adventure.
I'm using MailGun to allow Beacon to receive emails, and the ActiveStorage feature in Rails 5 to handle file uploads to a Google Cloud bucket. All of the documentation that I could find for ActiveStorage assumed an upload-via-form use case, which wasn't applicable for me.
I already had a service class called EmailProcessorService to handle processing incoming email and creating an Issue object, so now I needed to add a method to process attachments. The service initializes with an Email instance, provided by MailGun. So I added the new method:
# app/services/email_processor_service.rb
class EmailProcessorService
attr_reader :email
def initialize(email)
@email = email
end
...
def process_attachments
email.attachments.each do |attachment|
# do something
end
end
end
I actually did TDD for this feature, so the next thing I did was to extend my Email factory (using FactoryBot of course) to include an attachment:
# spec/factories/email.rb
FactoryBot.define do
factory :email, class: OpenStruct do
to {
[
{
full: 'sample_project@issues.coc-beacon.com',
email: 'sample_project@issues.coc-beacon.com',
token: 'sample_project', host: 'coc-beacon.com',
name: nil
}
]
}
from { {
token: 'reporter',
host: 'foo.com',
email: 'reporter@foo.com',
full: 'Reporter ', name: 'Reporter'
} }
subject { 'CoC Issue' }
body { 'Something bad happened.' }
attachments { [] }
trait :with_valid_attachment do
attachments {
[
ActionDispatch::Http::UploadedFile.new(
filename: 'img_1.png',
type: 'image/png',
tempfile: File.new(
"#{File.expand_path(File.dirname(__FILE__))}/fixtures/img_1.png"
)
)
]
}
end
I needed to add an image to spec/fixtures/, so I took a screenshot and saved it there as img_1.png.
Now, in my spec, I could write an example for an email with an attachment:
# spec/services/email_processor_service_spec.rb
require 'rails_helper'
describe EmailProcessorService do
let!(:project) {
FactoryBot.create(
:project,
account_id: FactoryBot.create(:account).id,
name: "Sample Project",
accept_issues_by_email: true
)
}
let!(:issue) {
FactoryBot.create(:issue, project_id: project.id, issue_number: 101)
}
context "with attachment" do
context "valid image attachment" do
let(:email_issue) { FactoryBot.build(:email, :with_valid_attachment) }
let(:service) { EmailProcessorService.new(email_issue) }
it "creates file attachments" do
expect{ service.process }.to_not raise_error
issue = project.issues.last
expect(issue.uploads.size).to eq(1)
end
end
end
After some iterating, I got the spec to green. Yay!
As I process the email attachments, I want to attach image-type attachments as uploads on an Issue. Here is where that is defined:
# app/models/issue.rb class Issue < ApplicationRecord has_many_attached :uploads, dependent: :destroy end(This is straight from the ActiveStorage docs.) To process the attachments and create the associations, I used the
attach method:
# services/email_processor_service.rb
class EmailProcessorService
...
def process_attachments
email.attachments.each do |attachment|
issue.uploads.attach(
io: attachment,
filename: attachment.original_filename,
content_type: attachment.content_type
)
end
end
end
Next, I wanted to make sure that the file attachments were valid images. I couldn't rely on the validations that I had in the Issue model, since there would be no way to alert the user (in this case, the email sender) of a validation error. So I decided to do inline validation in the service, and only process attachments from an allowed set of MIME types. So in the service class:
# services/email_processor_service.rb
class EmailProcessorService
VALID_MIME_TYPES = [
"image/gif",
"image/jpeg",
"image/png"
].freeze
...
def process_attachments
email.attachments.each do |attachment|
next unless VALID_MIME_TYPES.include?(attachment.content_type)
...
end
end
end
Nice! Now to create an example to prove that an unsupported MIME type wouldn't be attached to an Issue.
First the factory:
# spec/factories/email.rb
...
trait :with_invalid_attachment do
attachments {
[
ActionDispatch::Http::UploadedFile.new(
filename: 'random.csv',
type: 'text/csv',
tempfile: File.new(
"#{File.expand_path(File.dirname(__FILE__))}/fixtures/random.csv"
)
)
]
}
end
...
Again, the random.csv file must exist in spec/fixtures.
Then, in the service spec:
# spec/services/email_processor_service_spec.rb
...
context "invalid attachment" do
let(:email_issue) { FactoryBot.build(:email, :with_invalid_attachment) }
let(:service) { EmailProcessorService.new(email_issue) }
it "does not file attachments" do
expect{ service.process }.to_not raise_error
issue = project.issues.last
expect(issue.uploads.size).to eq(0)
end
end
...
And the test was green! The final case I wanted to test was an email with a mixture of valid (image) attachments and invalid (non-image) attachments:
In the factory:
# spec/factories/email.rb
...
trait :with_valid_and_invalid_attachments do
attachments {
[
ActionDispatch::Http::UploadedFile.new(
filename: 'img_1.png',
type: 'image/png',
tempfile: File.new(
"#{File.expand_path(File.dirname(__FILE__))}/fixtures/img_1.png"
)
),
ActionDispatch::Http::UploadedFile.new(
filename: 'random.csv',
type: 'text/csv',
tempfile: File.new(
"#{File.expand_path(File.dirname(__FILE__))}/fixtures/random.csv"
)
)
]
}
end
...
And then the example:
# spec/services/email_processor_service_spec.rb
...
context "valid and invalid attachment" do
let(:email_issue) { FactoryBot.build(
:email, :with_valid_and_invalid_attachments
) }
let(:service) { EmailProcessorService.new(email_issue) }
it "creates only valid file attachments" do
expect{ service.process }.to_not raise_error
issue = project.issues.last
expect(issue.uploads.size).to eq(1)
end
end
...
Another green spec! I was pretty happy with myself. So I pushed the branch up to GitHub, opened the PR, waited for CI to turn green, and deployed. Time to test in production!
I sent an email to Beacon with file attachments. The issue got created and was visible in the UI, but the attachments weren't there. And there was an error in the logs: Google::Apis::ClientError: Invalid upload source
I looked at the source for the Google API client, and saw that it was expecting an IO object. But the file attachments from a real email were an ActionDispatch::Http::UploadedFile, which is _not_ an IO object. That's why my tests passed, but the feature failed in production.
I thought the issue might have to do with temporary file storage locally vs on Heroku, so I went down that rabbit hole for about an hour, setting up a :mirror strategy in my ActiveStorage configuration file to simultaneously write to Google Cloud and a local tempfile. No luck.
After a _lot_ of Googling, I finally ended up in the source code for Http::Upload and saw this method:
def to_io @tempfile.to_io end
That smelled right, so I updated the method in the service class:
# services/email_processor_service.rb
class EmailProcessorService
...
def process_attachments
email.attachments.each do |attachment|
next unless VALID_MIME_TYPES.include?(attachment.content_type)
issue.uploads.attach(
io: attachment.to_io,
filename: attachment.original_filename,
content_type: attachment.content_type
)
end
end
end
So now instead of io: attachment, it's io: attachment.to_io. Specs were green locally, so I (gasp) pushed to the release branch and sent it up to Heroku. Time to test in production again! And this time, it worked!
Feature complete.
I decided to blog about this because figuring out the right incantation to make this work was a real journey, and I wanted to save others in this situation from having to work it out on their own. I hope someone finds this useful!