Emailer & Messaging Service

 Back in the mid 2000's Marketpath deployed a custom email service that more easily allowed custom applications to send emails with two primary goals in mind:

1. Create a common API for apps to send emails with just a couple lines of code. This avoids all the custom configuration and potential points of risk common to those early SMTP senders.
2. Allow for gracefull fallback if an email fails to send for some reason.

In the old days with IIS we'd set up an individual SMTP server on each server hosting web applications. Web apps would drop emails into a queue and send them one at a time. This was a generic service that worked fairly reliably but as SPAM services became more prominent we began having deliverability issues.

So, we centralized all email sending from one server, one IP address, and one email domain. The goal was to build email and IP reputation. The downfall is that all emails came from "emailer@marketpath.com". Recipient email headings looked like this:

To: somebody@mydomain.com
From: ABC Company [emailer@marketpath.com]

The From name could easily be changed but the from email had to be emailer@marketpath.com. Replies were handled using the reply-to field set to the customer's preference.

SPAM and Email Verification

Anti-SPAM initiatives really began turning up the heat with SPF and DKIM in the early 2010's. So we switched from a generic SMTP sender to Sendgrid. I don't actually think I set up DKIM for several years but eventually that was added in addition to the SPF.

This was so long ago (now being 2023) I don't remember all the timing and details. But needless to say, this single from email address method had to go. Our customers wanted to send emails from their own domains. Fair enough.

Security Issues

The old emailer app also used one single secret, shared between all apps that used it. Our intent was to change this eventually, but we never got around to it. We were really great at growing technical debt at the time (and sometimes still are).

We wanted unique authentication for each application using the service so we could turn it on or off in case of a big email sending spike.

Architecture

The old emailer was built with

• ASP.Net, .Net Framework, C# (for the API)
• Windows NT Service (for the agent)
• SQL Server Database

 

In November 2020, I completely rebuilt the app from the ground up using .Net Core. Every app that used it was required to have its own credentials. The high-level architecture was similar to the old Emailer but what it did under the hood was very different.

Architecture

The new messaging service was built with

• ASP.Net Core WebAPI, C#
• MongoDB (and then converted to Azure SQL)
• Azure Web Job

MongoDB back to Azure SQL

Initially, I used MongoDB as the database. Partly, I wanted the challenge of learning the popular database platform. I also thought it was a good fit for handling a large number of emails.

I used this service through Azure Cosmos DB and ran direct MongoDB queries. I don't recall where I read this but I thought MongoDB (or was it Cosmos DB in general) performed auto-indexing on its data. But that isn't really true. I had to specify an index on any field I wanted to use for lookups. I also wasn't a big fan of conducting searches across containers. Maybe SQL server is so ingrained into my brain that I just couldn't make the connection, but I struggled running queries on data between partitions. MongoDB is not relational, so there's that.

About a year later, when MongoDB began having epileptic fits, I converted this to an Azure SQL database. Ahhh, what a relief. I can see MongoDB for big data dumps of single track data. But it just didn't serve me well for related and normalized data.

Multiple Senders

This time, I built in the ability for each custom app to have a different transactional sender. I built in capability for both Sendgrid and Postmark. We had some issues with Sendgrid allowing spam across its shared IP addresses. We made the mistake of not purchasing dedicated IP's. Lesson learned.

Custom From Addresses

The new service allowed each sending application to have a specific from email address, rather than the old emailer@marketpath.com address. For each email, we have our customers set up both SPF and DKIM on their DNS servers.

Tracking Email Events

All emails are tracked - opens, link clicks, etc. We set up webhooks that allowed both Postmark and Sendgrid to report these events back to the Messaging service. The Messaging service also allowed each custom app to provide its own webhooks for this reporting, if desired.

We immediately put this to action with our support service. Now we know when support messages were opened or if they bounced or had some other issue.

Future Updates

The new messaging service is pretty reliable. The old one had to be restarted every 4-6 months or so and I never figured out exactly why. The new service hasn't been restarted since it's last update in March 2022, although it is possible Azure performed updates requiring a restart.

The only updates I can see making are providing some sort of UI to view sent emails and their deliverability statuses. For now, the new service is secure, efficient, and reliable.