background

When an email is sent to a non-existent email account, or the email inbox cannot receive the email due to a problem with the email account, the email server will respond with the message that cannot be received. This situation is called bounce email. The corresponding measure is bounce rate. Bounce email is an important factor in the email delivery rate. According to Sendgrid statistics, the bounce rate is more than 5% and the delivery rate is 71%. But if the bounce rate is 2 percent or less, the average delivery rate can increase to 91 percent.

At present, our platform sends tens of millions of emails every month, including notification emails and marketing emails, of which marketing campaigns account for the majority. Because marketing Campaigns will allow clients to customize contacts, this is a risk for potential bounce emails, so check whether the recipient’s email address is deliverable before sending the email and filter out junk and invalid email addresses. It can help reduce bounce rates. In this article we will detail how to improve delivery rates by validating email addresses and best practices.

Why does Bounce affect email delivery rates

According to Sendgrid’s statistics on bounce email above, the correlation between bounce rate and delivery rate can be clearly seen. But the correlation is not just that bounce makes up a large proportion of the total number of emails sent, but that bounce makes up a large proportion of the total number of emails sent.

Each email account has a sender reputation score, which helps the recipient’s email service provider (ESP) determine the quality of the message. The higher the score, the higher the delivery rate, and vice versa. Frequent bounce will cause the receiving Email Server to “question” whether the sender’s Email account is the real one. When it reaches a certain point, the sender account will be listed in the spam index of various ESPs, and finally sent to other users, it will be blocked. And bounce affects sender domain and IP reputation.

Therefore, email bounces can be a “nightmare” for marketing campaigns. Verifying email addresses helps send emails to the right recipients, keeps email accounts available, and improves Reputation. For business, it will also improve the quality of email campaign.

How do I verify email addresses

The complete email address verification process mainly includes the following four dimensions:

  1. Syntax checking
  2. Check whether the mailbox is disposable
  3. Check whether the DNS record of the domain exists
  4. Ping Recipient Email addresses

Syntax checking

Grammatical errors in spelling are one of the most common problems in email address checking. According to the common email address regular expression, you can determine whether the address is formatted. The expression for a general check is similar to [email protected] and consists of three parts: local part, @ delimiter, and domain.

The local part shall be inspected with emphasis, which consists of the following three parts:

  • Alphanumeric – A to Z (both uppercase and lowercase), 0-9
  • Printable character —! # $% & '* + - / =? ^ _~ `
  • A punctuation mark.– The local part cannot be used.Beginning or end, or continuous, for exampleexample.. [email protected]It is an invalid email address.

Different email service providers have different regulations on local Part. Here, Mailgun provides common ESP verification rules.

Domain is the same as the naming convention for domain names: only alphanumeric and – symbols are used, and cannot be used continuously.

In addition to checking email addresses against regular expressions, some points to consider are IETF standard and non-ASCII Domains.

Check whether the mailbox is a disposable mailbox

Disposable emails are those email addresses that are valid for a short period of time and are called disposable emails. Disposable Email is an address with correct syntax and can receive and send emails. Normally, it can only be used once. It is generally used to register a new account to bypass login and send malicious emails.

Common Disposable Email providers include Nada and Mailinator, etc. The way to identify them is to determine whether the domain is a Disposable domain. Currently the open source community maintains a number of disposable Domain lists that are updated in real time and can quickly filter Out Diposable email by searching for Domian in the list.

Check whether the DNS record of the domain exists

DNS query is the process of requesting DNS records from the DNS server. DNS records include a variety of domain records. MX Record is used for confirmation. This resolution record is used to specify that the email server receives emails for a specific domain name. For example, we can query DNS records for Aftership.com as follows:

You can see that aftership.com has four MX records for each pair. If the MX record exists, the ESP corresponding to the domain exists. Otherwise, it is not a valid email address.

Ping Recipient Email addresses

After confirming the existence of the MX Record, you can establish a connection with the SMTP server to verify the validity of the email address. As shown in the previous step, there are usually multiple MX records (some SMTP servers will set the weight value of record). The ADDRESS of the SMTP server is: MX records + SMTP Relay port. The principle of Ping recipient email addresses is to use SMTP to connect to a valid SMTP server and request the recipient address. The server responds to the request and determines whether the recipient address exists based on the response information.

If the SMTP server responds with 250 OK, the email address may be a valid address. An invalid address is returned if an error similar to 550-5.1.1 is returned.

Take [email protected] as an example, the following is a complete SMTP connection authentication process.

First, Telnet is connected to the recipient’s SMTP server and ehLO is used to identify the sender. Mail is used to set the email sender. Finally, RCPT is used to set the email recipient. RCPT can only set the email address of the SMTP domain. Otherwise, the SMTP rewriter overwrites the email address in the mail to match the primary SMTP address. If RCPT does not reject the request, the SMTP server verifies the address and adds the recipient to the mail list. The following is a complete flow chart of sending email using SMTP:

The SMTP Ping method is probably the most effective part of the email address verification process. The SMTP server can help you verify whether the recipient exists and is reachable. Note that the “possible” here is that [email protected] is actually an invalid address, RCPT response 250, and the email address is not necessarily reachable.

A catch-all Email Server, also called an accept-all Server, is an SMTP Server configured to accept All Email sent to the domain, regardless of whether the specified Email address exists. Catch-all redirects the incorrect address to a generic mailbox for periodic review, but the problem is the risk of improving bounce rates and catch-all addresses not being validated correctly. (E.g. Gmail is a catch-all email server)

Therefore, to verify the validity of an address, Ping the recipient’s EMAIL address to ensure that the SMTP server of the recipient is not a Catch-all email server and the RCPT command responds to 250, indicating that the address is deliverable. Otherwise, the reacability cannot be verified.

For more details about the verification process after connecting to the SMTP server, such as why RCPT is used instead of other commands, see Email Box pinging.

When do I need to verify the email address

Verification of email addresses may not be a regular process. It is recommended that verification be performed in the following situations:

  1. New Email addresses: As mentioned above, the email addresses added to the contacts recipient list must be verified during the marketing campaign to filter out invalid and illegal recipient accounts
  2. Email addresses that have not been re-verified for more than one month
  3. Bounce rate reaches or exceeds 2% : Set the bounce rate threshold to ensure the rate of mail delivery and improve the Sender reputation
  4. The probability of email being opened is low

Local authentication of emIL addresses vs using a third-party email authentication service

After the above steps to verify the entire email list, even if there is only one address verification will take a lot of time. However, manual verification is not necessary because there are many third-party email verification services that provide apis for verifying addresses. Several similar services (such as EmailChecker) were investigated and provided the following features:

  1. Domain calibration
  2. Verify a single email address
  3. Batch email address verification
  4. Syntax checking
  5. SMTP verification (Ping recipient email address)
  6. Provide validation APIS
  7. Bounce email detection
  8. GDPR data protection

So which of these two validation schemes is the better choice? Local validation email addresses are definitely preferred because self-validation is actually faster and better supports batch validation mailing lists; Note a lot of good third party validation service is paid, the online authentication need to confirm whether the service has GDPR data protection to ensure that won’t share users’ personal data with a third party, or security problems, but the third party check there will be no restrictions (most isps prohibit outbound connections on port 25), The IP address segment and domain name reputation are not affected.

email-verifier

For local verification of email addresses, there are several open source verification tools available in the community, with the largest number of stars being the Trumail project, which provides an ADDRESS verification API. But this project has two problems, one is slow verification, some performance problems; Second, the verification of Disposable domain is not supported, and archived of this project is no longer maintained.

In the development and maintenance of the Messages Platform, as the Platform side, in addition to providing highly available, simple and easily accessible email message channel service for business, we also reduce the bounce rate and improve the rate of mail delivery is one of our important indicators. Therefore, we need to have an efficient mail verification service to filter out illegal email addresses (the average amount of mail sent on the platform is 1000+ W letters/month), so as to improve the delivery rate. Based on our technology stack is Go, after investigating other open source email verification tools in git community, it is found that the Go project is relatively lacking in the construction of email verifier. Currently, there is no tool that provides multi-dimensional email checking (including Diposable domain, Role Account, etc.) and check address reachability.

Since TRUMail is no longer maintained, we have internally implemented a new email verification library – email-Verifier, which is currently running on the online environment. Compared with TRUMail, email address verification is more efficient and has more dimensions.

Compared with other existing email address verification tools, we provide high-performance, multi-dimensional and high-accuracy free email address verification solutions to solve the pain points of email address verification in different scenarios. Expect email-Verifier to help the community solve similar problems in a better way.

conclusion

This article introduces bounce Email. It explains how to verify email addresses without sending them. It also gives some suggestions for checking email addresses at appropriate times. Compare the advantages and disadvantages of local and third-party verification services and why we choose self-built verification services. Finally, we incubated a testing tool based on the verification principle during this process.

Generally speaking, after Marketing Campaigns are launched, bounce email will surely affect the quality of Campaigns. In this case, it is easy to understand the advantages of checking the effectiveness of the address before sending: first, improve the delivery rate of mail; Second, maintaining and improving sender account reputation is necessary for both the business side and the platform side.

reference

  • AfterShip/email-verifier
  • How to Verify Email Address Before Sending Cold Emails
  • Sendgrid Bounces
  • 202 Email Deliverability Guide
  • Amazon SES and delivery rates
  • How to Verify Email Address Without Sending an Email
  • mailgun/flanker
  • disposable/disposable
  • Recipient resolution in Exchange Server
  • The MAIL, RCPT, and DATA verbs
  • What is a Catch-All Email Server
  • SMTP protocol Explained