Email Hygiene and Spamtrap Removal Process

Verifications.io offers

Each client can scrub 10 million records per hour.

We designed the system with tethers to give each client an equal amount of time to remove threats and verify data in bulk.

We have an mx verification process that checks mail servers for signs of threats to remove bounces and spamtraps. Since we do not send email advertisements or blank emails with this server, we cannot verify top level domains like Hotmail, AOL, Gmail, Yahoo!, MSN and other free email service providers. When our mx system asks about a certain domain, their server will send us an IP and we check it against our existing blacklisted IP’s (spam fighters). If you want top level domain verification, we have that in a separate package.

Sometimes mail server IP’s can go undetected because we only received the first priority IP and it didn’t match our records, thus you may get different results. Our unlimited policy allows you to keep scrubbing to get all of the traps through the IP verification process if you didn’t the first time. We also have a reverse engineering tool that can find new traps only available for unlimited clients.

Our reporting may vary from each scrub because we add and remove new and old IP’s of spam fighters daily. Why – because they do the same. Every day, we have to manually go through hundreds of thousands to look for false positives. This is a daunting task and a process we cannot do programmatically because there are 36 common characteristics of spam fighters that we look for when adding their servers to our blacklist. It definitely takes a human eye.

We can remove bounces with our mx verification and static list (not not all). 70% of our 150 million hard bounces are legit. The other 30% were soft bounces at one time and were re-activated for several reasons:

Again, both the static list and mx verification combined usually removes up to 70% of all hard bounces. It does not remove top level domain bounces. That is our verification server that comes with a completely different price.

We were purposely built our validation server for companies who:

1. Harvest data

2. Have old databases

3. Purchased, traded, or leased their data

4. Email to general Internet data.

5. Are out of touch with their data

Examining the results:

botclickers.csv

Domain Bot Clickers: Malicious emails that are from machines that click on ads; computers that programmatically open email to create false clicks. Considered to be click fraud and can get you banned from Affiliates.

bounces.csv

Matched and removed against a static list of 150 million hard/soft bounces. The bounces were qualified by these attributes:

1. A non-existent email address: The email address could have a typo, the contact gave a false email address or the person with the address may have left the organization.

2. Undeliverable email: The receiving email server is temporarily unavailable, was overloaded, or couldn’t be found. A server that can’t be found could have crashed or be under maintenance, so this may just mean waiting to send the email to the address again. However, if this email address repeatedly bounces on multiple emails, it may mean the server is gone for good.

3. Mailbox Full: The contact has too many emails in their inbox and they cannot receive more. These emails will bounce back until there’s space for them. Sometimes, this can mean that the contact is no longer using that email address.

4. Vacation/Auto Reply: If someone goes on vacation or can’t check their email, emails to them will sometimes bounce. Usually this type of bounce means your email was successfully sent to the inbox. Sometimes when months go by, or the person hasn’t returned from vacation, they turn into bounces.

5. Blocked email: The receiving server has blocked the incoming email. This is often the case among spam fighters, government institutions or schools, where servers can be stricter when it comes to receiving emails.

6. Other: Bounces that don’t give the server a reason for bouncing, so it can mean that the email bounced for one of the reasons above or something else entirely.

If you test this bounce suppression file, you will find that there are good emails inside of this. The reasons are explained above.

Your file that is clean and ready to mail. The name of the file will have the word “clean” in front of it followed by the name of the original file and the date that we cleaned it.

departments.csv

Departmental Emails: Most of these emails are not a person, rather a group or department of several persons. In example, info@, webmaster@ or team@. Spamtraps are created daily using these so called “dictionary spam attacks”.

foreign_emails.csv

Foreign Domain Endings: Matched and removed against international domains. i.e. test.com.ar or test.com.ru and emails with foreign endings like .da, .ta, .ca, too. These are not likely threats but nice to have removed if you’re only mailing into the United States.

Email marketers combined all suppressions together and never separated them until around 2006. Unsubscribes, bounces, protestors, traps, litigators and more are located inside these old and bulky lists totaling 9 GB of data (billions).

mx.csv

A mail exchange record (MX record) is responsible for accepting email messages on behalf of a recipient’s domain name (DNS), and the value used to prioritize the delivery. The MX record specifies how emails should be routed via Simple Mail Transfer Protocol (SMTP). The MX mechanism provides the ability to run multiple mail servers for a single domain, and allows administrators to specify an order in which they should be tried. Bad MX emails have slow response times, 4xx errors, full load distribution, the mail exchange isn’t configured correctly, has incorrect port numbers, no backup server, is parked or the website doesn’t follow typical mail exchange setup standards. MX does remove bounces but not from top level domains.

protestors.csv

The industry used to call them “screamers” or “complainers”. We disagreed because of political correctness, so we changed the name to “protestors”. These are people who gripe directly to the advertiser, ISP or spam fighters. We have a static list of 60 million that we match and remove from. They are not unsubscribes.

report_filename_date.pdf

An “at a glance” PDF report explaining every detail about your list and what we removed.

spamcop_traps.csv

SpamCop is an email spam reporting service, allowing recipients of unsolicited bulk or commercial email to report IP addresses found by SpamCop’s analysis to be senders of the spam to the abuse reporting addresses of those IP addresses. SpamCop emails can only be hit instead of found. We work with hundreds of mailers who can detect these emails with their unique links (code) in which we update our static list daily.

spamtraps.csv

Traps matched against 465 spam fighters and the mail servers they use to trap email marketers. These are spam traps that our system removed from matching spam fighter IPs in the mx scrubbing option. There are many types of spam fighters. The first type is made up of a single person or group of people who created an email server in order to capture unsolicited spam (black holes). They use this data to study and interpret advertising language to build better spam filtering systems. Once they have collected the data, they start blocking text, domains and IPs, as well as openly posting their findings online. The common goal of the advisories, whether large or small, is to create a 99.999 percent spam-free network. In order for their filtering system to work, they must receive spam on a daily basis. This is why many publishers hit spam traps and get reported. They end up finding themselves in an advisories’ black hole.

For a detailed explanation on each spam advisory, go to

http://www.ehygienics.com/spam-advisories.pdf

threat_domains.csv

These are domains of protestors, spam fighters, temporary emails, misspelled domains, rotating IPs and dozens of other threats. Yes, there
are some good domains in there. You can choose what you want to mail to or not. If a domain is added to the blacklist domains, it means they can do harm to bulk email marketers.

threat_endings.csv

Matched and removed against federal, state, local governmental email addresses, education facilities (colleges) and other proprietary threats.

threat_string.csv

Swear words, spam fighter codes, bots, misspellings and other threats.

False Positives & Competition

The suppressions of other hygiene companies or ESP’s are of their own technology and research. Every hygiene process (company) is different. What they consider a trap, we may not or vice versa. That’s the funny thing about email list hygiene. In example:

Our system is a collection of old/new traps and honeypots with the exception of some false positives (but not many) along with private investigative work. Let’s say you want to test the suppressions by calling and/or emailing them (what we removed). You may confuse yourself because the honeypots and traps are in there but have not reported you “yet”. The suppressions you are testing are or were:

• Complainers that led to traps (contacted the ISP admin about spam and the ISP reported to blacklist). We can’t help the fact that a threat you called can’t remember they sparked such a complaint. Would you remember reporting spam let’s say back in 1999? • Legacy traps at one point since 1996 were part of some spamfighting group. Many spam-fighting associations have recruited and requested common people to enter their email to contribute. 5 years later, they can’t even remember they did that. • Emails on a shared server with spam fighter control (spam fighters use their clientele traffic to report spam). The consumer is unaware that they ever were being used for spam fighting. This is quite common. • At one point hit the report spam button on some email UI and it escalated into a trap. • A spam fighter lying to you because they do not want you to know they are a trap or set them. Most spam fighters will monitor your
traffic for a while to investigate you BEFORE they put the hammer down. Their goal is to find others that you work with and not just you.

Traps and honeypots are not always what you think they are. To determine a trap or honeypot is accurate, you will need the following knowledge:

• Understand that what doesn’t appear to be a trap could harm your business unless you understand the above bullet points. • In order to find honeypots or traps, you will need to do private investigative work to uncover spam fighter networks. This requires being a mole/friend and slowly (undercover infiltration) press for information or by researching the groups and friends they belong to. This requires phone calls, emails, texts and friendship building along with a false identity to keep anonymity. • Patience because most spam fighters are already using false identities in order to hide from angry spammers. You need time to build trust. • The information you receive from the spam fighter could be real or false. You would need to understand the 23 common characteristics of spam fighters in order to determine if the information is adequate for suppression or not.

You can easily find traps and honeypots by setting code inside the “click here” link. When a feedback loop sends you the complaints, you can gather the email that was triggered through code in the link. Remember, that spam fighters or blacklists will redact the email because they do not want you to know the traps so you can suppress them.

Also, spamtraps are not all email related. Spam fighters use these techniques to trap an emailer:

• Email (hiding email addresses on websites) • Filling out forms • Buying databases and reselling them (planting seeds and traps) • Pay data providers to seed lists or has some form of agreement to do the needful • Buying email addresses and activating them for traps purposes • Uses inactive email accounts • System administrator activates hard bounces • Hides on shared servers and uses unaware consumer/business emails as bait • Monitors whois, privacy and date the domain that is sending bulk email’s age
• System administrators monitor traffic from each domain or IP range and can report spam without even hitting a trap based on the numbers of email coming from one alleged source • Blackmail ESP’s, ISP’s and the like for information • Activation of the SMTP from expired domains (real common)

We do not trust the majority of other hygiene processes only because they haven’t seen the side that we have from all of our research. Traps are not just email addresses. The data that we collect and remove follow all of our guidelines from the research that we have done on how spam fighters setup their networks and the 23 common characteristics of their behaviors in code and system integration