742889_59607844a

This is an extra, out-of-normal-time blog post to explain the new anti-spam policies that I’ve been forced to implement here at Campaign Mastery.

The real price of Spam

Spam is an unfortunate reality. It will never go away.

Most of the time, Spam is like an itch that has to be scratched. But there are times when excessive spam floods in, and it becomes the equivalent of an attempted denial-of-service attack on a small one-man website, consuming hours of what would otherwise be productive time. And, even more rarely, its a direct attack aimed at bringing a website down, an attempt to discover and utilize a vulnerability of the website’s architecture.

To cope with these last two situations, a formal anti-spam response policy has had to be put in place, and will be subject to change without notice if it doesn’t work, or if something better comes along.

Spam Alert Level: Green

A reasonable level of Spam comments being submitted to any public website is both expected and will be tolerated. As a general rule, Spam comments will be deleted and will never be visible to the public. Given the number of hits CM receives, that number is somewhere in the vicinity of 20 every 6 hours or so, or 80 a day.

Spam Alert Level: Amber

From time to time, Spam levels – mostly driven by spambots – will get out of hand and the number of spam submissions will skyrocket. Again as a general rule of thumb, a day or two will be allotted for things to calm down of their own accord, which happens about half the time, in my experience. During this period, the only change from the normal state of affairs is that there will be too many spam comments for me to go through them all checking that none are ‘false positives’ from the spam test. The occasional genuine comment might get tagged as spam, and that’s unfortunate, but it’s the best compromise that I can make with the policy of keeping the site itself as open as possible.

If the problem is too major (more than 250-300 a day), or persists for too long, I’ll go to Spam Alert Level Red.

Spam Alert Level: Red

About half the time, the problem will not go away as a result of timely action on the part of the people responsible for the servers on which the spam originates. That’s when it’s time to get serious.

On the theory that the worst offenders will be more prevalent in any given “slice” of the spam being received, a number of originating IPs will be blocked. Typically, this will be 20-60. Every 6 hours or so, a new batch of 20 will be added to the blocked list, until the spam count reduces to the manageable level, triggering a shift to Spam Alert Level Blue.

These IP addresses are those which originated a spam comment.

Any server so blocked that has produced fewer than 2 hits in the preceding 48 hours will get immediately unblocked, because blocking so few is not worth the imposition on the public.

The remainder are assessed periodically. I have drawn up a table (later in this post) of allowable blocked hits in given time frame relative to the number of hits in the 48 hours prior to the commencement of blocking; if this number of attempted blocked hits or less is received, the site will be unblocked. If more than the tolerable level are received, the IP stays blocked.

Any blocked site that goes 24 hours without attempting to access the site will also be unblocked.

Spam Alert Level: Blue

This is exactly the same as Red except that I stop adding new IPs to the blocked list. It signifies that the response has achieved its goal of stopping the spam deluge, and that it’s time to start inching back from the draconian blocking of IPs. One by one, as the targeted VIPs stop delivering spam, the blocks get lifted. If a resurgence in spam levels follows, I’ll go back into Alert Level Red mode again.

Eventually, only the worst offenders will remain. If spam levels have remained at the tolerable level for 48 hours with everyone else unblocked, I’ll also unblock these – but be ready to reinstate the blocks if necessary and restart that clock.

Past experience has shown that Alert Level Red typically lasts for 24-48 hours, and Blue for another 2-3 days. I try to err on the side of keeping access open, and restore it s quickly as possible.

Spam Alert Levels: Violet and Black

I’ve never had to go this far, but if Red persists for a week, I’ll go to Alert Level Black. If Blue persists for a week with no prospect of an imminent reduction in Alert level, I’ll go to Violet.

Violet

Violet means that the worst offenders – those with more than say, 100 blocked hits in a 24 hour period for multiple days running – will be permanently blocked and – with the exception of that blocking – the rest of the site will go back to Green.

Black

This indicates that this anti-spam policy has failed, and left me with only one recourse: closing posts older than a couple of weeks to comments. If this produces the immediate reduction in Spam expected, comments may be reopened in a week or two on a trial basis. If necessary, the prohibition will remain permanent.

Because this will change the level of opportunity for spambots to affect the site, while the prohibition remains in effect, a less-tolerant set of spam figures will be devised.

The nitty-gritty

What are the numbers that I’m using to assess unblocking?

  • <2 hits prior to blocking: immediate unblock.
  • 6-8 hours after blocking:
    • <4 hits prior to blocking and no blocked hits: unblock.
    • 4-8 hits prior to blocking and no blocked hits: check again at end of 8 hours. If still no blocked hits, unblock.
    • >9 hits prior to blocking: IP remains blocked.
  • 12-16 hours after blocking:
    • Any hits prior to blocking and no blocked hits: unblock.
    • <5 hits prior to blocking and 1 blocked hit total: unblock.
    • 5-10 hits prior to blocking and 1 blocked hit total more than 6 hours old: unblock.
    • 11-15 hits prior to blocking and 1 blocked hit total more than 10 hours old: unblock.
    • >16 hits prior to blocking: IP remains blocked.
  • 16-20 hours after blocking:
    • <11 hits prior to blocking and 1 blocked hit total more than 4 hours old: unblock.
    • <11 hits prior to blocking and 2 blocked hit total, last one more than 6 hours old: unblock
    • <11 hits prior to blocking and 3-4 blocked hits total, last one more than 10 hours old: unblock
    • <11 hits prior to blocking and >4 blocked hits total: IP remains blocked.
    • 11-15 hits prior to blocking and 1 blocked hit total more than 8 hours old: unblock.
    • 11-15 hits prior to blocking and 2-4 blocked hits total, last one more than 10 hours old: unblock.
    • 11-15 hits prior to blocking and >4 blocked hits total: IP remains blocked.
    • 16-25 hits prior to blocking and <3 blocked hits total, last one more than 8 hours old: unblock.
    • 16-25 hits prior to blocking and 3-5 blocked hits total, last one more than 10 hours old: unblock.
    • 16-25 hits prior to blocking and 6-10 blocked hits total, last one more than 12 hours old: unblock.
    • 16-25 hits prior to blocking and >10 blocked hits total: IP remains blocked.
    • >25 hits prior to blocking: IP remains blocked.
  • 20-24 hours after blocking:
    • <20 hits prior to blocking and 1 blocked hit total more than 4 hours old: unblock.
    • 20-30 hits prior to blocking and <6 blocked hits total, last one more than 6 hours old: unblock
    • 20-30 hits prior to blocking and 6-8 blocked hits total, last one more than 8 hours old: unblock
    • 20-30 hits prior to blocking and 9-12 blocked hits total, last one more than 12 hours old: unblock
    • 20-30 hits prior to blocking and >13 blocked hits total: IP remains blocked.
    • 30-50 hits prior to blocking and <6 blocked hit total more than 8 hours old: unblock.
    • 30-50 hits prior to blocking and 6-12 blocked hits total, last one more than 12 hours old: unblock.
    • 30-50 hits prior to blocking and >13 blocked hits total: IP remains blocked.
    • >50 hits prior to blocking and <6 blocked hits total, last one more than 12 hours old: unblock.
    • >50 hits prior to blocking and 6-10 blocked hits total, last one more than 16 hours old: unblock.
    • >50 hits prior to blocking and >11 blocked hits total, last one more than 20 hours old: unblock.
  • apr. 48 hours after blocking and every 12 hrs thereafter:
    • <30 hits prior to blocking and last blocked hit more than 8 hours old: unblock.
    • 30-50 hits prior to blocking and last blocked hit more than 12 hours old: unblock
    • >50 hits prior to blocking and last blocked hit more than 16 hours old: unblock

This policy focuses on weeding out the trivial contributions to the spam count early on, and thereafter matching a required “spam free” period with a scale of initial traffic.

What This Means For You:

Hopefully, nothing. The odds are pretty huge that any real-life user will notice any difference whatsoever. There is always an outside chance that the originating IP belongs to something critical to your ability to see the website, in which it may become inaccessible to you, temporarily. I would expect you to see a “403” error message, “access refused”, if that happens. In which case you know that your ISP has been whacked on the head because I’ve detected spam coming from it. Complain to your technical department (politely, because they may already know about the problem and be in the process of pest control), point them at this article, and ask them to see if spam coming from their server is the cause of your problem. When the spam stops, normal service will be resumed.

I can’t police the entire internet, and shouldn’t have to. It’s up to each individual customer of each internet provider to police their own little corner of the Netiverse.

And remember that I’ll only block access as something close to a last resort.

Where are we now?

Last week, following an update to the latest version of all plugins and WordPress itself, Spam began to skyrocket. Within 24 hours, it was running at 15 times the usual rate, or about 20 an hour. I immediately went to Alert Level Amber, and things stabilized for a while. Gradually, though, the spam levels continued to climb, and over a six hour period on July 10, topped 300 for the first time during this Amber Level. Accordingly, I indicated in a footnote to Thursday’s post that I was instituting the blocking of servers identified as spamming the website.

Since I wanted to allow a little time for the word to get out, so that if the site went dark for someone there would be people out there who would know why, I delayed instituting Alert Level Red for several hours. At 6 AM this morning, an initial batch of 50 IP addresses were blocked, 10 of which were immediately unblocked as making a trivial contribution to the problem.

Six VIPs (and I’m not going to list them) made an immediate impression. They were responsible for, respectively (in order of blocking) 68, 105, 840, 70, 80, and 92 attempted accesses to the site over the 48 hours prior to the blocking. There were a number of others in the 10-20, 20-30, 30-40, and 40-50 hit range, and about a third of the blocked sites had 9 or less, but those six were the big attention-getters. In the eight hours since, these six produced 5, 3, 134, 60, 1, and 6 blocked attempts to access the site. That tells me that two, perhaps 4 of the big six now have their spam problem under control, but the other two are still running at an unacceptable level. In addition, another site that was in the 10-20 range yielded a noteworthy 15 blocked attempts to access the site. By far, the majority was 1, 2, or 3 failed attempts, indicating that many of these blocked sites will have access restored less than a day after it was blocked. So we’re heading for Alert Level Blue at the moment, but aren’t there yet. There was a definite drop in spam levels – to 200 in that eight hours – but it’s still way over the threshold.

At the same time as these numbers were being checked and documented, another 31 VIPs were blocked. I don’t have numbers yet for blocked attempts from those, but there are three attention-getters which logged 97, 60, and 452 attempts to access the site over the preceding 48 hours. So three or four of the initial big six may be about to drop out of the hostile category, but there appear to be three more to take their place. I will continue to monitor the situation, but as of right now, I’m still in Spam Alert Level Red.

An Update:

At the 12-hour mark, half of the initial 40 VIPs that were blocked were released. At the 8-hour mark for the second batch of blocked VIPs, 11 of the 31 were unblocked, and another 14 blocked. Spam dropped from over 30 per hour to about 10 an hour. That’s right, less than 1 day of this protocol and my spam problem has been cut by more than 60%, and I am now officially at Alert Level Blue – unless there’s a big spike overnight. And there’s been no visible dent whatsoever in my real-person hits as a result, indicating that to most of you, this whole “war” has been invisible – exactly as it should be!

I’m not actually going to post this until after the 18-hour update for batch one. So there will be one more update before anyone gets to read this – due about 2.5 hours from now. I’m predicting a spam count at that time of 20-30 at worst, and a more likely result of 15-20. Keep reading to see how accurately I’ve called it…

A Second Update:

One more of the original blocked batch has been released, and the total spam received: 19.I’m just about ready to declare victory!

Update 17 July 2014

Twice now I’ve dropped the alert level to blue and twice the spam level has rocketed back up to unaccepteable levels within 24 hours. That’s fine, it didn’t surprise me too much. But now I’m starting to see recidivists – IP numbers that have been blocked for spam, then cleared, now showing up in the spam list once again, some of them quite heavily. As a result, I’m taking a slightly harsher line when it comes to clearing IPs from the blocked list, instead of clearing them at the first opportunity.

It’s also interesting to observe that there are some IPs that, once blocked, have never earned their way back – one of them making 850 attempts to access the site in a single 24-hour period. All told, 1141 attempts to access the site have been blocked for spam reasons in the last 24 hours, divided among 29 different IP numbers – an average of about 40 attempts each. Most have been from Chinese servers, but the worst offenders have been from some Romanian servers, some Ukrainian servers, a handful of servers in the US, and – the worst offender of all – one server in Poland.

Overall, though, the strategy appears to be working; it’s just taking longer than I would have hoped.

Update 28 July 2014

Slowly but progressively, the anti-spam policy is working, as more and more ISPs get on top of the spambots running on their servers. Every day, more servers get released from the blocked list than get added, without incurring a fresh wave of spam. It’s still too early to call it a victory, but spam is now down to about 200% of what it was before the wave struck, a huge improvement from the 3000% that it reached at its worst.

This has given me a little time to think about the implications of this emergency strategy, and the risks involved.

First, I don’t like the idea that I can be forced to function as a weapon in denying people access to the site. Most of the blocked servers have identified themselves as being in China. It would be very easy for someone who wanted to restrict a population’s access to independent perspectives to get the webmaster to do their work for them by getting the site to block service, simply by running a state-sponsored spambot on their key infrastructure. I don’t think that will ever happen, as there are more efficient ways of blocking such access, so this is by no means an accusation. Just a concern. But, by extension, cyber warfare between any two groups can rope in any site employing this anti-spam technique simply by hacking the enemy and releasing a spambot.

Secondly, I believe in the benefits of an open internet, and this policy doesn’t sit well alongside that principle. The policy forces me to compromise my ideals, and however necessary that might be, it’s still something that leaves a bad taste in my mouth. This is only the second or third time that I’ve had to do something like this, and it will always be a policy of last resort – or close to it – as a result.

I’m always worried about one bad apple causing the site to be blocked for a much larger number of ordinary visitors. One reason why my initial sensitivity levels erred on the side of openness and spam tolerance is to minimize the impact on real users. The traffic numbers tell me that the policy works on that front, at least, but I never block a server without worrying about it.

It’s a concern that some of the earliest servers blocked have still not been released. The problem is that once a site has been blocked, I can no longer evaluate which traffic from that site are attempts to spam me, and which are genuine attempts by users trying to reach the site. The only way to find out is to release the block, and see what happens. This is more of a concern for servers located in a country from which I get a lot of traffic, like the US. So the final stages of Condition Blue need further thought. At the moment, the plan is to start releasing these one at a time at eight-hour intervals as soon as Spam levels return to pre-crisis standards. If the Spam goes back up, so does the block. Choosing which blocked servers to prioritize also needs a little more thought.

Finally, I’m always a little concerned that it provides an avenue for a direct attack on the site, simply by (potentially) getting me to block one of the servers on which I depend, or even the server on which the site resides. I don’t know what safeguards are in place within the plugin used to prevent that, and it makes me uncomfortable. If I inadvertently block a piece of my ISP’s key infrastructure, I can solve that problem by using a cybercafe to undo the change. If I unwittingly block one of the servers that the site itself depends on, there may be NO solution except to restore the site from a backup – a process that is always fraught with danger, and is never guaranteed of success.

As a result of all of these considerations, I am seriously contemplating a technological solution that automatically zaps anything it thinks is from a spambot – something that I have resisted in the past, due to the potential for false positives, but which may be the lesser of two evils. No decision has been made on the subject, and more research is needed before one can be made; a key question will be how well it plays with the existing infrastructure relating to comment management. Compatibility is not enough, I need to understand how they will work as a 1-2 punch.

The Penultimate Update: 10 Aug 2014

Things are slowly getting back to normal in terms of the Spam levels. 9 IPs remain blocked and one IP range from which truly horrid amounts of activity were resulting. In some cases we’re talking hundreds of spam attempts in a 24-hour period, in others we’re talking thousands.

I have decided on an addendum to the antispam policy to deal with the possibility that at least some of the blocked activity represents genuine attempts to use the site, however unlikely that might be. When the blocked list stabilizes, in any 24-hour period in which no new IPs are either blocked or released from blocking and in which spam levels are low, I will rank the remaining blocked IPs according to the reported levels of activity, and release the blocks on the least active. If this results in a return to unacceptable spam levels, the IP will be relisted and it will go to the back of the queue. Currently still blocked are (in ranking order least to most active):

1. 162.244.x.x Unknown location 133 hits (down from >400)
2. 112.111.x.x Shanghai, China 178 hits (down from >900)
3. 112.5.x.x Beijing, China 269 hits (down from >2500)
4. 91.200.x.x Ukraine 315 hits (down from >700)
5. 222.76.x.x Fuzhou, China 347 hits
6. 91.200.x.x Ukraine 494 hits
7. 91.200.x.x Ukraine 921 hits
8. 62.210.x.x France 1156 hits

note that the above list has been censored to avoid public identification of the owners of the servers in question, as per previous statements regarding the spam policy.

And it’s probably worth noting that five of these remaining eight were blocked in the initial day or two of the introduction of the new spam policy. Others, such as the French site, are far more recently listed. Others which I thought would be part of this final list such as a certain server in Las Vegas which had over 1100 hits listed against them managed to clear themselves eventually under the existing policies.

The problems and concerns with the current spam policy remain, but all have to be balanced against this: the policy works, at least for now.

I don’t expect to update this article again until I can announce that the last site has been released from blocking without ill-effects, and this particular spam war is over. That could be in as little as 9 days, or it could be weeks. It will be good not to lose a full day each week deleting and documenting spam sources again…

The Final Update: 27 Feb 2015

At long last, a cease-fire and general amnesty has been declared, which is to say, no-one is being blocked for excessive spam! In the course of this particular Spam War (and not counting future outbreaks), more than 750,000 attempts to spam the site were blocked. In one particular 8-hour period, more than 1000 pieces of spam were submitted as comments! This is the end of 7 long months of fighting this menace, at times consuming 3 or 4 extra hours a day of what might otherwise have been productive time.

In the end, I varied the policies described above slightly, opting for a sharper hair-trigger and faster release for high-volume sources like the US, Canada, Australia, New Zealand, and various European countries. These sources easily account for 98% of traffic to the site, so these were the sources at greatest risk of legitimate readers having their access blocked (hence the faster release).

The major source seemed to start in the Ukraine, migrate to China, move to Russia, then to the US, back to China, back to the Ukraine, over into Serbia, into North Korea, back into Russia, into England, back to Russia, back to China, back to Russia, and finally into the US. With fringes here, there, and everywhere. Germany was in there somewhere, too, as was France. They even got into Campaign Mastery’s Host server at one point (fortunately wordpress wouldn’t let me block myself)!

While it was hard at first to see any patterns, as incidents began to become more sporadic, it became clear that those responsible were not only in the US, but had a regular job of some sort or were students – it was early evening, US West Coast time, or late evening East Coast time, that the majority of attacks came in during the week. On the weekends, and on US public holidays, anything could happen at any time of the day.

Over the last week or so, things have definitely been winding down. Yesterday, we almost reached this point, but one IP remained (in Buffalo, New York) blocked. One final gasp, and it was done. So it’s been a hard fight, but I am declaring a victory for the good guys – at least for now!!


Discover more from Campaign Mastery

Subscribe to get the latest posts sent to your email.