We’ve known for a while that over half of all internet traffic comes from bots. We humans are a dying breed (on the internet), now representing just 47% of web traffic.
What’s more concerning though is the sheer amount of "bad bot" activity, which the latest Thales Bad Bot Report suggested makes up 40% of all internet traffic. In three years that number has grown by 10%.
This is obviously a problem.
We, and you, our customers, are caught in the middle of an arms race that’s costing all of us quite a lot of time and money. To some of you that will be visible in server resources, monitoring patterns, or a degraded customer experience.
We've been fighting back against this for years. We're about to take another major step, which is more aggressive, and more visible, so we want to explain a little about what we're doing and what you'll see.
An ongoing battle
Old wisdom suggested that bot traffic = bad, human traffic = good. It’s an outdated mindset. Bots power AI tools, search engines and countless other essential functions on the modern internet.
Sounding familiar? That’s because we’ve been writing a whole series of blog articles about this topic. We looked into the severity of the bad vs. good bot problem in part one of the series, and the engineering behind our day-to-day bot filtering in part two.
Now in this third part of our bot protection series, we’re explaining how and why any bad bot caught on one server is shut across our entire fleet, and beyond.
We're adding a new layer to our defences that we haven’t talked about before. In fact, it's already live on some of our servers, including this website, but hopefully none of you have seen it or noticed.
IP addresses have stopped being enough
With our new protection layer, we’re no longer judging visitors primarily by the IP address they turn up on. We’re instead judging them by the way they actually behave.
We saw this coming three years ago when we wrote about an aggressive bot directing thousands of Azure IPs at our servers. At the time Quintin, our technical director, described it as a game of whack-a-mole, where IP addresses pop up and you don't know which one’s coming next.
"There's an industry problem here, and we're going to have to change the way that IP reputation databases work or how fast they are updated," he said, back in 2023.
There’s a diminishing amount of info we can get from IP addresses. Cloud platforms let you rent out IP addresses by the hour. Bots, both good and bad, can take advantage of this and use thousands of different IPs in the space of an hour. If we used blunt force and banned the IP range of the cloud provider renting out the IPs, we’d be shutting out a lot of perfectly legitimate real humans and helpful bots. All it takes is one bad bot among them to poison the well.
Therefore, a set-and-forget ban list built blindly on IP addresses ends up punishing the many for the sins of the few. It's forever a step behind, needing to catch up with the bot’s newest IP.
It’s become an uncomfortable truth that IP reputation, on its own, is breaking down (we’ll come back to this).
So, we needed a way to recognise a bad bot by what it does, not just by where it happens to be coming from.
Catching bad bots by what they do
We’ve deployed an open-source security engine that’s built for exactly this kind of work. It doesn’t just watch IP addresses; it reads the same web server logs your sites already produce and looks for behaviour that gives an attacker away.
Consider what a malicious bot actually does. It rattles through hundreds of URLs, hunting for a login page or a forgotten admin panel. It probes for files that should never have been public. Things like .env files full of passwords, a .git folder, a stray wp-config.php. It hammers a WordPress login hundreds of times a minute to guess a password. It walks straight up to known software vulnerabilities to see whether you’ve patched them yet.
None of that looks anything like a person reading a website.
Back in part two, we described how we were assessing traffic based on factors like the average duration between requests on a site. This work was backed by the idea that robots behave on a metronome (you could even say they behave robotically), whereas humans behave messily and non-methodically.
We’ve added another dimension to the picture, recognising the shape of actual attacks in progress. And our new system achieves that with a pleasingly simple, but effective, mechanic.
Filling the bucket
The new tool we’re using deploys what could be considered a “leaky bucket” approach.
It uses a mechanism that judges every interaction a visitor makes as either good (humans, good bots) or bad (nefarious or poorly-designed bots). “Bad” actions tip a little bit of water into a metaphorical bucket. A normal human visitor might tip a bit in once or twice before the water safely drains away, anomalies that we can safely ignore. A less well-behaved (and less human) visitor who trips the same alarm dozens of times in a few seconds fills the bucket until it overflows. We flag bad bots that tip over the bucket, and ban them. Genuine human traffic and good bots sail through without noticing they’d tipped any water into the bucket, while the abusive stuff effectively removes itself.
When we talk about "banning" IPs, what we're actually doing is putting an automatic CAPTCHA challenge in front of the visitor, preventing them from accessing the site until they've solved it. We'll go further into this later in the article.
No longer welcome
To qualify for the ban treatment, bad bots need to repeatedly and very quickly fill that bucket of water. And when they do, we don’t just want these bad eggs banned on one server, we want them banned across the board.
Behind the scenes, the logs from across our fleet feed a single engine. Those logs are combined to produce a ban list that’s shared straight back out to every server we run. (This log system was the meat and bones of our most recent bad bots article).
The moment a bad bot is spotted causing trouble on one customer’s site, the door quietly closes on them everywhere else. One strike on any server anywhere, and the bad bot is banned across the whole fleet.
Community immunisation
We’ve teased the idea of “community immunisation” before. Our previous bad bot detection work was building towards it—sharing the IPs of abusive bots on our network back with a community-sourced database.
Now we’ve started doing this at a scale that makes the metaphor more well-earned.
That’s because we’re not the only ones running this trustworthy “leaky bucket” approach. In fact, there are tens of thousands of servers around the world running it, and they're all sharing the addresses of bad bots attacking them. Because everyone’s sharing information about threats they’ve faced, all of the security engine’s users are helping protect each other.
By joining that network we get two things:
A curated, constantly updated picture of the internet’s worst offenders (bots already caught attacking other people’s servers).
A way to share our list of bad bots back with the community, so everyone else can be protected from them too.
Hence the community immunisation metaphor. Every attack absorbed by one member makes everyone else a little more resistant. Malicious bots who’ve been busy causing trouble elsewhere on the internet are banned before they’ve sent us a single request.
Over 140,000 addresses, and counting
This new methodology, combined with our previous bot filtering work, means we now concurrently ban ~140,000 IP addresses from our servers. The list is stitched together from three sources.
Our own detection rules e.g. the leaky bucket system.
The shared list of other organisations using this system.
A further 30,000 IPs pulled from another community-sourced database (we talked about this in our last bad bots article).
All added up, it’s a lot of IPs.
But remember earlier when we were talking about IPs being less useful than ever?
Bots can change IPs incredibly quickly, and cloud providers like Azure reallocate those IPs to other users once the bot is done with them. It means mass banning IPs inevitably leads to banning innocent human traffic. Flagging IPs as abusive isn’t a permanent solution.
We’re using a ban list that’s a living, evolving, and constantly changing list—not a static one. Addresses are removed from the ban list when the abuse stops. Fresh ones are added the moment an attack is spotted. And it’s just as important that the list reaches every server as quickly as it can.
When we wrote our previous update, we were changing server banning rules every couple of minutes. Now with these new tools, updates are being pushed out every ~30 seconds. An abusive bot caught anywhere on our network, or any other community contributor’s network, is shut out almost everywhere before it has finished attacking its first target.
Keeping the good guys
We have to be pretty careful about who we’re banning. It’s not just humans who we want to let through. Accidentally banning Googlebot would be an SEO disaster. Banning a customer’s own office or their monitoring tools would be even worse.
In the same way we build and update our ban list, we maintain curated allowlists.
Obvious ones like search engine crawlers (Google, Bing etc.) are exempted. Major content delivery networks, uptime and status monitors, and of course our customers’ own list of trusted addresses, are all waved straight through. The idea is to be ruthless with bad actors, and completely invisible to everyone else.
Rolling out the careful way
If you’ve been with us for a while you’ll know we don’t suddenly make big changes across the whole fleet on a Friday afternoon. This new protection layer is being introduced with the same necessary level of caution as every other change we make.
So we’ve shipped this tool set switched off. As we roll it out, we’re initially enabling it in just one region at a time, on a single server at a time, and watching closely for a few days. We want to be certain no legitimate traffic is getting banned before rolling it out more widely.
It’s definitely not a set-and-forget one-and-done job.
What it means for your sites
We're already running these new protections on a small number of selected servers. Each attack encountered and every signal the wider community shares tunes the system a little further, keeping defences sharpened and updated. We've also successfully tested it on our internal Cloud Container infrastructure.
With more bad traffic kept at bay, your sites will spend less time and fewer resources serving unwanted bots.
So far our initial testing has been extremely positive, with one server banning over 1,700,000 bad requests in under 24 hours. For some servers we're seeing ~30% of all traffic being banned. During a spike, some of our servers peaked at a 66.3% ban rate.
What banned visitors see
Sometimes the system is going to get it wrong and real users will be banned. We'll work hard to minimise this, and the initial data is extremely positive. But it will happen, and so we want you to be aware of what the banned users (or even you) will see in that scenario:
A page with a simple CAPTCHA challenge (not the kind with impossible letters or traffic light grids), that tries to verify the authenticity of traffic. Your browser should automatically solve the challenge in a few seconds, but it’s designed to be computationally expensive for a bot to solve. When the challenge is completed, the user will be taken to the requested website and page as expected.
If for some reason the challenge is failed, or not available, they will be taken to a page which explains what's happened, links to our knowledge base, and provides some steps to take.
We're going to be enabling the new protection layer across our Cloud Container platform starting this week. We also know some of you will want to manually tweak these behaviours, or potentially opt out entirely, so we're actively working on customer-facing controls and customisation. We'll have more to show about this as we integrate these tools more widely.
If you're accidentally being banned, or want to know more about our process, check out our Knowledge Base for more information.
Always more to be done
Our plan is to roll out this new protection layer over more services and servers in the coming weeks and months.
This work isn't a silver bullet, but it's not a band aid either. It’s another layer in a defensive effort that we've been continuously building and improving.
Every web host in the world faces the same flood of bad bots. But not every host puts the same amount of energy into dealing with it, and there's a growing gap between the hosts who do this work quietly in the background and the ones who don't.