Bot requests to a CMS search page slow an entire site because every search URL is uncacheable, and each one forces the server to run a database query instead of serving a stored file. A normal page can be delivered from cache in milliseconds, but `/?s=blue+widget` is unique, so the CMS boots its full application, hits the database, and builds the result from scratch — and crawlers can invent an unlimited number of those URLs.
The damage is not confined to the search page. PHP worker processes, database connections, and memory are shared across the whole install, so once search requests occupy them, ordinary visitors queue behind work they never asked for. On shared or small VPS hosting, a few hundred such requests a minute can be enough to push the site into 502 or 504 errors.
Official resource:
- Read Google's crawl budget and host load documentation — Explains how Googlebot calculates a crawl capacity limit from your server's response times and errors, so you can see why heavy bot crawling degrades site performance.
Table of Contents
- What makes a search URL different from every other page
- Why the database work is disproportionate
- How bots end up hammering search in the first place
- Confirming it is actually search before you change anything
- What actually reduces the load
- The trade-offs to weigh before you block
- Frequently Asked Questions
What makes a search URL different from every other page
Most CMS pages have a fixed URL and stable output. A post at `/how-to-fix-a-leak/` returns the same HTML to everyone, so a page cache, a CDN, or a static-file plugin can store one copy and serve it thousands of times without touching PHP or MySQL. A search URL carries a query string — the `?s=` parameter in WordPress, `/search/node?keys=` in Drupal — and its value is part of the identity of the page.
Cache layers key on the full URL, so `?s=a`, `?s=aa` and `?s=aaa` are three separate entries. Most cache plugins go further and exclude search results by default, on the reasonable grounds that storing an unbounded set of one-off pages fills the cache without ever producing a hit. That leaves search as one of the few routes on a content site that is guaranteed to reach the origin application. Full page load, plugin and module initialization, session handling, and a `LIKE`-style query across the content tables — every single time.
Why the database work is disproportionate
Default CMS search is usually a wildcard text match across post titles and bodies. In MySQL, a `LIKE '%term%'` comparison cannot use a normal index, because the wildcard sits at the front of the pattern. The database reads through the content table rather than jumping straight to matching rows. On a site with a few hundred posts that is invisible.
On one with fifty thousand posts, plus revisions and meta rows joined in, a single search can take hundreds of milliseconds to several seconds. Multiply that by concurrent crawler requests and the database's connection limit becomes the bottleneck — at which point pages that have nothing to do with search start timing out too. Search results also tend to be expensive after the query runs. Themes commonly render excerpts, featured images, and related-post widgets for each result, so an empty or near-empty result set is cheap, but a broad term matching hundreds of posts is the worst case. Bots reliably find the worst case, because they submit terms drawn from your own content.
How bots end up hammering search in the first place
Search pages are rarely targeted deliberately. They get discovered and then amplified, usually through one of a handful of routes: The combinatorial problem is the important part.
Two parameters that each accept arbitrary values produce an effectively infinite URL space, and a crawler with no rate limit will keep walking it. This is the same mechanism behind faceted-navigation crawl traps on e-commerce sites; search is simply the version that exists on every content site by default.
- A theme or plugin outputs a search form whose results link into internal navigation, and a crawler follows those links.
- A search results page includes tag, category, or pagination links that themselves carry the `?s=` parameter, producing `?s=term&paged=2`, `&paged=3`, and so on without limit.
- Someone links to a search URL externally, and crawlers treat it as a legitimate entry point worth revisiting.
- Scrapers and AI training crawlers use search as a cheap site map, feeding it words harvested from your pages.
- Spam bots submit search queries containing URLs or scripts, hoping the term is echoed back into an indexable page.
Confirming it is actually search before you change anything
Slow-site symptoms have many causes, so identify the pattern in logs rather than assuming. In a raw access log, count requests containing the search parameter and group them by user agent and IP: One caveat worth respecting: not everything claiming to be Googlebot is Googlebot. Verify by reverse DNS lookup on the IP before you build rules around a user-agent string, and be careful about blocking legitimate search engine crawlers outright.
- Filter for `?s=`, `&s=`, or your platform's search path, and look at request volume per minute.
- Check whether the heavy agents self-identify as bots, and whether their IPs resolve back to the network they claim.
- Compare response times on search URLs against normal pages — a large gap points at the database query, not at network or asset delivery.
- Look for the same IP walking incrementing `paged` values, which is the signature of a crawl trap rather than human use.
- In Google Search Console, check Crawl Stats for a rising share of crawl requests returning pages you would not want indexed.
What actually reduces the load
The fixes fall into three layers, and the cheapest ones come first. Stop advertising search URLs to crawlers. Add `noindex` to search result pages — most SEO plugins do this by default, so confirm rather than assume — and disallow the search path in `robots.txt` (`Disallow: /?s=` for WordPress, `Disallow: /search/` for Drupal). Well-behaved crawlers respect both. Neither stops an ignoring bot, and note that a `robots.txt` block prevents crawling, so a `noindex` on a blocked URL will never be read; use one or the other deliberately. Block or slow the requests before they reach PHP.
A rate limit at the CDN or reverse-proxy layer, applied specifically to the search path, costs almost nothing and caps the worst case. Cloudflare rate-limiting rules, an Nginx `limit_req` zone, or a WAF rule keyed on the search parameter all work. Blocking at the edge is what keeps the request from consuming a PHP worker, which is the resource that actually runs out. Make search itself cheaper. Moving search to a dedicated index — Elasticsearch, Algolia, Meilisearch, or a self-hosted equivalent — removes the query from your primary database entirely, which protects the rest of the site even under load. Failing that, requiring a minimum query length, rejecting queries containing URLs, and capping results per page all trim the tail. A short micro-cache on search responses, even sixty seconds, absorbs bursts where a bot repeats the same term.
The trade-offs to weigh before you block
Blocking search aggressively has costs. Real users search, and some of them arrive from a bookmarked or shared search URL; a blanket block returns errors to people, not just to crawlers. Rate limiting by IP also penalises users behind shared addresses — offices, universities, mobile carrier NAT — so set thresholds well above plausible human behaviour. There is a monitoring cost too.
If you rate-limit at the edge, the blocked requests no longer appear in your application logs or analytics, so the problem can look solved when it is merely hidden. Keep an eye on the edge's own rule-hit counters, or you will not notice when the traffic pattern changes. Finally, resist fixing this by raising resource limits alone. More PHP workers or a larger database instance buys headroom, and headroom is useful, but an unbounded URL space will consume whatever you provide. The durable fix is reducing the number of uncacheable, database-hitting requests that reach the application at all — capacity is the backstop, not the solution.
Frequently Asked Questions
Will adding a page cache plugin fix this?
Generally not on its own. Most page caches exclude search results deliberately, because each query string produces a unique entry that is unlikely to be requested twice, so search requests still reach PHP and the database.
Is `robots.txt` enough to stop the traffic?
Only for crawlers that obey it. Search engine bots do; scrapers and spam bots frequently do not, which is why an edge rate limit or WAF rule is the part that reliably caps the load.
Should I remove the search feature entirely?
Rarely necessary. Disabling search is a valid option on sites where nobody uses it, but a `noindex` plus an edge rate limit keeps search working for visitors while removing the crawl incentive.
Why does the whole site slow down rather than just the search page?
PHP worker processes, database connections, and memory are shared across the installation. When search requests occupy those pooled resources, every other request waits for one to free up.
