A CMS media library often gets more bot traffic than article pages because it is bigger, easier to predict and cheaper to crawl. One article can come with a dozen image files, several resized copies, an attachment page and an API record, and many bots treat each one as a separate thing to fetch.
The media library is where a content management system (CMS) such as WordPress or Drupal stores uploaded images, PDFs, video and other files. Bots that visit it range from useful search engine image crawlers to scrapers and vulnerability scanners. Knowing which kind is visiting tells you whether to allow it, slow it down or block it.
Official resources:
- Read Google's guide to managing crawl budget on large sites — Shows how Google decides how much of a site to crawl and how to steer crawlers away from low-value URLs like media attachment pages.
- Review Google's robots.txt guide for controlling crawler access — Explains what robots.txt can and cannot block, including media and resource files, before you restrict a media library.
Table of Contents
- What Makes Media URLs So Easy for Bots to Find?
- Why One Upload Turns Into Many Crawlable URLs
- Which Bots Are Actually Hitting the Media Library?
- Why the Load Can Cost More Than Article Traffic
- How to Reduce Unwanted Media Crawling
- Frequently Asked Questions
What Makes Media URLs So Easy for Bots to Find?
wordpress stores uploads in `/wp-content/uploads/` by default, in year and month folders. Drupal usually keeps public files under `/sites/default/files/`. Because these paths are the same across millions of sites, a bot can guess likely file locations without reading a single article. File names add to the problem.
Many images keep camera or export names, such as numbered sequences, or follow the article slug. A bot that finds one file can often guess the files uploaded next to it. If a server allows directory listing, the folder itself can be browsed like an index. A bot then gets a complete inventory in one request. Many hosts turn this off, but it is worth checking on any server you manage.
Why One Upload Turns Into Many Crawlable URLs
When you upload an image to WordPress, it creates several resized copies for thumbnails, medium and large displays, and responsive `srcset` markup. Themes and plugins often register extra sizes. So one upload can become many files, each with its own URL. WordPress also has attachment pages, which are standalone pages for single media items. Since WordPress 6.4, new installs turn them off by default, but older sites may still serve thousands of them. Each one is a thin page, and to a crawler it looks like another page to fetch.
The REST API adds another route. On a default WordPress install, `/wp-json/wp/v2/media` returns paged lists of media items with their file URLs. A bot can walk through the whole library this way without loading any page HTML. drupal creates image derivatives, or "image styles," the first time someone requests them. Drupal uses a token in derivative URLs to limit abuse. Still, bots that request many style variations can make the server do real image processing instead of serving a cached file.
Which Bots Are Actually Hitting the Media Library?
traffic to the media library usually comes from a few groups, and each group has different intent: Scanner traffic often shows up as requests for files that do not exist. A burst of 404 errors for `.php` files under the uploads path is a warning sign. It is not a normal image crawl.
- Search engine image crawlers, such as Googlebot-Image, index files for image search. They are usually worth allowing.
- Scrapers and dataset collectors download images in bulk, often to reuse them or train models.
- Hotlinkers are other sites that embed your file URLs, so their visitors download from your server.
- Vulnerability scanners probe upload folders for executable files, such as a PHP web shell planted through an old plugin flaw.
- SEO and monitoring tools check image status codes, file sizes and alt text.
Why the Load Can Cost More Than Article Traffic
Articles are usually served from a page cache, so repeat requests are cheap. Media files are larger, and a bot downloading full-size originals uses far more bandwidth per request than one fetching HTML. On metered hosting or a CDN billed by transfer, that shows up on the bill. Uncached requests cost more still.
Attachment pages that skip the page cache, API calls for media lists and on-demand Drupal derivatives all run PHP and database queries. A crawl that looks modest in request counts can still slow the site for real visitors. Analytics tools can hide all of this. JavaScript-based analytics record page views, not direct file downloads, so media bot traffic often appears only in server logs or CDN logs.
How to Reduce Unwanted Media Crawling
Start with your server or CDN logs. Group requests by path and user agent before you block anything, so you don't cut off image search by mistake.
Practical steps, roughly in order of effort: Robots.txt rules only work for bots that choose to follow them. Well-behaved crawlers will respect a disallow rule for a media path, but scrapers and scanners usually ignore it. Those need firewall, CDN or server-level rules.
- Turn off directory listing for upload folders.
- Block PHP execution inside `/wp-content/uploads/` or Drupal's files directory.
- On older WordPress sites, turn off attachment pages or redirect them to the parent post or the file.
- Remove WordPress image sizes your theme never uses so fewer copies get created.
- Limit or require authentication for the media REST endpoint if nothing public depends on it.
Frequently Asked Questions
Will blocking bots from my media folder hurt image search visibility?
It can. If you block Googlebot-Image or disallow the uploads path in robots.txt, your images can drop out of image search. Target specific user agents or behaviour instead.
Does hotlink protection affect social media previews?
Strict referrer rules can break link previews on platforms that fetch images directly. Most CDNs let you allowlist known preview fetchers or empty referrers.
Should I make the whole media library private?
Usually not. Public posts need public image URLs. Keep private protection for files that shouldn't be public, such as gated downloads, and store those outside the public uploads directory.
