The reliable way to handle scrapers that keep requesting deleted CMS attachments is to make your server answer them as cheaply and clearly as possible: return a correct 404 or 410 status, serve it without invoking the CMS, and block only the clients that ignore those signals. A removed attachment — an image, PDF, or other media file once served by WordPress, Drupal, or a similar system — leaves behind URLs that bots collected long ago, and those bots will retry them for months or years regardless of what you do.
The real cost of these requests is rarely bandwidth. It is that many CMS setups route every missing file through the full application stack — PHP, database, theme — to render a styled "not found" page, turning a request that should cost almost nothing into one of the most expensive hits on the site. Fixing that routing usually matters more than fighting the scraper itself.
Official resources:
- Review Google's guidance on how crawlers handle 404 and 410 responses — Readers can verify how Google's crawlers treat each status code for removed URLs and choose the right response for deleted attachments.
- Read the HTTP specification's definition of the 410 Gone status code — RFC 9110 defines 410 Gone as the response that tells clients a resource is permanently removed and requests should stop.
Table of Contents
- Why scrapers keep asking for files you deleted
- Send the right status code — 410 when the file is gone for good
- Stop the CMS from rendering your 404s
- When and how to block the scraper itself
- Clean up the sources that keep the URLs alive
- Frequently Asked Questions
Why scrapers keep asking for files you deleted
Scrapers work from stored URL lists, not from your current site. Search engine crawlers revisit URLs from their index, archive services replay old snapshots, hotlinking sites embed your file URLs directly in their pages, and bulk content scrapers reuse crawl databases assembled long before you removed anything. None of these systems checks whether a link is still advertised anywhere; they simply request what they recorded. CMS attachment URLs make this worse because one file generates many URLs.
WordPress, for example, creates multiple resized copies of every uploaded image (`photo-300×200.jpg`, `photo-768×512.jpg`, and so on) plus an attachment page for the file itself. Deleting the media item can leave a dozen distinct URLs in circulation, each of which a scraper may retry independently. There is also a self-inflicted variant: your own pages may still reference the removed file. An old post embedding a deleted image, a cached page, or an RSS feed entry will cause legitimate crawlers to keep finding the dead URL. Before blaming a scraper, search your own content and templates for the file path.
Send the right status code — 410 when the file is gone for good
The HTTP status code is your main channel for talking to well-behaved bots. A 404 (Not Found) says the resource is missing but might return, so crawlers keep rechecking it. A 410 (Gone) says the removal is permanent, and major search crawlers treat it as a stronger signal to drop the URL and slow or stop retries. For attachments you deleted deliberately and permanently, 410 is the better answer.
You can return 410 at the web server layer without touching CMS code. In Apache, a few lines in the site configuration or `.htaccess` handle it: Two cautions apply. First, never return 200 with a custom "file removed" page at the old URL — that tells every bot the resource exists, guaranteeing indefinite recrawls and creating soft-404 problems in search. Second, avoid redirecting dead attachments to your homepage; crawlers treat mass redirects to an irrelevant page as soft 404s, and human visitors find them confusing. Redirect only when a genuine replacement file exists.
- `Redirect gone /wp-content/uploads/2022/old-catalog.pdf` for a single file
- `RedirectMatch gone ^/wp-content/uploads/2022/06/` for a whole removed directory
- In Nginx, a `location` block with `return 410;` does the same job
Stop the CMS from rendering your 404s
On a default WordPress or Drupal install, a request for a missing file under the uploads directory can fall through to the CMS front controller, which boots the application and queries the database just to say "not found." A scraper hammering fifty dead attachment URLs per minute is then effectively load-testing your PHP and MySQL stack. This is the mechanism that turns nuisance traffic into slow pages and hosting alerts. The fix is to short-circuit at the web server. For static file extensions under your media paths, tell the server to answer directly and never pass the request to PHP. In Nginx, that looks like a location block matching the uploads path with `try_files $uri =404;`.
In Apache, rewrite rules can serve a plain 404 or 410 for missing files matching image and document extensions before WordPress's catch-all rewrite runs. The response drops from a full application boot to a few microseconds of static handling. Verify the change with `curl -I` against a known-dead attachment URL: you should see the bare server 404/410 headers, not your theme's styled error page. Check response time too — a static error should return in single-digit milliseconds. Your access logs will still record the requests, but they will no longer cost you meaningful resources.
When and how to block the scraper itself
Blocking is worth the effort only when a client keeps hammering after receiving clear 410s, or when the volume is high enough to pollute logs and analytics. Identify candidates from your access logs: group requests by IP and user agent, and look for clients whose traffic is mostly errors. A client with thousands of 404s and few successful requests is not a search engine behaving normally.
Match the tool to the offender: Be conservative with anything automatic. Corporate networks, VPNs, and mobile carriers put many users behind shared IPs, so an aggressive ban threshold will eventually block real visitors. And never block by user agent strings belonging to search crawlers you rely on — if Googlebot is requesting dead attachments, the correct response is accurate status codes and time, not a block that damages your indexing.
- `robots.txt` `Disallow` rules work only for compliant crawlers — useful for search and archive bots, useless against scrapers that ignore the file
- Web server rules can deny by user agent string, though scrapers can trivially change theirs
- Firewall or fail2ban rules can ban IPs that accumulate rapid 404s, which catches persistent offenders automatically
- A CDN or proxy layer such as Cloudflare can rate-limit or challenge requests to dead paths before they reach your origin at all
Clean up the sources that keep the URLs alive
Cheap error responses handle the symptom; removing the URL from circulation shrinks the problem. For URLs still in search indexes, a consistent 410 leads crawlers to drop them on their own, and search console removal tools can accelerate it for individual URLs that expose something sensitive. For hotlinking sites embedding your dead image URLs, hotlink protection at the server or CDN — refusing image requests whose referrer is another domain — removes their incentive entirely. Inside your own CMS, search posts and custom fields for the removed file's path and update or delete the references.
In WordPress, a database search for the filename across `wp_posts` finds embedded uses; Drupal's file usage tracking shows where a managed file is referenced before you delete it. Regenerate or purge any page caches afterward, since a cached page can keep advertising the dead URL long after the content is fixed. Expect a long tail no matter what you do. Some scrapers never expire their URL lists, and a trickle of requests for files deleted years ago is normal background noise on any established site. The success condition is not zero requests — it is that each remaining request costs you a static 410 and nothing more.
Frequently Asked Questions
Should I return 404 or 410 for a deleted attachment?
Use 410 when the file is permanently gone — crawlers drop the URL faster and retry less. Use 404 only when the absence might be temporary.
Do these requests hurt my search rankings?
Ordinary 404s and 410s for removed files do not harm rankings. The risk comes from mishandling them — serving 200 pages or mass-redirecting dead files to the homepage creates soft-404 signals.
Why do requests for a deleted image use several different filenames?
The CMS generated resized copies at upload (thumbnail, medium, large), each with its own URL. Scrapers recorded whichever sizes your pages embedded, so cover the whole size set when adding 410 rules.
Is it worth redirecting a removed attachment to a replacement?
Yes, when a genuine successor exists — a new version of a PDF, an updated image at a new path. A 301 there preserves the visitor and any link value. Without a real replacement, 410 is more honest and more effective.
