Topic: How to stop annoying/abusive crawlers in 2.4

[Steffen]

User Xing has posted a nice list how he stopped quite some crawlers based on User Agent in 2.4. He excluded Google, Bing and others which he wants to index.

When you have a User Agent string which is worth to block, please post it here.

Steffen

[DnvrSysEngr]

This is what I have in my httpd.conf to stop crawling by unwanted crawlers/bots.
<Directory>
SetEnvIfNoCase User-Agent "AhrefsBot*" BadBot
SetEnvIfNoCase User-agent “Baidu” BadBot
SetEnvIfNoCase User-Agent "BaiDuSpider*" BadBot
SetENvIfNoCase User-Agent "Choopa" BadBot
SetEnvIfNoCase User-Agent "Cityreview*" BadBot
SetEnvIfNoCase User-Agent "crawl" BadBot
SetEnvIfNoCase User-Agent "Dotbot*" BadBot
SetEnvIfNoCase User-Agent "Java" BadBot
SetEnvIfNoCase User-Agent "MJ12bot" BadBot
SetEnvIfNoCase User-Agent "NG\ 1.x (Exalead)" BadBot
SetEnvIfNoCase User-Agent "Sogou" BadBot
SetEnvIfNoCase User-Agent "Sosospider" BadBot
SetEnvIfNoCase User-Agent "spider" BadBot
SetEnvIfNoCase User-Agent "Twiceler" BadBot
SetEnvIfNoCase User-agent “Yandex” BadBot
SetEnvIfNoCase User-Agent "YandexBot" BadBot
SetEnvIfNoCase User-Agent "Yandex*" BadBot
<RequireAll>
Require all granted
<RequireNone>
Require env BlockCountry (REMOVED for this POST)
<Limit GET POST PUT HEAD>
Require env BadBot
</Limit>
</RequireNone>
</RequireAll>
</Directory>

[glsmith]

I've gone a different route, save the 403s polluting the error log

RewriteEngine on

# [Multi-Useragent]
RewriteCond %{HTTP_USER_AGENT} black.widow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^e?mail.?(collector|magnet|reaper|siphon|sweeper|harvest|collect|wolf) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^IE\ \d\.\d\ Compatible.*Browser$" [OR]
RewriteCond %{HTTP_USER_AGENT} ^net.?(ants|carta|mechanic|spider|vampire|zip) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^site.?(searcher|snagger|valet) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^web.?(auto|bandit|catcher|collage|collector|copier|copy|core|devil|downloader|fetch??|image|inator|hook|layers|linker|log|mole|miner|mirror|quest|reaper|sauger|site|snake|snarf|stolperer|stripper|sucker|vac|walk??|watch|whacker|weasel|zinger|zip) [NC,OR] # ODs

# [HTTP_USER_AGENT]
# ::ELNSB50 EmailHarvesting & GuestbookSpamming
RewriteCond %{HTTP_USER_AGENT} ^::ELNSB50 [NC,OR]

RewriteCond %{HTTP_USER_AGENT} ^Acrobat [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^anarchie [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ASPSimply [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Atomz [OR]
RewriteCond %{HTTP_USER_AGENT} ^cherry.?picker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "compatible ; MSIE 6.0?" [OR]
RewriteCond %{HTTP_USER_AGENT} crescent [NC,OR] # OD
RewriteCond %{HTTP_USER_AGENT} "^DA \d\.\d+" [OR] # OD
RewriteCond %{HTTP_USER_AGENT} ^DataCha0s [OR]
RewriteCond %{HTTP_USER_AGENT} "^DTS Agent" [OR] # OD
RewriteCond %{HTTP_USER_AGENT} ^Download [OR] # OD
RewriteCond %{HTTP_USER_AGENT} ^EasyDL/\d\.\d+ [OR] # OD
RewriteCond %{HTTP_USER_AGENT} "^ABCdatos BotLink" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Acme.Spider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Ahoy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Alkaline [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ananzi [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Anthill [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Arachnophilia [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Arale [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Araneo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^AraybOt [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ArchitextSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Aretha [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ARIADNE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^arks [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ASpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^ATN Worldwide" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
RewriteCond %{HTTP_USER_AGENT} ^AURESYS [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BackRub [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BackWeb [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^BatchFTP [OR]
RewriteCond %{HTTP_USER_AGENT} "^Bay Spider" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Big Brother" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bjaaland [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Bloodhound [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Borg-Bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Bot\ mailto:craftbot@yahoo.com" [OR]
RewriteCond %{HTTP_USER_AGENT} ^BoxSeaBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^bright.net [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Buddy [OR]
RewriteCond %{HTTP_USER_AGENT} ^CACTVS [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Calif [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Cassandra [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Checkbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChristCrawler.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^churl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^cIeNcIaFiCcIoN.nEt [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^CMC/0.01 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Collective [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Collector [OR]
RewriteCond %{HTTP_USER_AGENT} "^Combine System" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^ComputingSite Robi/1.0" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Conceptbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ConfuzzledBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^CoolBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Copier [OR]
RewriteCond %{HTTP_USER_AGENT} ^crawlpaper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Cusco [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^CyberSpyder [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^CydralSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^DA [OR]
RewriteCond %{HTTP_USER_AGENT} "^Desert Realm " [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^DeWeb(c) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Die Blinde Kuh" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^DienstSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Digger [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Digimarc MarcSpider" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Digimarc Marcspider/CGI" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Digital Integrity Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Direct Hit Grabber" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^DISCo\ Pump" [OR]
RewriteCond %{HTTP_USER_AGENT} ^DNAbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^DownLoad Express" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Download\ Demon" [OR]
RewriteCond %{HTTP_USER_AGENT} "^Download\ Wonder" [OR]
RewriteCond %{HTTP_USER_AGENT} ^Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^DragonBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Drip [OR]
RewriteCond %{HTTP_USER_AGENT} ^DWCP [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EbiNess [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^e-collector [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} "^EIT Link Verifier Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ELFINBOT [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Emacs-w3 Search Engine" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^esculapio [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Esther [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Evliya Celebi" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Express\ WebPictures" [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FastCrawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Felix IDE" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FetchRover [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^fido [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FileHound [OR]
RewriteCond %{HTTP_USER_AGENT} "^Fish search" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} "^Fluid Dynamics Search Engine" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Fouineur [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Freecrawl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^FunnelWeb [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^gammaSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^gazz [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GCreep [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GetBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetSmart [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetterroboPlus [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^GetURL [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^Golem [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^gotit [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grapnel/0.01 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Griffon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Gromit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Gulper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Hämähäkki [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^HamBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Harvest [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^havIndex [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^HI.*Search [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^HKU WWW Octopus" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} "^Hometown Spider Pro" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ht://Dig [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^html_analyzer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^HTMLgobble [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^HTTrack [OR]
RewriteCond %{HTTP_USER_AGENT} ^Hyper-Decontextualizer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^I, Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^iajaBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^IBM_Planetwide [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^image.kapsi.net [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Imagelock [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^IncyWincy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Informant [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InfoSeek [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InfoSpiders [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Ingrid [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Inspector Web" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^IntelliAgent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} "^Internet Cruiser Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Internet Shinchakubin" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Internet\ Ninja" [OR]
RewriteCond %{HTTP_USER_AGENT} ^Iria [OR]
RewriteCond %{HTTP_USER_AGENT} ^Iron33 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Israeli-search [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JavaBee [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JCrawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Jeeves [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JoBo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Jobot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC [OR]
RewriteCond %{HTTP_USER_AGENT} ^JoeBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JumpStation [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^JustView [OR]
RewriteCond %{HTTP_USER_AGENT} ^Katipo [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^KDD-Explorer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Kilroy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^KIT-Fireball [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^KO_Yappo_Robot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^LabelGrabber [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^legs [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^lftp [OR]
RewriteCond %{HTTP_USER_AGENT} ^likse [OR]
RewriteCond %{HTTP_USER_AGENT} "^Link Validator" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkScan [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Lockon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^logo.gif Crawler" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Lycos [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Mac WWWWorm" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Magnet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mag-Net [OR]
RewriteCond %{HTTP_USER_AGENT} ^Magpie [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^marvin/infoseek [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Mass\ Downloader" [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mattie [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MediaFox [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Memo [OR]
RewriteCond %{HTTP_USER_AGENT} ^MerzScope [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^MIDown\ tool" [OR]
RewriteCond %{HTTP_USER_AGENT} ^MindCrawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mirror [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister [OR]
RewriteCond %{HTTP_USER_AGENT} ^mnoGoSearch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^moget [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MOMspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Monster [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Motor [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MSNBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Muncher [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Muninn [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Muscat [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mwd.Search [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NDSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NEC-MeshExplorer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Nederland.zoek [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NetMechanic [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NetScoop [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZip [OR]
RewriteCond %{HTTP_USER_AGENT} ^newscan-online [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^NHSE Web Forager" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Nomad [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Northern Light Gulliver" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "Novarra-Vision/unknown" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^NPBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^nzexplorer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ObjectsSearch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Occam [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} "^Offline\ Explorer" [OR]
RewriteCond %{HTTP_USER_AGENT} ^OntoSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Open Text Index Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Openfind data gatherer" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Orb Search" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Pack Rat" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PageBoy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} "^Papa\ Foto" [OR]
RewriteCond %{HTTP_USER_AGENT} ^ParaSite [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Patric [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^pegasus [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PerlCrawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^PGP Key Agent" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Phantom [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PhpDig [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PiltdownMan [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Pimptrain.com's" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Pioneer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^PlumtreeWebAccessor [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Pockey [OR]
RewriteCond %{HTTP_USER_AGENT} ^Poppi [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Popular Iconoclast" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Portal Juice Spider" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^PortalB Spider" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Pump [OR]
RewriteCond %{HTTP_USER_AGENT} "Python-urllib" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Raven Search" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^RBSE Spider" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Reaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Recorder [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} "^Resume Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^RixBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Road Runner" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^RoadHouse [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Robbie [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^RoboCrawl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^RoboFox [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Robot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Robozilla [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Roverbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^RuLeS [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SafetyNet [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Scooter [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Search.Aus-AU.COM [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SearchProcess [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Senrigan [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SG-Scout [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^ShagSeeker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Shai'Hulud [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Sift [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Simmany [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteTech-Rover [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Skymob.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SLCrawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Sleek [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Smart Spider" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snake [OR]
RewriteCond %{HTTP_USER_AGENT} ^Snooper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Solbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SpaceBison [OR]
RewriteCond %{HTTP_USER_AGENT} ^Spanner [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Speedy [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^spider_monkey [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SpiderBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Spiderline [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SpiderMan [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SpiderView [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Spry [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} StumbleUpon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^Suke [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^suntek [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Sven [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Sygol [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Tarantula [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^tarspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Tcl W3 Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^TechBOT [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Templeton [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^TeomaTechnologies [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^The Jubii Indexing Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^The NorthStar Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^The NWI Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^The Peregrinator" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^The Python Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^The TkWWW Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^The Web Moose" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^The Web Wombat" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^The Webfoot Robot" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^the World Wide Web Wanderer" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^The World Wide Web Worm" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^TITAN [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^TitIn [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^TLSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^TurnitinBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^UCSD Crawl" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^UdmSearch [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^UptimeBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^URL Check" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^URL Spider Pro" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Vacuum [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Valkyrie [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Verticrawl [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Victoria [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^vision-search [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^void-bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Voyager [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^VWbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^w.pSpider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^W3M2 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^w3mir [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Walhello [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WallPaper [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Webcapture [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Webster [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStolperer [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Whacker [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^whatUseek [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^WhoWhere [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Wild Ferret" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^Wired Digital" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} "^WWWC Ver 0.2.5" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^XGET [NC,OR] [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^XYLEME [NC]

# Send them home to mama or a dead connection
RewriteRule ^/(.*) http://localhost/$1 [L,R]