Topic: How to mitigate teleport pro automation webcrawler attack?

[maskego]

Is there anyone can provide useful way to mitigate automation crawler attack to apace2.2.x site?

I use modsecurity can't stop it.

[tdonovan]

Teleport Pro isn't really an attack - it is a website browser which makes a lot of connections and requests as it scans your site.

If you want to block it in Apache 2.2, you could deny access based on its User Agent string, like this:

[maskego]

tom,

Thank your reply.

Teleport is just a browser,but I think it is a malware scanner.It will increase your site's loading by illegal usage.And,it scans all your site.

I try your way to deny its user-agent at .htaccess.But,it doesn't take effects.Is there any good idea to block it via modsecurity?(the newest teleport pro1.65)

[Steffen]

The following rule set checks the user Agent:

modsecurity_crs_35_bad_robots.conf , it already checks "teleport pro".


Steffen

[maskego]

The strange thing is that I make a test to use teleport pro to mirror my live site,but the teleport pro take effects.(teleport pro download link:http://www.tenmax.com/teleport/pro/download.htm)

I am sure to apply this rule of mod_security CRS.

Does it mean that mod_security can't stop teleport pro automation web crawler? Or what setting should I revise?

[Steffen]

I stops with the above rules.
I tried it with curl:

curl 127.0.0.1/test -A "bla bla Teleport pro bla bla"

ModSecurity: Access denied with code 403 (phase 2). Pattern match "(?i?:c(?:o(?:n(?:t(?:entsmartz|actbot/)|cealed defense|veracrawler)|mpatible(?: ;(?: msie|\\\\.)|-)|py(?:rightcheck|guard)|re-project/1.0)|h(?:ina(?: local browse 2\\\\.|claw)|e(?:rrypicker|esebot))|rescent internet toolpak)|w(?:e(?:b(?: (?:downloader|by ..." at REQUEST_HEADERS:User-Agent. [file "D:/servers/apache/conf/httpd.conf"] [line "523"] [id "990012"] [rev "2.2.4"] [msg "Rogue web site crawler"] [data "teleport pro"] [severity "WARNING"] [tag "AUTOMATION/MALICIOUS"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] [hostname "127.0.0.1"] [uri "/test"] [unique_id "T3Gah8CoAQQAABcEN08AAABz"]

Steffen

ps.
You quote a lot in your posts, see the forum rules. Unnecessary use of quotes (for a topic history, scroll up). I removed a lot in your posts.

[maskego]

I doubt the newest teleport pro doesn't use teleport pro user-agent.Actually,I use teleport pro 1.65 to mirror a site,there is no error shows or be stopped.

I don't see those detection messages at logs.

Can you tell me where should I revise?

And,what steps can I do the same test like you did under windows os?

[Steffen]

Look in your logs to see what user agent is used.

A curl.exe you can find in ApacheStats.zip at the additional download page.

Btw.
A lot of scanners/crawlers are around and most use a "normal" user agent. To stop is mostly hard, and have to live with it.

Steffen

[maskego]

The bad teleport pro uses the "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT)" and "User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" to crawler.I find it at logs.

How to block it?

[Steffen]

That are valid/normal user agents. When you block based on these user agents then you block also users.

Only way to block a IP you know is using teleport, but that make not so much sense.

So, i think you have to live with it. Here crawlers sucking the site daily a few Gigs, most webmasters dealing with it.


Steffen

[glsmith]

maskego,

You say "bad Telepoert pro" like there is just the one. Does this bad teleport pro have just one IP? If so, block it's IP address instead.

[maskego]

gl:

There are many web crawlers not just one.
I call "teleport pro" as bad one,it means the teleport pro making abuse net freely in spite of webmasters.

steffen:

As you says,web crawlers suck the sites always.It not perfectly to ban ips.I feel it's a blind way to anti malicious web crawlers or scanners or robots.

Is it possible to find the web crawlers specific character to stop it before sucking the site?(When it search the robots.txt,how can we use modsecurity to ban it's ip automatically?Where can I find this rule?)

Can someone report the need to the modsecurity? Maybe it makes disgusting web crawlers to go away.