Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: blocking Applebot |
|
| Author |
|
Doug22
Joined: 02 Jun 2013
Posts: 57
Location: Houston TX
|
 Posted: Thu 12 Jan '23 5:13 Post subject: blocking Applebot |
 |
|
So I'm trying to block web bots, and my robots.txt file seems to be doing a pretty good job of that EXCEPT for Applebot. I've put
| Code: | User-agent: Applebot
Disallow: / |
in my robots.txt file,but Applebot seems to be grabbing my stuff anyway. In fact, when I search my archive for robots.txt, Applebot never seems to be accessing it. Googlebot and Yandex at least do so regularly.
What's going on? Does Applebot check on robots.text once a week,or once a month,or what? |
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7371
Location: Germany, Next to Hamburg
|
| Posted: Thu 12 Jan '23 13:40 Post subject: |
|
|
If it doesn't you can block it by the user agent
| Code: |
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^.*Applebot.*$
RewriteRule . - [R=403,L]
|
|
|
| Back to top |
|
Doug22
Joined: 02 Jun 2013
Posts: 57
Location: Houston TX
|
| Posted: Thu 12 Jan '23 15:24 Post subject: |
|
|
| Yes, thank you. I understand that is a straightforward possibility. But Apple formally says that its AppleBot obeys robots.txt. Not seeing it. |
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7371
Location: Germany, Next to Hamburg
|
| Posted: Thu 12 Jan '23 17:09 Post subject: |
|
|
| Doug22 wrote: | | But Apple formally says that its AppleBot obeys robots.txt. Not seeing it. |
So what is your question now? |
|
| Back to top |
|
Doug22
Joined: 02 Jun 2013
Posts: 57
Location: Houston TX
|
| Posted: Thu 12 Jan '23 19:56 Post subject: |
|
|
| Just pointing out that Apple says that AppleBot pays attention to robots.txt, but it really doesn't. I was originally asking if anyone knew something I didn't know about why that seems to be. |
|
| Back to top |
|
|
|
|
|
|
