logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: %{HTTP_REFERER} Two different strings for the same referrer
Author
walt



Joined: 24 Oct 2015
Posts: 25
PostPosted: Fri 08 Jan '16 8:28    Post subject: %{HTTP_REFERER} Two different strings for the same referrer Reply with quote
Hello, the variable %{HTTP_REFERER} sometimes returns http://www.domainname.com, and sometimes just http://domainname.com.

Is there a significance to this difference? It's causing me some trouble with mod_rewrite rules.

Thanks!
Back to top
covener



Joined: 23 Nov 2008
Posts: 59
PostPosted: Fri 08 Jan '16 23:51    Post subject: Reply with quote
It's just showing you the header sent by different clients, presumably accessing your server w/ and w/o the www. prefix.
Back to top
walt



Joined: 24 Oct 2015
Posts: 25
PostPosted: Sat 09 Jan '16 1:18    Post subject: Reply with quote
Thanks covener!

If I understand your answer correctly, it could be the remote server using a different style for the referrer field?

What I am trying to do is redirect referrers like:
http://domainname.com, and http://www.domainname.com, to a php script, and log the visit. From my apache (and awstat) logs, it looks like these are usually spam referrals. The php script also displays a button that can be pressed to enter the site.

I added an exception if the visitor followed a link on my site. The mod_rewrite rule is not applied if %{HTTP_REFERER} = http://www.my_domainname.com

This works fine (catching all the .ru), however the file is also catching %{HTTP_REFERER} = http://my_domainname.com

My concern is that it might be inconveniencing legitimate visitors. However, when I browse my own site, everything works fine. The 'push button to enter' page never pops up.
Back to top
walt



Joined: 24 Oct 2015
Posts: 25
PostPosted: Sun 10 Jan '16 3:00    Post subject: Reply with quote
I am trying to use the code below, to log the referrer spam. First redirecting with code 412, then catching the 412 error on the last line, and redirecting to my log file. It's generating the error shown below.

Is it a syntax error, or is what I am trying to do not possible? The 'E=SPAM' variable is from previous attampts, and is not doing anything.

Code:
<IfModule mod_rewrite.c>
RewriteEngine on
Options +FollowSymLinks

RewriteCond %{HTTP_REFERER} !^http://www.my_domain.com/?$
RewriteCond %{HTTP_REFERER} ^http://[^/]*\.[a-zA-Z]{1,4}/?$ [NC]
RewriteRule .* - [E=SPAM:1,R=412,L]

</IfModule>

ErrorDocument 412 /system/services/referrer/filter.php?ref=%{Referer}i

Code:
Precondition Failed
The precondition on the request for the URL / evaluated to false.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.


I also tried this (after changing to R=403 in rewrite rule), but it generated a server error 500:
Code:
<If "%{ENV:SPAM} =~ /1/ | %{ENV:REDIRECT_SPAM} =~ /1/">
    ErrorDocument 403 /system/services/referrer/filter.php?ref=%{Referer}i
</If>
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7373
Location: Germany, Next to Hamburg
PostPosted: Mon 11 Jan '16 11:03    Post subject: Reply with quote
What is in your error log about that?
Back to top
walt



Joined: 24 Oct 2015
Posts: 25
PostPosted: Mon 11 Jan '16 19:33    Post subject: Reply with quote
Hello James, unfortunately it just shows a custom 500 error page from my service provider. I have a shared hosting account, and I don't see anywhere in my directories where the Apache error logs might be, if at all. The only ting I have available are access logs.

However, even when no rules are being triggered in the htaccess file, just un-commenting
Code:
<If "%{ENV:SPAM} =~ /1/ | %{ENV:REDIRECT_SPAM} =~ /1/">
    ErrorDocument 403 /system/services/referrer/filter.php?ref=%{Referer}i
</If>
will generate the 500 error. My service provider might not have a custom page for a 413 error so, for that error, the full text was shown in the browser.

I have also found another possible solution. Apparently Apache variables that were available to the page that caused an error, are also available to the page handling the error*. So, I should be able to drop the query string from this:
Code:
ErrorDocument 412 /system/services/referrer/filter.php?ref=%{Referer}i

I have not had th chance to try ths yet. Still, it would be nice to know how to make the Apache <if></if> work.

Thanks!

*
https://httpd.apache.org/docs/2.4/custom-error.html
Quote:
Redirecting to another URL can be useful, but only if some information can be passed which can then be used to explain or log the error condition more clearly.

To achieve this, when the error redirect is sent, additional environment variables will be set, which will be generated from the headers provided to the original request by prepending 'REDIRECT_' onto the original header name. This provides the error document the context of the original request.
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7373
Location: Germany, Next to Hamburg
PostPosted: Wed 13 Jan '16 13:28    Post subject: Reply with quote
The if Syntax is available in Apache 2.4 or newer. It might be that your provider still uses Apache 2.2 and then your 2.4 syntax causes the error since 2.2 can't process it.
Back to top
walt



Joined: 24 Oct 2015
Posts: 25
PostPosted: Wed 13 Jan '16 16:02    Post subject: Reply with quote
Hello James, yes that appears to be the problem. None of the standard ways to get the Apache version seem to be working (never realized how limited a shared account is). Had to look on their web site: Apache
2.2.24 and 2.2.25

Fortunately there seem to be multiple ways of doing the same thing Smile. I was able to generate a 412 error by putting this at the top of my filter.php file:
Code:
header("HTTP/1.0 412 Not Found");


I cannot check my Awstats at the moment, to see if the spam referrals are now being excluded from the visitor count. The stats take all morning to update, and will not be available till the afternoon.

Just to summarize:
'ErrorDocument 4xx /filter.php' does work on its own, just can't trigger it from a rewrite rule. I suspect my base path is getting altered when triggering a redirect from a rewrite rule. Cannot use an absolute path: http://..../filter.php because Apache will not pass the original (pre-redirect) environment variables to an absolute path.*

Putting a header on first line of filter.php:
'header("HTTP/1.0 412 Not Found")' appears to have the same effect. Also has the advantage of simplifying the htaccess file. Will update post as soon as I can check the updated Awstats.

Thanks!

*https://httpd.apache.org/docs/2.4/custom-error.html
Quote:
REDIRECT_ environment variables are created from the environment variables which existed prior to the redirect. They are renamed with a REDIRECT_ prefix, i.e., HTTP_USER_AGENT becomes REDIRECT_HTTP_USER_AGENT.

REDIRECT_URL, REDIRECT_STATUS, and REDIRECT_QUERY_STRING are guaranteed to be set, and the other headers will be set only if they existed prior to the error condition.

None of these will be set if the ErrorDocument target is an external redirect (anything starting with a scheme name like http:, even if it refers to the same host as the server).
Back to top
James Blond
Moderator


Joined: 19 Jan 2006
Posts: 7373
Location: Germany, Next to Hamburg
PostPosted: Wed 13 Jan '16 17:48    Post subject: Reply with quote
As you can see in your url, there is a 2.4 Wink

The older one for 2.2 is a bit different

https://httpd.apache.org/docs/2.2/custom-error.html
Back to top
walt



Joined: 24 Oct 2015
Posts: 25
PostPosted: Sat 16 Jan '16 21:16    Post subject: Reply with quote
Thanks James, I hadn't noticed that.

This doesn't answer the original question, however just wanted to post the results of the referrer spam filter. The referrals from http://my_domain.com (without www) have disappeared.

Over a period of 24 hrs, the filter has redirected 45 spam referrals (about 15 in triplicate). The Awstats visitor count has dropped from daily mid-50's to low 30's.

Had to add only 2 exceptions for google and facebook. It appears that listing allowed referrers, instead of banned ones, results in much shorter code. On the other hand, all 45 spam referrals come from only 4 ip's.

Code:
RewriteCond %{HTTP_REFERER} !^http://m\.facebook\.com/?$
RewriteCond %{HTTP_REFERER} !^http://www\.google\.[a-z]{1,2}/?$
RewriteCond %{HTTP_REFERER} !^http://www\.my_domain\.com/?$
RewriteCond %{HTTP_REFERER} ^http://[^/]*\.[a-zA-Z0-9_-]+/?$ [NC]
RewriteRule .* /system/services/referrer/filter.php?ip=%{REMOTE_ADDR}&ref=%{HTTP_REFERER} [L]


filter.php*
Code:
<?php
header("HTTP/1.0 412 Precondition Failed");
.
. Code to log ip and referrer
. Code to display apology page with enter button,
. in case of real visitor
?>


* filter.php really has only an 'include' statement. Actual file is outside of the public directory.
Back to top


Reply to topic   Topic: %{HTTP_REFERER} Two different strings for the same referrer View previous topic :: View next topic
Post new topic   Forum Index -> Apache