Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
Topic: Error Logs - Dynamically Changing LogLevel |
|
Author |
|
apishdad
Joined: 01 Jul 2019 Posts: 44 Location: Canada, Toronto
|
Posted: Fri 19 Feb '21 5:12 Post subject: Error Logs - Dynamically Changing LogLevel |
|
|
How can you make apache change the level of logging of its error log during run time?
For example, you want for the most part to have a
Loglevel error
But then if a situation happens where you get an HTTP 404, or 500 error then you want the log level to change to debug.
Is there a way to make the Error Log do that, I know you can do it for custom log, but haven't figured it out for Error Log |
|
Back to top |
|
tangent Moderator
Joined: 16 Aug 2020 Posts: 348 Location: UK
|
Posted: Fri 19 Feb '21 16:39 Post subject: |
|
|
I'm not aware of any method to change the error log level dynamically as you ask, but even if you could, what subsequent event would lower the log level back to what it was before? So what's the real problem that needs extra logging?
40x errors should be straightforward enough to understand and analyse what's causing them; missing content, auth issues, etc.
If you're running a proxy environment, then in my experience 50x errors are normally down to the back end. You may well need to use ProxyErrorOverride to map such events to suitable local error documents, rather than sending a terse error message to the client exposing back end details.
If you do need raised log levels, then you could always pipe the logging to an external script process (Perl or the like), and use that to dynamically filter detail to separate files, but I would be careful about doing this in a production environment.
If you're just trying to reduce the amount of unwanted 40x/50x errors being logged, load balancer health check probes, etc., then you can use some simple tricks, e.g.
Code: | Redirect 404 /favicon.ico
RedirectMatch 404 ^/(MSOffice|_vti_bin|_vti_inf\.html$)
BrowserMatchNoCase HTTP-Monitor DontLog
SetEnvIfNoCase Request_URI ^/favicon.ico$ DontLog
SetEnvIfNoCase Request_URI ^/(MSOffice|_vti_bin|_vti_inf\.html$) DontLog
RewriteCond %{HTTP_USER_AGENT} ^$ [NV]
RewriteRule ^/$ - [E=DontLog:true,NE]
|
Here, the DontLog variable can be used to prevent logging with a CustomLog statement ending in 'env=!DontLog'. |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Fri 19 Feb '21 19:18 Post subject: |
|
|
tangent wrote: | I'm not aware of any method to change the error log level dynamically as you ask |
The LogLevel is a global Server config or a vhost config. Both require a a restart of the webserver.
Since httpd Apache 2.4 does not log the page errors in the error log as it used to be with 2.2 I wrote a script[1] to parse the errors and find out which occur the most to have some hints.
Also Apache 2.4 allows you to configure the LogLevel per module
example
Code: |
LogLevel warn ssl:error auth_digest:error socache_shmcb:warn mpm_worker:warn fcgid:info
|
So maybe you can debug some of the issues without flooding your logs completely
[1] https://github.com/JBlond/apache_log_parser |
|
Back to top |
|
apishdad
Joined: 01 Jul 2019 Posts: 44 Location: Canada, Toronto
|
Posted: Sat 20 Feb '21 8:18 Post subject: |
|
|
Thanks Tangent and James Blond.
I really appreciate your input in this matter.
One final question is that to what extent would debug level logging slow down a server.
I am thinking maybe the best solution in my case is to increase Ram and Processor and get lots of disk space and just put log level at debug level. Just a plain debug. Something like:
LogLevel debug
No module debugging
We keep loglevel at an error level, to keep log files manageable and dont run out of space on the 50 Gig allocated VM machine that we have been given. The issue is that when serious production problems happen, we dont have enough information in the logs to analyze, so at that time we switch to debug level to get more information. |
|
Back to top |
|
tangent Moderator
Joined: 16 Aug 2020 Posts: 348 Location: UK
|
Posted: Sun 21 Feb '21 12:29 Post subject: |
|
|
You're asking a question for which we're unable to give you a simple yes/no answer.
Yes, you could increase the server capacity and storage to help with increased logging, and for a given VM suspect your service will be limited by the I/O throughput.
However, what is the nature of your 'production problems' that makes you think debug rather than error level logging will help solve things?
Do you suspect problems with Apache resource limits, or its functionality and configuration?
Is Apache acting as a proxy to back end services, in which case a whole raft of additional issues come into play, e.g. session affinity, request timeouts, etc.?
What if any load balancers are present in your service configuration? In my time working with Middleware, I've spent many an hour defending Apache services when the real problem turned out to be back-end application servers, e.g. service response delays, inadequate error handling, concurrency and database issues, locked resources, etc.
Appreciating it might be difficult for you to do so, but can you anonymize more detail over the basis of your production problems? |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Sun 21 Feb '21 18:20 Post subject: |
|
|
apishdad wrote: |
I am thinking maybe the best solution in my case is to increase Ram and Processor and get lots of disk space and just put log level at debug level. Just a plain debug. Something like:
LogLevel debug
No module debugging
|
That sets everything in debug, including the modules. That is why I was suggesting to set different levels to prevent log flooding. |
|
Back to top |
|
apishdad
Joined: 01 Jul 2019 Posts: 44 Location: Canada, Toronto
|
Posted: Mon 22 Feb '21 7:07 Post subject: |
|
|
Many thanks to both of you Tangent and James Blond,
To answer Tangent comments here is a bit more information:
In my, scenario Apache is acting as a Reverse Proxy and Load balancing 4-6 JBoss Applications.
Our JBoss applications make a series of API calls to third party vendors to send documents to our clients OR upload/download documents from our vendors
The issues that we have started to happen when the load builds up between 10 am - 2 pm.
Some documents don't get uploaded or downloaded properly.
In such cases, I see a lot of HTTP 403 and 404 errors in my custom logs.
Although I believe much of what I am seeing has to do with the way document management is being handled by the JBoss Application,
but I thought maybe if I put the loglevel at a debug level, I can see more information and be able to trace more of what's happening
by looking at Apache logs and coming to a better conclusion when comparing them with the JBoss Logs.
Sometimes this is not possible so I resort to Wireshark to see what is happening between my Apache server and the back end JBoss applications.
At times this technique works, but most of the time I have to resort to the JBoss logs, which give very limited information as to what is happening.
And as usual, in such times the vendor or the makers of the JBoss application take forever to give proper diagnostics.
So I really need all the info I can get off different systems to put together what is happening.
Thanks |
|
Back to top |
|
tangent Moderator
Joined: 16 Aug 2020 Posts: 348 Location: UK
|
Posted: Mon 22 Feb '21 22:58 Post subject: |
|
|
Thanks for the extra information. So there are TWO service layers behind Apache, either one of which could be the bottleneck here as much as Apache.
If you increase the Apache logging further, using debug, I doubt it's going to tell you very much more, since you already know you're getting 403 and 404 errors, and those error log entries will tell you what user requests they relate to, in both cases no doubt being served from the back end. Going into debug might be a case of going too deep, and not being able to see the wood for the trees.
What proxy balancer algorithm (and session affinity) are you using between Apache and the JBoss servers, and are you allowing session failover if one of the JBoss servers doesn't accept a connection?
If so, then unless your JBoss servers are clustered with session replication, that might explain some of your 403 errors, i.e. under load user requests are being switched to a less heavily loaded JBoss server which doesn't accept their current session id.
Equally, you've got to consider similar session related connections from the JBoss servers through the API calls to your third party vendor services, since presumably they equally require some form of authentication token with user requests. Also, are they presented as a single point of access behind a load balancer, or multiple hosts?
If it were me, my next step would be to run some scripts to capture netstat connection counts (and the time) every few seconds on the Apache and JBoss servers, so you can see what is happening on each tier when problems start. This should help you decide where to look first.
You don't say what OS your servers are running on (Windows or Linux I guess), to know what scripting tools you have to hand or prefer, but there are any number of scripts out there you could adapt, or if you've time write your own of course.
A quick look on the net revealed this Perl script on Stack Overflow, https://stackoverflow.com/questions/32382216/script-that-only-counts-established-time-wait-and-closed-wait-connections-usin, which can easily be updated to report the time (and connections counts by IP if needed), e.g.
2021-02-22 - 19:58:03
ESTABLISHED - 37
TIME_WAIT - 0
CLOSED_WAIT - 0
SYN_SENT - 0
SYN_RECV - 0 If you go down this route, let us know how you get on. |
|
Back to top |
|
apishdad
Joined: 01 Jul 2019 Posts: 44 Location: Canada, Toronto
|
Posted: Thu 25 Feb '21 13:15 Post subject: |
|
|
Hi Tangent,
Thanks for the reply above.
To answer your questions one by one
You mention:
What proxy balancer algorithm (and session affinity) are you using between Apache and the JBoss servers, and are you allowing session failover if one of the JBoss servers doesn't accept a connection?
Answer:
I use the following declaration for my balancer:
<Proxy balancer://MY_BALANCER>
BalancerMember http://SERVER1:8080 loadfactor=1 max=100 connectiontimeout=300 timeout=300 retry=60 route=SERVER1
BalancerMember http://SERVER2:8080 loadfactor=3 max=150 connectiontimeout=300 timeout=300 retry=60 route=SERVER2
BalancerMember http://SERVER3:8080 loadfactor=3 max=150 connectiontimeout=300 timeout=300 retry=60 route=SERVER3
BalancerMember http://SERVER4:8080 loadfactor=2 max=100 connectiontimeout=300 timeout=300 retry=60 route=SERVER4
proxyset lbmethod=byrequests scolonpathdelim=On stickysession=JSESSIONID|jsessionid
</Proxy>
You Mention:
You don't say what OS your servers are running on (Windows or Linux I guess), to know what scripting tools you have to hand or prefer, but there are any number of scripts out there you could adapt, or if you've time write your own of course.
Answer:
Apache is running on windows servers and JBoss is accessing Microsoft SQL servers.
Thanks for all your help Tangent |
|
Back to top |
|
tangent Moderator
Joined: 16 Aug 2020 Posts: 348 Location: UK
|
Posted: Thu 25 Feb '21 21:30 Post subject: |
|
|
In answer to your question, if you really mean the database is suffering from deadlocks (rather than blocking), then I'd suggest your problem lies with the application code combined with the database schema, which your DBA's and code developers need to investigate and resolve.
Periods of heavy load from multiple user requests may well cause blocking within an application, but shouldn't cause deadlocks. Deadlocks will obviously compromise the application functionality, and cause user requests to hang and pile up. In this situation, the connection count in Apache will continue to rise as more user requests come in, until it eventually runs out of resource, which is the effect you're seeing.
Beyond this key issue, I would also revisit your proxy balancer session stickyness configuration in Apache.
I note in one of you earlier posts you mentioned "Our JBoss applications", so is more than one application deployed on a given JBoss server (with different context roots)? If so, unless JBoss has been configured to support session sharing across its applications, then each application will have it's own JSESSIONID. This is separate to any session replication that may be taking place horizontally across your four JBoss servers.
Indeed, for these reasons, I prefer to let Apache manage the proxy balancer routing using it's own cookie, rather than relying on one coming from the application layer, to work correctly. Similarly, I don't like the idea of mixing cookie and URL stickyness which you currently have enabled (scolonpathdelim=On stickysession=JSESSIONID|jsessionid).
In the following configuration, I have used mod_rewrite to set a variable with the current date and time (to the nearest second), which is then prepended to the chosen worker route, e.g. ROUTEID: 20210225185612.s1. Having the client first connect time in the cookie can be quite useful too, should you choose to log it.
Note, proxy balancer documentation says "if the Apache web server finds a dot in the value of the stickyness cookie, it only uses the part behind the dot to search for the route."
Code: | # Define date and time environment variable, for use in session cookies.
#
RewriteEngine On
RewriteCond %{TIME} (.+)
RewriteRule .* - [E=DT:%1,NE]
Header add Set-Cookie "ROUTEID=%{DT}e.%{BALANCER_WORKER_ROUTE}e; path=/; HttpOnly" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://MY_BALANCER>
ProxySet lbmethod=byrequests stickysession=ROUTEID
BalancerMember http://SERVER1:8080 loadfactor=1 max=100 connectiontimeout=300 timeout=300 retry=60 route=s1
BalancerMember http://SERVER2:8080 loadfactor=3 max=150 connectiontimeout=300 timeout=300 retry=60 route=s2
BalancerMember http://SERVER3:8080 loadfactor=3 max=150 connectiontimeout=300 timeout=300 retry=60 route=s3
BalancerMember http://SERVER4:8080 loadfactor=2 max=100 connectiontimeout=300 timeout=300 retry=60 route=s4
</Proxy>
|
You don't say if the JBoss servers are clustered with session replication horizontally, such that user requests can be transferred from one JBoss server to another. If this feature is not present, you should really add nofailover=On to the proxy settings. |
|
Back to top |
|
apishdad
Joined: 01 Jul 2019 Posts: 44 Location: Canada, Toronto
|
Posted: Fri 26 Feb '21 7:35 Post subject: |
|
|
Hi Tangent,
Appreciate your kindness and help in this matter.
To answer some of your questions in regards to the application running on JBOSS:
The JBoss Application is a financial application , same application on 4 different servers. I am not sure what sort of clustering is in place by the vendor, but I know Sessions cannot be passed from one server to the other. You are right, each application has its own JSESSIONID.
As far as the code you have shown me in the previous post, the issue is that with the configuration that you have proposed our firewall strips out the cookies from the header, so the only way it works is with what I have shown you. I had tried similar code to your Stickysession=ROUTEID, but all kind of errors were popping on the screen
I was not aware of the scolonpathdelim=On is causing cookies to be mixed with URL Stickyness. I will try to remove that and test it to see if it breaks anything and I will put your recommendation of NoFailOver=On on the load balancer. |
|
Back to top |
|
tangent Moderator
Joined: 16 Aug 2020 Posts: 348 Location: UK
|
Posted: Fri 26 Feb '21 21:11 Post subject: |
|
|
I can assure you, the cookie based sticky session method detailed above does work, providing that cookie has a dot in front of the chosen route (hence the DT prefix). If you simply set the cookie as "ROUTEID=%{BALANCER_WORKER_ROUTE}e", it won't work.
Next, without having the Apache based sticky session cookie method tested and working first, I wouldn't advocate removing the mixed cookie and URL based stickiness options, unless your code developers confirm this is acceptable for the application. According to the documentation, it's the statement "stickysession=JSESSIONID|jsessionid" that enables this URL based stickiness feature, whilst "scolonpathdelim=On" simply says it's acceptable to have a semicolon in that URL based session ID, e.g. something of the form http://www.example.com/context;JSESSIONID=12345?query_param=ABC
Beyond knowing it's JSESSIONID, you've not said how that session detail is being provided by the application. To which end I'm very surprised to hear you say the firewall is stripping out cookies, since that implies JSESSIONID would have to be passed in the URL, which is significantly less secure than a cookie based approach.
However. if your application firewall does allow a JSESSIONID cookie, then surely you can request it also passes a ROUTEID cookie too?
Also, don't you have a test/pre-production environment where these issues can be reviewed, tested and ironed out?
Being an Apache forum, based on the the details posted so far, I'm not sure we can go much further with advice on where to go next. You're facing problems that require Middleware expertise across the whole application stack; not just the web layer. Indeed, if I was being asked to configure and support this Apache service, I'd expect a bit more flexibility and co-operation from the rest of the application team. |
|
Back to top |
|
apishdad
Joined: 01 Jul 2019 Posts: 44 Location: Canada, Toronto
|
Posted: Tue 02 Mar '21 19:53 Post subject: |
|
|
Hi Tangent,
Thanks a lot for all your help. You have done great mentoring and guidance in my understanding of this.
I think I have enough information and help to go and spend the rest of the time to do testing to improve our proxy servers.
Thanks again |
|
Back to top |
|
|
|
|
|
|