Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: Apache stopped working permanently after error "Server |
|
| Author |
|
leocda
Joined: 29 Jun 2016
Posts: 2
Location: Brazil
|
 Posted: Wed 29 Jun '16 9:31 Post subject: Apache stopped working permanently after error "Server |
 |
|
Hi.
I have been working with Apache for more than 8 years, but now I'm facing a severe problem and I couldn't find a solution on internet so far.
I'm running Apache 2.2.15 32bit on a Windows 2008 R2 64bit with Service Pack 1. I always used a 32bit Apache because my CGI codes (exe files) are 32bit.
The computer has a 3.10 GHz processor and 16GB RAM. The server load is always very low. The access log showed a normal request amount when the problem happened.
Since the installation many months ago, everything was running correctly, but now my Apache stopped working and I don't know how to fix this problem.
It has been many weeks I don't change any configuration, but suddenly the Apache stopped serving the HTTP responses. All requests get timed out. In a practical sense, my site is offline.
By using Windows Event Viewer, I have found an error log when Apache stopped working. It's the error event:
| Code: |
Log Name: Application
Source: Application Error
Date: 6/28/2016 8:08:33 PM
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: PictureToPeople
Description:
Faulting application name: httpd.exe, version: 2.2.15.0, time stamp: 0x4b8fed95
Faulting module name: ntdll.dll, version: 6.1.7601.17514, time stamp: 0x4ce7ba58
Exception code: 0xc0000005
Fault offset: 0x00037188
Faulting process id: 0xbbc
Faulting application start time: 0x01d1cdfef0f8148c
Faulting application path: C:\Apache2.2\bin\httpd.exe
Faulting module path: C:\Windows\SysWOW64\ntdll.dll
Report Id: 3c1e0a43-3d85-11e6-aad3-001e8cf18221
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Application Error" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>100</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2016-06-28T23:08:33.000000000Z" />
<EventRecordID>693</EventRecordID>
<Channel>Application</Channel>
<Computer>PictureToPeople</Computer>
<Security />
</System>
<EventData>
<Data>httpd.exe</Data>
<Data>2.2.15.0</Data>
<Data>4b8fed95</Data>
<Data>ntdll.dll</Data>
<Data>6.1.7601.17514</Data>
<Data>4ce7ba58</Data>
<Data>c0000005</Data>
<Data>00037188</Data>
<Data>bbc</Data>
<Data>01d1cdfef0f8148c</Data>
<Data>C:\Apache2.2\bin\httpd.exe</Data>
<Data>C:\Windows\SysWOW64\ntdll.dll</Data>
<Data>3c1e0a43-3d85-11e6-aad3-001e8cf18221</Data>
</EventData>
</Event>
|
It's what Apache error log recorded near to the time of error above (6/28/2016 8:08:33). It's possible to see the
error message "Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting".
After this message, Apache never worked again.
(Previous Apache error log messages were recorded very earlier and are not related.)
| Code: |
[Tue Jun 28 20:08:30 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
[Tue Jun 28 20:08:31 2016] [notice] Child 3004: Exit event signaled. Child process is ending.
[Tue Jun 28 20:08:32 2016] [info] Child 3004: Accept thread exiting.
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: Released the start mutex
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: Terminating 99 threads that failed to exit.
[Tue Jun 28 20:08:32 2016] [notice] Child 3004: All worker threads have exited.
[Tue Jun 28 20:08:32 2016] [info] [client 1.23.96.161] (OS 10038)An operation was attempted on something that is not a socket. : core_output_filter: writing data to the network
[Tue Jun 28 20:08:34 2016] [notice] Parent: Child process exited successfully.
[Tue Jun 28 20:08:34 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=284)
[Tue Jun 28 20:08:54 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 20:08:54 2016] [notice] Server built: Mar 4 2010 11:27:46
[Tue Jun 28 20:08:54 2016] [notice] Parent: Created child process 2468
[Tue Jun 28 20:08:54 2016] [info] Parent: Duplicating socket 308 and sending it to child process 2468
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Child process is running
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Acquired the start mutex.
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Starting 100 worker threads.
[Tue Jun 28 20:08:54 2016] [notice] Child 2468: Starting thread to listen on port 80.
[Tue Jun 28 20:08:56 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
|
After the error above, restarting Windows and Apache service didn't solve the problem (same error message).
After reinstalling Apache and booting the machine, the problem was still there (same error message).
So, I increased the "MaxRequestsPerChild" parameter to 300, but all worker threads were consumed in 12 seconds.
So, I increased the "MaxRequestsPerChild" parameter to 500 and rebooted the machine, but all worker threads were consumed in 9 seconds as the log shows below.
| Code: |
[Tue Jun 28 21:55:01 2016] [notice] Parent: Received shutdown signal -- Shutting down the server.
[Tue Jun 28 21:55:01 2016] [notice] Child 864: Exit event signaled. Child process is ending.
[Tue Jun 28 21:55:02 2016] [info] Child 864: Accept thread exiting.
[Tue Jun 28 21:55:02 2016] [notice] Child 864: Released the start mutex
[Tue Jun 28 21:55:20 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [error] [client 1.23.96.161] request failed: error reading the headers
[Tue Jun 28 21:55:31 2016] [notice] Parent: Forcing termination of child process 368
[Tue Jun 28 21:55:31 2016] [info] removed PID file C:/Apache2.2/logs/httpd.pid (pid=2252)
[Tue Jun 28 22:10:13 2016] [notice] Apache/2.2.15 (Win32) configured -- resuming normal operations
[Tue Jun 28 22:10:13 2016] [notice] Server built: Mar 4 2010 11:27:46
[Tue Jun 28 22:10:13 2016] [notice] Parent: Created child process 1452
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Child process is running
[Tue Jun 28 22:10:13 2016] [info] Parent: Duplicating socket 308 and sending it to child process 1452
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Acquired the start mutex.
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Starting 500 worker threads.
[Tue Jun 28 22:10:13 2016] [notice] Child 1452: Starting thread to listen on port 80.
[Tue Jun 28 22:10:22 2016] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild setting
|
Below you can see my complete original httpd.conf. It had been working correctly for months.
(I removed just all the comment lines to make it smaller and more readable, and also masked some minimal sensible data with "xxxxx"):
| Code: | ServerRoot "C:/Apache2.2"
Listen 80
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule include_module modules/mod_include.so
LoadModule isapi_module modules/mod_isapi.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
<IfModule !mpm_netware_module>
<IfModule !mpm_winnt_module>
User daemon
Group daemon
</IfModule>
</IfModule>
ServerAdmin xxxxxxxxxxxxxxxxxxxxxxxx
ServerName xxxxxxxxxxxxxxxxxxxxxxxx:80
DocumentRoot "xxxxxxxxxxxxxxxxxxxxxxxx"
<Directory />
Options FollowSymLinks
AllowOverride None
Order deny,allow
Deny from all
</Directory>
<Directory "xxxxxxxxxxxxxxxxxxxxxxxx">
Options FollowSymLinks
Options +Includes
Options -Indexes
AllowOverride None
Order allow,deny
Allow from all
LimitRequestBody 10485760
SetOutputFilter DEFLATE
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip
</Directory>
<IfModule dir_module>
DirectoryIndex index.html
</IfModule>
<FilesMatch "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</FilesMatch>
<IfModule mod_headers.c>
Header always append X-Frame-Options SAMEORIGIN
</IfModule>
<IfModule mod_expires.c>
ExpiresActive on
ExpiresDefault "access plus 1 week"
ExpiresByType text/css "access plus 1 week"
ExpiresByType application/atom+xml "access plus 1 hour"
ExpiresByType application/rdf+xml "access plus 1 hour"
ExpiresByType application/rss+xml "access plus 1 hour"
ExpiresByType application/json "access plus 0 seconds"
ExpiresByType application/ld+json "access plus 0 seconds"
ExpiresByType application/schema+json "access plus 0 seconds"
ExpiresByType application/vnd.geo+json "access plus 0 seconds"
ExpiresByType application/xml "access plus 0 seconds"
ExpiresByType text/xml "access plus 0 seconds"
ExpiresByType image/vnd.microsoft.icon "access plus 1 week"
ExpiresByType image/x-icon "access plus 1 week"
ExpiresByType text/html "access plus 1 day"
ExpiresByType application/javascript "access plus 1 week"
ExpiresByType application/x-javascript "access plus 1 week"
ExpiresByType text/javascript "access plus 1 week"
ExpiresByType application/manifest+json "access plus 1 year"
ExpiresByType application/x-web-app-manifest+json "access plus 0 seconds"
ExpiresByType text/cache-manifest "access plus 0 seconds"
ExpiresByType audio/ogg "access plus 1 month"
ExpiresByType image/bmp "access plus 1 month"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/svg+xml "access plus 1 month"
ExpiresByType video/mp4 "access plus 1 month"
ExpiresByType video/ogg "access plus 1 month"
ExpiresByType video/webm "access plus 1 month"
<Directory xxxxxxxxxxxxxxxxxxxxxxxx>
ExpiresByType image/gif "access plus 0 seconds"
ExpiresByType image/jpeg "access plus 0 seconds"
ExpiresByType image/png "access plus 0 seconds"
</Directory>
ExpiresByType application/vnd.ms-fontobject "access plus 1 month"
ExpiresByType font/eot "access plus 1 month"
ExpiresByType font/opentype "access plus 1 month"
ExpiresByType application/x-font-ttf "access plus 1 month"
ExpiresByType application/font-woff "access plus 1 month"
ExpiresByType application/x-font-woff "access plus 1 month"
ExpiresByType font/woff "access plus 1 month"
ExpiresByType application/font-woff2 "access plus 1 month"
ExpiresByType text/x-cross-domain-policy "access plus 1 week"
</IfModule>
ErrorLog "|bin/rotatelogs logs/error.%Y-%m-%d.log 50M"
LogLevel info
<IfModule log_config_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
<IfModule logio_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
</IfModule>
CustomLog "|bin/rotatelogs logs/access.%Y-%m-%d---%H.log 300M" common
</IfModule>
<IfModule alias_module>
ScriptAlias /p2p/ "xxxxxxxxxxxxxxxxxxxxxxxx"
</IfModule>
<IfModule cgid_module>
</IfModule>
<Directory "xxxxxxxxxxxxxxxxxxxxxxxx">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
DefaultType text/plain
<IfModule mime_module>
TypesConfig conf/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
AddType text/html .html
AddHandler server-parsed .html
</IfModule>
<IfModule mpm_winnt_module>
ThreadsPerChild 100
MaxMemFree 100
MaxRequestsPerChild 5000
</IfModule>
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>
RewriteEngine On
RewriteCond %{HTTP_HOST} !^www
RewriteRule (.*) http://www.%{HTTP_HOST}$1 [L,R]
|
As shown above, my Apache stops responding almost instantly after launch.
My site doesn't connect to databases. It just serves dynamic pages by using CGI.
Is there a way to clean up the system to make Apache work again as it used to work before this mysterious crash?
Could anyone give me any advice to solve this issue or at least to get a workaround?
Thanks in advance. |
|
| Back to top |
|
jraute
Joined: 13 Sep 2013
Posts: 188
Location: Rheinland, Germany
|
| Posted: Thu 30 Jun '16 8:51 Post subject: Re: Apache stopped working permanently after error "Ser |
|
|
| Quote: |
| Code: |
<IfModule mpm_winnt_module>
ThreadsPerChild 100
MaxMemFree 100
MaxRequestsPerChild 5000
</IfModule>
|
|
Why using MaxMemFree 100 ?
I would suggest this for a test:
| Code: |
<IfModule mpm_winnt_module>
ThreadsPerChild 250
MaxConnectionsPerChild 0
MaxRequestsPerChild 0
</IfModule>
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
EnableMMAP Off
EnableSendfile Off
AcceptFilter http none
AcceptFilter https none
|
And if you get problems with the memory-usage then you can try to limit the MaxConnectionsPerChild and MaxRequestsPerChild values.
Best regards
JR |
|
| Back to top |
|
leocda
Joined: 29 Jun 2016
Posts: 2
Location: Brazil
|
| Posted: Thu 30 Jun '16 12:26 Post subject: |
|
|
Thanks for trying to help. Very appreciated.
| Quote: | | Why using MaxMemFree 100 ? |
Because over the year this configuration made my Apache faster for this application. I never had memory problems.
| Quote: | <IfModule mpm_winnt_module>
ThreadsPerChild 250
MaxConnectionsPerChild 0
MaxRequestsPerChild 0
</IfModule> |
As you can see at my first post, I have tried even "ThreadsPerChild 500", but it didn't help.
Apache documentation recommends "MaxRequestsPerChild 0" on Windows, but changing this value made Apache work for some seconds before hanging again, so I let this change for now. In fact, testing several values for MaxRequestsPerChild over the years, I could see it cause problems mainly if you rotate logs, because when new Apache processes are created, the child rotatelogs are left behind. If I can solve the issue, I intend to return this to zero.
| Quote: | EnableMMAP Off
EnableSendfile Off
AcceptFilter http none
AcceptFilter https none
|
I have seen people recommending this configuration above, but always for a completely different problem. Anyway, I will try it.
| Quote: | KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
|
Since the symptom is that Apache threads never return, what could be a explanation for the immediate out of threads, it looks like this config could make things even worse. Anyway, if the previous change does't help, I will try this one as well.
I replaced the original web server for an Apache 2.2.31 32bits. I also tried the Apache 2.2.31 64bits. They were compiled with Visual Studio (downloaded from Apache Haus), but the result was the same in all cases: Apache got out of threads in a few seconds.
It's very sad to see that restarting Apache or even rebooting the machine doesn't solve the issue not even for a moment. Apache starts, so immediately generates the "Server ran out of threads to serve requests" error and hangs. |
|
| Back to top |
|
jraute
Joined: 13 Sep 2013
Posts: 188
Location: Rheinland, Germany
|
| Posted: Fri 01 Jul '16 17:05 Post subject: |
|
|
Btw why not using a 2.4.20 version?
If you ask the experts in the forum they would recommend that version, for sure. |
|
| Back to top |
|
|
|
|
|
|
