Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: [doubts] HIGH traffic website on ASF Apache + JBoss |
|
| Author |
|
alabastida
Joined: 15 Aug 2008
Posts: 1
Location: Mexico
|
 Posted: Fri 15 Aug '08 2:10 Post subject: [doubts] HIGH traffic website on ASF Apache + JBoss |
 |
|
Hello, how are you my fellow engineers and development fanatics?
Im working currently on a high traffic website, developed for internet bus ticket sales.
We deployed our application, wich is done with Java on the backend and mostly JQuery on the front-end, over the next structure:
Apache SSL Version: apache_2.2.9-win32-x86-openssl-0.9.8h-r2 downloaded from ASF
JBoss Version: jboss-4.0.2 from jboss.org 
Apache serves all the static content, JBoss takes care of dynamic content.
The server tech Specs are:
Intel(R) Xeon(R) CPU
E5405 @ 2.00GHz
2.00 GHz, 3.99 GB of RAM
(8 CPU's)
The OS is:
Microsoft Windows Server 2003 R2
The website currently has an average of 400 recurrent users.
So, we deployed our new application, yesterday Wednesday Agst. 13, at 20:00 hours.
Today @ noon, we recieved a notification that the website was not responding.
So i connected remotely to the server, and noticed that the httpd.exe process was still running and also the java.exe for JBoss.
Although httpd was not responding, the simple solution was to restart the apache service and it was working fine again.
I went to the w2k3 Event Viewer and i saw this, it happened today @ 12:54 hrs :
| Code: |
Application Error
Faulting application httpd.exe, version 2.2.9.0, faulting module unknown, version 0.0.0.0, fault address 0x00000000.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. |
after almost 2 hours from the first error, i was monitoring the apache's behavior, and suddenly @ 14:58 hrs the same issue happened again:
| Code: |
Application Error
Faulting application httpd.exe, version 2.2.9.0, faulting module unknown, version 0.0.0.0, fault address 0x00000000.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. |
After that ocurred, i restarted the apache service one more time, and until this very moment im posting this (19:10 hrs), im not having any similar issue again.
So my question is:
Do you guys have any idea of what could ever happened?
Some things that i verified are:
- The memory was very stable from start to crash both times @ 150mb in Task Manager, and right now it is too @ that level.
- The Apache error log DIDNT capture anything related to the issue.
- JBoss is not the problem, its working flawlessly, latest GC Log:
PSPermGen: 49839K->49839K(131072K).
- Im guessing that it might be the AV Software running on the server (i remember that one time the sysadmin told me that the AV runs from 13:00hrs to 15:00hrs everyday, so, could that be a coincidence?), the one that is being used there is:
Trend Micro OfficeScan 8.0
And now, let me show you the sintetized httpd.conf contents im using:
| Code: | ServerRoot "C:/Program Files/Apache Software Foundation/Apache2.2"
ServerSignature Off
ServerTokens Prod
Listen 80
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dir_module modules/mod_dir.so
LoadModule env_module modules/mod_env.so
LoadModule include_module modules/mod_include.so
LoadModule isapi_module modules/mod_isapi.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule mime_module modules/mod_mime.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule cache_module modules/mod_cache.so
<IfModule mod_cache.so>
LoadModule disk_cache_module modules/mod_disk_cache.so
<IfModule mod_disk_cache.so>
CacheRoot C:\Program Files\Apache Software Foundation\Apache2.2\cache
CacheSize 256
CacheEnable disk /Webapp/fb
CacheEnable disk /Webapp/css
CacheEnable disk /Webapp/rb
CacheEnable disk /Webapp/tools
CacheEnable disk /Webapp/html
CacheEnable disk /Webapp/img
CacheDirLevels 5
CacheDirLength 3
</IfModule>
</IfModule>
Win32DisableAcceptEx
<IfModule mpm_winnt_module>
EnableMMAP off
EnableSendfile off
ThreadsPerChild 768
MaxRequestsPerChild 0
</IfModule>
ServerAdmin webmaster@customerdomain.com
ServerName portal.customerdomain.com
DocumentRoot "C:/Program Files/Apache Software Foundation/Apache2.2/htdocs"
<Directory />
Order Allow,Deny
Allow from all
Options -Indexes +FollowSymLinks -SymLinksIfOwnerMatch
AllowOverride None
</Directory>
<Directory "C:/Program Files/Apache Software Foundation/Apache2.2/htdocs">
Options -Indexes +FollowSymLinks -SymLinksIfOwnerMatch
AllowOverride None
Order allow,deny
Allow from all
</Directory>
<IfModule dir_module>
DirectoryIndex index.html
</IfModule>
<FilesMatch "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</FilesMatch>
ErrorLog "logs/error.log"
LogLevel warn
<IfModule log_config_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
<IfModule logio_module>
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
</IfModule>
CustomLog "logs/access.log" combined
</IfModule>
<IfModule alias_module>
ScriptAlias /cgi-bin/ "C:/Program Files/Apache Software Foundation/Apache2.2/cgi-bin/"
</IfModule>
<Directory "C:/Program Files/Apache Software Foundation/Apache2.2/cgi-bin">
AllowOverride None
Options None
Order allow,deny
Allow from all
</Directory>
DefaultType text/plain
<IfModule mime_module>
TypesConfig conf/mime.types
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz
</IfModule>
Listen 443
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
SSLSessionCache "shmcb:C:/Program Files/Apache Software Foundation/Apache2.2/logs/ssl_scache(512000)"
SSLMutex default
<VirtualHost *:443>
SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile "C:/Program Files/Apache Software Foundation/Apache2.2/conf/Portal.crt"
SSLCertificateKeyFile "C:/Program Files/Apache Software Foundation/Apache2.2/conf/Portal.key"
ErrorLog "C:/Program Files/Apache Software Foundation/Apache2.2/logs/ssl_error.log"
TransferLog "C:/Program Files/Apache Software Foundation/Apache2.2/logs/access.log"
ProxyRequests Off
ProxyPass /Webapp/css !
ProxyPass /Webapp/fb !
ProxyPass /Webapp/html !
ProxyPass /Webapp/img !
ProxyPass /Webapp/jackbe !
ProxyPass /Webapp/jf !
ProxyPass /Webapp/rb !
ProxyPass /Webapp/tools !
ProxyPass /Webapp/jsp/galerias !
ProxyPass /Webapp/jsp/bus !
ProxyPass /Webapp/jsp/Scripts !
ProxyPassMatch /Webapp/jsp/*.swf !
ProxyPassMatch /Webapp/jsp/*.pdf !
ProxyPassMatch /Webapp/jsp/*.xml !
ProxyPassMatch /Webapp/jsp/*.ppt !
ProxyPassMatch /Webapp/jsp/*.avi !
ProxyPassMatch /Webapp/jsp/*.jpg !
ProxyPassMatch /Webapp/jsp/*.JPG !
ProxyPassMatch /Webapp/jsp/*.html !
ProxyPass /Webapp http://portal.customerdomain.com:8080/Webapp
ProxyPassReverse /Webapp http://portal.customerdomain.com:8080/Webapp
ProxyPass /presto http://portal.customerdomain.com:8080/presto
ProxyPassReverse /presto http://portal.customerdomain.com:8080/presto
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory "C:/Program Files/Apache Software Foundation/Apache2.2/cgi-bin">
SSLOptions +StdEnvVars
</Directory>
BrowserMatch ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
</virtualHost>
<VirtualHost *:80>
ProxyRequests Off
ProxyPass /Webapp/css !
ProxyPass /Webapp/fb !
ProxyPass /Webapp/html !
ProxyPass /Webapp/img !
ProxyPass /Webapp/jackbe !
ProxyPass /Webapp/jf !
ProxyPass /Webapp/rb !
ProxyPass /Webapp/tools !
ProxyPass /Webapp/jsp/galerias !
ProxyPass /Webapp/jsp/bus !
ProxyPass /Webapp/jsp/Scripts !
ProxyPassMatch /Webapp/jsp/*.swf !
ProxyPassMatch /Webapp/jsp/*.pdf !
ProxyPassMatch /Webapp/jsp/*.xml !
ProxyPassMatch /Webapp/jsp/*.ppt !
ProxyPassMatch /Webapp/jsp/*.avi !
ProxyPassMatch /Webapp/jsp/*.jpg !
ProxyPassMatch /Webapp/jsp/*.JPG !
ProxyPass /Webapp http://portal.customerdomain.com:8080/Webapp
ProxyPassReverse /Webapp http://portal.customerdomain.com:8080/Webapp
ProxyPass /presto http://portal.customerdomain.com:8080/presto
ProxyPassReverse /presto http://portal.customerdomain.com:8080/presto
</virtualHost>
Include conf/expires.conf
LoadModule deflate_module modules/mod_deflate.so
SetOutputFilter DEFLATE
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule> |
The expires.conf content its very simple so im not putting it right now, only if you guys request it for something.
Please let me know if you need anything else to try and see what could be happening here.
Thanks a lot in advance for your helpful and needed advices.
Edit: Also im guessing that i maybe running out of threads, what do you think about that? Thanks!
Adrian Labastida |
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7373
Location: Germany, Next to Hamburg
|
| Posted: Fri 15 Aug '08 15:23 Post subject: |
|
|
| Which Event id do you see in the event log? |
|
| Back to top |
|
tdonovan
Moderator
Joined: 17 Dec 2005
Posts: 611
Location: Milford, MA, USA
|
| Posted: Sat 16 Aug '08 16:41 Post subject: |
|
|
Whoa! too much config file posted...
The AV is a good guess - there have been vulnerabilities in Trend Micro OfficeScan recently. A good starting point is to go to the U.S. National Vulnerability Database and search for "Trend Micro OfficeScan" for the last 3 years. You should make sure your copy has patches for all the recent 8.0 vulnerabilities.
Another good utility to diagnose this kind of problem is Microsoft's Process Explorer. Run Process Explorer on your system and view the DLLs in the lower window for the httpd process (the second, most indented httpd process) and check the modules which are loaded. Since your Event showed "faulting module unknown, version 0.0.0.0", look for a module without a version number as a likely culprit.
Finally, you may want to enable the Dr. Watson for Windows (drwtsn32) postmortem debugger on your system. See the Windows Server 2003 Help for how to do this. The \Documents and Settings\All Users\Application Data\Microsoft\Dr Watson\Drwtsn32.log file might contain better information than the Windows Event Log if this problem happens again.
Hope this helps,
-tom- |
|
| Back to top |
|
|
|
|
|
|
