Author |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
Posted: Sun 10 May '15 14:42 Post subject: Apache and PHP-FPM fastcgi timeout |
|
|
I've noticed that my web server has occasional 1-5 minute outages every few hours..
I've checked the Apache error log and found the following:
Code: | [Sun May 10 14:13:19.299784 2015] [fastcgi:error] [pid 2599:tid 139669761148672] [client 174.34.156.130:13278] FastCGI: comm with server "/usr/lib/cgi-bin/php5-fcgi" aborted: idle timeout (30 sec)
[Sun May 10 14:13:19.299855 2015] [fastcgi:error] [pid 2599:tid 139669761148672] [client 174.34.156.130:13278] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php5-fcgi"
[Sun May 10 14:14:00.782370 2015] [fastcgi:error] [pid 2473:tid 139669735970560] [client 82.103.128.63:45704] FastCGI: comm with server "/usr/lib/cgi-bin/php5-fcgi" aborted: idle timeout (30 sec)
[Sun May 10 14:14:00.782432 2015] [fastcgi:error] [pid 2473:tid 139669735970560] [client 82.103.128.63:45704] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php5-fcgi"
[Sun May 10 14:14:19.124915 2015] [fastcgi:error] [pid 2473:tid 139669786326784] [client 188.138.118.184:34672] FastCGI: comm with server "/usr/lib/cgi-bin/php5-fcgi" aborted: idle timeout (30 sec)
[Sun May 10 14:14:19.124962 2015] [fastcgi:error] [pid 2473:tid 139669786326784] [client 188.138.118.184:34672] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php5-fcgi"
[Sun May 10 14:14:33.978792 2015] [fastcgi:error] [pid 2473:tid 139669643650816] [client 82.103.128.63:11778] FastCGI: comm with server "/usr/lib/cgi-bin/php5-fcgi" aborted: idle timeout (30 sec)
[Sun May 10 14:14:33.978853 2015] [fastcgi:error] [pid 2473:tid 139669643650816] [client 82.103.128.63:11778] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php5-fcgi"
[Sun May 10 14:14:34.380783 2015] [fastcgi:error] [pid 2598:tid 139669744363264] [client 174.34.156.130:46479] FastCGI: comm with server "/usr/lib/cgi-bin/php5-fcgi" aborted: idle timeout (30 sec)
[Sun May 10 14:14:34.380843 2015] [fastcgi:error] [pid 2598:tid 139669744363264] [client 174.34.156.130:46479] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php5-fcgi"
[Sun May 10 14:15:19.518501 2015] [fastcgi:error] [pid 2598:tid 139669685614336] [client 76.164.194.74:28967] FastCGI: comm with server "/usr/lib/cgi-bin/php5-fcgi" aborted: idle timeout (30 sec)
[Sun May 10 14:15:19.518575 2015] [fastcgi:error] [pid 2598:tid 139669685614336] [client 76.164.194.74:28967] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php5-fcgi"
[Sun May 10 14:16:19.404843 2015] [fastcgi:error] [pid 2598:tid 139669727577856] [client 50.23.94.74:23923] FastCGI: comm with server "/usr/lib/cgi-bin/php5-fcgi" aborted: idle timeout (30 sec)
[Sun May 10 14:16:19.404894 2015] [fastcgi:error] [pid 2598:tid 139669727577856] [client 50.23.94.74:23923] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php5-fcgi"
[Sun May 10 14:17:19.210294 2015] [fastcgi:error] [pid 2598:tid 139669769541376] [client 85.17.156.99:15068] FastCGI: comm with server "/usr/lib/cgi-bin/php5-fcgi" aborted: idle timeout (30 sec)
[Sun May 10 14:17:19.210368 2015] [fastcgi:error] [pid 2598:tid 139669769541376] [client 85.17.156.99:15068] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php5-fcgi" |
I've looked in the syslog and php5-fpm.log but couldn't find any errors.
in my php.ini I had set: error_log = /var/log/php_errors.log
But this file is not being generated, while log_errors is turned to on.
Also with error_log = syslog no errors are being reported in the syslog regarding PHP.
Any idea what I can do to resolve this problem?
I'm using Ubuntu server 15.04 x64 |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
|
Back to top |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
Posted: Sun 10 May '15 23:56 Post subject: |
|
|
Does this mean that a visitor has been waiting 30 sec or longer for his request? If so doesn't this mean there is a problem with my PHP application?
I didn't have this problem on a similar virtual machine (hosting the same site but on Ubuntu 14.10) |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Mon 11 May '15 18:01 Post subject: |
|
|
Yes, from the start of the request where wasn't any output (or writing to the disc) within 30 seconds. |
|
Back to top |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
Posted: Mon 11 May '15 18:14 Post subject: |
|
|
How can I troubleshoot that? I would like to know why that is happening. |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Tue 19 May '15 17:54 Post subject: |
|
|
Slow internet connection? Slow dos attack? Something else? hard to tell without a log analyse. |
|
Back to top |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
Posted: Fri 06 May '16 20:54 Post subject: |
|
|
James Blond wrote: | Slow internet connection? Slow dos attack? Something else? hard to tell without a log analyse. |
I'm still having the same problem even with PHP 7 on Ubuntu 16.04 and the latest Joomla 3.5.1 version.
The only errors appear in the Apache3 error.log:
Code: | [Fri May 06 20:44:20.668626 2016] [fastcgi:error] [pid 6064:tid 139747573860096] [client 5.199.157.1:32057] FastCGI: comm with server "/usr/lib/cgi-bin/php7.0" aborted: idle timeout (30 sec), referer: https://www.xgclan.com/administrator/index.php
[Fri May 06 20:44:20.668682 2016] [fastcgi:error] [pid 6064:tid 139747573860096] [client 5.199.157.1:32057] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php7.0", referer: https://www.xgclan.com/administrator/index.php
[Fri May 06 20:44:20.789615 2016] [fastcgi:error] [pid 6063:tid 139747715245824] [client 5.199.157.1:61446] FastCGI: comm with server "/usr/lib/cgi-bin/php7.0" aborted: idle timeout (30 sec), referer: https://www.xgclan.com/administrator/index.php
[Fri May 06 20:44:20.789671 2016] [fastcgi:error] [pid 6063:tid 139747715245824] [client 5.199.157.1:61446] FastCGI: incomplete headers (0 bytes) received from server "/usr/lib/cgi-bin/php7.0", referer: https://www.xgclan.com/administrator/index.php |
I've checked the syslog but it doesn't show any errors. The php7.0-fpm.log also doesn't show anything odd.
I've enabled all logging functions in PHP:
Code: | error_reporting = E_ALL
display_errors = On
display_startup_errors = On
log_errors = On
log_errors_max_len = 0
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = On
html_errors = On
error_log = /var/log/php_errors.log
|
This all doesn't help, I only get a 500 server error page and the apache log, which isn't useful.
Do you have any suggestions or idea's on what to do to get the needed log information which can tell me why there are problems?
I'm able to reproduce the error in Joomla when visiting the check extensions or check Joomla version pages. It's most likely related to a server side setting (in PHP) being wrong. I've already checked that curl and allow_url_fopen are turned on.
On some other pages the error appears every now and then, reloading the page a few times usually resolves it and on some pages it never even happens afaik. |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Mon 09 May '16 20:09 Post subject: |
|
|
Do you use mod_fcgid? Mod_fastcgi or mod_proxy_fcgi? |
|
Back to top |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
Posted: Mon 09 May '16 20:27 Post subject: |
|
|
I'm hosting the same website on Windows and there the pages with issues render in 1-2 seconds tops.
Those pages shouldn't take 30 seconds or longer to load, so something must be wrong.
I should mention that I'm using mod_fcgid on windows with a config that was suggested here.
And that I'm not using mod_fcgid on Linux, instead I'm using mod_fastcgi on Linux.
Why I'm using fastcgi on Linux when I used fcgid on Windows?
Here it says that fastcgi is faster than fcgid: https://superuser.com/questions/228173/whats-the-difference-between-mod-fastcgi-and-mod-fcgid
Some old article's also say I can't use PHP-FPM with mod_fcgid. But I'm not sure if that situation has changed since late 2011.
So that's why I'd like to get mod_fastcgi working correctly.
My current mod_fastcgi config looks like this:
Code: | <IfModule mod_fastcgi.c>
AddType application/x-httpd-fastphp7.0 .php
Action application/x-httpd-fastphp7.0 /php7.0-fcgi
Alias /php7.0-fcgi /usr/lib/cgi-bin/php7.0
FastCgiExternalServer /usr/lib/cgi-bin/php7.0 -socket /var/run/php/php7.0-fpm.sock -pass-header Authorization
<Directory /usr/lib/cgi-bin>
Require all granted
</Directory>
</IfModule>
|
|
|
Back to top |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Sat 25 Jun '16 22:49 Post subject: |
|
|
You can use mod_proxy_fcgid for PHP FPM.
On my server I tried mod_fcgid vs mod_proxy_fcgid with FPM. There wasn't a speed difference.
Now over years using mod_fcgid on windows and linux I don't see the reason to use mod_fastcgi. Speed? I don't think so.
When was the last code change in mod_fastcgi. 2011 / 2012 ??
I say give my setup a try https://github.com/JBlond/debian_build_apache24/
Run the self compile apache on a port of your choise and compare. |
|
Back to top |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
Posted: Sat 25 Jun '16 23:34 Post subject: |
|
|
So we can use mod_fcgid with PHP-FPM? I'm really confused because a lot of sources say it can't.
I'm trying the proxy approach but I've heard it's less secure and it has some issues for me, which I assume can be resolved with configuration options. |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Sun 26 Jun '16 18:35 Post subject: |
|
|
gijs wrote: | So we can use mod_fcgid with PHP-FPM? I'm really confused because a lot of sources say it can't.
|
Nope, you can't. I said mod_proxy_fcgi.
gijs wrote: |
I'm trying the proxy approach but I've heard it's less secure and it has some issues for me, which I assume can be resolved with configuration options. |
Issues? What kind of issues?
You mean this?
Quote: |
Unlike mod_fcgid and mod_fastcgi, mod_proxy_fcgi has no provision for starting the application process; fcgistarter is provided (on some platforms) for that purpose. Alternatively, external launching or process management may be available in the FastCGI application framework in use.
|
The thing I like most about that module is that you can run several instances aka clustering and load balancing. |
|
Back to top |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
Posted: Sun 26 Jun '16 19:04 Post subject: |
|
|
Thank you for the fast reply James
My goal is to have the best performance, which requires UNIX sockets and PHP-FPM if I understood correctly. But security is a top priority together with stability and then comes performance/efficiency and maintainability.
I'll explain the issues/concerns I have right now.
I couldn't get UNIX sockets to work with php files from subdirectories. (seems like the $1 part is ignored with sockets?) As explained here: https://wiki.apache.org/httpd/PHP-FPM
So I had to resort to using TCP. After some reading I found that it's unsafe to use the fcgi proxy approach. Because the proxy is publicly accessible.
I edited my configuration to:
Code: | ProxyPassMatch ^/(.*\.php(/.*)?)$ fcgi://127.0.0.1:9000/var/www/xgclan.com/public_html/$1 | Which I believe is safe. (PHP-FPM is only bound to 127.0.0.1 as well, so it's no longer publicly accessible?)
The major issue I still have is that SEF URL's don't load. For example my Joomla front end and admin page load fine. But other URL's don't.
Because they don't have the index.php included in the URL I think. But obviously I don't want the index.php in a Search Engine Friendly URL.
I also know very little about the configuration of this proxy approach at the moment.
So far it seems that I need to add the above code for each virtual host, but I don't know how to override the PHP version in a .htaccess file for one application (which runs inside a folder of my virtualhost).
I'm also concerned/worried about the footnote on: https://wiki.apache.org/httpd/PHP-FPM
It states security risks (and performance issues). And by the looks of it the default settings are not secure and to resolve that it requires a complex configuration with rewrites? |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
Posted: Tue 28 Jun '16 14:07 Post subject: |
|
|
if you start it with -b 127.0.0.1:9000 it is safe. |
|
Back to top |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
Posted: Tue 28 Jun '16 14:35 Post subject: |
|
|
Ubuntu starts it automatically for me, but in the /etc/php/7.0/fpm/pool.d/www.conf file I've set:
listen = 127.0.0.1:9000
Which should do the same and thus is just as safe, I assume.
I still have the following questions/issues:
1. The major issue I still have is that SEF URL's don't load. For example my Joomla front end and admin page load fine. But other URL's don't.
Because they don't have the index.php included in the URL I think. But obviously I don't want the index.php in a Search Engine Friendly URL.
2. from: https://wiki.apache.org/httpd/PHP-FPM
Quote: | Caveats
One might be tempted to point out that a greedy ProxyPassMatch directive might allow some malicious content uploaded by a HTTP client to be served.
This is by no means a comprehensive security document, but instead will point out a possible injection vector that could be generated from the directives in this document.
Take, for example:
/uploads/malicious.jpg/lalalaalala.php
Would lead php-fpm to process that file (/uploads/malicious.jpg), and without certain sanity check, possibly lead to a compromised server.
This, of course, is not recommended. Content uploaded using php should be saved safely outside the DocumentRoot, and the pathinfo should be scrutinized.
Additionally, php-fpm should check if the script being invoked is allowed.
If such restrictions cannot be implemented easily, then checks could be performed prior to proxying with a RewriteCond or FallbackResource to ensure that the URI is not altered by the HTTP client. |
What would be the best way to protect my self against these attack vectors? Some of the applications I use don't allow me to move the temp and upload folders out of my document root..
I think the rewritecond will be harder to maintain than a fallbackresource? |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7371 Location: Germany, Next to Hamburg
|
|
Back to top |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
Posted: Fri 01 Jul '16 0:15 Post subject: |
|
|
I put this into the main apache config, exactly as you said.
Code: | <IfModule mod_rewrite.c>
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^([^\.]+\.(php|phtml))$ fcgi://127.0.0.1:9000/$1 [P,L]
</IfModule> |
Doesn't work, I only see:
Code: | setStart($startTime, $startMem)->mark('afterLoad') : null; // Instantiate the application. $app = JFactory::getApplication('site'); // Execute the application. $app->execute(); |
As for the other rewrite rule, it doesn't work in apache's config. Which means I'd have to put it in each virtualhost?[/quote] |
|
Back to top |
|
gijs
Joined: 27 Apr 2012 Posts: 189 Location: The Netherlands
|
Posted: Fri 01 Jul '16 2:40 Post subject: |
|
|
Well, after hours of troubleshooting I found that my apache2.conf file had changed. Restoring the old one resolved my problem with the SEF url's..
(Probably caused by .htaccess file being ignored)
I'm currently using the:
ProxyPassMatch ^/(.*\.php(/.*)?)$ fcgi://127.0.0.1:9000/var/www/xgclan.com/public_html/$1
I'm still having the same problem with some pages being very slow (just like I had with fastcgi, so the problem I had was actually a PHP problem and not a fastcgi problem ) |
|
Back to top |
|