Author |
|
Renaud SALVIGNAC
Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
|
Posted: Fri 14 Jun '24 8:36 Post subject: Apache Proxy errors AH01102 + AH00898 - Request too long |
|
|
Hello,
I have an error on Apache + proxy with balancer.
When an url exceeds a certain length, I have an Proxy error 502 :
Code: | Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request
Reason: Error reading from remote server |
With this message in error log :
Code: | [Fri Jun 14 08:18:33.010605 2024] [proxy_http:error] [pid 47783:tid 139883905533632] (104)Connection reset by peer: [client 84.103.4.88:59302] AH01102: error reading status line from remote server 192.168.202.41:80
[Fri Jun 14 08:18:33.010830 2024] [proxy:error] [pid 47783:tid 139883905533632] [client 84.103.4.88:59302] AH00898: Error reading from remote server returned by /test.php
|
Apache log :
Code: | 84.103.xxx.xxx - - [14/Jun/2024:08:18:33 +0200] "GET /favicon.ico HTTP/1.1" 404 2005 "https://dev.xxxxxx.cloud/test.php?code=0.AQwAJDMu3tBPA0i7lCy2LEWy9QpYbsxwWF5Hrz2Vu76zvO4MANc.AgABBAIAAADnfolhJpSnRYB1SVj-Hgd8AgDs_wUA9P_3Y6zJdaJIrc4scFbbFErN7BCueQ-2bx3G7cA-6dLRBgyIe_23b8xAx1wIQIWBNf45-5b67bkHSc56oWE5XdIP0BJffyrMJAOrTIhaHO95tQ8Rpmnluhzx3zqBjgi-Q3U6LBd__e49k0Qe48L0RGuIIysOJ2UcpRrgpmIp2AeLez51JjqfEKKsxB4jvhIxH5YMtTqnXxoFKuIiG8XzDFpXGPDxSWeZ1zhKcKgfB3yLsP2GtxJ0g97JSKLyFtft5CfYs0UHP0r_kduwGw--dDpnq1tmTQE1vqZhiX9mebWtALoqaM1ykbeD2RDL8bvVAy3Yu3vKmwOu4RwJjEXLojR1hrf8nsDLSTskxLRZWbAKFi7rZjxo30xaxTRPKXpv5Xrnd9MMrBQa8xKU52IdLUMvgEKD1mkvpKH871s1K2mRoB5LqVDamIgKcMpRqrfECIPCmVBDPfDjZJU12wzi5-S3965qkWWb1dIJNgQ4J0ar9THVBxCICrH-I2CgHfFuJ3Y7JXJIHG9tLcpyDDHWeXe5bKUBfvGRNEx0rsEp9L-ZpMeknx2ReCqToLFu_j-SL8LGYMGqSULtNJt2y5heRnns4vboAV5k_GX51y_fSvgPX-FhKqj9ERfirvXLQSwPJqpVdr88mGX9_9yKd-PP9MCg4LD6Mkd2oD8mpcOa6WmRN3EahoeMVM5qkS2Bt5Q4ATOOeWhJ_U_LPva_DRToswcKaTINSff6wWH5GCl7_be3Hk-gbZHtPNl-NaJsjuDqZcy8bjx5eU2mf3IARZWUFM7DeuOniPh3cP2t6MMT5Q1rqVmnJ2P_Me9xHFFMPqVt3-PIU5xlK7OpHO-tMVnYrVUBEHcG5zhlIy3oIeQSMgzhtkXdQF0b-r6yd6Db2ZHvMxn7eLedJtj_zoD111111111111111" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"
|
My Apache : Apache/2.4.57
OS : Debian 12
I'm desperate to find a solution to this problem...
Thanks for your help |
|
Back to top |
|
tangent Moderator
Joined: 16 Aug 2020 Posts: 348 Location: UK
|
Posted: Fri 14 Jun '24 16:43 Post subject: |
|
|
You don't say what your upstream load balanced servers are running, but presume you've checked to see what corresponding error is being recorded there?
Your request URL alone is over 1K in length, and request headers in general have grown in size over time (multiple cookies, authentication tokens, X-headers, etc.), so is it possible you've exceeded some buffer size for some of your client requests?
In the past I've seen this problem with Tomcat backend servers, where we needed to increase maxHttpHeaderSize in server.xml. Other backend services will equally have some finite defaults.
Also, at the Apache level, check out ProxyIOBufferSize which defaults to 8192 bytes. |
|
Back to top |
|
Renaud SALVIGNAC
Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
|
Posted: Sat 15 Jun '24 8:17 Post subject: RE |
|
|
Hello,
Sorry. I have an other apache serveur behind.
If I test directly the url with local IP in replacement of domain, it works.
Errors are logged on Proxy server, not on server behind.
I've already tested ProxyIOBufferSize directive.
It doesn't change anything. |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7373 Location: Germany, Next to Hamburg
|
Posted: Mon 17 Jun '24 10:53 Post subject: |
|
|
You may set Code: | connectiontimeout=[time in seconds] timeout=[time in seconds] |
for example 60 seconds (1 min)
Code: |
ProxyPass / http://localhost:4382/ connectiontimeout=60 timeout=60 |
|
|
Back to top |
|
Renaud SALVIGNAC
Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
|
Posted: Wed 19 Jun '24 8:53 Post subject: |
|
|
Also tested without any result... |
|
Back to top |
|
James Blond Moderator
Joined: 19 Jan 2006 Posts: 7373 Location: Germany, Next to Hamburg
|
Posted: Fri 21 Jun '24 8:48 Post subject: |
|
|
Please post your reverse proxy config. |
|
Back to top |
|
Renaud SALVIGNAC
Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
|
Posted: Fri 21 Jun '24 10:32 Post subject: |
|
|
Below the config
Code: |
<VirtualHost *:443>
ServerName www.xxxxxx.cloud
ServerAlias xxxxxx.cloud
ErrorLog /var/log/apache2/error.xxxxxx.cloud.log
CustomLog /var/log/apache2/xxxxxx.cloud.log combined
<Directory /var/www/xxxxxx.cloud/>
Options Indexes FollowSymLinks
AllowOverride All
DirectoryIndex index.htm index.php index.html
</Directory>
Header add Set-Cookie "ROUTEID=.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
<Proxy balancer://xxxxxx_prod>
include /etc/apache2/xxxxxx_prod.conf
ProxySet lbmethod=byrequests
ProxySet stickysession=ROUTEID
ProxySet nofailover=On
</Proxy>
ProxyPass "/" "balancer://xxxxxx_prod/"
ProxyPassReverse "/" "balancer://xxxxxx_prod/"
<Location /balancer-manager>
SetHandler balancer-manager
Order deny,allow
Allow from all
</Location>
ProxyRequests Off
ProxyPreserveHost On
Include /etc/letsencrypt/options-ssl-apache.conf
ServerAlias beta.xxxxxx.cloud
ServerAlias oweba.xxxxxx.cloud
SSLCertificateFile /etc/letsencrypt/live/www.xxxxxx.cloud/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.xxxxxx.cloud/privkey.pem
</VirtualHost>
|
And balancer settings
Code: |
BalancerMember "http://192.168.202.101" loadfactor=1 route=route1
BalancerMember "http://192.168.202.102" loadfactor=1 route=route2
BalancerMember "http://192.168.202.103" loadfactor=1 route=route3
|
|
|
Back to top |
|
tangent Moderator
Joined: 16 Aug 2020 Posts: 348 Location: UK
|
Posted: Fri 21 Jun '24 21:32 Post subject: |
|
|
Looking at your proxy configuration, and the error log detail in the original post, I notice the error relates to remote server 192.168.202.41:80 whereas your balancer member IP's are 192.168.202.101, 102 and 103. Is this relevant, or did you change the balancer IP's for the posted config?
Otherwise, if you've already tried increasing the buffer sizes, I suspect the problem lies somewhere with the proxied headers being fragmented / truncated, or relates to compression, but to analyse this further believe you need more detail to crack where the exact problem lies.
There are two things I'd try next here.
Firstly try removing the Accept-Encoding request header, so that the back end servers send an uncompressed response to Apache. Put the following in your virtual host block or global configuration (if acceptable).
Code: |
RequestHeader unset Accept-Encoding |
Secondly, I'd use network capture tools such as tcpdump on your Debian platforms (Wireshark if the backend servers are Windows based), to see exactly what's going on here during the client request.
Per chance, is there a firewall between your Apache and the backend servers? |
|
Back to top |
|
Renaud SALVIGNAC
Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
|
Posted: Tue 25 Jun '24 9:35 Post subject: |
|
|
We have 1 proxy and 3 environments (prod / dev /beta).
The configuration given is about Prod.
IP address 192.168.202.41 is for dev environment.
I've added "RequestHeader ..." but it doesn't change anything.
I'm checking with tcpdump but not simple to analyze. |
|
Back to top |
|
tangent Moderator
Joined: 16 Aug 2020 Posts: 348 Location: UK
|
Posted: Tue 25 Jun '24 12:55 Post subject: |
|
|
Ok, so your one proxy covers multiple environments. Presumably, this error occurs across all three, albeit inconsistently?
Assuming you can narrow the problem to say the dev environment, you should be able to craft a filter for tcpdump to restrict the captured traffic (written to a pcap file). I'd personally just capture and analyse the traffic between the proxy and corresponding/potential back-end servers (hence the recommendation to remove the Accept-Encoding request header, so the proxied content isn't compressed).
I tend to transfer the pcap files over to a Windows platform, where it's easier to analyse them with Wireshark.
Let us know how you get on. |
|
Back to top |
|
Renaud SALVIGNAC
Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
|
Posted: Tue 25 Jun '24 14:12 Post subject: |
|
|
Yes it's the same on each environment.
I try your method and give you feedback. |
|
Back to top |
|
Renaud SALVIGNAC
Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
|
Posted: Wed 26 Jun '24 10:31 Post subject: |
|
|
We found a workaround.
We use proxy server in "normal web mode" for one subdomain.
We store parameters then we redirect to other script that recover the paramaters.
Thanks for your help |
|
Back to top |
|
tangent Moderator
Joined: 16 Aug 2020 Posts: 348 Location: UK
|
Posted: Wed 26 Jun '24 21:03 Post subject: |
|
|
Not sure I fully follow your explantion, but am glad you've resolved the problem with a workaround.
Apache Lounge has numerous subscribers with many years of experience. Don't hesitate to post queries for help, or equally your expertise in solving problems. |
|
Back to top |
|
Renaud SALVIGNAC
Joined: 13 Jun 2024 Posts: 8 Location: FRANCE - Cognac
|
Posted: Thu 27 Jun '24 8:14 Post subject: |
|
|
I will not miss it.
Thanks for your advices. |
|
Back to top |
|