Topic: Reverse-Proxy incorrectly formatted calls to api (400 error)

[gunderwood]

Hi,

I have been trying to configure a reverse proxy for a back-end embedded IIS server. The back-end is a Power BI Report Server. The Apache server is version 2.4.43. Looking at comparable traces between a direct call that is successful and going through the reverse proxy where I am getting 400 errors there are the following differences. All the errors are api calls that are throwing the errors. Once the page hits the 400 error the BE throws an error message and stops loading.

Example 400 response: GET /reports/api/v2.0/System/ReportServerRelativeUrl HTTP/1.1" 400 - "https://..."


- “br” is added to the Accept-Encoding header
- The following headers are added:
sec-ch-ua header
sec-ch-ua-mobile header
sec-ch-ua-platform header
sec-Fetch-Site header
sec-Fetch-Mode header
sec-Fetch-Dest header

I have tried to use the "Header always unset" without success. The following modules are in the httpd.conf file.

LoadModule log_config_module ${APACHE_INSTROOT}/modules/mod_log_config.so
LoadModule status_module ${APACHE_INSTROOT}/modules/mod_status.so
LoadModule setenvif_module ${APACHE_INSTROOT}/modules/mod_setenvif.so
LoadModule version_module ${APACHE_INSTROOT}/modules/mod_version.so
LoadModule mime_module ${APACHE_INSTROOT}/modules/mod_mime.so
LoadModule unixd_module ${APACHE_INSTROOT}/modules/mod_unixd.so
LoadModule autoindex_module ${APACHE_INSTROOT}/modules/mod_autoindex.so
LoadModule alias_module ${APACHE_INSTROOT}/modules/mod_alias.so
LoadModule env_module ${APACHE_INSTROOT}/modules/mod_env.so
LoadModule socache_shmcb_module ${APACHE_INSTROOT}/modules/mod_socache_shmcb.so
LoadModule negotiation_module ${APACHE_INSTROOT}/modules/mod_negotiation.so
LoadModule include_module ${APACHE_INSTROOT}/modules/mod_include.so
LoadModule dir_module ${APACHE_INSTROOT}/modules/mod_dir.so
LoadModule headers_module ${APACHE_INSTROOT}/modules/mod_headers.so
LoadModule authz_core_module ${APACHE_INSTROOT}/modules/mod_authz_core.so
LoadModule authz_host_module ${APACHE_INSTROOT}/modules/mod_authz_host.so
LoadModule proxy_module ${APACHE_INSTROOT}/modules/mod_proxy.so
LoadModule proxy_http_module ${APACHE_INSTROOT}/modules/mod_proxy_http.so
LoadModule proxy_balancer_module ${APACHE_INSTROOT}/modules/mod_proxy_balancer.so
LoadModule lbmethod_byrequests_module ${APACHE_INSTROOT}/modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bybusyness_module ${APACHE_INSTROOT}/modules/mod_lbmethod_bybusyness.so
LoadModule slotmem_shm_module ${APACHE_INSTROOT}/modules/mod_slotmem_shm.so
LoadModule filter_module ${APACHE_INSTROOT}/modules/mod_filter.so

Any help is appreciated!

[James Blond]

A 400 can be

- invalid Destination Header
- invalid Depth Header
- invalid If Header
- invalid Overwrite Header
- invalid Translate Header
- invalid Request Body
- invalid Content-length
- invalid Timeout
- invalid Lock Token

Is there more in your log files?

[gunderwood]

There isn't much more I can share with the logs other than the example in my original post as I am not allowed to share any confidential information in the logs.

The Security part of the header - Anyone know which module would be manipulating he security part of the header?

Security
Authorization: NTLM TlRMTVNT...
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin

Thanks

[tangent]

I'd suggest those response headers are being generated by your back end server, not Apache.

Can you post your reverse proxy configuration and related settings, suitably anonymised of course.

[gunderwood]

The standard config file used by our company has a large amount of header edits. I have removed all of them to see if that was the issue. The current config file is as simple as it could be. This has been very frustrating to say the least.


<?xml version="1.0" encoding="UTF-8"?>
<pr:projectconfig xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
…
…
<serverconfig>
<includeconfig>
<include filename="${APACHE_PROJ}/vhost.conf"/>
</includeconfig>

<logconfig>
<loglevel severity="debug"></loglevel>
</logconfig>


<proxyconfig>
<proxyrequests>off</proxyrequests>
<proxypreservehost>on</proxypreservehost>
<proxypassblock>
<proxypass url="https://powerbiserver.net/" path="/"/>
</proxypassblock>
<proxypassreverseblock>
<proxypassreverse url="https://powerbiserver.net/" path="/"/>
</proxypassreverseblock>

</proxyconfig>
<sslproxyengine>on</sslproxyengine>

<sslconfig>
<sslproxycheckpeercn>off</sslproxycheckpeercn>
<sslproxycheckpeername>off</sslproxycheckpeername>
</sslconfig>

</serverconfig>


vhost.conf file:

<VirtualHost *:7874>
ServerName powerbiserver-rp.net
SSLEngine on
ProxyPreserveHost On
ProxyPass / https://powerbiserver.net/
ProxyPassReverse / https://powerbiserver.net/
</VirtualHost>

[tangent]

Your proxied site is using HTTPS, but I can't see an "SSLProxyEngine on" directive in your configuration (by default, it's off).

You might also need to include the SSLProxyProtocol directive along with appropriate SSL proxy certificate entries.

[gunderwood]

It's in there....

There are additional SSL directives in another file.

</proxyconfig>
<sslproxyengine>on</sslproxyengine>

<sslconfig>
<sslproxycheckpeercn>off</sslproxycheckpeercn>
<sslproxycheckpeername>off</sslproxycheckpeername>
</sslconfig>

[tangent]

Ok, it might be in another file, but that directive and related SSL proxy settings need to be present in in the vhost block where the proxy definitions are.

Are some of these problems down to the fact you appear to be using an XML structure to generate your Apache configuration? I've not seen this approach before.

[gunderwood]

Hi tangent,

Our company uses xml to stage the configuration so it deploys to multiple nodes for various reasons. It is very frustrating that this one particular department only has a contract with the support vendor for operations even though all products are supposed to have a devops structure. There are multiple xml files that end up generating the http.conf file.

I did just notice that Wireshark is showing the 400 responses from the server as http and the successful responses are showing as http/json. I redirected the reverse proxy to use port 80 on the server so I can see all the data with Wireshark. I can see the request some into the server and the response but I am not able to see any other differences.

[tangent]

Ok, if the proxied content is still a problem with HTTP, then at least you're able to troubleshoot things with Wireshark for the back-end connection. It's a good strategy, and one I've used before. So what happens if you Follow => TCP Stream on one of the /reports/api requests, and compare that to a direct connection that bypasses the proxy? Presume those two traces aren't similar, since if they are then the problem is something to do with the remaining upstream Apache configuration.

In your earlier post you said the Accept-Encoding request header was getting Brotli (br) added, but surely that encoding option must come from the client browser, and not be gratuitously appended by Apache. So is this a problem with "double compression", meaning your proxied response is compressed, and Apache is then adding it's own compression to that response back to the client. This problem has been discussed on this site before.

As a quick test, try removing the compression options from the Accept-Encoding request header, by adding this to your virtual host block.

[gunderwood]

I added the code you suggested and it did not resolve the 400 errors. I noticed another section of the code from the reference post and it looks like it may fix the issue based on what I am seeing in the Wireshark traces. I am rather ignorant on how the substitute command works so when I added that section apache is throwing an error. I posted a response in that other thread regarding the code and the error "Bad Substitute format, must be an s/// pattern".

Thanks for everyone's help so far.

[tangent]

Ok, let's rewind a little.

So removing support for compressed content in the proxy request didn't solve the problem, but you say you believe you've identified the 400 error problem, based on what you see from the Wireshark traces.

Can you elaborate exactly what that detail is (suitably anonymized), and whether it's a problem with response headers or the response body.

Without that detail, you're not going to be able to identify what needs editing in the response, be that header edits or mod_subsitute (or mod_proxy_html) changies to the response body.

[gunderwood]

Wireshark is showing a difference in the protocol column from the Wireshark trace.

Success Response is showing HTTP/JSON as the protocol

No / Time / Source / Destination / Protocol / Length / Info

73 25.415964 <IP Address> <IP Address> HTTP/JSON 580 HTTP/1.1 200 OK , JavaScript Object Notation (application/json)

Failed Response is only showing HTTP as the protocol

3821 43.665391 <IP Address> <IP Address> HTTP 181 HTTP/1.1 400 Bad Request

[tangent]

I think we're too deep down the rabbit hole at the moment, and need to retrace our steps a little.

If the Wireshark trace of the proxied connection is showing a 400 error response, then it rather suggests the request being passed through the proxy to the back-end is the cause of the problem.

So what does Wireshow show is the difference between a direct connection to your back-end server, and the one coming out of Apache? Do this test with a browser set with compression disabled, i.e. the accept-encoding header is blank. This is easy to do with Firefox (in about:config edit network.http.accept-encoding and network.http.accept-encoding.secure), whilst in Chrome you'll need something like the ModHeader plugin. You will of course need HTTP rather than HTTPS for each connection path.

Performing similar browser requests, use the Wireshark "Follow TCP Stream" feature for both cases, up to the point where the 400 error occurs. At that point compare the request headers (including cookies), query strings, POST details, etc. There has to be a difference in the proxied request that's causing your 400 error.

Only then can you decide what needs fixing.

[gunderwood]

Request hitting the server directly (succesful)

GET /reports/api/v2.0/System/ReportServerRelativeUrl HTTP/1.1
Host: http:example.net
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-XSRF-TOKEN: deprecated
Connection: keep-alive
Referer: http://example.net/reports/
Cookie: XSRF-NONCE=H2VzNSR2ICuD8%2FTm2G5UNflr3jbdJ

Dy1nI3V4wb9cRg%3D; XSRF-TOKEN=
….

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 132
Content-Type: application/json; odata.metadata=minimal
Server: Microsoft-HTTPAPI/2.0
X-Content-Type-Options: nosniff
Path=/reports; HttpOnly
Set-Cookie: XSRF-TOKEN=deprecated; path=/reports
OData-Version: 4.0
Date: Tue, 08 Mar 2022 21:04:10 GMT

Request going through the RP (failed)

GET /reports/api/v2.0/System/ReportServerRelativeUrl HTTP/1.1
Host: example.net
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:97.0) Gecko/20100101 Firefox/97.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
X-XSRF-TOKEN: deprecated
Referer: https://example.net/reports/browse/
displayedContent=%7B%22hidden%22%3Afalse%7D;
ai_user=Tir1a|2022-03-08T20:07:31.788Z; ai_session=MZvij|1646770051787|1646771691865
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-
….
X-Forwarded-For: xxx.xxx.xxx.xxx
X-Forwarded-Host: example.net

X-Forwarded-Server: example.net
Connection: Keep-Alive

HTTP/1.1 400 Bad Request
Content-Length: 0
Server: Microsoft-HTTPAPI/2.0
Date: Tue, 08 Mar 2022 20:34:51 GMT


Differences:

Good call has this even though I followed your recommended Firefox settings:

Accept-Encoding: gzip, deflate
X-RBT-SCAR: xx.xxx.x.xxx:1765913794:3000 (xx.xxx.x.xxx is an IP address)


Bad call has this added

Referer: https://example.net/reports/browse/
displayedContent=%7B%22hidden%22%3Afalse%7D;
ai_user=Tir1a|2022-03-08T20:07:31.788Z;
ai_session=MZvij|1646770051787|1646771691865
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Authorization:
. .
X-Forwarded-For: xxx.xxx.xxx.xxx (IP Address)
X-Forwarded-Host: example.net
X-Forwarded-Server: example.net

[gunderwood]

I ended up being able to remove the 400 errors by changing the back-end nodes to not use NTLM and use basic authentication. Next step is to configure Kerberos to hopefully remove the 2nd authorization challenge that has popped up after changing the default authorization method. Thanks for everyone's input and assistance!

[tangent]

Sorry, I've been offline for a couple of days, so hadn't seen your latest post over NTLM.

When I saw your previous post with the Wireshark traces, my first thoughts were to question the topology of your network, and where those traces were taken. Were they both on your IIS based report server per chance, and where does the Apache reverse proxy sit within that topology.

Reason was, some of the header details you originally posted (since removed) made me suspect that Riverbed/SteelHead appliances were in the connectivity loop somewhere. I've had run-ins with these devices in the past, since they think they know best about optimizing WAN traffic over private networks. They will manipulate HTTP headers, applying compression on the fly, rarely transparently.

I'd also noticed your two NTLM auth tokens were of different format, and knowing 400 errors normally related to authentication/authorization, was going to suggest this an area for investigation.

Good luck with the Kerberos approach to solving your problem.

[James Blond]