Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: mod_xsendfile with fastcgi/cgi ? |
|
| Author |
|
jabaltie
Joined: 02 Aug 2006
Posts: 12
Location: Brazil
|
 Posted: Wed 02 Aug '06 16:11 Post subject: mod_xsendfile with fastcgi/cgi ? |
 |
|
Mr Steffen kindly provided us with a Windows version for mod_xsendfile, that has been originally published at
http://celebnamer.celebworld.ws/stuff/mod_xsendfile/
I installed the module, declared it's load by using the directive below :
LoadModule xsendfile_module modules/mod_xsendfile.so
And also turned it on, by setting
XSendFile on
However, it's not working. Some points :
- My scripts are issuing the http header as described. That I can see from FireFox using it's Livehttpheaders extension.
- I'm issuing that header with a file path unix-style (/ instead of \)
- That file exists, of course.
- When I use Lighthttpd, which also has a similar directive (it was the inspiration for Apache's one), it works perfectly.
What could be wrong with Apache ?
I guess I'm putting it's directives in a wrong place. I'm also using FastCGI, so I dont know if it's compatible or not.
Just in case you want to see my configuration file, get it from here, please :
http://www.des.online.unimep.br/au/pub/apache2.conf
Thanks in advance 4 your support ![/list] |
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7373
Location: Germany, Next to Hamburg
|
| Posted: Wed 02 Aug '06 16:24 Post subject: |
|
|
How does your PHP script look? Tried headers_sent() ?
The version is 0.8 may be still some errors in the code
Which version of Apache do you use? I see that x-sendfile uses apr_* it has changed in Apache 2.1 against 2.0!
For that contanct the author Nils Maier, MaierMan(%at%)web(%dot%)de |
|
| Back to top |
|
jabaltie
Joined: 02 Aug 2006
Posts: 12
Location: Brazil
|
| Posted: Wed 02 Aug '06 16:41 Post subject: |
|
|
| James Blond wrote: | How does your PHP script look? Tried headers_sent() ?
The version is 0.8 may be still some errors in the code
|
I dont use PHP. I have a FastCGI server of my own.
However, I can see from the browser side that the http header was issued indeed.
| Quote: |
Which version of Apache do you use? I see that x-sendfile uses apr_* it has changed in Apache 2.1 against 2.0!
For that contanct the author Nils Maier, MaierMan(%at%)web(%dot%)de |
I'm using 2.0.58. |
|
| Back to top |
|
MaierMan
Joined: 02 Aug 2006
Posts: 7
|
| Posted: Thu 03 Aug '06 4:55 Post subject: hi from the author. |
|
|
I just got mod_xsendfile working with fastcgi/cgi.
Was a bit tricky (and the code at least "feels" somewhat dirty).
Why in hell do those cgi-provided headers go into err_headers_out instead of headers_out... Argh. Took some time to figure it out.
Will prepare a new release and instructions the next few hours/days 
Nils
PS. Thanks Steffen for pointing me at here. |
|
| Back to top |
|
jabaltie
Joined: 02 Aug 2006
Posts: 12
Location: Brazil
|
| Posted: Thu 03 Aug '06 14:37 Post subject: Re: hi from the author. |
|
|
| MaierMan wrote: |
Will prepare a new release and instructions the next few hours/days
Nils
PS. Thanks Steffen for pointing me at here. |
Thank you so much Nils.
I also have a suggestion of enhancement for your module :
Consider this situation : a script runs some sort of command line tool to generate a huge graphical file. That file is generated on a RAMDRIVE.
Thanks to your extension, the script may instruct the http server to send that file, preventing the script to re-read it again right away. An enormous time saver, I think.
So far, so good.
Now, my suggestion is : I'd like to have one more option for X-SENDFILE : an instruction for the http server to REMOVE that file as soon it is delivered to to the browser. Because then we would clean-up the RAMDRIVE automatically (intrinsically).
It could be another module, say
X-SENDFILE-AND-REMOVE-IT-AFTERWARDS
As for the complexity of doing it, I really dont know. I dont know the internals of Apache. But I have done a similar thing using an API of the http server I'm working on : WebSite PRO from Oreilly (discontinued product).
The problem, I think, is to "buble-up" that order of removing the file back inside to the kernel of the http server. Cause removing should be done after the deliver. I guess this is far away from your current code. But, again, this is a guess.
Maybe it's on the tip of your fingers ...
Let me know what do you think of this idea, OK ?
BTW, I'm anxiously waiting for the newest version of mod_xsendfile for Windows (Steffen ?) ! |
|
| Back to top |
|
MaierMan
Joined: 02 Aug 2006
Posts: 7
|
| Posted: Fri 04 Aug '06 10:02 Post subject: |
|
|
Ok, here is a test-release of the new version.
Zip includes the source, the slighty updated manual and a windows binary (d'oh, I compiled with SSE, but what CPU nowadays does not support SSE :p)
Not really "official" yet. But kinda RC.
http://celebnamer.celebworld.ws/stuff/mod_xsendfile/mod_xsendfile-0.9.bin.zip
Usage hint:
You'll have to enable XSendFile for the actual fastcgi handler and not for any locations/scripts it handles.
Something like this
| Code: | <Location /php-fcgi/php-cgi.exe>
Options +ExecCGI
SetHandler fastcgi-script
XSendFile On
</Location> |
I hope to add a second filter later so that I can grab the original request path so that no subrequest is required when used with cgi/fcgi.
But I had no time to look into this yet.
But it's planned, as I will propably use the module in a apache2-worker + php-fcgi environment.
(Rant: Can somebody tell those "professional" php "application" developers that there is no point to implement serving static files from within php with fpassthru/fread and without even thinking of cache-management. Beside the fact that this will simply increase memory usage and will make nice stuff like sendfile() useless... :p)
I won't implement your REMOVE-FILE sugguestion:
- I have no direct gains from this, and not enough spare time and dedication for this.
- nor is this filter IMO the right place to implement this
- nor is it easy to do. Your "bubble" statement is perfectly valid. The filter will replace the contents, and then pass to the next filter in chain, like the transcoding filters and the final network pump. You'd have to do "nasty" tricks that break the filter stack by keeping the filter alive to remove that file afterwards.
- (nor do I really think this is from a security perspective; stealing files is something bad, but removing them :p)
|
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU
|
| Posted: Fri 04 Aug '06 13:12 Post subject: |
|
|
The version 0.9 does not build against Apache 2.2, get an error at the APR_BRIGADE_FOREACH block.
APR_BRIGADE_FOREACH is deprecated.
Steffen |
|
| Back to top |
|
jabaltie
Joined: 02 Aug 2006
Posts: 12
Location: Brazil
|
| Posted: Fri 04 Aug '06 14:10 Post subject: |
|
|
| MaierMan wrote: | Usage hint:
You'll have to enable XSendFile for the actual fastcgi handler and not for any locations/scripts it handles.
Something like this
| Code: | <Location /php-fcgi/php-cgi.exe>
Options +ExecCGI
SetHandler fastcgi-script
XSendFile On
</Location> |
|
I really need some help to configure this. I'm newbie with Apache and dont understand it's configuration file. Think it's too complicated cause I dont know the concepts.
I used WebSite PRO from OReilly and it had all of a Configuration Panel (Windows Dialogs). Easy, very well organized.
Now I have to understand this configuration.
Anyway, the problem is that I want to add X-SENDFILE for all requests that are processed through my FastCGI external server.
This is the piece of the config file :
FastCgiExternalServer c:\apache2\htdocs\api\gtwebsiteapi.wsa -host localhost:101 -appConnTimeout 120 -idle-timeout 300
CheckSpelling on
<Location /api/gtwebsiteapi.wsa>
XSendFile On
</Files>
The whole config file can be get from
www.des.online.unimep.br/au/pub/apache3.conf
When I leave the configuration like that, it will issue a 404. If I remove the location part and keep the XSendFile On, it will process but will not interpret the xsendfile http header issued by the backend.
What do I need to do ?
| Quote: |
I hope to add a second filter later so that I can grab the original request path so that no subrequest is required when used with cgi/fcgi.
But I had no time to look into this yet.
But it's planned, as I will propably use the module in a apache2-worker + php-fcgi environment.
|
I hope you DO NOT prevent the requests to pass through the script !
In my case, the script will access a mainframe database in order to see if the user has rights (or not) to see a JPG file which is stored on a folder. As a matter of fact , the mainframe response will tell if the user has rights or not and, also , the location (path) of the JPG file.
So, I REALLY need that ALL the requests GO THROUGH the script.
You CANNOT bufferize or cache it anyhow.
Got it ?
| Quote: |
(Rant: Can somebody tell those "professional" php "application" developers that there is no point to implement serving static files from within php with fpassthru/fread and without even thinking of cache-management. Beside the fact that this will simply increase memory usage and will make nice stuff like sendfile() useless... :p)
|
I understand perfectly. What is a static hit should be left as simple as that. But sometimes we need a processing before, mostly to check rights.
| Quote: |
I won't implement your REMOVE-FILE sugguestion:
|
| Quote: |
- I have no direct gains from this, and not enough spare time and dedication for this.
- nor is this filter IMO the right place to implement this
- nor is it easy to do. Your "bubble" statement is perfectly valid. The filter will replace the contents, and then pass to the next filter in chain, like the transcoding filters and the final network pump. You'd have to do "nasty" tricks that break the filter stack by keeping the filter alive to remove that file afterwards.
- (nor do I really think this is from a security perspective; stealing files is something bad, but removing them :p)
|
OK. Understood your points. I'm trying to make Lighty's author to do it for me but, he has not even read my email yet !
If I dont get it working inside the http server, I'll build a queue of my own. But it is too costy and will have a calculated delay to remove the files. Very likely it will remove the files 5 minutes after creation. I guess this is time enough for it to be delivered to the browser.
Better, of course, it would be to have the http server removing it ASAP.
Anyway, thank you so much !
Hope I can make at least my configuration work.
Thanks again  |
|
| Back to top |
|
MaierMan
Joined: 02 Aug 2006
Posts: 7
|
| Posted: Sat 05 Aug '06 5:02 Post subject: |
|
|
Try turning on |XsendFileAllowAbove on| as well.
Otherwise only files below the request-path (in your case c:\apache2\htdocs\api\) will be allowed to be send out.
Some general notes:
In your particular case it makes more sense to send those files directly from the DB without "buffering" them to the disk first, as they are "one-time" sends anyway if I understood you correctly...
X-Sendfile (or lightys equivalant) are designed to handle files that are stored on disk anyway or for kinda "perisistent" cache files.
To your remarks about the lighty and it's author:
I already noticed that you must be the same guy that is positng to the lighty list as well.
Just a hint: You're asking for not-so-general-prupose features and configuration advice, but you (usually) won't get premium support or custom development for free. This kind of work is usually paid work |
|
| Back to top |
|
Steffen
Moderator
Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU
|
| Posted: Sat 05 Aug '06 11:43 Post subject: |
|
|
| Steffen wrote: | The version 0.9 does not build against Apache 2.2, get an error at the APR_BRIGADE_FOREACH block.
APR_BRIGADE_FOREACH is deprecated. |
Nils, you overlooked this post ?
Steffen |
|
| Back to top |
|
MaierMan
Joined: 02 Aug 2006
Posts: 7
|
| Posted: Sat 05 Aug '06 15:12 Post subject: |
|
|
| Steffen wrote: | | Steffen wrote: | The version 0.9 does not build against Apache 2.2, get an error at the APR_BRIGADE_FOREACH block.
APR_BRIGADE_FOREACH is deprecated. |
Nils, you overlooked this post ?
Steffen |
yep, I did. (I actually overlooked it twice. First your post and then I didn't meant to keep it in the code.)
The code will get some final cleanup/review anyway.
The 0.9.bin.zip from above was posted for testing pruposes only. |
|
| Back to top |
|
jabaltie
Joined: 02 Aug 2006
Posts: 12
Location: Brazil
|
| Posted: Sat 05 Aug '06 15:33 Post subject: |
|
|
I logged on this Saturday only to see if there were some answer. And I see there is !
Thank you Mr Miers !
| MaierMan wrote: | Try turning on |XsendFileAllowAbove on| as well.
Otherwise only files below the request-path (in your case c:\apache2\htdocs\api\) will be allowed to be send out.
|
I tried. But it doesnt work 
Again, here's the configuration part :
| Code: |
LoadModule fastcgi_module modules/mod_fastcgi.dll
LoadModule xsendfile_module modules/mod_xsendfile.so
FastCgiExternalServer c:\apache2\htdocs\api\gtwebsiteapi.wsa -host localhost:101 -appConnTimeout 120 -idle-timeout 300
XsendFileAllowAbove on
|
All I want is to have everything that is processed through my FastCGI server to interpret X-SENDFILE. Just that.
Notice that the script IS issuing the header cause I can see it on FireFox. Also, it doesnt matter if I use normal windows slashes (\) or (/). It doesnt work with none of them. Firefox shows me an empty (blank) page.
When it works for you, which are the headers ? My guess is that maybe I'm issuing the headers on the wrong order (or adding headers that will disable your module). Can you tell me in your case which are the headers received by your browser ? Can you paste them here ?
Here's a screen shot of my headers :
http://www.des.online.unimep.br/au/pub/headers.jpg
| Quote: |
Some general notes:
In your particular case it makes more sense to send those files directly from the DB without "buffering" them to the disk first, as they are "one-time" sends anyway if I understood you correctly...
X-Sendfile (or lightys equivalant) are designed to handle files that are stored on disk anyway or for kinda "perisistent" cache files. |
They are NOT on a DataBase. They're simply files which are stored on disk. Each student/employee/teacher has its own folder (and subfolders).
On this context, X-SENDFILE makes all the sense cause the script will check if the request has the rights or not to see those files (if who's viewing is the owner of the document or say, the boss wants to see his employee document for instance). Then, the script simply points the http server to send that file which is already on disk.
Again, there is no database. Files are spread amongst zillions of folders on a SCSI disk.
Simple and effective.
But I also use the same resource for processed files. My application is external and, it generates the response files on a RAMDRIVE. Then, I use X-SENDFILE to simply instruct the http server to send those already generated files. Why is it like that ? Because my application has a kind of a cache too. If it sees that it doesnt need to re-process the request, it simply talks back the file path (and FastCGI server will in its turn also talk back the file path via your module).
Further, I plan to do the http server itself to check if the cache is valid or not, thus saving it to have to launch the script over and over again, just to see if the cache is valid.
Lighty has something like that. It can prevent scripts from being triggered again. It is something regarded to CML (cache markup language) and cache module.
But this is LATER. Now I must replace my current http server and EVERYTHING should work as it is today. I shall not make this migration more complicated than what it is already.
| Quote: | To your remarks about the lighty and it's author:
I already noticed that you must be the same guy that is positng to the lighty list as well.
Just a hint: You're asking for not-so-general-prupose features and configuration advice, but you (usually) won't get premium support or custom development for free. This kind of work is usually paid work |
Yes, I'm an over-talkative member of their Forum.
Apache is working more properly , although I do prefer Lighty.
But Lighty has had many problems :
- When using an external FastCGI server, it has some problems regarded to large form submissions, say when you upload files.
- Also, you cannot use PATH INFO cause when you do that, it wont find the script (404).
But I'm still around Lighty.
As for the file removal, I have built a secondary thread on my FastCGI server that will run after the files to be removed. But I think this is more complicated and more costy than having the http server by removing it itself.
I'm taking too much from your time here but please, I do need your help to configure Apache !
This is the only missing part on my FastCGI server. That is, in order to put my kit (Apache/FastCGI server) into PROduction site, I only need to solve the X-SENDFILE mysthery !
Can you help me ?
Here's how my configuration file right now :
http://www.des.online.unimep.br/au/pub/apache4.conf |
|
| Back to top |
|
MaierMan
Joined: 02 Aug 2006
Posts: 7
|
| Posted: Sat 05 Aug '06 16:01 Post subject: |
|
|
Jose,
where exactly do you enable XSendFile in your configuration?
I guess nowhere (if you didn't put it in some .htaccess file).
You need to do |XSendFile on| AND |XSendFileAllowAbove on| in your particular case.
Former enables XSendFile and latter allows sending of files from everywhere not just below the original request folder.
But latter does not imply former.
It's like most stuff in apache.
No RewriteRule without RewriteEngine
Header order is not important at all.
The header name (X-SENDFILE) is not even case sensitive.
PS: Not that this is important, or that I care a lot about it, but in fact my family name is spelled Maier. |
|
| Back to top |
|
jabaltie
Joined: 02 Aug 2006
Posts: 12
Location: Brazil
|
| Posted: Sat 05 Aug '06 16:13 Post subject: |
|
|
Thanks again !
Thought only me would work on Saturdays...
| MaierMan wrote: | Jose,
where exactly do you enable XSendFile in your configuration?
I guess nowhere (if you didn't put it in some .htaccess file).
|
I've added right after the LoadModules declaration.
| Quote: |
You need to do |XSendFile on| AND |XSendFileAllowAbove on| in your particular case.
Former enables XSendFile and latter allows sending of files from everywhere not just below the original request folder.
But latter does not imply former.
|
Let's add both then....
Perfect ! Now it's working !!!!!!!!!!!!!!
| Quote: |
It's like most stuff in apache.
No RewriteRule without RewriteEngine
Header order is not important at all.
The header name (X-SENDFILE) is not even case sensitive.
PS: Not that this is important, or that I care a lot about it, but in fact my family name is spelled Maier. |
Sorry !
Now, one more thing : you dont care about headers but I do !
I can see from the browser perspective that X-SENDFILE will be there.
This is very, very risky here.
This will reveal the original file path. For cache generated files is not a problem, however, for those other official documents, it is a big vulnerability.
For instance, student's grades report are stored like that. As well as teachers and employees copies of their pay-checks (holleriths).
So, I NEVER, EVER let end users know the original file path. One could see the other documents. Of course those folders are not windows shareable but it is not good for people out there to know where they are.
And believe-me, students are very, very curious. I can tell N stories of what they've done here. Some very creative and bizarre artistic works with teachers photos on PhotoShop for instance. Teachers photos are stored on those folders , simply like JPG files too.
So, is there a way to make it be swallowed by the http server ?
Thanks again !
Admin note:
Please open new topic for new questions |
|
| Back to top |
|
|
|
|
|
|
