Keep Server Online
If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.
or
A donation makes a contribution towards the costs, the time and effort that's going in this site and building.
Thank You! Steffen
Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
| |
|
 Topic: Sporadic hangs with Event MPM? |
|
| Author |
|
neilgunton
Joined: 12 Feb 2016
Posts: 2
Location: Albany, Oregon
|
 Posted: Fri 12 Feb '16 23:27 Post subject: Sporadic hangs with Event MPM? |
 |
|
I am using Apache 2.4.18 (front end caching reverse proxy, Event MPM, SSL) and 2.2.31 (mod_perl backend, prefork MPM), both compiled from source, both on the same dedicated server, to host a community website that I have been developing and running for the last 15 years. I have noticed recently that with the Event MPM enabled on the front end, I would see occasional, sporadic hangs when trying to load the website in my browser. It would just sit there saying "connecting". Then it would often just clear up by itself, sometimes after a few seconds, sometimes more. Sometimes I would have to restart apache, but I don't know if it would have eventually come out of it if I had just left it - but being a live website, when I see this my first priority is to get the site working again. When I am at my console and able to access the server, I run htop during those moments, and can see that Apache is not serving requests. Tail the server log, nobody is getting any pages. And, no activity in terms of CPU either, according to htop. It's just sitting there. So it's not just a browser or DNS issue on the client side. Then suddenly it would all clear up, and you'd see the requests coming through as usual, as if nothing had happened.
Then one day recently I had a brainstorm and decided to try switching the build on the front end to Worker MPM instead of Event. Nothing else changed, just the MPM. And lo and behold, no more hangs since then.
I am aware that the Event MPM behaves like Worker when SSL is in use. But not everybody accesses my site via SSL, so I thought it might help at least some of the time. Shouldn't matter in any case, should it?
Now I look back, I can remember when I first started using Event MPM (sometime last summer, I think it was, right about when I was getting slammed by hits from China and had to revamp my config to be able to handle the traffic better), when I set MaxConnectionsPerChild for the front end to anything but 0, then eventually the server that had those threads would just sit there in the status for a while. It's like it had a problem closing down the threads. The server was still serving requests, but these "zombie" threads were still showing up in the status page. I worked around it by simply making MaxConnectionsPerChild 0. I thought, from what I read, that Event was a better choice than Worker given the heavy traffic I was getting from China (it was really bad, I think some DNS got redirected over to my server for whatever reason, because I was getting tons of requests that were clearly for other domains, and related to bittorrent etc, which I don't run). Anyway, I eventually solved the DDoS problem by banning China completely at the firewall level, so that's not so much of an issue any more. They still hit me daily, but not at a volume that presents any issue with performance. I handle using iptables + ipset, and ip blocks from http://www.ipdeny.com/ipblocks/.
So the question is: Has anyone else noticed any issues with Event MPM in terms of occasional, sporadic hangs? I won't bother posting my entire configuration here for now, because it's quite large and I'm not sure it would be relevant, given that I am seeing such a difference between Event and Worker MPM's. Should there be any difference at all if you switch between them, but leave everything else the same? Surely not, but I can't really be of much help in tracing what the code is doing, since the hangs are so occasional and so unpredictable, and nothing crashes (I think), it just hangs for a while and then eventually recovers. There's never anything in the log files afterwards. During the hangs, I am still able to access the server remotely via SSH without any issues, so it's not a network problem. It's just Apache that stops responding.
Just wondering if anyone else has noticed this as being an issue with Event vs Worker. Everything's good now with Worker, so I'm happy to leave it like that (and I'm finding that the status page is a bit more informative now too), I just thought I'd bring this up in case the Apache people weren't aware of a bug somewhere in there.
My server is 64-bit Debian Wheezy. The website gets around 500,000 to 700,000 or more http requests per day (around 100,000 - 150,000 page requests). So, not that busy, but constantly getting reasonable traffic. It's a bicycle touring journal community website with forums, so basically a lot of text and pics, but no video or anything like that. The server also runs DNS, MySQL and Sendmail. When the hangs happen, the server is nowhere near its capacity or hitting any limits with traffic.
Thanks for any insights,
Neil |
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7371
Location: Germany, Next to Hamburg
|
| Posted: Wed 17 Feb '16 11:33 Post subject: |
|
|
If it comes to SSL connection I didn't notice any difference event vs worker.
You wrote that you use it as reverse proxy and caching. It might be an issue of t he caching cleaning process. At least I had that. |
|
| Back to top |
|
neilgunton
Joined: 12 Feb 2016
Posts: 2
Location: Albany, Oregon
|
| Posted: Wed 16 Mar '16 22:38 Post subject: |
|
|
| James Blond wrote: | | If it comes to SSL connection I didn't notice any difference event vs worker. |
Thanks. I think I remember now what happened with Event MPM when I had MaxConnectionsPerChild set to non-zero. The Apache Status page, which has the scoreboard of all the processes and threads, seems to use one line per process (at least for my setup, I do ThreadsPerChild 64). And when I had MaxConnectionsPerChild non-zero, with Event MPM, I would eventually see a whole line of the thread status go to all 'G', which means 'gracefully finishing'. It would open up a new process, and be serving it, but the old one would just sit there with every thread in 'G' state. So I don't know if that is related to the actual hangs I was seeing later on, but it could be.
| Quote: | | You wrote that you use it as reverse proxy and caching. It might be an issue of t he caching cleaning process. At least I had that. |
Yes, I had issues with the caching reverse proxy, especially when combined with SSL. Basically with the default setting of 2 for CacheDirLevels and CacheDirLength, the cache would grow very quickly. Two characters makes for a LOT of directories at each level. I also believe it grew quickly because SSL made for a lot of cache misses - the stored headers (which were used to key the cache) are different for every request by different clients, even for the same page. So two people requesting the same page via SSL would get two different stored pages in the cache. Kind of negates the whole point of having the cache. I don't know if it still works that way, but that's what was happening a few years ago when I first enabled SSL on my site.
I found that the default number of directories being generated by "CacheDirLength 2" quickly made it very slow to traverse the cache when cleaning. Something about so many files over a hierarchy (albeit only 2 levels deep, though I think the default used to be 3, but I can't be sure). I use ext4 filesystem. Maybe it could have been better handled by one of the other file systems, I don't know. In any case, the way I handled that was to set CacheDirLevels to 2, and CacheDirLength to 1. This helps control the exponential explosion of directories a bit. Then I also run a cron job every 10 minutes, which does the following:
htcacheclean -t -p/var/cache/www -l100M
I also have a script I wrote called prune_cache_headers which traverses the cache, removing empty directories left over by htcacheclean.
This helps to stop the cache from ever getting out of hand size-wise. I haven't had any issues with cache performance since.
My website is perhaps a little unusual in that it has a large amount of dynamically generated content. It's bicycle tour journals, with over 2 million pictures, 350,000 journal pages, and 400,000 forum posts. There are a lot of potential dynamic pages that could be cached, obviously we don't want to cache everything people browse during the day, so it is constantly being pruned back.
The main reason for having the caching at all is to be able to handle specific pages being linked to from other popular sites like reddit or facebook (used to be slashdot). If hundreds of people are all clicking on the same link, then it would be good to have that page be cached so it doesn't have to hit the mod_perl backend for every request. That's it; not much point in caching everything else. I obviously exclude the actual pic images from the cache (why bother, they are static files anyway and so only hit the front end proxy), and everything else can be cycled out of the cache pretty quickly. Even if a popular page that is the subject of a slashdotting gets expired from the cache every minute, well, having to redo that page from the back-end every minute is no big deal. In the meantime, having thousands of people get the cached copy is the big win. So, really no need to be obsessed over having a huge cache; the more important job there is actually to LIMIT the cache as much as possible.
Anyway, thanks for the feedback. I haven't seen any other commentary on this issue (Event MPM hangs), so I wonder if it's not something weird about my build config settings (I built apache and the APR from source), or runtime config. In any case, I think I have solved it by switching to the Worker MPM, and since I use SSL extensively on my site, I guess that won't matter much (since Event acts like Worker with SSL). I was just curious if these hangs were something other people had been noticing. I guess not. So there could also be something to do with the relative complexity of my site, I don't think it's too common to have this large a config (I run several websites off the same server), front-end caching reverse proxy to mod_perl back end etc. Maybe there's some subtle interaction between components that is only becoming apparent with my setup, I don't know. Oh well.
Neil |
|
| Back to top |
|
James Blond
Moderator
Joined: 19 Jan 2006
Posts: 7371
Location: Germany, Next to Hamburg
|
|
| Back to top |
|
|
|
|
|
|
