Topic: scoreboard is full, not at MaxRequestWorkers.Increase Server

[guanggevwu]

The system and software information are here:
https://drive.google.com/file/d/1_ghH_vWvXGGAqOe2T6Db12msfrMqm-8j/view?usp=sharing
The settings for mpm_event is:
https://drive.google.com/file/d/1Kj8dqNWa-t9vm5jX4-clR42MS_M3Frtc/view?usp=sharing
As can be seen from the image, there are a lot of PIDs "Stopping; yes (old gen)". In details, every day, it generates 4 PIDs but the old ones seemed not killed.
Since I set the ServerLimit to 32, eventually it should lead to the error in the title. I restarted apache 6 days ago and thus occupied 24 slots/PID (4 PIDs each day * 6 days).
Any solutions?

[tangent]

Your post details don't provide any real clues as to why the "old gen" processes have stopped accepting connections. Presume you've checked the Apache error log for clues (noting you're using mod_wsgi). Are the processes blocking on some connection or resource that's timed out, or needed keepalive?

Other than timeout settings, I'd consider changing MaxConnectionPerChild to something non-zero, to cause new child processes to be launched periodically, and see what if anything changes.

[guanggevwu]

tangent, thanks for looking into my problem.

Your post details don't provide any real clues as to why the "old gen" processes have stopped accepting connections.
I actually don't know much about how apache works, but I checked the error log and somehow at midnight everyday in the error log file:
[Mon Feb 26 00:00:01.049613 2024] [mpm_event:notice] [pid 1021636:tid 140429992023936] AH00493: SIGUSR1 received. Doing graceful restart
Does this cause apache2 to generate new PIDs everyday?I think so.
I Just did some test. I manually did a "systemctl restart apache2" and then kept doing "apachectl -k graceful", the old PIDs were killed successfully by monitoring the server-status page. Howevers, as long as I set up a connection to my website (by visiting the index page and then close the page), "apachectl -k graceful" can't kill all the 4 old PIDs (usually it kills 3 of them).
In summary, it seems the log configuration caused a graceful restart everyday. But the reason why the old PIDs can't be killed is unclear.

Are the processes blocking on some connection or resource that's timed out, or needed keepalive?
As can be seen from the image links, I have 4 slots (25 threads each) accepting requests, and most of them are idle (the one that is not idle I suppose is the server-status page itself). This means I don't have a lot of visitors and there should not be any connections that refuse to close.

Other than timeout settings, I'd consider changing MaxConnectionPerChild to something non-zero, to cause new child processes to be launched periodically, and see what if anything changes.
Can you please explain more about the timeout settings? Is there a way Apache can hard stop an old PID automatically?
Is there a suggested MaxConnectionPerChild value?

[tangent]

Your Apache log entry showing a graceful restart at midnight, suggests someone's set up a cron task to do this, and if so hopefully they've added a comment to the cron entry explaining why. One possible reason for this might be to force Apache to pick up WSGI application code changes that would otherwise be cached in process memory. This depends on your various Apache mod_wsgi configuration settings, which you've not disclosed.

Either way, your simple index page access test rather confirms the Apache process is holding onto a resource, which is preventing the graceful restart from killing the process.

Since your status page is showing there are no open/pending network connections, I'd consider this to be WSGI resource related, so for me that's where I'd start looking further.

Other things I'd consider:

a) Raise the Apache loglevel to debug or higher and repeat your index page graceful restart test.

[guanggevwu]

Thanks, tangent. It seems this post has the same problem as mine.
https://github.com/GrahamDumpleton/mod_wsgi/issues/383#issuecomment-449555949
First, the reason the "graceful" happens in midnight is because of logrotate.
Second, the reason why the "graceful stop" doesn't work correctly is because the information can't be passed to the WSGIDaemonProcess.
However, I don't quite know how to fix the problem. Fixing either of the two issues should solve my problem. For the second issue, I don't know how to set the "display-name" even I read the documentation (https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html).
For the first issue, I don't think I have made changes to use an external log script. Maybe it is the Ubuntu default. Is there a way I can use Apache's own rotatelog?

[tangent]

I can't help with the WSGI side of things, but you could reconfigure the logging setup on Ubuntu to avoid the graceful restart triggered through /etc/logrotate.d/apache2.

With some effort you could change the Apache configuration to using the rotatelogs utility (or third party cronolog), both of which will rotate log files automatically based date and time, e.g.

[guanggevwu]

After disabling the default logrotate and using rotatelogs, the server didn't try to graceful restart in the midnight. Although the real problem should be in the mod_wsgi side which causes a graceful restart can not be completed, I believe this solved the "scoreboard is full" problem. Thank you again, tangent.