logo
Apache Lounge
Webmasters

 

About Forum Index Downloads Search Register Log in RSS X


Keep Server Online

If you find the Apache Lounge, the downloads and overall help useful, please express your satisfaction with a donation.

or

Bitcoin

A donation makes a contribution towards the costs, the time and effort that's going in this site and building.

Thank You! Steffen

Your donations will help to keep this site alive and well, and continuing building binaries. Apache Lounge is not sponsored.
Post new topic   Forum Index -> Apache View previous topic :: View next topic
Reply to topic   Topic: Watchdog hang with mod_md renew with 2.4.32
Author
pbhq



Joined: 17 Mar 2013
Posts: 37
Location: Germany
PostPosted: Wed 14 Mar '18 19:39    Post subject: Watchdog hang with mod_md renew with 2.4.32 Reply with quote

Split from www.apachelounge.com/viewtopic.php?p=36585


Hallo Steffen,

Steffen wrote:


*) mod_md is added as an experimental module, not advised to use in production yet, we need more success stories.
Also at Let's encrypt there are new features around the corner, like a new ACMEv2 protocol and wildcard. So better to wait.



Yes, you better take this part seriously ... Shocked

Yesterday was the day for the Renew of the test certificate and of course it did not work again. Instead, mod_watchdog hangs in an endless loops loop and writes the logfile full. Rolling Eyes


Code:

[Mon Mar 12 10:08:13.393493 2018] [md:debug] [pid 41696:tid 624] mod_md.c(760): AH10055: md watchdog run, auto drive 1 mds
[Mon Mar 12 10:08:13.393493 2018] [md:debug] [pid 41696:tid 624] mod_md.c(782): AH10107: next run in 12:00:00 hours
[Mon Mar 12 22:16:11.642746 2018] [md:debug] [pid 41696:tid 624] mod_md.c(760): AH10055: md watchdog run, auto drive 1 mds
[Mon Mar 12 22:16:11.642746 2018] [md:debug] [pid 41696:tid 624] mod_md.c(782): AH10107: next run in 12:00:00 hours
[Tue Mar 13 10:17:05.874747 2018] [md:debug] [pid 41696:tid 624] mod_md.c(760): AH10055: md watchdog run, auto drive 1 mds
[Tue Mar 13 10:17:05.874747 2018] [md:debug] [pid 41696:tid 624] mod_md.c(782): AH10107: next run in  7:17:18 hours
[Tue Mar 13 17:40:02.722855 2018] [md:debug] [pid 41696:tid 624] mod_md.c(760): AH10055: md watchdog run, auto drive 1 mds
[Tue Mar 13 17:40:02.722855 2018] [md:debug] [pid 41696:tid 624] mod_md.c(704): AH10053: md(ftp.pbhq.com): is complete, cert expires Sun, 15 Apr 2018 16:34:24 GMT
[Tue Mar 13 17:40:02.722855 2018] [md:debug] [pid 41696:tid 624] mod_md.c(782): AH10107: next run in  0:-5:-38 hours
[Tue Mar 13 17:40:02.822861 2018] [md:debug] [pid 41696:tid 624] mod_md.c(760): AH10055: md watchdog run, auto drive 1 mds
[Tue Mar 13 17:40:02.822861 2018] [md:debug] [pid 41696:tid 624] mod_md.c(704): AH10053: md(ftp.pbhq.com): is complete, cert expires Sun, 15 Apr 2018 16:34:24 GMT
[Tue Mar 13 17:40:02.822861 2018] [md:debug] [pid 41696:tid 624] mod_md.c(782): AH10107: next run in  0:-5:-38 hours
[Tue Mar 13 17:40:02.922867 2018] [md:debug] [pid 41696:tid 624] mod_md.c(760): AH10055: md watchdog run, auto drive 1 mds
[Tue Mar 13 17:40:02.922867 2018] [md:debug] [pid 41696:tid 624] mod_md.c(704): AH10053: md(ftp.pbhq.com): is complete, cert expires Sun, 15 Apr 2018 16:34:24 GMT
[Tue Mar 13 17:40:02.922867 2018] [md:debug] [pid 41696:tid 624] mod_md.c(782): AH10107: next run in  0:-5:-38 hours
[Tue Mar 13 17:40:03.022873 2018] [md:debug] [pid 41696:tid 624] mod_md.c(760): AH10055: md watchdog run, auto drive 1 mds
[Tue Mar 13 17:40:03.022873 2018] [md:debug] [pid 41696:tid 624] mod_md.c(704): AH10053: md(ftp.pbhq.com): is complete, cert expires Sun, 15 Apr 2018 16:34:24 GMT
[Tue Mar 13 17:40:03.022873 2018] [md:debug] [pid 41696:tid 624] mod_md.c(782): AH10107: next run in  0:-5:-38 hours


The bug is in Apache/Win32 v2.4.30 and v2.4.32.
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU
PostPosted: Wed 14 Mar '18 19:58    Post subject: Reply with quote
Thanks.

I am not using it, too much magic and issues with my config. I use win-acme (formerly called letsencrypt-win-simple) from https://github.com/PKISharp/win-acme . For me no need to use mod_md for Apache and my other (mail)servers.


Looking at the log it looks like a loop in mod_md.

Reported at the dev list. Do not how and where they reply.

Thanks for reporting !
Back to top
DnvrSysEngr



Joined: 15 Apr 2012
Posts: 226
Location: Denver, CO USA
PostPosted: Wed 14 Mar '18 21:19    Post subject: Reply with quote
Thanks for that info Steffen. Guess that explains why my LetsEncrypt certs have not updated yet (supposed to have auto updated last weekend).

I was / still am using mod_md, but guess I will have to manually update my certs using win-acme / letsencrypt-windows.

mod_md as a 3rd party module worked with 2.4.29, but sounds like it is not working with 2.4.32?

-S
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU
PostPosted: Wed 14 Mar '18 21:34    Post subject: Reply with quote
I cannot confirm. I stopped testing about a month ago.

Till which version of mod_md was it working ?
Back to top
DnvrSysEngr



Joined: 15 Apr 2012
Posts: 226
Location: Denver, CO USA
PostPosted: Wed 14 Mar '18 22:08    Post subject: Reply with quote
I was using V1.1.8 + mod_ssl patched for it
2018-01-20 with httpd 2.4.29. However, the last time mod_md updated automatically was back in December, so the version of omd_md may have been 1.1.5 or 1.1.6.
Back to top
pbhq



Joined: 17 Mar 2013
Posts: 37
Location: Germany
PostPosted: Wed 14 Mar '18 23:02    Post subject: Reply with quote
DnvrSysEngr wrote:
mod_md as a 3rd party module worked with 2.4.29, but sounds like it is not working with 2.4.32?


I had the same error once in January with the v2.4.29 and then the update from mod_md v1.1.8 (from v1.0.Cool fixed the problem. Why the problem occurred again yesterday completely identical, no idea.

However, I also used LetsEncrypt-Win-Simple for the productive domains. Wink
Back to top
DnvrSysEngr



Joined: 15 Apr 2012
Posts: 226
Location: Denver, CO USA
PostPosted: Thu 15 Mar '18 1:12    Post subject: Reply with quote
UGGHHH!!!! Now when I try to run LetsEncrypt, I get the following error:

Unable to access http://mydomainname.com/.well-known/acme-challenge/xxxxxx

I have an alias for .well-known to go to /document/root/.wellknown/ ---
but it is not allowing me to get to http://mydomainname.com/.well-known/acme-challenge/xxxxxx

any ideas as to what I am missing here?

It used to work long ago before i started using mod_md
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU
PostPosted: Thu 15 Mar '18 8:51    Post subject: Reply with quote
Mod_md still running ?
Back to top
DnvrSysEngr



Joined: 15 Apr 2012
Posts: 226
Location: Denver, CO USA
PostPosted: Thu 15 Mar '18 16:43    Post subject: Reply with quote
Yes, I am still running mod_md. I just ended up renaming my managed domain folder, restarting Apache to get new certs created and then restarting Apache to get the newly created certs to take effect.

Not sure why letsencrypt was throwing 404 errors when trying to validate.
Back to top
pbhq



Joined: 17 Mar 2013
Posts: 37
Location: Germany
PostPosted: Thu 15 Mar '18 18:40    Post subject: Reply with quote
DnvrSysEngr wrote:
Yes, I am still running mod_md. I

Not sure why letsencrypt was throwing 404 errors when trying to validate.


You have to use the older version of mod_ssl (without mod_md support) from the Apache v2.4.29 release for LetEncrypt-Win-Simple, because mod_md reserves internally the ".well-know" -URL.
Back to top
DnvrSysEngr



Joined: 15 Apr 2012
Posts: 226
Location: Denver, CO USA
PostPosted: Thu 15 Mar '18 21:31    Post subject: Reply with quote
Thank you PB. I gave up and just ended up updating my certs manually (since auto-renew function in mod_md is not behaving) by using the steps I mentioned in my post.
Back to top
pbhq



Joined: 17 Mar 2013
Posts: 37
Location: Germany
PostPosted: Sun 18 Mar '18 20:46    Post subject: Reply with quote
pbhq wrote:
I had the same error once in January with the v2.4.29 and then the update from mod_md v1.1.8 (from v1.0.Cool fixed the problem. Why the problem occurred again yesterday completely identical, no idea.


So, now I'm just as smart as in January Question Question

activated mod_md again and the certificates were renewed immediately. In January, I thought I had solved the problem with a newer version of md_mod. Since the binaries are identical, there must be a problem somewhere between the Renew-Detect with Default-Option "33%" and the actual Renew request of the certificates (here more than 2-3 days, I think).

I noticed something else: Today MDNotifyCMD did not work (possibly because SSLEngine was still "Off" in the test host).

Second, the WatchdogInterval option had no effect on the frequency of log file entries.
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU
PostPosted: Sun 18 Mar '18 21:07    Post subject: Reply with quote
Notify comes after 24 hours. When you restart Apache before that 24 then no notification.
Back to top
pbhq



Joined: 17 Mar 2013
Posts: 37
Location: Germany
PostPosted: Sun 18 Mar '18 21:13    Post subject: Reply with quote
This is new, or?
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU
PostPosted: Sun 18 Mar '18 21:19    Post subject: Reply with quote
I saw it always. In the how-to there is also the note.
Back to top
pbhq



Joined: 17 Mar 2013
Posts: 37
Location: Germany
PostPosted: Sun 18 Mar '18 23:32    Post subject: Reply with quote
I like you there and I think so synonymous your info from 01.11.2017 to, but when the function of MDNotifyCMD was new, I tested my script and the script was executed directly after the Renew. Otherwise, I would never have been able to write & test my script so fast.

Anyway, I've set the Renew to "84d" now. Let's see what happens next week. Very Happy
Back to top
Steffen
Moderator


Joined: 15 Oct 2005
Posts: 3092
Location: Hilversum, NL, EU
PostPosted: Mon 19 Mar '18 11:53    Post subject: Reply with quote
Fix for your reported watchdog https://www.apachelounge.com/viewtopic.php?p=36635 :


mod_md: fixes error in renew window calculation that may lead to mod_md running
watchdog in a tight loop until actual renewal becomes necessary.

http://svn.apache.org/viewvc?view=revision&sortby=date&revision=1827180

Thanks again for reporting !
Back to top


Reply to topic   Topic: Watchdog hang with mod_md renew with 2.4.32 View previous topic :: View next topic
Post new topic   Forum Index -> Apache