Let's encrypt certificate error - existing certificate renewal - post upgrade

[Scott MacDonald]

Following the upgrade from v1.0.6 to v2.0.1 the LE certificate that existed and was being renewed under v1.0.6 is now generating errors on renewal attempts with v2.0.1.

#!

There was error when issuing new Let's encrypt certificate

[2018-05-04 06:30:00] LEScript.INFO: Account already registered. Continuing.
[2018-05-04 06:30:00] LEScript.INFO: Starting certificate generation process for domains
[2018-05-04 06:30:00] LEScript.INFO: Requesting challenge for mail.********.com
[2018-05-04 06:30:00] LEScript.INFO: Sending signed request to /acme/new-authz
[2018-05-04 06:30:00] LEScript.INFO: Got challenge token for mail.********.com
[2018-05-04 06:30:00] LEScript.INFO: Token for mail.********.com saved at /opt/www//.well-known/acme-challenge/PnAR56pobgGE66FV5zofRVK0FYfi7lRHU9TtLPOFPU8 and should be available at http://mail.********.com/.well-known/acme-challenge/PnAR56pobgGE66FV5zofRVK0FYfi7lRHU9TtLPOFPU8

The token appears to be accessible from what I can see. Contrary to what the following error (cron job) that is also occurring:

#!

/etc/cron.daily/lets-encrypt-renew:
06:31:01 ERROR     [app] Please check http://mail.********.com/.well-known/acme-challenge/PnAR56pobgGE66FV5zofRVK0FYfi7lRHU9TtLPOFPU8 - token not available [] []
06:31:01 ERROR     [app] #0 /opt/admin/src/AppBundle/Handler/LeHandler.php(55): Analogic\ACME\Lescript->signDomains(Array) [] []
06:31:01 ERROR     [app] #1 /opt/admin/src/AppBundle/CommandInternal/RenewCommand.php(22): AppBundle\Handler\LeHandler->renew() [] []
06:31:01 ERROR     [app] #2 /opt/admin/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php(252): AppBundle\CommandInternal\RenewCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) [] []
06:31:01 ERROR     [app] #3 /opt/admin/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(964): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) [] []
06:31:01 ERROR     [app] #4 /opt/admin/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(248): Symfony\Component\Console\Application->doRunCommand(Object(AppBundle\CommandInternal\RenewCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) [] []
06:31:01 ERROR     [app] #5 /opt/admin/src/AppBundle/CommandInternal/Application.php(66): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) [] []
06:31:01 ERROR     [app] #6 /opt/admin/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(148): AppBundle\CommandInternal\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) [] []
06:31:01 ERROR     [app] #7 /opt/admin/bin/mailserver(33): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput)) [] []
06:31:01 ERROR     [app] #8 {main} [] []

[SH]

Do you expose port 80 or what kind of setup http/https do you have?

[Scott MacDonald]

Yes, port 80 and 443 are exposed thru the container -- the admin (and webmail) interfaces are both available so I'm fairly certain everything is working that way. This certificate existed post upgrade to V2.0.x and had been working without issue. It was only post upgrade that this issue started.

So to confirm -- the log file line listing the "token" should be available at.. I'm able to copy&paste and download that token so I'm sure that port 80 is working.

[SH]

Hmm. If you open token url at browser is it redirected to https or it stays at http?

[Scott MacDonald]

I think I recall encountering this issue previously in some testing I'd done back with Poste.io v1... it seems that even though the docker container exposes ports 80 and 443 and that these ports are opened for inbound communication that the docker container cannot (itself) access either port 80 or 443 (similar to a router loopback) based on the rules put in place by docker that ports 80 and 443 must also be opened up thru the firewall.

I've made the adjustment and will now wait for the cron job to execute tomorrow morning.

Scott

[Scott MacDonald]

Alright.. so the docker container is unable to access itself if there is a local firewall running that configures iptables. I had to open port 80 and now the container can access itself. I gather that Poste.io LE must check to see that the token is actually accessible and was failing because it was unable to loopback onto itself with port 80 not opened.

However, the certificate did not renew and I still got an error despite the token now being accessible.

This is the one email received, says error let's encrypt certificate issue exception, but looks like the process was successful.

#!
There was error when issuing new Let's encrypt certificate

[2018-05-08 06:40:43] LEScript.INFO: Account already registered. Continuing.
[2018-05-08 06:40:43] LEScript.INFO: Starting certificate generation process for domains
[2018-05-08 06:40:43] LEScript.INFO: Requesting challenge for mail.xxxxxxxx.com
[2018-05-08 06:40:43] LEScript.INFO: Sending signed request to /acme/new-authz
[2018-05-08 06:40:44] LEScript.INFO: Got challenge token for mail.xxxxxxxx.com
[2018-05-08 06:40:44] LEScript.INFO: Token for mail.xxxxxxxx.com saved at /opt/www//.well-known/acme-challenge/PnAR56pobgGE66FV5zofRVK0FYfi7lRHU9TtLPOFPU8 and should be available at http://mail.xxxxxxxx.com/.well-known/acme-challenge/PnAR56pobgGE66FV5zofRVK0FYfi7lRHU9TtLPOFPU8
[2018-05-08 06:40:44] LEScript.INFO: Sending request to challenge
[2018-05-08 06:40:44] LEScript.INFO: Sending signed request to https://acme-v01.api.letsencrypt.org/acme/challenge/odl3QuXdm6dFrbHLQrw-ce5RXyIbdz_Ex2IF_4TDagU/4482506807
[2018-05-08 06:40:44] LEScript.INFO: Verification pending, sleeping 1s
[2018-05-08 06:40:45] LEScript.INFO: Verification ended with status: valid
[2018-05-08 06:40:45] LEScript.INFO: Sending signed request to /acme/new-cert
[2018-05-08 06:40:46] LEScript.INFO: Got certificate! YAY!
[2018-05-08 06:40:46] LEScript.INFO: Requesting chained cert at https://acme-v01.api.letsencrypt.org/acme/issuer-cert
[2018-05-08 06:40:46] LEScript.INFO: Saving fullchain.pem
[2018-05-08 06:40:46] LEScript.INFO: Saving cert.pem
[2018-05-08 06:40:46] LEScript.INFO: Saving chain.pem
[2018-05-08 06:40:46] LEScript.INFO: Done !!§§!

Furthermore, these are subsequent errors received on what seems to be a failed certificate renewal:

#!

/etc/cron.daily/lets-encrypt-renew:
06:40:46 ERROR     [app] The command "'sudo' '-u' 'root' '/etc/cont-init.d/21-ssl-certificate.sh'" failed.

Exit Code: 1(General error)

Working directory: /

Output:
================
                 [1;33m* initalizing certificates [0m
* processing certificate uploaded from web admin\033[0m
                 [1;33m* processing certificate from /data/ssl [0m


Error Output:
================
 [] []
06:40:46 ERROR     [app] #0 /opt/admin/src/AppBundle/Server/MailServerManager.php(144): Symfony\Component\Process\Process->mustRun() [] []
06:40:46 ERROR     [app] #1 /opt/admin/src/AppBundle/Server/MailServerManager.php(47): AppBundle\Server\MailServerManager->run('root', '/etc/cont-init....') [] []
06:40:46 ERROR     [app] #2 /opt/admin/src/AppBundle/Handler/LeHandler.php(62): AppBundle\Server\MailServerManager->restartAllAfterCertificateChange() [] []
06:40:46 ERROR     [app] #3 /opt/admin/src/AppBundle/CommandInternal/RenewCommand.php(22): AppBundle\Handler\LeHandler->renew() [] []
06:40:46 ERROR     [app] #4 /opt/admin/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php(252): AppBundle\CommandInternal\RenewCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) [] []
06:40:46 ERROR     [app] #5 /opt/admin/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(964): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) [] []
06:40:46 ERROR     [app] #6 /opt/admin/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(248): Symfony\Component\Console\Application->doRunCommand(Object(AppBundle\CommandInternal\RenewCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) [] []
06:40:46 ERROR     [app] #7 /opt/admin/src/AppBundle/CommandInternal/Application.php(66): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) [] []
06:40:46 ERROR     [app] #8 /opt/admin/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php(148): AppBundle\CommandInternal\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) [] []
06:40:46 ERROR     [app] #9 /opt/admin/bin/mailserver(33): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput)) [] []
06:40:46 ERROR     [app] #10 {main} [] []
/etc/cron.daily/qmail-delivery-delete-old-logs:
find: '/data/log/delivery.*/*': No such file or directory

[Scott MacDonald]

Alright.. It seems that opening the firewall ports inbound for port 80 and 443 (possibly just 80 but I've not tested and simply opened both) is all that is required -- even though the docker container exposes 80/443 the container isn't able to loopback onto itself with any firewall/iptables rules in place.

As this was primarily user configuration -- I'm closing this ticket as the issue is resolved.

[Scott MacDonald]

With any iptables firewall running (ufw, firewalld or similar) port 80/443 need to be opened to allow the docker container to loopback on itself.