systemd service file doesn't work with recent versions of systemd

[Steve Mokris]

systemd switched to stricter PIDFile handling, and after updating systemd, I get this error when landcareresearch’s Solr module tries to start Solr:

New main PID … does not belong to service, and PID file is not owned by root. Refusing.

…then it shuts down Solr.

If I change the module’s service file to this simpler version, it’s able to successfully launch Solr:

[Unit]
Description=Apache SOLR
After=syslog.target network.target remote-fs.target nss-lookup.target
[Service]
Environment=SOLR_INCLUDE=/etc/sysconfig/solr
ExecStart=/opt/solr/bin/solr start -noprompt -f
User=solr
[Install]
WantedBy=multi-user.target

‌

[Steve Mokris]

Here’s a patch. (I’d submit a PR, but I don’t have permission to submit PRs to this repo.)

diff --git a/templates/solr.service.epp b/templates/solr.service.epp
index 4852caf..5f35807 100644
--- a/templates/solr.service.epp
+++ b/templates/solr.service.epp
@@ -11,14 +11,8 @@
 Description=Apache SOLR
 After=syslog.target network.target remote-fs.target nss-lookup.target
 [Service]
-PIDFile=<%= $solr_pid_dir %>/solr-<%= $solr_port %>.pid
 Environment=SOLR_INCLUDE=<%= $solr_env %>
-ExecStart=<%= $solr_bin %>/solr start -noprompt
-ExecRestart=<%= $solr_bin %>/solr restart -noprompt
-ExecStop=<%= $solr_bin %>/solr stop -noprompt
-ExecStatus=<%= $solr_bin %>/solr status -noprompt
-ExecReload=/bin/kill -s HUP $MAINPID
+ExecStart=<%= $solr_bin %>/solr start -noprompt -f
 User=solr
-PrivateTmp=true
 [Install]
 WantedBy=multi-user.target

‌

[Michael Speth]

Thank you for reporting Steve. I will have a look into integrating your changes!

[Michael Speth]

I have attempted to apply the changes in the patch. However, these changes break both Ubuntu 16.04 and 18.04. WIthout the PID specified as well as start,restart,stop,status, and reload, solr directly fails right away and SystemD doesn’t seem to know what to do. I am looking into the Stricter PIDFile handling for a possible solution that will also working on older versions of systemd.

[Michael Speth]

What operating system are you encountering this bug? I would like to try to replicate. Thank you.

[Steve Mokris]

Thanks for investigating. I ran into the problem on a CentOS 7 VM (a SmartOS LX Zone).

$ cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)
$ uname -a
Linux … 3.13.0 BrandZ virtual linux x86_64 x86_64 x86_64 GNU/Linux
$ yum info systemd
Name        : systemd
Arch        : x86_64
Version     : 219
Release     : 67.el7_7.1.0.1
…

‌

[Michael Speth]

Hi Steve. I have updated the master branch of this project with a working Vagrant testing environment. Its using Centos7.7. The way I was able to replicate this issue was to provision the vm with vagrant and using puppet (here are the steps).

cd vagrant
scripts/run.bash

I logged in using ‘vagrant ssh’ and then issued sudo service solr status. This returned the error described in this issue.

The work around I found was to do the following

sudo service solr restart

Solr seems to be working without issue:

sudo service solr status
Redirecting to /bin/systemctl status solr.service
* solr.service - Apache SOLR
   Loaded: loaded (/etc/systemd/system/solr.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2019-10-29 02:12:08 UTC; 5min ago
 Main PID: 1252 (java)
   CGroup: /system.slice/solr.service
           └─1252 /usr/lib/jvm/jre-1.8.0/bin/java -server -Xms512m -Xmx5

Can you try to manually restart the service and see if you still get the same error?

[Steve Mokris]

Can you try to manually restart the service and see if you still get the same error?

Yes, I still get the same error. I suspect the problem is specific to the SmartOS LX environment (which doesn’t support some syscalls used in recent systemd versions).

Would you be open to either 1. using the above simpler systemd service file only when on SmartOS LX ($::virtual=='lxc'), or 2. exposing a parameter to optionally prevent this Puppet class from automatically creating a systemd service file?

[Darryl Friesen]

I came across this issue while investigating some problems I was having with RHEL 8.4 and Solr 8.9.0. I tried Steve’s stripped down service file, and that seemed to resolve my problems.

I was initially having issues with the PID file location, similar to what Steve mentioned in his original message. Once that was dealt with however, Solr would start, then immediately shutdown. The key to solving this issue seems to be the -f flag he used on the ExecStart command.

I’ll also mention that ExecRestart and ExecStatus do not seem to be standard systemd service file options, and cause warnings on my RHEL 8 systems. The stripped down version of the service file given by Steve fixes those issues as well.

[Darryl Friesen]

There is a proposed solr.service file in this Apache Solr issue, so systemd support may be added to install_solr_service.sh in the future.

https://issues.apache.org/jira/browse/SOLR-14410

Short term, I’ve been creating /etc/systemd/system/solr.service.d/override.conf to override all the settings in this module’s version of the service file.

[Michael Speth]

Thank you Darryl, I will check out the solr.service and look to applying fix.

[Michael Speth]

Fixed in 6.0.9