Issue : Current Systemd implementation may not support workload management per Splunk Docs Systemd configuration

[lmnogues]

Configuration of the systemd Splunkd.service is using the following values :

    - { option: "ExecStart", value: "{{ splunk_home }}/bin/splunk start --accept-license --answer-yes --no-prompt" }
    - { option: "ExecStop", value: "{{ splunk_home }}/bin/splunk start stop" }
    - { option: "ExecReload", value: "{{ splunk_home }}/bin/splunk restart" }
    - { option: "Restart", value: "on-failure" }
    - { option: "RestartSec", value: "30s" }
    - { option: "TimeoutStopSec", value: "10min" }
    - { option: "Type", value: "forking" }
    - { option: "RemainAfterExit", value: "False" }
    - { option: "User", value: "{{ splunk_nix_user }}" }
    - { option: "Group", value: "{{ splunk_nix_group }}" }
    - { option: "PIDFile", value: "{{ splunk_home }}/var/run/splunk/splunkd.pid" }
    - { option: "LimitNOFILE", value: "1024000" }
    - { option: "LimitNPROC", value: "512000" }
    - { option: "LimitFSIZE", value: "infinity" }
    - { option: "LimitDATA", value: "infinity" }
    - { option: "LimitCORE", value: "infinity" }
    - { option: "TasksMax", value: "infinity" }

when the splunk documentation is asking for :

#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.

[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network.target

[Service]
Type=simple
Restart=always
ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=360
LimitNOFILE=65536
SuccessExitStatus=51 52
RestartPreventExitStatus=51
RestartForceExitStatus=52
User=splunker
Group=splunker
Delegate=true
CPUShares=1024
MemoryLimit=<value>
PermissionsStartOnly=true
ExecStartPost=/bin/bash -c "chown -R splunker:splunker /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R splunker:splunker /sys/fs/cgroup/memory/system.slice/%n"

[Install]
WantedBy=multi-user.target

The current configuration does not allow for the Workload management to be enabled via an ansible installation (at least the ExecStart and the Restart need to be changed.

I'm not a Unix expert so I don't really know the impact of this.

[mason-splunk]

Good catch. I came across this in the docs as well, but only after the refactor to rely on service restarts for everything was done. I have not tested with workload management to confirm that the current configuration does not work with it, but I agree that we generally should be aligned with the best practices from the docs.

This is going to require some additional refactoring, as ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd does not accept additional arguments, namely --accept-license which means we'll need to make a new task that does splunk ftr --accept-licence --no-prompt --answer-yes and include it at the appropriate points (ie after install/upgrade but before starting splunk) during the installation and upgrade tasks.

[mason-splunk]

Need to refactor to comply with the docs guidance: https://docs.splunk.com/Documentation/Splunk/8.1.3/Admin/RunSplunkassystemdservice