[Bug] Ansible Systemd is running continously even after finished

[NanoCode012]

Hello, sorry to bother you again. I have found something interesting. I have upgraded my account to PayG and noticed that there were costs from Object Storage. After a while (and some misunderstandings), I finally realized that it came from Requests. It made around 2000 requests per day!

I checked the Ansible scripts and disabled the backup-to-pihole.{timer|service} to no avail. It turns out that the cloudblock-ansible-state.{timer|service} was actually running all this time (for two whole months)! (Downloading the backup.tar.gz from buckets over and over). After running journalctl -u cloudblock-ansible-state.service, I can retrieve the logs below. Unfortunately, it's been so long, so only the recent logs are here.

It should be noted that the VM already works (can connect via Wireguard) and the files are already stored in the bucket. As a workaround, I disabled the ansible service script as well. I looked at the source and believe that it should not have ran at all every two minutes, only at boot.

One conjecture I have is that I rebooted the VM once to apply some security updates. Since the service script runs on boot, there could be a chance that it re-ran then met an error causing this infinite loop. If so, I propose changing the service script so that it disables later or find a way to run only once.

journalctl logs (this is repeated every two minutes in logs) Iteration: 24957

-- Logs begin at Fri 2022-04-15 12:26:21 UTC, end at Fri 2022-04-15 14:31:08 UTC. --
Apr 15 12:26:22 cloudblock-instance systemd[1]: cloudblock-ansible-state.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 15 12:26:22 cloudblock-instance systemd[1]: cloudblock-ansible-state.service: Failed with result 'exit-code'.
Apr 15 12:26:52 cloudblock-instance systemd[1]: cloudblock-ansible-state.service: Scheduled restart job, restart counter is at 24957.
Apr 15 12:26:52 cloudblock-instance systemd[1]: Stopped cloudblock-ansible-state.
Apr 15 12:26:52 cloudblock-instance systemd[1]: Started cloudblock-ansible-state.
Apr 15 12:26:52 cloudblock-instance cloudblock-ansible-state.sh[2700532]: Hit:1 http://{removed}-1-ad-1.clouds.archive.ubuntu.com/ubuntu focal InRelease
Apr 15 12:26:52 cloudblock-instance cloudblock-ansible-state.sh[2700532]: Hit:2 http://{removed}-1-ad-1.clouds.archive.ubuntu.com/ubuntu focal-updates InRelease
Apr 15 12:26:52 cloudblock-instance cloudblock-ansible-state.sh[2700532]: Hit:3 http://{removed}-1-ad-1.clouds.archive.ubuntu.com/ubuntu focal-backports InRelease
Apr 15 12:26:52 cloudblock-instance cloudblock-ansible-state.sh[2700532]: Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease
Apr 15 12:26:56 cloudblock-instance cloudblock-ansible-state.sh[2700532]: Reading package lists...
Apr 15 12:26:57 cloudblock-instance cloudblock-ansible-state.sh[2700792]: Reading package lists...
Apr 15 12:26:58 cloudblock-instance cloudblock-ansible-state.sh[2700792]: Building dependency tree...
Apr 15 12:26:58 cloudblock-instance cloudblock-ansible-state.sh[2700792]: Reading state information...
Apr 15 12:26:58 cloudblock-instance cloudblock-ansible-state.sh[2700792]: git is already the newest version (1:2.25.1-1ubuntu3.3).
Apr 15 12:26:58 cloudblock-instance cloudblock-ansible-state.sh[2700792]: python3-pip is already the newest version (20.0.2-5ubuntu1.6).
Apr 15 12:26:58 cloudblock-instance cloudblock-ansible-state.sh[2700792]: The following package was automatically installed and is no longer required:
Apr 15 12:26:58 cloudblock-instance cloudblock-ansible-state.sh[2700792]:   libfwupdplugin1
Apr 15 12:26:58 cloudblock-instance cloudblock-ansible-state.sh[2700792]: Use 'apt autoremove' to remove it.
Apr 15 12:27:01 cloudblock-instance cloudblock-ansible-state.sh[2700792]: 0 upgraded, 0 newly installed, 0 to remove and 22 not upgraded.
Apr 15 12:27:02 cloudblock-instance cloudblock-ansible-state.sh[2700859]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
Apr 15 12:27:02 cloudblock-instance cloudblock-ansible-state.sh[2700859]:   from cryptography.utils import int_from_bytes
Apr 15 12:27:02 cloudblock-instance cloudblock-ansible-state.sh[2700859]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
Apr 15 12:27:02 cloudblock-instance cloudblock-ansible-state.sh[2700859]:   from cryptography.utils import int_from_bytes
Apr 15 12:27:03 cloudblock-instance cloudblock-ansible-state.sh[2700859]: Requirement already satisfied: pip in /usr/local/lib/python3.8/dist-packages (22.0.4)
Apr 15 12:27:03 cloudblock-instance cloudblock-ansible-state.sh[2700859]: WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]:   from cryptography.utils import int_from_bytes
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]:   from cryptography.utils import int_from_bytes
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: ansible in /usr/local/lib/python3.8/dist-packages (5.6.0)
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: oci in /usr/local/lib/python3.8/dist-packages (2.63.0)
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: ansible-core~=2.12.4 in /usr/local/lib/python3.8/dist-packages (from ansible) (2.12.4)
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: circuitbreaker<2.0.0,>=1.3.1 in /usr/local/lib/python3.8/dist-packages (from oci) (1.3.2)
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: pytz>=2016.10 in /usr/local/lib/python3.8/dist-packages (from oci) (2021.3)
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: python-dateutil<3.0.0,>=2.5.3 in /usr/local/lib/python3.8/dist-packages (from oci) (2.8.2)
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: cryptography<=3.4.7,>=3.2.1 in /usr/local/lib/python3.8/dist-packages (from oci) (3.4.7)
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: pyOpenSSL<=19.1.0,>=17.5.0 in /usr/lib/python3/dist-packages (from oci) (19.0.0)
Apr 15 12:27:05 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from oci) (2019.11.28)
Apr 15 12:27:06 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: jinja2 in /usr/lib/python3/dist-packages (from ansible-core~=2.12.4->ansible) (2.10.1)
Apr 15 12:27:06 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from ansible-core~=2.12.4->ansible) (21.3)
Apr 15 12:27:06 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: PyYAML in /usr/lib/python3/dist-packages (from ansible-core~=2.12.4->ansible) (5.3.1)
Apr 15 12:27:06 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: resolvelib<0.6.0,>=0.5.3 in /usr/local/lib/python3.8/dist-packages (from ansible-core~=2.12.4->ansible) (0.5.4)
Apr 15 12:27:06 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.8/dist-packages (from cryptography<=3.4.7,>=3.2.1->oci) (1.15.0)
Apr 15 12:27:06 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.5.3->oci) (1.14.0)
Apr 15 12:27:06 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: pycparser in /usr/local/lib/python3.8/dist-packages (from cffi>=1.12->cryptography<=3.4.7,>=3.2.1->oci) (2.21)
Apr 15 12:27:06 cloudblock-instance cloudblock-ansible-state.sh[2700864]: Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->ansible-core~=2.12.4->ansible) (3.0.7)
Apr 15 12:27:06 cloudblock-instance cloudblock-ansible-state.sh[2700864]: WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Apr 15 12:27:07 cloudblock-instance cloudblock-ansible-state.sh[2700869]: Starting galaxy collection install process
Apr 15 12:27:07 cloudblock-instance cloudblock-ansible-state.sh[2700869]: Nothing to do. All requested collections are already installed. If you want to reinstall them, consider using `--force`.
Apr 15 12:27:08 cloudblock-instance cloudblock-ansible-state.sh[2700873]: fatal: destination path '/opt/cloudblock' already exists and is not an empty directory.
Apr 15 12:27:08 cloudblock-instance cloudblock-ansible-state.sh[2700881]: Already up to date.
Apr 15 12:27:09 cloudblock-instance cloudblock-ansible-state.sh[2700882]: [WARNING]: No inventory was parsed, only implicit localhost is available
Apr 15 12:27:09 cloudblock-instance cloudblock-ansible-state.sh[2700882]: [WARNING]: provided hosts list is empty, only localhost is available. Note that
Apr 15 12:27:09 cloudblock-instance cloudblock-ansible-state.sh[2700882]: the implicit localhost does not match 'all'
Apr 15 12:27:11 cloudblock-instance python3[2700926]: ansible-ansible.legacy.setup Invoked with gather_subset=['all'] gather_timeout=10 filter=[] fact_path=/etc/ansible/facts.d
Apr 15 12:27:14 cloudblock-instance python3[2701032]: ansible-apt Invoked with pkg=['docker.io', 'libcap2-bin', 'python3-pip', 'ssl-cert'] state=latest update_cache=True package=['docker.io', 'libcap2-bin', 'python3-pip', 'ssl-cert'] update_cache_retries=5 update_cache_retry_max_delay=12 cache_valid_time=0 purge=False force=False upgrade=no dpkg_options=force-confdef,force-confold autoremove=False autoclean=False fail_on_autoremove=False only_upgrade=False force_apt_get=False allow_unauthenticated=False allow_downgrade=False lock_timeout=60 deb=None default_release=None install_recommends=None policy_rc_d=None
Apr 15 12:27:19 cloudblock-instance python3[2701321]: ansible-pip Invoked with name=['docker'] executable=pip3 state=latest virtualenv_site_packages=False virtualenv_command=virtualenv editable=False version=None requirements=None virtualenv=None virtualenv_python=None extra_args=None chdir=None umask=None
Apr 15 12:27:22 cloudblock-instance python3[2701353]: ansible-sysctl Invoked with name=net.ipv4.ip_forward value=1 state=present reload=True sysctl_set=True sysctl_file=/etc/sysctl.conf ignoreerrors=False
Apr 15 12:27:24 cloudblock-instance python3[2701381]: ansible-systemd Invoked with name=docker state=started enabled=True daemon_reload=False daemon_reexec=False scope=system no_block=False force=None masked=None
Apr 15 12:27:25 cloudblock-instance python3[2701410]: ansible-file Invoked with path=/opt/cloudflared state=directory recurse=False force=False follow=True modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S unsafe_writes=False _original_basename=None _diff_peek=None src=None modification_time=None access_time=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None
Apr 15 12:27:25 cloudblock-instance python3[2701436]: ansible-file Invoked with path=/opt/pihole state=directory recurse=False force=False follow=True modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S unsafe_writes=False _original_basename=None _diff_peek=None src=None modification_time=None access_time=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None
Apr 15 12:27:26 cloudblock-instance python3[2701461]: ansible-file Invoked with path=/opt/pihole/etc state=directory recurse=False force=False follow=True modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S unsafe_writes=False _original_basename=None _diff_peek=None src=None modification_time=None access_time=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None
Apr 15 12:27:26 cloudblock-instance python3[2701486]: ansible-file Invoked with path=/opt/pihole/dnsmasq.d state=directory recurse=False force=False follow=True modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S unsafe_writes=False _original_basename=None _diff_peek=None src=None modification_time=None access_time=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None
Apr 15 12:27:27 cloudblock-instance python3[2701511]: ansible-file Invoked with path=/opt/webproxy state=directory recurse=False force=False follow=True modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S unsafe_writes=False _original_basename=None _diff_peek=None src=None modification_time=None access_time=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None
Apr 15 12:27:27 cloudblock-instance python3[2701536]: ansible-file Invoked with path=/opt/wireguard state=directory recurse=False force=False follow=True modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S unsafe_writes=False _original_basename=None _diff_peek=None src=None modification_time=None access_time=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None
Apr 15 12:27:28 cloudblock-instance python3[2701562]: ansible-ansible.legacy.stat Invoked with path=/opt/webproxy/httpd-ssl.conf follow=False get_checksum=True checksum_algorithm=sha1 get_md5=False get_mime=True get_attributes=True
Apr 15 12:27:29 cloudblock-instance python3[2701573]: ansible-ansible.legacy.file Invoked with owner=root group=root mode=292 dest=/opt/webproxy/httpd-ssl.conf _original_basename=httpd-ssl.conf recurse=False state=file path=/opt/webproxy/httpd-ssl.conf force=False follow=True modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S unsafe_writes=False _diff_peek=None src=None modification_time=None access_time=None seuser=None serole=None selevel=None setype=None attributes=None
Apr 15 12:27:29 cloudblock-instance python3[2701599]: ansible-ansible.legacy.stat Invoked with path=/opt/webproxy/httpd.conf follow=False get_checksum=True checksum_algorithm=sha1 get_md5=False get_mime=True get_attributes=True
Apr 15 12:27:30 cloudblock-instance python3[2701610]: ansible-ansible.legacy.file Invoked with owner=root group=root mode=292 dest=/opt/webproxy/httpd.conf _original_basename=httpd.conf recurse=False state=file path=/opt/webproxy/httpd.conf force=False follow=True modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S unsafe_writes=False _diff_peek=None src=None modification_time=None access_time=None seuser=None serole=None selevel=None setype=None attributes=None
Apr 15 12:27:31 cloudblock-instance python3[2701636]: ansible-blockinfile Invoked with create=True mode=0644 path=/opt/cloudflared/endpoints block=adguard https://dns.adguard.com/dns-query
                                                      cloudflare https://cloudflare-dns.com/dns-query
                                                      applied-privacy https://doh.applied-privacy.net/query
                                                      google https://dns.google/dns-query
                                                      hurricane-electric https://ordns.he.net/dns-query
                                                      pi-dns https://doh.pi-dns.com/dns-query
                                                      quad9-recommended https://dns.quad9.net/dns-query
                                                      libre-dns https://doh.libredns.gr/dns-query
                                                      opendns https://doh.opendns.com/dns-query
                                                       state=present marker=# {mark} ANSIBLE MANAGED BLOCK backup=False marker_begin=BEGIN marker_end=END unsafe_writes=False insertafter=None insertbefore=None validate=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None
Apr 15 12:27:32 cloudblock-instance python3[2701663]: ansible-ansible.legacy.command Invoked with _raw_params=awk '/cloudflare/ { print $2 }' /opt/cloudflared/endpoints
                                                       _uses_shell=True warn=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Apr 15 12:27:37 cloudblock-instance python3[2701692]: ansible-oci_key_management_decrypted_data Invoked with auth_type=instance_principal ciphertext={removed} service_endpoint=https://{removed}-crypto.kms.{removed}-1.oraclecloud.com key_id=ocid1.key.oc1.{removed}-1.{removed}.{removed} force_create=False state=present config_file_location=None config_profile_name=None api_user=None api_user_fingerprint=NOT_LOGGING_PARAMETER api_user_key_file=None api_user_key_pass_phrase=NOT_LOGGING_PARAMETER tenancy=None region=None key_by=None associated_data=None logging_context=None key_version_id=None encryption_algorithm=None
Apr 15 12:27:42 cloudblock-instance python3[2701719]: ansible-oci_object_storage_object Invoked with auth_type=instance_principal namespace_name={removed} bucket_name=cloudblock-bucket object_name=wireguard/wireguard_backup.tar.gz dest=/opt/wireguard_backup.tar.gz force=True state=present config_file_location=None config_profile_name=None api_user=None api_user_fingerprint=NOT_LOGGING_PARAMETER api_user_key_file=None api_user_key_pass_phrase=NOT_LOGGING_PARAMETER tenancy=None region=None src=None content_length=None expect=None content_md5=None content_type=None content_language=None content_encoding=None content_disposition=None cache_control=None opc_sse_customer_algorithm=None opc_sse_customer_key=NOT_LOGGING_PARAMETER opc_sse_customer_key_sha256=NOT_LOGGING_PARAMETER opc_sse_kms_key_id=None storage_tier=None opc_meta=None version_id=None
Apr 15 12:27:44 cloudblock-instance python3[2701770]: ansible-stat Invoked with path=/opt/wireguard/server follow=False get_md5=False get_checksum=True get_mime=True get_attributes=True checksum_algorithm=sha1
Apr 15 12:27:46 cloudblock-instance python3[2701801]: ansible-oci_object_storage_object Invoked with auth_type=instance_principal namespace_name={removed} bucket_name=cloudblock-bucket object_name=pihole/pihole_backup.tar.gz dest=/opt/pihole_backup.tar.gz force=True state=present config_file_location=None config_profile_name=None api_user=None api_user_fingerprint=NOT_LOGGING_PARAMETER api_user_key_file=None api_user_key_pass_phrase=NOT_LOGGING_PARAMETER tenancy=None region=None src=None content_length=None expect=None content_md5=None content_type=None content_language=None content_encoding=None content_disposition=None cache_control=None opc_sse_customer_algorithm=None opc_sse_customer_key=NOT_LOGGING_PARAMETER opc_sse_customer_key_sha256=NOT_LOGGING_PARAMETER opc_sse_kms_key_id=None storage_tier=None opc_meta=None version_id=None
Apr 15 12:27:48 cloudblock-instance python3[2701828]: ansible-stat Invoked with path=/opt/pihole/etc/pihole-FTL.db follow=False get_md5=False get_checksum=True get_mime=True get_attributes=True checksum_algorithm=sha1
Apr 15 12:27:49 cloudblock-instance python3[2701859]: ansible-blockinfile Invoked with create=True mode=0644 path=/opt/cloudflared/Dockerfile block=FROM ubuntu:bionic
                                                      EXPOSE 53/tcp
                                                      EXPOSE 53/udp
                                                      RUN apt-get update && \
                                                          apt-get install -y wget && \
                                                          wget https://github.com/cloudflare/cloudflared/releases/download/2021.9.2/cloudflared-linux-amd64 --directory-prefix=/usr/local/bin/ && \
                                                          chmod 555 /usr/local/bin/cloudflared-linux-amd64
                                                          ENTRYPOINT ["/usr/local/bin/cloudflared-linux-amd64", "proxy-dns","--address", "172.18.0.2", "--port", "53", "--upstream", "https://cloudflare-dns.com/dns-query"]
                                                       state=present marker=# {mark} ANSIBLE MANAGED BLOCK backup=False marker_begin=BEGIN marker_end=END unsafe_writes=False insertafter=None insertbefore=None validate=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None
Apr 15 12:27:50 cloudblock-instance python3[2701888]: ansible-docker_network Invoked with name=cloudblock driver=bridge driver_options={'com.docker.network.driver.mtu': 1500} ipam_config=[{'subnet': '172.18.0.0/24', 'gateway': '172.18.0.1', 'iprange': None, 'aux_addresses': None}] docker_host=unix://var/run/docker.sock api_version=auto timeout=60 tls=False use_ssh_client=False validate_certs=False debug=False connected=[] state=present force=False appends=False labels={} tls_hostname=None ca_cert=None client_cert=None client_key=None ssl_version=None ipam_driver=None ipam_driver_options=None enable_ipv6=None internal=None scope=None attachable=None
Apr 15 12:27:52 cloudblock-instance python3[2701929]: ansible-docker_container Invoked with name=cloudflared_doh image=cloudflared_doh networks=[{'name': 'cloudblock', 'ipv4_address': '172.18.0.2', 'ipv6_address': None, 'aliases': None, 'links': None}] purge_networks=True restart_policy=always docker_host=unix://var/run/docker.sock api_version=auto timeout=60 tls=False use_ssh_client=False validate_certs=False debug=False cleanup=False container_default_behavior=no_defaults force_kill=False ignore_image=False keep_volumes=True networks_cli_compatible=True output_logs=False pull=False recreate=False restart=False state=started tls_hostname=None ca_cert=None client_cert=None client_key=None ssl_version=None auto_remove=None blkio_weight=None capabilities=None cap_drop=None cgroup_parent=None command=None comparisons=None command_handling=None cpu_period=None cpu_quota=None cpus=None cpuset_cpus=None cpuset_mems=None cpu_shares=None default_host_ip=None detach=None devices=None device_read_bps=None device_write_bps=None device_read_iops=None device_write_iops=None device_requests=None dns_servers=None dns_opts=None dns_search_domains=None domainname=None entrypoint=None env=None env_file=None etc_hosts=None exposed_ports=None groups=None healthcheck=None hostname=None init=None interactive=None ipc_mode=None kernel_memory=None kill_signal=None labels=None links=None log_driver=None log_options=None mac_address=None memory=None memory_reservation=None memory_swap=None memory_swappiness=None mounts=None network_mode=None oom_killer=None oom_score_adj=None paused=None pid_mode=None pids_limit=None privileged=None publish_all_ports=None published_ports=None read_only=None removal_wait_timeout=None restart_retries=None runtime=None security_opts=None shm_size=None stop_signal=None stop_timeout=None storage_opts=None sysctls=None tmpfs=None tty=None ulimits=None user=None userns_mode=None uts=None volume_driver=None volumes=None volumes_from=None working_dir=None
Apr 15 12:27:52 cloudblock-instance systemd[1]: cloudblock-ansible-state.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 15 12:27:52 cloudblock-instance systemd[1]: cloudblock-ansible-state.service: Failed with result 'exit-code'.
Apr 15 12:28:23 cloudblock-instance systemd[1]: cloudblock-ansible-state.service: Scheduled restart job, restart counter is at 24958.

[chadgeary]

There are a few problems:

• From the log it appears your playbook is failing to start either

  • the cloudflared_doh container or,
  • the pihole container (which comes immediately after doh)
  • see ansible stuff here

• The systemd service doesnt limit failures (runs indefinitely)

  • fixed via limit systemd service via StartLimitInterval and StartLimitBurst #55

[chadgeary]

Can you review the output of /var/log/cloudblock.log to determine the failure in the ansible playbook?

[NanoCode012]

Hi @chadgeary , thanks for the reply!

Here are the logs for /var/log/cloudblock.log. It would seem that the systemd service (potentially when I rebooted) is trying to start the cloudflared_doh container despite it already being up due to the restart_policy: "always" set in the docker policy .

TASK [cloudflared_doh container] ***********************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Error starting container 24c65edf340f84ef3b6ae40ec79854e365bf81785fca00fe2ad871913d0e6905: 403 Client Error for http+docker://localhost/v1.41/containers/24c65edf340f84ef3b6ae40ec79854e365bf81785fca00fe2ad871913d0e6905/start: Forbidden (\"Address already in use\")"}

PLAY RECAP *********************************************************************
localhost                  : ok=16   changed=2    unreachable=0    failed=1    skipped=3    rescued=0    ignored=0

I think limiting the service would fixes the problem, although could it disable itself after it succeeds once even across reboots?