Skip to navigation

On-Prem Agent Troubleshooting

View as Markdown

Installation Errors

These errors occur while running setup_agent.sh during initial setup or upgrade. For errors after the agent is running, see Diagnosing Common Issues below.

🔴 Docker service fails to start

Failed to enable unit: Unit file docker service does not exist
[ERROR] Error enabling Docker service to start on boot
Failed to start Docker service: Unit docker service not found
[ERROR] Error starting Docker service

There may be an issue with the Docker installation. To confirm:

  1. Run docker logs --follow
  2. If you see exec /bin.sh: operation not permitted, uninstall Docker and reinstall using the Docker convenience script

🔴 Docker or Podman will not install

Error installing Podman. Install Podman and Rerun
Error installing Docker, Install Docker and Rerun

The script can only auto-install Docker or Podman on select Linux distributions. Manually install Docker or Podman and rerun the script with the same options.


🔴 Podman repository error (Ubuntu earlier than 22.04)

Error adding Podman repository to package sources. Install Podman and Rerun
Error adding GPG key for Podman repository. Install Podman and Rerun

Ubuntu versions earlier than 22.04 do not support Podman via the official repository. Manually install Docker or Podman and rerun the script.


🔴 yum-utils missing (RHEL)

Error installing yum-utils to install docker. Install docker and Rerun

RHEL-based distributions require yum-utils to install Docker. Manually install Docker and rerun the script.


🔴 loginctl not installed (Podman)

Error enabling linger
Loginctl not installed. Error enabling linger

If loginctl is not installed, the script continues. This is not an issue for Docker-based containers, but for Podman the agent may stop running when the user session ends.


🔴 systemd not installed

Warning: System does not use systemd. Docker service may not start on boot

Without systemctl, the Docker or Podman container may not restart automatically on boot.


🔴 Docker or Podman version too old

Podman version $podman_version does not meet the minimum required version $PODMAN_VERSION. Please upgrade Podman
Docker version $docker_version does not meet the minimum required version $DOCKER_VERSION. Uninstall Docker and rerun.

Upgrade Docker or Podman to meet the minimum version requirement, then rerun the script.


🔴 systemd directory missing (Podman)

Error creating directory /etc/systemd/system. Create the directory and Rerun
Error creating directory $HOME/.config/systemd/user. Create the directory and Rerun

Manually create the directory and rerun the script with the same options.


🔴 Podman container naming or systemd unit error

Error renaming Podman container CONTAINER_NAME to moveworks_agent_{i}
Error generating systemd unit file for Podman container moveworks_agent_

Clean up old containers, networks, and volumes: podman system prune


🔴 systemd daemon reload error (Podman)

Error reloading systemd daemon for Podman containers
Error enabling service for Podman container moveworks_agent_
Error disabling or stopping service for Podman container
Error reloading systemd daemon for Podman containers.

Verify that systemd is correctly installed and functional.


🔴 Failed to pull agent image from ECR

Error: Failed to pull agent image with version '$AGENT_VERSION' using podman from ECR repository '$ECR_URL'.

Verify the version tag is correct at gallery.ecr.aws/moveworks/agent. If the VM cannot reach ECR due to firewall restrictions, see Fetching the Agent Image Without ECR in the Installation Guide.


Diagnosing Common Issues

This page covers the most common configuration and runtime issues encountered when running the Moveworks On-Prem Agent.

Configuring LDAP over SSL (LDAPS, Port 636)

To set up the certificate, see Configure an LDAP SSL certificate for the On-Prem Agent, which also lists common certificate errors.

To show the server’s certificate, the names it covers, and its expiry:

openssl s_client -connect <ldap-host>:636 -showcerts </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -enddate -ext subjectAltName

Validating the LDAP Service Account

Via AD Explorer

AD Explorer lets you verify service account access to the LDAP server.

  1. Open AD Explorer and connect using the host, service account username, and password. You may need to connect to VPN first.
  2. Once connected, navigate to groups to validate access as needed.

Passwords containing special characters can cause authentication failures. If other steps check out, try resetting the password to one without special characters.

Via command line

Find the LDAP hostname:

nslookup -q=SRV _ldap._tcp.<DOMAIN_NAME>.com

Test service account credentials:

# Test authentication
ldapwhoami -x -H <LDAP_SERVER_HOSTNAME_OR_IP> -w <PASSWORD> -D "<NETBIOS_DOMAIN>\<SVC_ACCOUNT>"
# Search for objects
ldapsearch -x -H ldaps://<LDAP_SERVER_HOSTNAME_OR_IP>:636 -W -D "<NETBIOS_DOMAIN>\<SVC_ACCOUNT>" -b "<BASE_DN>"
# Return users (LDAPS example)
ldapsearch -x -LLL -H ldaps://ldap.company.com:636 -D "username" -w "password" -b "dc=company,dc=com" -s sub "(objectClass=user)" givenName

On Red Hat OS, install openldap first: yum install openldap*


Confirming the NETBIOS Domain for an AD Service Account

Some environments require the service account username in the format NETBIOS_DOMAIN\username (e.g. MOVEWORKS\svc_ad_moveworks).

To find the NETBIOS domain:

  1. Open Active Directory Users and Computers
  2. Click the Find Objects icon (folder with magnifying glass)
  3. Search for the service account and double-click it
  4. Select the Account tab — the value in User logon name is the NETBIOS domain

Update the agent config to use: NETBIOS_DOMAIN\username


Known Issues and Fixes

🔴 Moveworks access secret is invalid

Your org access secret is incorrect or has expired. Tokens expire if not used within 30 days of generation.

Generate a new secret in Moveworks Setup > Core Platform > On-Premise Agents > Generate Secret.

To validate a token manually:

curl --header "Content-Type: application/json" --request POST \
--data '{"access_secret":"<YOUR_TOKEN>", "access_key": "<YOUR_ORG_NAME>"}' \
https://agent.moveworks.com/api/v1/auth

🔴 Permission denied when starting the agent

open /home/moveworks/agent/conf/agent_config.yml: permission denied

Inside its container, the agent runs as the moveworks user in group 17540, the agent group. It reads its certificates and rewrites agent_config.yml on every start, so that group needs access, whatever your own account can open. The host needs no group of that name. From the install folder, fix the folders:

sudo ./setup_agent.sh --stop
sudo ./setup_agent.sh --permissions
sudo ./setup_agent.sh --validate
sudo ./setup_agent.sh --start

On Podman, run sudo chown -R <agent-user>:17540 moveworks instead of --permissions.

To fix individual files, give the agent group (17540) access. Run from the folder that contains setup_agent.sh:

sudo chgrp 17540 moveworks/conf/agent_config.yml moveworks/certs/<cert-file>.pem
sudo chmod 660 moveworks/conf/agent_config.yml
sudo chmod 640 moveworks/certs/<cert-file>.pem

The config needs 660 because the agent rewrites it on every start; certificates need only 640 (or 644). To copy a replaced file’s permissions instead, run sudo chown --reference=<old-file> <new-file> and sudo chmod --reference=<old-file> <new-file>. On Podman, also run sudo chown <agent-user> <file>.

Don’t use 600: it removes the group access the agent needs. Don’t use 777: it lets every account on the host read your passwords.


🔴 Permission denied when accessing a certificate file

A certificate copied in with scp keeps its original permissions, and --validate doesn’t check files. Give the agent group (17540) read access, then restart the agent:

sudo chgrp 17540 moveworks/certs/<cert-file>.pem
sudo chmod 640 moveworks/certs/<cert-file>.pem

🔴 Agent cannot communicate — proxy authentication required

rest_pool: Get "https://...": Proxy Authentication Required

Your proxy applies to one system but not another. Add the following to the REST config entry that needs to bypass the proxy:

do_not_use_rest_proxy: true

🔴 Agent cannot communicate with an on-prem system — Forbidden

rest_pool: Get "https://<base_url>/...": Forbidden

Same cause as above. Add do_not_use_rest_proxy: true to the relevant REST config entry.


🔴 401 Unauthorized when authenticating to Moveworks

retry: Received status code 401 while trying to authenticate (moveworks_user=ORGNAME)

The access secret is invalid or expired (tokens expire after 30 days without use). Generate a new one in Moveworks Setup and reconfigure the agent:

sudo ./setup_agent.sh --configure

Firewall tests:

ping google.com
curl https://agent.moveworks.com/api/v1/web

🔴 Agent fails to start — “Operation Not Permitted”

The container user lacks executable permissions. Work with your sysadmin to resolve at the VM level. As a workaround, remove --security-opt=no-new-privileges from the start script.


🔴 I/O timeout in agent logs

retry: Post "https://agent.moveworks.com/...": dial tcp: ... i/o timeout

The agent container cannot make an outbound connection to Moveworks. Common causes: firewall rules, DNS issues, or Docker networking.

Try running the agent in host network mode — see the Docker networking issue section below.


🔴 CA bundle certificate missing

context deadline exceeded (Client.Timeout exceeded while awaiting headers)

CA bundle certs are not being recognized by the container.

  1. Find the CA bundle path: openssl version -d (typically /etc/pki/tls/certs/ca-bundle.crt)
  2. Mount the volume in the start script: -v /etc/pki/:/etc/pki/
  3. Reference it in agent_config.yml:
moveworks_config:
path_to_cert: /etc/pki/tls/certs/ca-bundle.crt

🔴 Docker networking issue — No Route to Host

  1. Restart the Docker daemon: sudo systemctl restart docker
  2. Use telnet to verify the server can reach the LDAP domain controller
  3. If networking is the issue, enable host network mode by adding --net=host to the start script

🔴 Docker container shuts down on server restart

sudo systemctl enable docker

🔴 Docker log size error

unknown log opt 'max-size'/'max-file' for journald log driver

Edit the start script and add --log-driver json-file.


🔴 Container in “Restarting” state, logs not generating

🔴 Agent config file is empty when running cat agent_config.yml

The configurator likely couldn’t write to moveworks/conf/. Give the agent group (17540) write access, then configure again:

sudo chgrp 17540 moveworks/conf moveworks/conf/agent_config.yml
sudo chmod 770 moveworks/conf
sudo chmod 660 moveworks/conf/agent_config.yml
sudo ./setup_agent.sh --configure

🔴 x509: certificate signed by unknown authority — connecting to Moveworks API

retry: Post "https://agent.moveworks.com/...": x509: certificate signed by unknown authority

The system cert used to connect to the Moveworks API is not from a known CA. Ask your admins to install a cert from a trusted certificate authority.

As a last resort, copy the system CA bundle (/etc/ssl/certs/ca-certificates.crt) to the agent’s /certs folder, mount it, and reference it:

moveworks_config:
path_to_cert: /home/moveworks/agent/certs/ca-certificates.crt

Test with:

curl -vs --cacert <cert_path> https://agent.moveworks.com/api/v1/web

🔴 x509: certificate signed by unknown authority — connecting to LDAP

retry: DialURL: LDAP Result Code 200 "Network Error": x509: certificate signed by unknown authority

The cert used to connect to LDAP does not include the full chain. Ensure the .pem file contains all intermediate certificates, then add it to ldap_config:

ldap_config:
use_ssl: true
path_to_cert: /home/moveworks/agent/certs/cert.pem

🔴 No such file or directory: TLS cert path

The cert exists on the host but is not mounted into the container. Add the volume to the start script:

-v "$(pwd)/certs":/home/moveworks/agent/certs

🔴 x509: certificate signed by unknown authority — connecting to an on-prem REST service

x509: certificate signed by unknown authority

Use ca_cert_path in the REST config to point to the cert inside the container:

rest_configs:
MY_SERVICE:
service: MY_SERVICE
ca_cert_path: /home/moveworks/agent/certs/myCert.crt

If the cert is expired, you can temporarily disable verification (not recommended for production):

rest_configs:
MY_SERVICE:
tls_skip_verify: true

🔴 Unknown certificate authority — wrong file format

If the cert is a .p7b file:

openssl pkcs7 -print_certs -in certnew.p7b -out ca_chain.pem

If the cert is a .cer file:

openssl x509 -inform der -in cert.cer -out cert.pem

🔴 Certificate relies on legacy Common Name field

x509: certificate relies on legacy Common Name field, use SANs

The certificate only has a CN with no SAN, which is deprecated. Set tls_skip_verify: true in the agent config as a workaround, or request a new cert with a SAN from your CA.


🔴 404 for config request

received non-200 response status [404 Not Found] for config request

The config_url in agent_config.yml is incorrect. Verify it matches your region — for US Commercial: https://agent.moveworks.com/api/v1/config.


🔴 404 for authentication request

Received status code 404 while trying to authenticate

The auth_url is incorrect. Ensure there are no spaces or quotes around the value. For US Commercial: https://agent.moveworks.com/api/v1/auth.


🔴 Unable to read LDAP response packet — connection reset by peer

The LDAP server is denying the connection. Likely a bad cert or wrong port. Try port 389 as a workaround to rule out network issues, then work with your team to obtain the correct LDAPS cert.


🔴 TLS handshake error — first record does not look like a TLS handshake

Typically caused by a proxy configured for the wrong protocol (HTTP vs HTTPS). Check whether your HTTP_PROXY or HTTPS_PROXY setting matches the protocol the agent is using.


🔴 Container not visible in docker ps / podman ps after start

Check agent_config.yml for typos. Even a single indentation error will prevent the container from starting.


🔴 LDAP Result Code 200 — certificate valid for wrong host

x509: certificate is valid for NMH-DC01.domain.com

The agent is connecting to a load-balanced domain that resolves to multiple servers, but the cert is issued for a specific server. Point directly to one server in agent_config.yml:

ldap_config:
host: nmh-dc02.domain.com

🔴 LDAP Result Code 200 — server misbehaving (DNS)

LDAP Result Code 200 "Network Error": dial tcp: lookup ... server misbehaving

DNS resolver mismatch between host and container.

  1. Compare cat /etc/resolv.conf on the host and inside the container
  2. Stop and remove all containers:
    docker stop $(docker ps -a -q)
    docker rm $(docker ps -a -q)
  3. Restart Docker and start a new agent:
    sudo systemctl restart docker
    sudo ./setup_agent.sh --start

Alternatively, enable host networking mode in the start script.


🔴 Podman — user namespaces not enabled

user namespaces are not enabled in /proc/sys/user/max_user_namespaces

On CentOS 7, run as root:

echo "user.max_user_namespaces=10000" > /etc/sysctl.d/42-rootless.conf
sysctl --system

🔴 LDAP Result Code 200 — certificate expired

x509: certificate has expired or is not yet valid

Check the certificate file’s expiry on the agent host, the server or VM that runs the agent’s container. Run from the folder that contains setup_agent.sh:

sudo openssl x509 -enddate -noout -in moveworks/certs/<cert-file>.pem

Check the LDAP server’s certificate:

openssl s_client -connect <ldap-host>:636 </dev/null 2>/dev/null | openssl x509 -noout -enddate

Check the host’s clock with date -u. To replace the certificate, see Rotate a certificate.


🔴 LDAP connection reset by peer

The port is likely closed. If using port 389, switch to port 636 with a cert and configure LDAPS.


🔴 LDAP Result Code 8 — Strong Auth Required

The server requires LDAPS (port 636 with a cert), but the agent is configured for plain LDAP (port 389). Reconfigure using LDAPS.


🔴 ITSM_DISCONNECTED — consistent timeouts for ticketing

  • Check for WAFs throttling the connection between the agent and your ITSM
  • Lower environments often have reduced CPU/storage — verify the timeout is not caused by resource limits on the ITSM instance itself

Common LDAP Error Codes

Error CodeErrorDescription
525User not foundInvalid username supplied
52eInvalid credentialsValid username but wrong password. Also occurs on password expiry.
530Not permitted at this timeLogin restricted to certain hours
531Not permitted from this workstationLogin restricted to certain machines
532Password expiredPassword is valid but expired
533Account disabledAccount has been disabled
701Account expiredAccount has expired
773User must reset passwordPassword reset required before login
775Account locked outAccount is locked, regardless of whether password is correct

Debugging Tips

Check agent logs

# Follow live logs (Docker)
docker logs -f <container_name>
# Tail INFO logs from log files
tail -f /var/log/moveworks/*.INFO.log
# Access the container shell
docker exec -it $(docker ps --format '{{.Names}}') /bin/bash

Useful Docker commands

# Kill all running containers
docker kill $(docker ps -q)
# Remove all stopped containers
docker rm $(docker ps -aq)
# Remove a specific agent image
docker rmi <image_id>
# Test network connectivity
telnet <LDAP_DOMAIN> 636

Test connectivity

# Test outbound access
ping google.com
curl https://agent.moveworks.com/api/v1/web
# Test LDAP
ldapsearch -x -H ldaps://<host>:636 -D "DOMAIN\user" -W
# Test with a specific cert
curl -vs --cacert <cert_path> https://agent.moveworks.com/api/v1/web

See Also