Safety Mode & Zombie Reconciliation

ServOS continuously monitors the state of your servers to ensure the containers running on them match the expected state in the database. When it finds a container that ServOS manages (labeled servos.managed=true) but which does not correspond to any active deployment in the database, it considers it a Zombie Container.

Zombie Reconciliation

Under normal operation, ServOS automatically removes Zombie Containers to free up resources. This keeps your servers clean and prevents “drift”.

Safety Mode Protection

In certain scenarios, such as restoring the ServOS database from an older backup, there might be a massive discrepancy between the database (which thinks nothing is running) and the servers (which have 50 live containers).

If ServOS were to blindly follow the “Reconciliation” logic, it would delete all 50 containers, causing a catastrophic outage.

To prevent this, ServOS implements a Safety Mode:

Trigger Condition

Safety Mode is triggered automatically for a specific server if:

1. >50% of the managed containers on that server are detected as zombies.
2. There are more than 3 zombie candidates (prevents false positives on empty servers).

What Happens?

1. No Deletion: ServOS will skip the auto-pruning process for that server. The zombie containers will remain running.
2. Alert: A system alert will be generated (visible in the Dashboard status).
3. Logs: Use docker ps -a --filter label=servos.managed=true on the server to investigate.

How to Resolve

review the “zombies”.

• If they are indeed garbage: You can manually delete them via Docker CLI, or manually synchronize the database state (e.g. by re-deploying services). Once the ratio drops below 50%, ServOS will resume auto-cleanup.
• If the database is wrong: You might need to re-import or re-deploy the services so the database tracks them correctly again.