2 Runtime State, Watchdog and Self-Healing

Intranet devices often sit unattended, so "recovers by itself" beats "someone on call". This chapter covers reading state at a glance, when the watchdog steps in, and the order to investigate repeated restarts.

State at a Glance

lynxdrive-ctl status lists the main service state, watchdog state, listening port, data root and port-connectivity probe — one line each, ideal for screenshots. lynxdrive-ctl info adds the device's per-NIC access addresses, the form best forwarded to colleagues. For detail, lynxdrive-ctl logs prints the last 50 lines by default.

How the Watchdog Works

The watchdog inspects every 10 seconds: is the service process alive, can the port still be reached. Three consecutive failures count as a fault and trigger an automatic restart, usually recovering within 30 seconds to a minute. It starts with the main service and runs independently, so it can act even when the service hangs.

What Self-Healing Looks Like

A page that suddenly fails and later works again is often self-healing in action. Watchdog actions are logged — entries like "health check failed" or "service restarted" are the fastest evidence that a self-heal happened, no guessing required.

Investigating Frequent Restarts

Investigate in the order space, permissions, port, storage: is the disk full (uploads and writes fail when exhausted); were the data directory's owner and permissions altered; is the port occupied by another program; is the external disk flapping. Fix the cause — repeated manual restarts only bury it.

Manual Restart and Maintenance Downtime

lynxdrive-ctl restart restarts manually; stop / start serve maintenance windows; all act on both the service and the watchdog to avoid stop-and-pull-back tug-of-war. Don't kill processes directly: the watchdog relaunches them instantly and the problem stays. For long downtime, stop the watchdog first, then the service, and restore in reverse.

Reading the Logs

Logs are chronological: read the last few dozen lines for the most recent anomaly, then trace upward by timestamp. Watch keywords like "health check failed", "service restarted", "migration" and "error". Recording the key lines with timestamps is the most valuable material when contacting support.