Deploy / v0.1

Troubleshoot a customer-hosted server

Health endpoints, service logs, and what each installer, backup and restore error means and how to recover.

Reviewed September 27, 2026 · Current implementation

Health endpoints

Check	What it proves
http://127.0.0.1:8080/api/health	Console is up and reached the database
http://127.0.0.1:8081/health	Query API is up and reached the database; 503 when the database ping fails
http://127.0.0.1:8082/health	MCP listener is up; says nothing about the API or keys
http://127.0.0.1:8083/health	Relay listener is up; returns ok
https://your console name/api/health, with --cacert	TLS, the proxy and the console together
https://your console name/relay-health	The proxy reaches the relay

None of these shows that sign-in, enrollment, an API key or an endpoint question works.

Service logs

Terminal

systemctl status recursift-{webapp,api,mcp,relay,proxy}
journalctl -u recursift-webapp -u recursift-api -u recursift-mcp \
  -u recursift-relay -u recursift-proxy --since "1 hour ago"

All services log to the system journal. The relay's startup line includes its version and source commit. The proxy writes no access log. Review journal retention and access on your server; logs are kept when the server is removed.

Verification failures

Verification failures change nothing on the host. Never work around one by editing the release or replacing a file.

Message	Meaning and recovery
Invalid signature	The manifest was not signed by the pinned key. Confirm the key fingerprint out of band, then obtain the release again from the authenticated channel.
Missing or extra artifact files, Missing or altered artifact, Not a regular file	The staged set differs from its signed manifest. Copy the complete set again into a fresh directory that only you can write.
Unapproved artifact version or manifest	An API, relay or endpoint set is not the one pinned in the console release. Stage the matching set.
Missing pinned agent prerequisite (MR-23A), or Missing or invalid agent prerequisite (MR-23A)	The signed Linux endpoint set is absent or invalid. This is expected until agent#117 delivers it.
Required signed file missing	A required file is not listed in the manifest. Obtain a complete release.
Wrong target for	A binary is not built for Linux x86-64.
Unsupported composition or optional roles	The release is not the core profile or lists an optional role. Obtain a core release.
An Ed25519 public key is required, or Invalid base64 encoding	A pinned key file is in the wrong format. Use the published PEM or base64 key.

Installer failures

Message	Meaning and recovery
Run as root on the dedicated customer VM.	Run with sudo.
Another customer-stack operation is running.	Wait for the other install, backup or removal to finish.
Origin differs from the signed customer-specific release.	--origin must match the origin in the signed release exactly. Request a release built for your origin.
Linux x86-64 is required.	Use a Linux x86-64 server.
Offline platform package mismatch	Install the exact package version pinned in the release from verified platform media.
Trusted bootstrap runtime version mismatch.	Install the pinned runtime version from trusted media.
This exact release is already installed. No schema changes.	Not an error. Nothing changed.
Existing installation differs. Upgrades belong to MR-24.	A different release is installed. There is no upgrade path yet; see Back up, restore and remove.
Partial or existing installation found. Preserve it for recovery; refusing to overwrite.	An earlier attempt left state behind. Keep it and its logs for diagnosis. Gap: there is no supported cleanup of a partial install; start again on a fresh server.
Database exists. Fresh installation requires an empty stack; upgrades are out of scope.	A Recursift database is already present. Install only on an empty server.
Recursift database roles already exist; refusing reuse.	Leftover database roles. Install only on an empty server.
Database must listen only on localhost.	Set the database to listen on localhost only, then restart it.
Provide --origin and --tls with server.crt, server.key and ca.crt.	Supply both options and all three non-empty files.
A certificate verification error	server.crt does not chain to ca.crt or does not match the console name.
TLS private key does not match certificate.	Supply the key for this certificate.
Missing customer-local identity bootstrap (webapp#20 / MR-22).	The console release lacks local sign-in. Obtain a current release.
Unsafe preexisting service account	An account with a recursift- name exists and is not a no-login system account with its own group. Remove or fix it.
Health check failed	A service did not answer within 60 seconds. The installer stops the services it started. Read the journal for that service.

Backup and restore failures

Message	Meaning and recovery
Usage: backup.sh NEW_BACKUP.tar.gz	Give exactly one destination that does not exist yet.
No completed installation to back up.	The install never finished. There is nothing consistent to back up.
Restore requires an empty stack. Never overwrite a running installation.	Restore only onto a server with no Recursift state.
A checksum mismatch	The archive is damaged. Use another copy of the backup.
Backup release differs from verified artifacts.	Supply the same signed release the backup was taken from.

Running server

Symptom	Meaning and recovery
403 Forbidden from the console	The request's Host or Origin is not a configured origin. Use the console name exactly and make sure any proxy preserves Host.
503 Optional services are unavailable in the customer core profile.	Expected. Inference, model-catalog and vulnerability-feed routes are off in the core profile.
External services are disabled in the customer core profile. in the console log	Expected. A feature tried to reach an outside address and was refused.
API 401, 403, 404 or 429	See Troubleshooting for the Query API; the behavior is the same on your own origin.
Setup link rejected	It expired after 15 minutes, was already used, or a newer link replaced it. Issue a new one.

Continue readingData flows and retention →