Operate an installation

Use this guide to investigate a running installation, process queued work and rotate credentials. You need the application sources, the supported Node.js and npm versions, and access to the intended installation's runtime configuration and database. Provider investigations also require the corresponding Stripe or Resend account.

For initial setup, see hosting and configuration. For account closure and erasure, see data requests.

Select the installation and diagnose it

Run commands from the application directory. Replace /ABS/runtime.env with the absolute path to your private runtime file. The with-env launcher loads the file without evaluating shell code; do not source it in a shell.

npm run with-env -- /ABS/runtime.env data -- target
npm run with-env -- /ABS/runtime.env operations -- status

Check the reported mode, database and destination before any mutation. operations status reads SQL state without contacting Stripe or Resend. It reports worker history, queue ages, budgets, causes and recommended actions. Its exit code is 0 for ok, 2 for degraded state and 1 for a command failure. It reports runtimeConfiguration: "not_checked" because it does not validate all runtime secrets or public reachability.

Use the doctor to investigate configuration or database compatibility:

npm run with-env -- /ABS/runtime.env doctor
npm run with-env -- /ABS/runtime.env doctor -- --connection-only
npm run with-env -- /ABS/runtime.env doctor -- --database

The first checks mode-aware configuration. --connection-only checks the database connection, TLS and PostgreSQL version; --database also checks migrations.

For “Service unavailable. Try again later.”, inspect the failed request in the browser's Network panel and copy its response header X-Error-Reference. Search the web service logs for the request_failed event with the same reference. Its category distinguishes database from application failures; stage identifies where the request failed, such as installation, authentication or project_create. Use these fields to narrow the investigation; the event omits request contents and raw error details.

Inspect billing queues

npm run with-env -- /ABS/runtime.env billing:reconcile -- status

Diagnose missing email

npm run with-env -- /ABS/runtime.env email:public -- status

email:public requires public mode. Read email states, retained payload keys, suppression counts and budget warnings. An email accepted by Resend is not necessarily delivered to the recipient; inspect its provider status and provider evidence when delivery matters.

If a password recovery email is missing, the public acknowledgement does not prove that an account exists or that email was queued. An absent or closed account, an exhausted installation daily or recipient hourly budget, a suppressed recipient, or a failure to record the email can produce the same neutral response without a job. Check dailyRequested against dailyLimit, the configured recipientHourlyLimit, suppressed, queue states and worker history with the commands above.

These are aggregate results. For an authorized recipient-specific investigation, compute the recipient's hash with recipientHash (src/server/email/settings.ts) for the selected installation. Use that hash to inspect the recipient's current UTC hourly bucket in public_email_budgets and any suppression reason in public_email_suppressions. If a job exists, follow its failure/provider status and Resend delivery events. Keep recipient details private; do not clear budgets or suppression records merely to force a resend.

After a fix, read status again and check that the affected work progresses, including delayed retries and work left unresolved by an interruption.

Answer a user's support request

Identify the user's account and selected team, then use the matching action. After the correction, ask the user to repeat the failed action.

Keep background work running

The Render worker runs billing reconciliation, email dispatch, email payload cleanup and monitoring. Keep it running alongside the web service; see hosting. For another process supervisor, the supplied foreground schedulers are:

npm run with-env -- /ABS/runtime.env operations:schedule -- workers
npm run with-env -- /ABS/runtime.env operations:schedule -- monitor

Run these as separate supervised processes. Billing and purge are scheduled every 60 seconds; email and monitoring every 30 seconds.

For a bounded manual pass on the selected public installation:

Email dispatch processes at most 10 jobs per pass. Retries preserve the original message and provider idempotency key; they are limited to three attempts and a 23-hour retry window, subject to earlier message expiry. For operations -- email, check ok and progressed: they report whether the pass succeeded and how many emails Resend accepted.

For targeted email work, email:public -- run-once uses the same worker reservation and history as operations -- email. It also reports accepted, retryable, abandoned and stale; skipped: true means another email pass holds the reservation. After a dispatch failure, an idle pass does not establish recovery. Correct the cause and confirm a subsequent successful dispatch.

The direct command billing:reconcile -- run-once is also available, but does not update the worker history shown by operations -- status.

Payload cleanup runs in batches of at most 100. It retains delivery history, suppression records, budgets and replay evidence. It is not an account-erasure command.

Receive operational alerts

Monitoring emits JSON triggered and resolved events to stdout, with a five-minute repeat interval for active causes. Configure your log handling if you need these events to reach an operator.

The Render worker also supports an optional Better Stack heartbeat through KIT_COLLECTOR_ENABLED=1 and KIT_COLLECTOR_HEARTBEAT_URL. The URL must use the supported https://incidents.betterstack.com/api/v1/heartbeat/... form. The collector checks SQL status every 30 seconds and sends a failure heartbeat when the status is unhealthy. Keep the token-bearing URL private. Verify incident and recovery receipt with your monitoring service.

When stopping the worker, allow it to drain. Its final drained event confirms its shutdown path completed; drain-unconfirmed requires investigation. Inspect queue status and outstanding work before retrying after an interruption.

Recover an uncertain Checkout

Use recovery when Stripe created an original Checkout session but the application did not retain its result. Identify the team in the selected database first. In your database client, bind the verified owner's email as $1 and the team name as $2 in this read-only query:

SELECT t.id AS team_id, t.name, b.attempt_id, b.customer_id, b.created_at
FROM teams t
JOIN memberships m ON m.team_id = t.id AND m.role = 'owner'
JOIN users u ON u.id = m.user_id
LEFT JOIN stripe_billing b ON b.team_id = t.id
WHERE lower(u.email) = lower($1) AND t.name = $2;

Names are not unique identifiers. Confirm the intended team with the requester and its billing history if several rows match; keep these results private. Use its team_id as TEAM_UUID:

npm run with-env -- /ABS/runtime.env billing:recover -- status TEAM_UUID

Use the returned attempt_id and the exact original Checkout session from Stripe. In the matching Stripe account and environment, inspect the stored customer_id and its Checkout sessions or request logs. The original request uses idempotency key ATTEMPT_UUID-checkout; its session metadata contains kit=atelier, team=TEAM_UUID and attempt=ATTEMPT_UUID. Match the original customer, time, configured price and application return URLs, then copy the session's cs_... ID, not its bearer URL. Do not choose a session merely because its amount or customer name looks similar.

Recovery requires the original customer to be durably bound locally. If the customer or local attempt is missing, stop and investigate the original request; this command does not reconstruct them from provider metadata.

Prepare a new operation UUID, the team UUID, the current attempt UUID, the original cs_... Checkout ID and a non-secret reference to your evidence. Replace every placeholder in this example:

npm run with-env -- /ABS/runtime.env billing:recover -- recover-checkout '{"operation":"OPERATION_UUID","team":"TEAM_UUID","attempt":"ATTEMPT_UUID","reason":"Recover original Checkout","proof":"incident:checkout-001","checkout":"cs_ORIGINAL"}'

The reason is 8–199 characters and the proof reference is 8–159 characters. Use plain letters, digits and :._/-; spaces are allowed in the reason. Do not include personal data, secrets or bearer URLs.

Recovery reads provider evidence, validates the original binding, attaches the original session and synchronizes billing. It does not create another Checkout, cancel a subscription or issue a refund. A result of duplicate: false records a new recovery; duplicate: true resumes the same recorded request. If interrupted, inspect team status and retry with the identical operation UUID and payload. Changing the payload under an existing UUID is refused.

Read billing:recover status TEAM_UUID and billing:reconcile status afterwards, then verify the team's billing view. Keep the original attempt until its outcome is resolved; creating a replacement subscription can leave the customer paying for both.

Rotate credentials

Identify every consumer first: web, worker and operator runtime files. Preserve the installation identity, billing bindings, budgets and pending work. Update consumers consistently, verify the replacement works and verify the revoked credential no longer works.

Email encryption keys

1. Add a newly generated independent key under a new ID in PUBLIC_EMAIL_KEYS, retaining the old keys. Set PUBLIC_EMAIL_ACTIVE_KEY to the new ID. Use the format in configuration; never replace key bytes under an existing ID.

2. Update all web, worker and operator consumers so no process continues creating payloads with the old active key.

3. Re-encrypt stored payloads through the updated configuration:

   npm run with-env -- /ABS/runtime.env email:public -- reencrypt
   npm run with-env -- /ABS/runtime.env email:public -- status
   

4. Repeat the bounded re-encryption pass as needed. It handles at most 100 payloads and skips active leases. Before removing an old key from runtime configuration, confirm that keys reports no remaining payloads under that ID and that all producers use the new key. A pass returning zero alone is insufficient if leased payloads remain.

Keep any keys required to read retained backups in your protected backup records. Removing a key from current runtime configuration does not resolve backup retention.

Authentication secret in public mode

Changing AUTH_SECRET requires an explicit local invalidation step. It signs out all users, invalidates verification/reset links and cancels pending invitations.

1. Stop this installation's web and worker processes and confirm that they have stopped. Email leases last 30 seconds; allow that interval after the last dispatcher stops. In your database client, bind APP_INSTALLATION_ID as $1 and check:

   SELECT count(*) AS active_leases, max(lease_until) AS latest_expiry
   FROM email_outbox
   WHERE installation_id = $1 AND lease_until > now();
   

   Continue only when active_leases is 0. Keep the processes stopped during the change; do not clear lease values to bypass this check.

2. Set the new independent AUTH_SECRET in every consumer's configuration.

3. Run the command with the new configuration:

   npm run with-env -- /ABS/runtime.env email:public -- rotate-auth
   

4. Expect rotated: true; false means the stored fingerprint already matches. Restart the web and worker with the same configuration. Check status, confirm old sessions and links are rejected, and verify a new sign-in flow.

The operation abandons pending identity email, clears sessions and verification records, and cancels pending invitations. It preserves email budgets and business-message retry identity. Do not use it just to rotate the Resend API key.

Provider and database credentials

Replace RESEND_API_KEY, webhook secrets, Stripe credentials or database credentials through the respective provider and every affected runtime consumer. Preserve pending request identities and retained encryption keys. Where a provider offers overlapping credentials, choose the cutover sequence supported by that provider; the kit has no universal rotation command.

For Stripe settings, run the supplied preflight after updating configuration:

npm run with-env -- /ABS/runtime.env billing:recover -- preflight

This contacts Stripe to validate the catalog and webhook endpoint, then activates compatible settings in the local database. It is not a read-only status command. Follow it with reconciliation and a check of the affected workflow. For database changes, run the doctor with --database before restarting normal work.

Maintain and recover the installation

Before an application update, keep the acquired sources, dependency lockfile, applied migration checksums and the runtime settings needed by that version. Add higher-numbered migrations for schema changes; do not edit migrations already applied. Use the verification workflow in customization, then the deployment procedure in hosting.

This version supplies no backup, database restore or rollback command. Arrange backup retention and restore procedures with your PostgreSQL provider or your own operating tools. A useful restore exercise needs a separate database, the matching application version and required secrets. Keep restored web, email and billing workers from running against real providers until you have checked the restored state and decided how to handle work already performed since the backup. Restoring SQL cannot undo a sent email or a Stripe operation.

Check the restored database with the doctor, then verify the application and the provider reconciliation required for your installation before reopening it.