Abrova
☰ Docs menu · Abrova Trace agent (Docker)

Abrova Trace Agent (Docker)

The Abrova Trace agent streams logs from every Docker container on your servers, with container management, live stats, multi-host support, secret masking and automatic log level detection. Deploy it with one command.

DockerAgent 0.2.1linux/amd64 + arm64Last updated: 27 Oct 2026

The Abrova Trace agent streams logs from all Docker containers on your server, with real-time container management, multi-host support, and automatic log level detection. Deploy it with a single docker run command.

What the agent gives you.
  • Live Docker logs in Abrova Console — every container, every host, one feed
  • Container management from the Console: start, stop, restart, remove
  • Live CPU / memory / network stats per container
  • Multi-host: one collector aggregates remote agents on additional servers
  • Automatic log level detection, multi-line joining (stack traces become one entry) and secret masking before anything leaves the host
Costs from your wallet.

Container logs count toward your Abrova Trace logs: 1 GB a month is free, then 15,000 Toman per GB from your wallet. See Pricing.

How it works#

The agent uses a collector + remote agent architecture:

  1. The collector runs on your main server. It watches local containers, connects to remote agents, and sends batched logs to Abrova Trace over HTTPS.
  2. Remote agents run on additional servers. They stream their container logs to the collector over a WebSocket.
  3. All containers (running and stopped) are registered in your project, so you can see and manage them.
  4. The connection works both ways, so container commands (start, stop, restart, remove, stats) reach any host.

With a single server you only need the collector; that is the default mode.

ModeCommandWhat it does
Default (collector)(none)Watches local Docker and connects to remote agents; sends logs to Abrova Trace. Needs the key
AgentagentWebSocket server that streams this host's container logs to a collector. No key needed
Collector onlycollectorConnects to remote agents only, without watching local Docker. Needs the key

Deploy on a single server#

bash
docker run -d \
  --name abrova-trace-agent \
  --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v abrova-trace-agent-data:/var/lib/abrova-trace-agent \
  -e ABROVA_TRACE_API_KEY=ab_srv_your_server_secret \
  ghcr.io/abrova-co/abrova-trace-agent:latest
Where the image comes from.

The agent image is pulled from ghcr.io (GitHub's container registry), so the server needs a connection that can reach it. If it cannot, pull the image on a machine that can, then move it with docker save ghcr.io/abrova-co/abrova-trace-agent:latest -o agent.tar and docker load -i agent.tar.

The volume keeps the agent's disk buffer and read positions when the container is re-created, so no lines are lost or sent twice.

Which key?

Give the agent your project's server secret (ab_srv_...) in ABROVA_TRACE_API_KEY. The app key (ab_live_...) also works as long as the project setting Server routes require the server secret is off, but it ships inside your apps, so it should not be what runs your agent. See Server secret & server routes.

The agent registers every running and stopped container in your project within a few seconds. Open Abrova Trace → Logs in Abrova Console to watch the logs stream in.

Multi-host setup#

For multiple servers, deploy a remote agent on each additional server and point a single collector at them.

Remote agent (on each additional server)

bash
docker run -d \
  --name abrova-trace-agent \
  --restart=always \
  -p 7008:7008 \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -e ABROVA_TRACE_HOSTNAME=server-2 \
  ghcr.io/abrova-co/abrova-trace-agent:latest agent

Collector (on the main server)

bash
docker run -d \
  --name abrova-trace-agent \
  --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v abrova-trace-agent-data:/var/lib/abrova-trace-agent \
  -e ABROVA_TRACE_API_KEY=ab_srv_your_server_secret \
  -e ABROVA_TRACE_REMOTE_AGENTS=server2:YOUR_SERVER2_HOST:7008,server3:YOUR_SERVER3_HOST:7008 \
  ghcr.io/abrova-co/abrova-trace-agent:latest

The ABROVA_TRACE_REMOTE_AGENTS value is a comma-separated list of name:host:port. Each remote agent streams logs to the collector over a WebSocket and reconnects by itself after a disconnect (every 5 seconds).

Deploy from the desktop app.

The Abrova desktop app (coming soon) will deploy the agent to your servers over SSH for you: add a server, trust its host key, deploy. Until then, use the docker run commands on this page.

Manage containers from the Console#

In Abrova Console, open Abrova Trace → Logs. The Sources panel lists every host and its containers; the menu next to a container has:

  • View stats — a live dialog with CPU %, memory used / limit and memory %, and network RX / TX (bytes received and sent across all network interfaces)
  • Start — start a stopped container
  • Restart — restart in place
  • Stop — gracefully stop a running container
  • Delete — stop and remove the container

Commands travel through Abrova Trace to the agent's WebSocket, so they work the same whether the container is on your main server or on a remote-agent host.

Start, stop, restart and delete need the admin or owner role; every role can see containers, logs and stats. See Roles & Permissions.

Configuration reference#

Collector environment variables

OptionTypeRequiredDescription
ABROVA_TRACE_API_KEYstringYesYour project server secret (ab_srv_..., recommended). The app key (ab_live_...) works only while the "Server routes require the server secret" setting is off
ABROVA_TRACE_URLstringNoAbrova Trace API URL (default: https://trace.abrova.ir)
ABROVA_TRACE_EXCLUDEstringNoContainer names to exclude, comma-separated (default: abrova-trace-agent)
ABROVA_TRACE_INCLUDEstringNoContainer names to include, comma-separated (empty = all)
ABROVA_TRACE_LABEL_FILTERstringNoOnly watch containers with this label (for example abrova-trace.project=abc123); takes priority over include/exclude
ABROVA_TRACE_REMOTE_AGENTSstringNoRemote agents in name:host:port format, comma-separated
ABROVA_TRACE_HOSTNAMEstringNoName this host is shown under (default: the Docker host name)

Remote agent environment variables

OptionTypeRequiredDescription
ABROVA_TRACE_AGENT_PORTnumberNoWebSocket port for the agent (default: 7008)
ABROVA_TRACE_HOSTNAMEstringNoName this host is shown under (default: the system host name)
ABROVA_TRACE_EXCLUDEstringNoContainer names to exclude, comma-separated (default: abrova-trace-agent)
ABROVA_TRACE_INCLUDEstringNoContainer names to include, comma-separated (empty = all)
ABROVA_TRACE_LABEL_FILTERstringNoOnly watch containers with this label

Log processing (all modes)

OptionTypeRequiredDescription
ABROVA_TRACE_MULTILINEbooleanNoJoin stack traces and multi-line output into one entry (default: true)
ABROVA_TRACE_REDACTbooleanNoMask secrets: tokens, passwords, API keys, credentials in URLs (default: true)
ABROVA_TRACE_REDACT_PIIbooleanNoMask e-mail addresses and phone numbers (default: true)
ABROVA_TRACE_HEALTH_SAMPLEnumberNoKeep 1 in N successful health-check lines; 1 = all, 0 = none (default: 20)
ABROVA_TRACE_DROPstringNoRegular expressions separated by ||; matching entries are not sent
ABROVA_TRACE_DATA_DIRstringNoDisk buffer and read positions (default: /var/lib/abrova-trace-agent; mount a volume to keep them)
ABROVA_TRACE_SPOOL_MBnumberNoDisk buffer cap in MB; the oldest unsent batches are dropped beyond it (default: 50)
ABROVA_TRACE_CATCHUP_SECONDSnumberNoAfter a restart, resume from the last line read if it is this recent (default: 900)

Log level detection#

The agent sets a level on every entry so the level filters in the Console work without code changes:

  • Explicit markers win — [ERROR], ERROR:, level=warn, [debug] and similar at the start of a line use the level written there. critical and panic count as fatal, notice as info, verbose as trace.
  • JSON logs — a level, severity, lvl or loglevel field is used, including the numeric levels of pino and bunyan.
  • NestJS lines ([Nest] PID - TIMESTAMP LEVEL ...) use the level NestJS printed.
  • Keywords otherwise — fatal or panic → fatal; error, exception, failed, unhandled, uncaught, ECONNREFUSED, ETIMEDOUT → error; warn, warning, deprecated → warn; debug, dbg → debug.
  • info — the default when nothing matches, so info: 0 errors found stays info.

What an entry carries#

Each log entry is one event, not one line. Besides the message and level, it carries extra fields in metadata:

FieldMeaning
fingerprint / templateThe message with ids, numbers and dates replaced by placeholders, and a hash of it, so identical problems group together
attributesFields parsed from JSON logs, key=value pairs and HTTP access lines (method, path, status, durationMs)
traceIdA request or trace id found in the entry, to follow one request across containers
linesNumber of raw lines joined (stack traces, printed objects)
kind: container_eventA Docker lifecycle event (start, exit code, out-of-memory kill, restart, health check) written into the same timeline
sampledThis health-check line stands for N lines
hostnameThe Docker host

Batching and delivery#

  • Batched — logs are sent in batches of up to 100 entries, or every 5 seconds, whichever comes first.
  • Retry on failure — failed batches are re-queued and retried, with up to 5,000 pending entries per source in memory (the oldest are dropped beyond that).
  • Disk buffer — if Abrova Trace cannot be reached, batches are written to ABROVA_TRACE_DATA_DIR and re-sent in order when it is back (up to ABROVA_TRACE_SPOOL_MB).
  • Resume after restart — the agent remembers the last line it read per container and continues from there.
  • Graceful shutdown — docker stop abrova-trace-agent flushes pending batches before exiting.

Key features#

Auto-discovery

New containers are detected and watched as they start

Container registration

All containers (running and stopped) are registered in your project on startup

Status reporting

Container start, stop, die and kill events are reported as they happen

Multi-host support

Monitor containers across several servers from one place

Auto-reconnect

Remote agent connections reconnect by themselves after a disconnect (5 s delay)

Secret masking

Tokens, passwords, API keys, e-mail addresses and phone numbers are masked before anything leaves the host

Security model#

Docker socket access

The agent mounts /var/run/docker.sock. The :ro flag only protects the socket file, not the Docker API, so treat the agent like any container with Docker access. It acts on containers only when a start / stop / restart / remove / stats command arrives from Abrova Trace, which authenticates it with your project credential.

Run the agent with the server secret

Use the server secret (ab_srv_...) for the agent and switch on "Server routes require the server secret" in Project settings → Products → Abrova Trace. The app key ships inside your apps; with the setting on, it can no longer be used to connect as an agent or to register containers.

Keep the key out of files

Pass the key with an environment variable or Docker secrets; never hard-code it in a compose file you commit.

Role-based command authorization

Container commands are made by a signed-in project owner or admin, pass through Abrova Trace, then reach the agent over its WebSocket. The agent never accepts commands from anywhere else.

Remote agent port (7008)

Should only be reachable from your collector. Configure your firewall so that only the collector's address can connect.

Troubleshooting#

Agent container keeps restarting

Check the agent logs: docker logs abrova-trace-agent. Common causes: an invalid key, an unreachable API URL, or the Docker socket not mounted.

Logs arrive, but containers are missing or can't be controlled

If the project setting Server routes require the server secret is on and the agent runs with the app key, log lines still arrive but the agent cannot register containers, report their status or receive commands. Redeploy the agent with ABROVA_TRACE_API_KEY set to the server secret (ab_srv_...). See Server secret & server routes.

Containers not showing in the Console

The agent registers containers on startup. If you added the agent after containers were running and they still do not show, restart the agent: docker restart abrova-trace-agent.

Stats dialog shows zeros or doesn't update

Stats are read from the local Docker daemon of the agent that owns the container. A host that only runs in agent mode streams logs to its collector; if stats stay empty for its containers, check that the collector is connected to it (see the next item).

Remote agent not connecting

Make sure port 7008 is reachable from the collector, for example ufw allow from COLLECTOR_IP to any port 7008. Double-check the ABROVA_TRACE_REMOTE_AGENTS format: name:host:port, comma-separated.

No logs appearing

Check that your containers are not in the exclude list (ABROVA_TRACE_EXCLUDE) and match ABROVA_TRACE_INCLUDE / ABROVA_TRACE_LABEL_FILTER if you set them. Verify the key is correct and the agent can reach the API URL. Logs are batched — wait up to 5 seconds for them to appear.

A line I need is missing

Check ABROVA_TRACE_DROP (matching entries are not sent) and ABROVA_TRACE_HEALTH_SAMPLE (only 1 in 20 successful health-check lines is kept by default; set it to 1 to keep all).

Was this page helpful?
Report a problem