Abrova

Ship container logs with the agent

At the end: logs from every container on a host, searchable in Abrova Trace.
Abrova TraceDockerIntermediate · 20 min

Before you start

ContainersAbrova Trace agentLogs page

The Abrova Trace agent is one container that reads the logs of every other container on the host and sends them to Abrova Trace in batches. You change nothing in your apps. In the Console you then search all hosts in one feed, filter by level and container, and start, stop or restart containers.

Costs from your wallet. Container logs count toward your Abrova Trace logs: 1 GB a month is free, then 15,000 Toman per GB from your wallet. Step 4 shows how to send less.

Get the server secret#

The agent runs on your server, so give it the project's server secret (ab_srv_…), not the app key. It is in Abrova Console under Project settings → Keys and is shown once, when you create or rotate it. Keep it in your server's secret store or environment.

Run one agent on the host#

bash
docker run -d \
  --name abrova-trace-agent \
  --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v abrova-trace-agent-data:/var/lib/abrova-trace-agent \
  -e ABROVA_TRACE_API_KEY=ab_srv_your_server_secret \
  ghcr.io/abrova-co/abrova-trace-agent:latest
  • The Docker socket lets the agent find containers and read their logs.
  • The abrova-trace-agent-data volume keeps its disk buffer and read positions when the container is re-created, so no lines are lost or sent twice.
  • With --restart=always it comes back after a reboot.

If you run your services with Docker Compose, add the agent to the same file and keep the key in .env, which you don't commit:

yaml
# docker-compose.yml
services:
  abrova-trace-agent:
    image: ghcr.io/abrova-co/abrova-trace-agent:latest
    container_name: abrova-trace-agent
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - abrova-trace-agent-data:/var/lib/abrova-trace-agent
    environment:
      - ABROVA_TRACE_API_KEY=${ABROVA_TRACE_API_KEY} # ab_srv_… from .env

volumes:
  abrova-trace-agent-data:
If the server can't reach ghcr.io. The image is pulled from GitHub's container registry. If your server can't reach it, pull the image on a machine that can and move it: docker save ghcr.io/abrova-co/abrova-trace-agent:latest -o agent.tar, copy the file, then docker load -i agent.tar on the server.

See the logs in the Console#

Within a few seconds the agent registers every running and stopped container. Open Abrova Console → Trace → Logs:

  • The Sources panel lists the host and its containers.
  • New lines appear in the feed within about 5 seconds: the agent sends batches of up to 100 entries, or every 5 seconds, whichever comes first.
  • Each entry has a level, even for plain text logs: [ERROR], level=warn, JSON level fields and words like exception or failed are read for you, so the level filters work without code changes.
  • Stack traces and other multi-line output arrive as one entry, and tokens, passwords, API keys, e-mail addresses and phone numbers are masked before anything leaves the host.

To check the agent itself, read its own output on the server:

bash
docker logs abrova-trace-agent

Search, saved searches and the query syntax are on Log search.

Choose what is sent#

Most of the log volume usually comes from a few noisy containers and from health checks. Trim them with environment variables on the agent:

VariableWhat it does
ABROVA_TRACE_EXCLUDEContainer names to skip, comma-separated. Default: abrova-trace-agent. A name matches when it is part of the container name.
ABROVA_TRACE_INCLUDEOnly these containers, comma-separated (empty = all).
ABROVA_TRACE_LABEL_FILTEROnly containers with this label, for example abrova-trace.project=shop. Takes priority over include and exclude.
ABROVA_TRACE_HEALTH_SAMPLEKeep 1 in N successful health-check lines (default 20; 1 = all, 0 = none).
ABROVA_TRACE_DROPRegular expressions separated by ||; matching entries are not sent.

For example, to skip a health-check sidecar and a debug container, and drop static-file access lines:

bash
docker rm -f abrova-trace-agent

docker run -d \
  --name abrova-trace-agent \
  --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v abrova-trace-agent-data:/var/lib/abrova-trace-agent \
  -e ABROVA_TRACE_API_KEY=ab_srv_your_server_secret \
  -e ABROVA_TRACE_EXCLUDE=abrova-trace-agent,healthcheck,debug-shell \
  -e 'ABROVA_TRACE_DROP=GET /static/||GET /favicon.ico' \
  ghcr.io/abrova-co/abrova-trace-agent:latest
Keep the agent in the exclude list. Setting ABROVA_TRACE_EXCLUDE replaces the default, so list abrova-trace-agent again, or the agent ships its own logs.

The first GB of logs each month is free; what you send above it is paid from your wallet. See Pricing.

Require the server secret#

Once the agent runs with the server secret, turn on Server routes require the server secret under Project settings → Products → Abrova Trace. Then the app key, which ships inside your apps, can no longer connect as an agent or register containers. Apps keep sending errors and logs with the app key as before. See Server secret & server routes.

Add more hosts#

With several servers, run the agent in agent mode on each extra host. It needs no key; it streams its containers to one collector over a WebSocket:

bash
# On each extra server
docker run -d \
  --name abrova-trace-agent \
  --restart=always \
  -p 7008:7008 \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -e ABROVA_TRACE_HOSTNAME=server-2 \
  ghcr.io/abrova-co/abrova-trace-agent:latest agent

On the main server, tell the collector where they are, as a comma-separated name:host:port list:

bash
-e ABROVA_TRACE_REMOTE_AGENTS=server2:YOUR_SERVER2_HOST:7008

Let only the collector's address reach port 7008, for example ufw allow from COLLECTOR_IP to any port 7008. Details: Abrova Trace agent.

Troubleshooting#

The agent container keeps restarting

Read docker logs abrova-trace-agent. The usual causes are a wrong key, a server that can't reach trace.abrova.ir, or the Docker socket not mounted.

No logs appear

Check that your containers are not in ABROVA_TRACE_EXCLUDE (names match on any part) and that they match ABROVA_TRACE_INCLUDE or ABROVA_TRACE_LABEL_FILTER if you set them. Logs are batched: wait up to 5 seconds.

Logs arrive, but containers are missing or can't be controlled

Server routes require the server secret is on and the agent runs with the app key. Log lines still arrive, but registration, status and commands are refused. Redeploy the agent with the server secret (ab_srv_…).

Containers started before the agent are not listed

The agent registers containers when it starts. Restart it: docker restart abrova-trace-agent.

A line I need is missing

Check ABROVA_TRACE_DROP (matching entries are not sent) and ABROVA_TRACE_HEALTH_SAMPLE (only 1 in 20 successful health-check lines is kept by default; set it to 1 to keep all).

Logs were lost while the network was down

When Abrova Trace can't be reached, the agent writes batches to its data folder and sends them in order later, up to 50 MB (ABROVA_TRACE_SPOOL_MB). Mount the abrova-trace-agent-data volume so that buffer survives a re-created container.

Pitfalls
  • Logs above 1 GB a month cost from your wallet.
  • Setting ABROVA_TRACE_EXCLUDE replaces the default: keep abrova-trace-agent in the list.

Where next: Logs overview