forgerunner/docs/troubleshooting.md

# Troubleshooting

Symptom-driven guide for common runner failures.

---

## Runner not appearing in Gitea admin panel

**Check:** Container logs for registration errors.

```bash
docker logs gitea-runner 2>&1 | grep -E "BOOTSTRAP|FATAL|ERROR"
```

**Common causes:**

| Cause | Fix |
|---|---|
| `GITEA_RUNNER_TOKEN` is invalid or expired | Generate a fresh token in Gitea admin → Runners |
| `GITEA_INSTANCE_URL` wrong or unreachable | Verify URL is reachable from inside the container: `docker exec gitea-runner curl -I "$GITEA_INSTANCE_URL"` |
| TLS cert error (self-signed Gitea) | Mount your CA cert to `/config/trusted-ca/` and restart |
| Runner name already registered | Use a unique `GITEA_RUNNER_NAME` or deregister the old one |

---

## `node: not found` in job logs

The runner image bundles Node.js 20 LTS. If you see this error it usually means:

- The job is running inside a container image that does not have Node.js (e.g. `alpine`).
- The `ubuntu-latest` label is mapped to a minimal image.

**Fix:** Ensure the label mapping uses a full Node image:

```
ubuntu-latest:docker://node:20-bullseye
```

Or use a host-mode label:

```
ubuntu-latest:host
```

---

## Docker socket permission error

**Symptom:**

```
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
```

**Causes and fixes:**

| Cause | Fix |
|---|---|
| Socket not mounted | Add `-v /var/run/docker.sock:/var/run/docker.sock` |
| Socket GID mismatch | Container runs as root in v1 — this should not occur. If using a non-root custom image, add the runner user to the `docker` group or set `--group-add $(stat -c '%g' /var/run/docker.sock)` |

---

## Workflow checkout failures

**Symptom:** `actions/checkout` fails with an auth or host error.

**Check:**
1. Gitea URL is accessible from inside the container.
2. If using self-signed TLS, the CA cert is trusted inside the container.

```bash
docker exec gitea-runner curl -v https://git.example.com
```

If you see certificate errors, add your CA cert to `/config/trusted-ca/` and restart.

---

## Registry login failures (`docker/login-action`)

**Symptom:** Login to `registry.alwisp.com` fails.

**Check:**
1. Registry URL is correct.
2. Credentials are valid.
3. If using self-signed TLS on the registry, the CA cert needs to be trusted inside the container AND the Docker daemon config must also trust the registry.

For Docker daemon CA trust on the Unraid host:

```bash
mkdir -p /etc/docker/certs.d/registry.alwisp.com
cp my-ca.crt /etc/docker/certs.d/registry.alwisp.com/ca.crt
```

Then restart the Docker daemon and the runner container.

---

## Container job image issues

**Symptom:** Jobs that run inside container images fail to pull or execute.

**Check:**
- The container image referenced in the label mapping is pullable from inside the runner container.
- If using a self-hosted registry image, the runner container can authenticate to it.

---

## CA / certificate trust issues

**Symptom:** TLS errors when connecting to Gitea or the registry.

**Fix:**
1. Copy your CA cert (`.crt` format, PEM encoded) to `config/trusted-ca/`.
2. Restart the container — the entrypoint installs certs from that directory on startup.
3. Verify: `docker exec gitea-runner curl -v https://git.example.com` should show cert chain OK.

---

## Unraid-specific: container exits immediately

**Check:** Container logs before exit:

```bash
docker logs gitea-runner
```

Look for `[FATAL]` lines. Common cause is a missing required env var (`GITEA_INSTANCE_URL`, `GITEA_RUNNER_TOKEN`, `GITEA_RUNNER_NAME`) when no config file exists.

**Fix:** Set required env vars in the Unraid container template.

---

## Startup validation failures

**Symptom:** Container exits with `[FATAL] Missing required binaries`.

This is an image build problem — the binary should always be present if the image was built correctly.

**Fix:** Rebuild the image from the `docker/runner/Dockerfile` and verify:

```bash
./scripts/verify-runtime.sh custom-gitea-runner:latest
```

---

## Orphan images piling up on the host (disk filling)

**Symptom:** The Unraid Docker page (or `docker images`) shows a growing list of
`(orphan image)` entries — both tagged (`registry.alwisp.com/<owner>/<repo>:latest`)
and untagged (bare image IDs) — that nothing created intentionally.

**Cause:** Jobs that run on the `host` label use the **host Docker daemon** via the
mounted socket. Every `docker build` leaves its output image on the host, and each
rebuild orphans the previous one (the moved tag leaves the old image dangling).
Multi-stage build layers add more. Nothing removes them automatically, so they
accumulate and consume the array.

**Fix:** Have Docker-build workflows clean up after pushing to the registry:

```yaml
      - name: Clean up build images on host
        if: always()
        run: |
          IMAGE="registry.alwisp.com/${{ gitea.repository }}"
          docker image rm -f "${IMAGE}:latest" 2>/dev/null || true
          docker image prune -f 2>/dev/null || true   # removes dangling layers (host-wide)
```

The image lives in the registry after `docker push`, so removing the local copy is
safe. `docker image prune -f` only removes **dangling** (untagged) images, not
tagged base images, so rebuilds stay fast. Note it acts host-wide — fine on a
dedicated runner host; be aware if the host runs unrelated build tooling.

**One-time cleanup** of images left by earlier builds (run on the host):

```bash
docker images --format '{{.Repository}}:{{.Tag}}' \
  | grep '^registry.alwisp.com/' | xargs -r docker rmi -f
docker image prune -f
```