forgerunner/forgerunner.src/AGENT.md

# AGENT.md

## Project

**Name:** custom-gitea-runner

**Purpose:**
Build a self-owned, versioned, Docker-first Gitea Actions runner platform that avoids opaque upstream behavior, is easy to debug, and is safe to operate on Unraid and general Linux hosts. This project should produce a maintainable runner image, startup logic, configuration templates, and documentation that make runner behavior explicit and reproducible.

This project exists because the stock/self-hosted runner path introduced hard-to-diagnose issues around nested containers, action runtime dependencies, container image behavior, and host/runtime compatibility. The replacement should favor observability, deterministic builds, explicit tooling, and low-surprise operations.

---

## Mission

Create a production-ready custom Gitea runner solution that:

- Runs reliably in Docker on Unraid and standard Linux hosts.
- Supports JavaScript-based actions by shipping the required runtime dependencies explicitly.
- Supports Docker build/push workflows without hidden image assumptions.
- Uses pinned versions wherever practical.
- Separates image build concerns from runtime registration/config concerns.
- Makes debugging first-class through logs, health checks, self-tests, and clear failure modes.
- Keeps all project knowledge versioned in-repo through AGENT.md, README.md, ROADMAP.md, CHANGELOG.md, and example configs.

---

## Product goals

1. Build a custom Docker image for a Gitea-compatible runner.
2. Provide an entrypoint/bootstrap script that prepares the runtime deterministically.
3. Support runner registration through environment variables and/or mounted config.
4. Support common CI workloads for the target environment: checkout, Node-based actions, Docker login, Docker build, Docker push, shell scripts, and future extensibility for additional tooling.
5. Provide a clean path for image tagging, publishing, rollback, and host deployment.
6. Minimize dependence on mutable upstream defaults like `latest`, floating action images, or runtime package installs during startup.
7. Make the image and scripts inspectable, testable, and easy to extend.

---

## Non-goals

- Do not build a full replacement for the Gitea server.
- Do not invent a proprietary CI protocol if standard Gitea runner registration is sufficient.
- Do not optimize first for Kubernetes.
- Do not assume multi-tenant hostile workloads at v1; this is a controlled self-hosted environment.
- Do not rely on runtime `apk add` or ad hoc container mutation as a permanent solution.

---

## Development philosophy

Treat this like an internal platform product, not a one-off container hack.

Principles:

- **Version everything.** Image versions, dependency versions, docs, sample configs, and release notes should all be explicit.
- **Document while building.** If behavior changes, update README.md and ROADMAP.md in the same workstream.
- **Prefer deterministic builds.** Install dependencies at image build time, not container startup.
- **Prefer explicit over magical.** If Node, Git, Docker CLI, certificates, or helper binaries are required, declare and verify them.
- **Design for diagnosis.** Every major startup step should emit useful logs.
- **Keep the bootstrap thin.** Complex logic should live in versioned scripts/modules, not giant one-line container commands.
- **Bias for maintainability.** Small, composable scripts beat clever entrypoints.

---

## User and environment assumptions

Primary operator profile:

- Advanced technical operator.
- Comfortable with Docker, Unraid, Node.js/TypeScript, Bash, and self-hosted infrastructure.
- Prefers documented roadmaps, structured project files, and reusable internal tooling.

Primary target environment:

- Unraid Docker host first.
- Standard Linux Docker host second.
- Gitea instance is self-hosted.
- Registry may be self-hosted Gitea Container Registry.
- Project repositories are hosted in Gitea.

---

## Required deliverables

The agent should scaffold and maintain at minimum:

```text
custom-gitea-runner/
├── AGENT.md
├── README.md
├── ROADMAP.md
├── CHANGELOG.md
├── LICENSE
├── .gitignore
├── .env.example
├── docker/
│   ├── runner/
│   │   ├── Dockerfile
│   │   ├── docker-entrypoint.sh
│   │   ├── healthcheck.sh
│   │   └── smoke-test.sh
│   └── compose/
│       ├── docker-compose.example.yml
│       └── unraid-template-notes.md
├── config/
│   ├── runner.example.yaml
│   ├── labels.example.env
│   └── trusted-ca/
├── scripts/
│   ├── register-runner.sh
│   ├── deregister-runner.sh
│   ├── verify-runtime.sh
│   ├── build-image.sh
│   ├── publish-image.sh
│   └── test-local-runner.sh
├── docs/
│   ├── architecture.md
│   ├── operations.md
│   ├── troubleshooting.md
│   ├── unraid.md
│   └── decisions/
│       └── 0001-base-image.md
├── test/
│   ├── fixtures/
│   ├── sample-workflows/
│   └── smoke/
└── src/
    └── optional/
```

Notes:

- `src/` is optional unless the project grows into a TypeScript control plane or helper CLI.
- If a CLI/helper app is introduced, prefer TypeScript for maintainability and consistency with the project stack.

---

## Recommended architecture

### Baseline architecture

Start with a Docker image that wraps or embeds the official `act_runner` binary but controls:

- base image selection,
- runtime package set,
- startup/registration flow,
- mounted config handling,
- Docker socket strategy,
- health checks,
- and diagnostics.

### Preferred v1 design

- Debian or Ubuntu slim base image, not an ultra-minimal image that requires runtime mutation.
- Install at build time:
  - ca-certificates
  - bash
  - curl
  - git
  - jq
  - nodejs
  - npm
  - docker CLI
  - tini
  - timezone data
- Copy in a pinned `act_runner` binary or install from a pinned release artifact.
- Use `tini` as PID 1.
- Use a real multi-line `docker-entrypoint.sh`.
- Validate runtime prerequisites before daemon start.
- Prefer mounted `/var/run/docker.sock` for v1 if the host is trusted and this matches the deployment model.
- Expose health check status through a lightweight script.

### Why this design

This directly addresses the observed failure class:

- runtime packages missing in the runner image,
- overreliance on upstream defaults,
- brittle nested-container behavior,
- and low observability when setup fails.

---

## Runner behavior requirements

The custom runner must:

1. Start deterministically with a documented set of environment variables.
2. Fail fast when required inputs are missing.
3. Verify and log the presence of key tools:
   - `git`
   - `node`
   - `npm`
   - `docker`
   - `act_runner`
4. Emit version information at startup.
5. Support both pre-registered config files and first-boot registration flows.
6. Avoid mutating the filesystem in surprising ways at runtime beyond known writable paths.
7. Support Docker-based build/push jobs cleanly.
8. Provide optional label mapping guidance for common targets like `ubuntu-latest`.
9. Make workspace and temp directories explicit and configurable.
10. Provide a smoke-test path that validates the runner image before production use.

---

## Configuration model

Support two modes.

### Mode A: Pre-generated config

Operator mounts:

- runner config yaml
- optional registration metadata
- optional CA bundle

This is the preferred stable production mode.

### Mode B: Bootstrap registration

Operator provides env vars such as:

- `GITEA_INSTANCE_URL`
- `GITEA_RUNNER_TOKEN`
- `GITEA_RUNNER_NAME`
- `GITEA_RUNNER_LABELS`
- `GITEA_RUNNER_WORKDIR`
- `GITEA_RUNNER_CONFIG_PATH`

Entrypoint performs:

1. validation,
2. one-time registration if config absent,
3. daemon startup.

The agent should design this so reboots are idempotent.

---

## Docker and security model

### v1 security posture

This runner is intended for a trusted, self-hosted environment. Simplicity and reliability are more important than extreme sandbox isolation in v1.

### v1 approach

- Support Docker socket mount for build/push workloads.
- Document the trust implications clearly.
- Keep the runner container itself as minimal and explicit as possible.
- Avoid privileged mode unless a specific feature requires it.
- Prefer read-only mounts where practical except for config/work directories.
- Run as non-root where feasible, but do not force complexity if Docker socket access or upstream tooling makes that unrealistic in v1.

### Future hardening track

Document as roadmap items:

- rootless variants,
- DinD variant,
- dedicated buildkit integration,
- ephemeral one-job runners,
- restricted labels by repo or org,
- network isolation strategies.

---

## Documentation requirements

The agent must keep these files current as the project evolves.

### README.md

Should include:

- project purpose,
- why this exists,
- supported features,
- architecture overview,
- quick start,
- environment variables,
- Docker compose example,
- Unraid deployment notes,
- sample workflow guidance,
- troubleshooting links,
- versioning and release model.

### ROADMAP.md

Must track phases, status, and next actions.

Recommended sections:

- Phase 0: Research and design
- Phase 1: Base image and bootstrap
- Phase 2: Registration/config flows
- Phase 3: Docker workflow support
- Phase 4: Test harness and smoke validation
- Phase 5: Unraid packaging and docs
- Phase 6: Hardening and enhancements

### CHANGELOG.md

Use Keep a Changelog style:

- Added
- Changed
- Fixed
- Removed

### docs/troubleshooting.md

Must be practical and symptom-driven.

Include sections for issues like:

- runner not registering,
- `node` not found,
- Docker socket permission errors,
- workflow checkout failures,
- registry login failures,
- container job image issues,
- CA/certificate trust issues,
- Unraid-specific gotchas.

---

## Functional scope for v1

### Must support

- `actions/checkout`
- `docker/login-action`
- `docker build`
- `docker push`
- shell-based build scripts
- Gitea registry auth patterns
- branch/tag based image builds

### Nice to support in v1 if straightforward

- `buildx`
- cache mounts
- custom CA installation
- repo/org label strategies
- self-test action workflow

### Defer unless easy

- autoscaling
- UI dashboard
- metrics exporter
- multi-runner orchestrator
- web control plane

---

## Testing strategy

The project should include real validation, not just build success.

### Required tests

1. **Image build test**
   - Docker image builds successfully from scratch.

2. **Runtime verification test**
   - Container starts and confirms key binaries exist.

3. **Registration smoke test**
   - Runner can register against a test Gitea instance or mocked flow.

4. **Workflow smoke tests**
   - checkout-only workflow,
   - Node-based action workflow,
   - Docker login workflow,
   - Docker build/push workflow against a test registry.

5. **Regression tests**
   - specifically capture previously observed failure classes around missing Node and fragile container behavior.

### Testing philosophy

Every bug that costs real debugging time should become:

- a documented issue,
- a troubleshooting entry,
- and ideally a regression test.

---

## Logging and observability

Startup logs should clearly print:

- image version,
- git commit if embedded,
- act_runner version,
- node version,
- docker version,
- registration mode,
- config path,
- work directory,
- labels,
- target Gitea URL.

Do not log secrets.

Health check should verify at minimum:

- runner process exists,
- config is present,
- Docker socket availability if required,
- core binaries exist.

Optional future enhancements:

- structured JSON logs,
- `/healthz` sidecar or helper,
- metrics export.

---

## Versioning and release policy

Use semantic-ish tags for the custom image.

Recommended tagging:

- `v0.1.0`
- `v0.1.1`
- `v0.2.0`
- `latest` only as a convenience alias to the newest stable release

Rules:

- Never deploy unpinned `latest` in production docs as the primary recommendation.
- Every release should map to a Git tag and CHANGELOG entry.
- Embed image metadata labels such as version, vcs ref, build date, and source URL.

---

## Agent operating instructions

When working on this project, the agent must:

1. Update `ROADMAP.md` when a phase changes or a major task is added/closed.
2. Update `README.md` when usage, env vars, image behavior, or deployment steps change.
3. Update `CHANGELOG.md` for user-visible changes.
4. Prefer small, reviewable commits/changesets.
5. Ask for clarification before making irreversible architecture choices when multiple valid paths exist.
6. Keep scripts POSIX/Bash-friendly and easy to audit.
7. Avoid hidden runtime package installation unless explicitly implementing a temporary debug path.
8. Prefer pinned versions in Dockerfile and examples.
9. Add troubleshooting notes whenever a new failure mode is discovered.
10. Preserve portability between Unraid and standard Linux where reasonable.

---

## Implementation preferences

Preferred languages and tooling:

- Bash for bootstrap and operational scripts.
- Dockerfile for image definition.
- TypeScript only if a helper CLI/control plane becomes necessary.
- Markdown for all project documentation.
- Git-based versioning hosted in Gitea.

Preferred project style:

- Modular files.
- Clean naming.
- Self-explanatory scripts.
- No giant inline command strings in container templates.
- Examples should be copy-paste friendly.

---

## Suggested first milestones

### Milestone 1: Foundation

- Initialize repository structure.
- Write README.md skeleton.
- Write ROADMAP.md.
- Write Dockerfile using pinned base image.
- Write entrypoint with startup validation.
- Add healthcheck script.

### Milestone 2: Registration and config

- Add env-driven bootstrap registration flow.
- Add mounted-config mode.
- Add `.env.example`.
- Add example runner config.

### Milestone 3: CI workload support

- Ensure Node, Git, Docker CLI, CA certs all present.
- Validate `actions/checkout` and `docker/login-action` support.
- Build sample workflow fixtures.

### Milestone 4: Unraid packaging

- Create compose example.
- Document Unraid template fields and recommended mounts.
- Add upgrade and rollback instructions.

### Milestone 5: Quality and regression safety

- Add smoke-test script.
- Add local test harness.
- Add troubleshooting guide.
- Capture prior bug classes as regression cases.

---

## Decision log expectations

Create ADR-style notes under `docs/decisions/` for major decisions such as:

- base image choice,
- root vs non-root default,
- socket mount vs DinD,
- bootstrap registration design,
- bundled tools list,
- release strategy.

Each decision note should include:

- context,
- decision,
- consequences,
- alternatives considered.

---

## Success criteria

This project is successful when:

- a fresh operator can build and run the image with documented steps,
- the runner reliably executes checkout + Docker login + build + push workflows,
- Node-based actions work without runtime hacks,
- failures are diagnosable from logs and docs,
- upgrades are explicit and reversible,
- and the entire runner behavior is version-controlled and understandable.

---

## Open questions for early clarification

Before heavy implementation, clarify:

1. Should v1 wrap the official `act_runner` binary or fork/build from source?
2. Is the preferred base image Debian slim or Ubuntu minimal?
3. Should v1 default to socket mount only, or support both socket mount and DinD variants from day one?
4. Should the repository include an Unraid XML/template artifact, or just docs plus compose examples?
5. Should the first release include a small helper CLI for registration/testing, or keep everything shell-only?
6. Should runner labels be opinionated by default, or minimal and operator-supplied?
7. Should custom CA trust be a first-class feature in v1?

If these are unanswered, the agent should pause and ask before overcommitting to a direction.

---

## Build order

When starting implementation, follow this order:

1. AGENT.md
2. README.md
3. ROADMAP.md
4. Dockerfile
5. entrypoint/bootstrap scripts
6. healthcheck/smoke tests
7. config examples
8. compose/unraid examples
9. troubleshooting docs
10. regression fixtures/tests

---

## Final instruction

Optimize for reliability, debuggability, and maintainability over cleverness.

This project should feel like a stable internal platform component that the operator can trust, inspect, extend, and version for years.