nyaa-crawler/AGENTS.md

# AGENTS.md

## Mission

Build a small, dockerized web service that lets the user:

- Search and select anime releases from Nyaa.si.
- Persist a personal “watch list” of shows and their release patterns.
- Poll Nyaa (via RSS or lightweight scraping / API wrapper) for new episodes.
- Automatically download the next .torrent file for each tracked show into a host-mounted download directory.
- Track which episodes are:
    - Automatically downloaded (auto-checked),
    - Manually checked as already downloaded by the user.

Target deployment is an Unraid server using a single Docker container with a simple web UI and a lightweight persistence layer (SQLite preferred).[^1]

***

## High-level Architecture

- **Frontend**: Minimal web UI (SPA or server-rendered) for:
    - Searching Nyaa.si.
    - Adding/removing shows from the watch list.
    - Viewing episodes per show with status (pending, downloaded).
    - Manually checking episodes as downloaded.
- **Backend**:
    - HTTP API for the UI.
    - Nyaa integration (RSS and/or search scraping).
    - Scheduler/worker to periodically poll Nyaa and enqueue downloads.
    - Torrent fetcher that downloads `.torrent` files to a host-mounted directory.
- **Data store**:
    - SQLite database stored on a bind-mounted volume for easy backup and migration.
- **Containerization**:
    - Single Docker image with app + scheduler.
    - Config via environment variables.
    - Unraid-friendly: configurable ports, volume mapping for DB and torrents.[^2][^1]

***

## Functional Requirements

### 1. Nyaa Integration

- Use Nyaa’s RSS endpoints for polling where possible (e.g. `https://nyaa.si/?page=rss` plus query parameters), falling back to HTML scraping or an existing wrapper library if necessary.[^3][^4][^5][^6][^7]
- Support user-driven search:
    - Input: search term (e.g. “Jujutsu Kaisen 1080p SubsPlease”).
    - Output: recent matching torrents with:
        - Title
        - Torrent ID
        - Category
        - Size
        - Magnet/torrent link URL if exposed in the feed or page.[^8][^9][^10]
- When a user “adds” an anime:
    - Store a normalized pattern to match future episodes (e.g. base title + quality/resolution + sub group).
    - Maintain reference to the Nyaa search or RSS query that defines this feed.[^6][^3]


### 2. Watch List \& Episodes

- Entities:
    - **Show**: id, display name, search/RSS query, quality filter, fansub group, active flag.
    - **Episode**: id, show_id, episode_code (string or parsed integer), nyaa_torrent_id, title, status (`pending`, `downloaded_auto`, `downloaded_manual`, `failed`), torrent_url, download_error, file_path, attempts, created_at, downloaded_at.
- Behavior:
    - Adding a show:
        - Run an immediate search and ingest existing feed items, then auto-download from the oldest episode upward (batch releases excluded).
    - Pausing a show:
        - Toggle the active flag off — episodes are preserved, polling stops.
    - Deleting a show:
        - Permanently remove the show and cascade-delete its episodes. `.torrent` files already written to disk are left in place.
    - Manual check:
        - User can mark an episode as already downloaded (`downloaded_manual`), no torrent action taken; or reset a downloaded/failed episode back to `pending`.


### 3. Auto-Download Logic

- Periodic job (configurable interval, default 15 min). A re-entrancy guard
  prevents overlapping runs (a slow poll or a manual "Poll Now" won't double up):
    - For each active show:
        - Query Nyaa using its stored RSS/search parameters.[^4][^3][^6]
        - Skip batch torrents via title heuristics (titles containing "Batch",
          "Complete", "Vol.", or an episode range).
        - Process every feed item not already downloaded (oldest episode first),
          plus any previously-`failed` episodes — including ones that have scrolled
          off the current feed, retried from their stored `torrent_url`.
        - For each candidate:
            - Insert a `pending` Episode record if new.
            - Download the `.torrent` file (NOT the media itself) into the mapped
              host directory as `<show-slug>-ep<episode>-<torrent-id>.torrent`.
            - On success, set status `downloaded_auto` and record the file path; on
              failure, set status `failed`, store the error, and increment `attempts`.
        - Stop retrying an episode once it reaches the attempt cap (5) so a
          permanently-dead torrent isn't re-attempted forever.
- Do not attempt to control or integrate directly with a torrent client (scope is "download the .torrent file" only).


### 4. Web UI

- Views:
    - **Shows list**:
        - Add show via a Nyaa search-first modal; clicking a result pre-fills the form.
        - Pause/resume polling per show.
        - Quick link to show detail.
    - **Show detail**:
        - Stats summary (total / pending / auto / manual / failed) and a failed-downloads panel with error details.
        - Table of episodes: episode number/title, Nyaa ID, status, timestamps.
        - Controls:
            - Manually mark individual episodes as downloaded, or reset them to pending (with confirmation).
            - Bulk “mark up to episode N as downloaded” helper.
            - "Poll Now" to trigger an immediate check; Pause/Resume; Delete (with confirmation).
    - **Settings**:
        - Poll interval.
        - Default quality / sub group preferences.
        - Torrent download directory (read-only display; actual path comes from environment/volume).
    - A health-warning banner appears on every page when the torrent output directory isn't writable.
- UX constraints:
    - Keep it extremely simple; focus is internal tool.
    - Assume a single user instance behind LAN.

***

## Non-Functional Requirements

- **Language/Stack**:
    - Prefer Node.js + TypeScript backend with a minimal React or server-rendered frontend to align with existing projects, unless you choose a simpler stack.
- **Security**:
    - App is assumed to run behind LAN; basic auth or reverse-proxy auth can be added later.
    - Do not expose any admin-only functionality without at least a simple auth hook.
- **Resilience**:
    - Polling should be robust to Nyaa timeouts and 4xx/5xx responses (retry with backoff, log errors).
    - Do not spam Nyaa with aggressive polling; default interval should be conservative (e.g. 15 minutes, configurable).
- **Observability**:
    - Minimal logging for:
        - Polling attempts.
        - New episodes found.
        - Torrent downloads started/completed or failed.

***

## Data Model (Initial)

### Tables

- `shows`
    - `id` (PK)
    - `name` (string)
    - `search_query` (string)
    - `quality` (string, nullable)
    - `sub_group` (string, nullable)
    - `rss_url` (string, nullable)
    - `is_active` (boolean, default true)
    - `created_at`, `updated_at`
- `episodes`
    - `id` (PK)
    - `show_id` (FK → shows.id, `ON DELETE CASCADE`)
    - `episode_code` (string, e.g. “S01E03” or “03”)
    - `title` (string)
    - `torrent_id` (string, Nyaa ID)
    - `torrent_url` (string)
    - `status` (enum: `pending`, `downloaded_auto`, `downloaded_manual`, `failed`)
    - `download_error` (string, nullable — last failure message)
    - `file_path` (string, nullable — saved `.torrent` path)
    - `attempts` (integer, default 0 — failed download count, capped at 5)
    - `downloaded_at` (datetime, nullable)
    - `created_at`, `updated_at`
    - `UNIQUE(show_id, torrent_id)`

***

## Container \& Unraid Integration

### Environment

- `PORT` – HTTP port to listen on (default 3000).
- `POLL_INTERVAL_SECONDS` – Polling frequency.
- `TORRENT_OUTPUT_DIR` – Inside-container path where `.torrent` files are written.
- `DATABASE_PATH` – Inside-container path to SQLite file.


### Volumes

- Map SQLite DB to persistent storage:
    - `/data/db.sqlite` → Unraid share: e.g. `/mnt/user/appdata/nyaa-watcher/db.sqlite`.[^1][^2]
- Map torrent output directory to a download share:
    - `/data/torrents` → e.g. `/mnt/user/downloads/torrents/nyaa/`.


### Ports

- Expose app port to LAN (bridge mode):
    - Container: `3000`, Host: `YOUR_PORT` (e.g. 8082).


### Example docker-compose snippet

```yaml
services:
  nyaa-watcher:
    image: your-registry/nyaa-watcher:latest
    container_name: nyaa-watcher
    restart: unless-stopped
    environment:
      - PORT=3000
      - POLL_INTERVAL_SECONDS=900
      - TORRENT_OUTPUT_DIR=/data/torrents
      - DATABASE_PATH=/data/db.sqlite
    volumes:
      - /mnt/user/appdata/nyaa-watcher:/data
      - /mnt/user/downloads/torrents/nyaa:/data/torrents
    ports:
      - "8082:3000"
```

This can be translated to an Unraid template or used via docker-compose with a Docker context pointing at the Unraid host.[^11][^2][^1]

***

## Implementation Roadmap

1. **Skeleton app**
    - Set up HTTP server, health endpoint, and a static web page.
    - Wire SQLite with migrations for `shows` and `episodes`.
2. **Nyaa client**
    - Implement RSS-based polling for a hard-coded query.
    - Parse feed, extract torrent IDs, titles, and links.[^5][^3][^4][^6]
    - Optionally evaluate an existing node `nyaa-si` wrapper as a shortcut.[^7]
3. **Watch list CRUD**
    - API endpoints + UI for managing shows.
    - Initial search → show add flow.
4. **Episode tracking**
    - When adding a show, ingest existing feed items into `episodes` as `pending`.
    - Implement manual check/mark endpoints and UI.
5. **Auto-download worker**
    - Background job to poll active shows and write `.torrent` files.
    - Update episode status to `downloaded_auto`.
6. **Dockerization \& Unraid deployment**
    - Dockerfile, volume mappings, environment configuration.
    - Test deployment on Unraid, ensure persistence and torrent file visibility.
7. **Polish**
    - Basic auth or IP allowlist if desired.
    - Guardrails against batch torrent downloads.
    - Minimal styling for the UI.

***

## Open Questions for Product Owner

- What poll interval do you consider acceptable by default (e.g. 5, 10, or 15 minutes)?
- Do you want any basic auth in front of the UI out of the box, or will this live behind an existing reverse proxy?