nyaa-crawler/AGENTS.md

# AGENTS.md

## Mission

Build a small, dockerized web service that lets a user:

- Search and select anime releases from Nyaa.si.
- Persist a personal “watch list” of shows and their release patterns.
- Poll Nyaa (via RSS or lightweight scraping / API wrapper) for new episodes.
- Automatically download the next .torrent file for each tracked show into a host-mounted download directory.
- Track which episodes are:
    - Automatically downloaded (auto-checked),
    - Manually checked as already downloaded by the user.

Target deployment is an Unraid server using a single Docker container with a simple web UI and a lightweight persistence layer (SQLite preferred).[^1]

***

## High-level Architecture

- **Frontend**: Minimal web UI (SPA or server-rendered) for:
    - Searching Nyaa.si.
    - Adding/removing shows from the watch list.
    - Viewing episodes per show with status (pending, downloaded).
    - Manually checking episodes as downloaded.
- **Backend**:
    - HTTP API for the UI.
    - Nyaa integration (RSS and/or search scraping).
    - Scheduler/worker to periodically poll Nyaa and enqueue downloads.
    - Torrent fetcher that downloads `.torrent` files to a host-mounted directory.
- **Data store**:
    - SQLite database stored on a bind-mounted volume for easy backup and migration.
- **Containerization**:
    - Single Docker image with app + scheduler.
    - Config via environment variables.
    - Unraid-friendly: configurable ports, volume mapping for DB and torrents.[^2][^1]

***

## Functional Requirements

### 1. Nyaa Integration

- Use Nyaa’s RSS endpoints for polling where possible (e.g. `https://nyaa.si/?page=rss` plus query parameters), falling back to HTML scraping or an existing wrapper library if necessary.[^3][^4][^5][^6][^7]
- Support user-driven search:
    - Input: search term (e.g. “Jujutsu Kaisen 1080p SubsPlease”).
    - Output: recent matching torrents with:
        - Title
        - Torrent ID
        - Category
        - Size
        - Magnet/torrent link URL if exposed in the feed or page.[^8][^9][^10]
- When a user “adds” an anime:
    - Store a normalized pattern to match future episodes (e.g. base title + quality/resolution + sub group).
    - Maintain reference to the Nyaa search or RSS query that defines this feed.[^6][^3]


### 2. Watch List \& Episodes

- Entities:
    - **Show**: id, display name, search/RSS query, quality filter, fansub group, active flag.
    - **Episode**: id, show_id, episode_number (string or parsed integer), nyaa_torrent_id, title, status (`pending`, `downloaded_auto`, `downloaded_manual`), torrent_url, created_at, downloaded_at.
- Behavior:
    - Adding a show:
        - Run an immediate search.
        - Populate existing episodes in DB as `pending` (no download) to let the user backfill by manually checking already downloaded ones.
    - Removing a show:
        - Leave episodes in DB but mark show as inactive (no further polling).
    - Manual check:
        - User can mark an episode as already downloaded (`downloaded_manual`), no torrent action taken.


### 3. Auto-Download Logic

- Periodic job (e.g. every 5–15 minutes, configurable):
    - For each active show:
        - Query Nyaa using its stored RSS/search parameters.[^4][^3][^6]
        - Determine the “next” episode:
            - Prefer simplest rule: highest episode number not yet marked downloaded.
            - Guard against batch torrents by using size or title pattern heuristics (e.g. skip titles containing “Batch”).
        - If the next episode’s torrent is not yet in DB:
            - Create an Episode record with status `downloaded_auto`.
            - Download the `.torrent` file (NOT the media itself) into the mapped host directory.
                - Filename suggestion: `<show-slug>-ep<episode>-<torrent-id>.torrent`.
- Do not attempt to control or integrate directly with a torrent client (scope is “download the .torrent file” only).


### 4. Web UI

- Views:
    - **Shows list**:
        - Add show (form: name, search query, quality, group).
        - Toggle active/inactive.
        - Quick link to show detail.
    - **Show detail**:
        - Table of episodes: episode number/title, Nyaa ID, status, timestamps.
        - Controls:
            - Manually mark individual episodes as downloaded.
            - Bulk “mark previous episodes as downloaded” helper (e.g. “mark up to episode N”).
    - **Settings**:
        - Poll interval.
        - Default quality / sub group preferences.
        - Torrent download directory (read-only display; actual path comes from environment/volume).
- UX constraints:
    - Keep it extremely simple; focus is internal tool.
    - Assume a single user instance behind LAN.

***

## Non-Functional Requirements

- **Language/Stack**:
    - Prefer Node.js + TypeScript backend with a minimal React or server-rendered frontend to align with existing projects, unless you choose a simpler stack.
- **Security**:
    - App is assumed to run behind LAN; basic auth or reverse-proxy auth can be added later.
    - Do not expose any admin-only functionality without at least a simple auth hook.
- **Resilience**:
    - Polling should be robust to Nyaa timeouts and 4xx/5xx responses (retry with backoff, log errors).
    - Do not spam Nyaa with aggressive polling; default interval should be conservative (e.g. 15 minutes, configurable).
- **Observability**:
    - Minimal logging for:
        - Polling attempts.
        - New episodes found.
        - Torrent downloads started/completed or failed.

***

## Data Model (Initial)

### Tables

- `shows`
    - `id` (PK)
    - `name` (string)
    - `search_query` (string)
    - `quality` (string, nullable)
    - `sub_group` (string, nullable)
    - `rss_url` (string, nullable)
    - `is_active` (boolean, default true)
    - `created_at`, `updated_at`
- `episodes`
    - `id` (PK)
    - `show_id` (FK → shows.id)
    - `episode_code` (string, e.g. “S01E03” or “03”)
    - `title` (string)
    - `torrent_id` (string, Nyaa ID)
    - `torrent_url` (string)
    - `status` (enum: `pending`, `downloaded_auto`, `downloaded_manual`)
    - `downloaded_at` (datetime, nullable)
    - `created_at`, `updated_at`

***

## Container \& Unraid Integration

### Environment

- `PORT` – HTTP port to listen on (default 3000).
- `POLL_INTERVAL_SECONDS` – Polling frequency.
- `TORRENT_OUTPUT_DIR` – Inside-container path where `.torrent` files are written.
- `DATABASE_PATH` – Inside-container path to SQLite file.


### Volumes

- Map SQLite DB to persistent storage:
    - `/data/db.sqlite` → Unraid share: e.g. `/mnt/user/appdata/nyaa-watcher/db.sqlite`.[^1][^2]
- Map torrent output directory to a download share:
    - `/data/torrents` → e.g. `/mnt/user/downloads/torrents/nyaa/`.


### Ports

- Expose app port to LAN (bridge mode):
    - Container: `3000`, Host: `YOUR_PORT` (e.g. 8082).


### Example docker-compose snippet

```yaml
services:
  nyaa-watcher:
    image: your-registry/nyaa-watcher:latest
    container_name: nyaa-watcher
    restart: unless-stopped
    environment:
      - PORT=3000
      - POLL_INTERVAL_SECONDS=900
      - TORRENT_OUTPUT_DIR=/data/torrents
      - DATABASE_PATH=/data/db.sqlite
    volumes:
      - /mnt/user/appdata/nyaa-watcher:/data
      - /mnt/user/downloads/torrents/nyaa:/data/torrents
    ports:
      - "8082:3000"
```

This can be translated to an Unraid template or used via docker-compose with a Docker context pointing at the Unraid host.[^11][^2][^1]

***

## Implementation Roadmap

1. **Skeleton app**
    - Set up HTTP server, health endpoint, and a static web page.
    - Wire SQLite with migrations for `shows` and `episodes`.
2. **Nyaa client**
    - Implement RSS-based polling for a hard-coded query.
    - Parse feed, extract torrent IDs, titles, and links.[^5][^3][^4][^6]
    - Optionally evaluate an existing node `nyaa-si` wrapper as a shortcut.[^7]
3. **Watch list CRUD**
    - API endpoints + UI for managing shows.
    - Initial search → show add flow.
4. **Episode tracking**
    - When adding a show, ingest existing feed items into `episodes` as `pending`.
    - Implement manual check/mark endpoints and UI.
5. **Auto-download worker**
    - Background job to poll active shows and write `.torrent` files.
    - Update episode status to `downloaded_auto`.
6. **Dockerization \& Unraid deployment**
    - Dockerfile, volume mappings, environment configuration.
    - Test deployment on Unraid, ensure persistence and torrent file visibility.
7. **Polish**
    - Basic auth or IP allowlist if desired.
    - Guardrails against batch torrent downloads.
    - Minimal styling for the UI.

***

## Open Questions for Product Owner

- What poll interval do you consider acceptable by default (e.g. 5, 10, or 15 minutes)?
- Do you want any basic auth in front of the UI out of the box, or will this live behind an existing reverse proxy?