jason/nyaa-crawler
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0429f1e263080bdf8633bd5c4feea379a584673d
nyaa-crawler/AGENTS.md
T
jason ee85151d76
Build and Push Docker Image / build (push) Has been cancelled
Details
Update AGENTS.md
2026-06-28 23:36:53 -05:00

10 KiB
Raw Blame History

AGENTS.md

Mission

Build a small, dockerized web service that lets the user:

  • Search and select anime releases from Nyaa.si.
  • Persist a personal “watch list” of shows and their release patterns.
  • Poll Nyaa (via RSS or lightweight scraping / API wrapper) for new episodes.
  • Automatically download the next .torrent file for each tracked show into a host-mounted download directory.
  • Track which episodes are:
    • Automatically downloaded (auto-checked),
    • Manually checked as already downloaded by the user.

Target deployment is an Unraid server using a single Docker container with a simple web UI and a lightweight persistence layer (SQLite preferred).[^1]

High-level Architecture

  • Frontend: Minimal web UI (SPA or server-rendered) for:
    • Searching Nyaa.si.
    • Adding/removing shows from the watch list.
    • Viewing episodes per show with status (pending, downloaded).
    • Manually checking episodes as downloaded.
  • Backend:
    • HTTP API for the UI.
    • Nyaa integration (RSS and/or search scraping).
    • Scheduler/worker to periodically poll Nyaa and enqueue downloads.
    • Torrent fetcher that downloads .torrent files to a host-mounted directory.
  • Data store:
    • SQLite database stored on a bind-mounted volume for easy backup and migration.
  • Containerization:
    • Single Docker image with app + scheduler.
    • Config via environment variables.
    • Unraid-friendly: configurable ports, volume mapping for DB and torrents.[^2][^1]

Functional Requirements

1. Nyaa Integration

  • Use Nyaa’s RSS endpoints for polling where possible (e.g. https://nyaa.si/?page=rss plus query parameters), falling back to HTML scraping or an existing wrapper library if necessary.[^3][^4][^5][^6][^7]
  • Support user-driven search:
    • Input: search term (e.g. “Jujutsu Kaisen 1080p SubsPlease”).
    • Output: recent matching torrents with:
      • Title
      • Torrent ID
      • Category
      • Size
      • Magnet/torrent link URL if exposed in the feed or page.[^8][^9][^10]
  • When a user “adds” an anime:
    • Store a normalized pattern to match future episodes (e.g. base title + quality/resolution + sub group).
    • Maintain reference to the Nyaa search or RSS query that defines this feed.[^6][^3]

2. Watch List & Episodes

  • Entities:
    • Show: id, display name, search/RSS query, quality filter, fansub group, active flag.
    • Episode: id, show_id, episode_code (string or parsed integer), nyaa_torrent_id, title, status (pending, downloaded_auto, downloaded_manual, failed), torrent_url, download_error, file_path, attempts, created_at, downloaded_at.
  • Behavior:
    • Adding a show:
      • Run an immediate search and ingest existing feed items, then auto-download from the oldest episode upward (batch releases excluded).
    • Pausing a show:
      • Toggle the active flag off — episodes are preserved, polling stops.
    • Deleting a show:
      • Permanently remove the show and cascade-delete its episodes. .torrent files already written to disk are left in place.
    • Manual check:
      • User can mark an episode as already downloaded (downloaded_manual), no torrent action taken; or reset a downloaded/failed episode back to pending.

3. Auto-Download Logic

  • Periodic job (configurable interval, default 15 min). A re-entrancy guard prevents overlapping runs (a slow poll or a manual "Poll Now" won't double up):
    • For each active show:
      • Query Nyaa using its stored RSS/search parameters.[^4][^3][^6]
      • Skip batch torrents via title heuristics (titles containing "Batch", "Complete", "Vol.", or an episode range).
      • Process every feed item not already downloaded (oldest episode first), plus any previously-failed episodes — including ones that have scrolled off the current feed, retried from their stored torrent_url.
      • For each candidate:
        • Insert a pending Episode record if new.
        • Download the .torrent file (NOT the media itself) into the mapped host directory as <show-slug>-ep<episode>-<torrent-id>.torrent.
        • On success, set status downloaded_auto and record the file path; on failure, set status failed, store the error, and increment attempts.
      • Stop retrying an episode once it reaches the attempt cap (5) so a permanently-dead torrent isn't re-attempted forever.
  • Do not attempt to control or integrate directly with a torrent client (scope is "download the .torrent file" only).

4. Web UI

  • Views:
    • Shows list:
      • Add show via a Nyaa search-first modal; clicking a result pre-fills the form.
      • Pause/resume polling per show.
      • Quick link to show detail.
    • Show detail:
      • Stats summary (total / pending / auto / manual / failed) and a failed-downloads panel with error details.
      • Table of episodes: episode number/title, Nyaa ID, status, timestamps.
      • Controls:
        • Manually mark individual episodes as downloaded, or reset them to pending (with confirmation).
        • Bulk “mark up to episode N as downloaded” helper.
        • "Poll Now" to trigger an immediate check; Pause/Resume; Delete (with confirmation).
    • Settings:
      • Poll interval.
      • Default quality / sub group preferences.
      • Torrent download directory (read-only display; actual path comes from environment/volume).
    • A health-warning banner appears on every page when the torrent output directory isn't writable.
  • UX constraints:
    • Keep it extremely simple; focus is internal tool.
    • Assume a single user instance behind LAN.

Non-Functional Requirements

  • Language/Stack:
    • Prefer Node.js + TypeScript backend with a minimal React or server-rendered frontend to align with existing projects, unless you choose a simpler stack.
  • Security:
    • App is assumed to run behind LAN; basic auth or reverse-proxy auth can be added later.
    • Do not expose any admin-only functionality without at least a simple auth hook.
  • Resilience:
    • Polling should be robust to Nyaa timeouts and 4xx/5xx responses (retry with backoff, log errors).
    • Do not spam Nyaa with aggressive polling; default interval should be conservative (e.g. 15 minutes, configurable).
  • Observability:
    • Minimal logging for:
      • Polling attempts.
      • New episodes found.
      • Torrent downloads started/completed or failed.

Data Model (Initial)

Tables

  • shows
    • id (PK)
    • name (string)
    • search_query (string)
    • quality (string, nullable)
    • sub_group (string, nullable)
    • rss_url (string, nullable)
    • is_active (boolean, default true)
    • created_at, updated_at
  • episodes
    • id (PK)
    • show_id (FK → shows.id, ON DELETE CASCADE)
    • episode_code (string, e.g. “S01E03” or “03”)
    • title (string)
    • torrent_id (string, Nyaa ID)
    • torrent_url (string)
    • status (enum: pending, downloaded_auto, downloaded_manual, failed)
    • download_error (string, nullable — last failure message)
    • file_path (string, nullable — saved .torrent path)
    • attempts (integer, default 0 — failed download count, capped at 5)
    • downloaded_at (datetime, nullable)
    • created_at, updated_at
    • UNIQUE(show_id, torrent_id)

Container & Unraid Integration

Environment

  • PORT – HTTP port to listen on (default 3000).
  • POLL_INTERVAL_SECONDS – Polling frequency.
  • TORRENT_OUTPUT_DIR – Inside-container path where .torrent files are written.
  • DATABASE_PATH – Inside-container path to SQLite file.

Volumes

  • Map SQLite DB to persistent storage:
    • /data/db.sqlite → Unraid share: e.g. /mnt/user/appdata/nyaa-watcher/db.sqlite.[^1][^2]
  • Map torrent output directory to a download share:
    • /data/torrents → e.g. /mnt/user/downloads/torrents/nyaa/.

Ports

  • Expose app port to LAN (bridge mode):
    • Container: 3000, Host: YOUR_PORT (e.g. 8082).

Example docker-compose snippet

services:
  nyaa-watcher:
    image: your-registry/nyaa-watcher:latest
    container_name: nyaa-watcher
    restart: unless-stopped
    environment:
      - PORT=3000
      - POLL_INTERVAL_SECONDS=900
      - TORRENT_OUTPUT_DIR=/data/torrents
      - DATABASE_PATH=/data/db.sqlite
    volumes:
      - /mnt/user/appdata/nyaa-watcher:/data
      - /mnt/user/downloads/torrents/nyaa:/data/torrents
    ports:
      - "8082:3000"

This can be translated to an Unraid template or used via docker-compose with a Docker context pointing at the Unraid host.[^11][^2][^1]

Implementation Roadmap

  1. Skeleton app
    • Set up HTTP server, health endpoint, and a static web page.
    • Wire SQLite with migrations for shows and episodes.
  2. Nyaa client
    • Implement RSS-based polling for a hard-coded query.
    • Parse feed, extract torrent IDs, titles, and links.[^5][^3][^4][^6]
    • Optionally evaluate an existing node nyaa-si wrapper as a shortcut.[^7]
  3. Watch list CRUD
    • API endpoints + UI for managing shows.
    • Initial search → show add flow.
  4. Episode tracking
    • When adding a show, ingest existing feed items into episodes as pending.
    • Implement manual check/mark endpoints and UI.
  5. Auto-download worker
    • Background job to poll active shows and write .torrent files.
    • Update episode status to downloaded_auto.
  6. Dockerization & Unraid deployment
    • Dockerfile, volume mappings, environment configuration.
    • Test deployment on Unraid, ensure persistence and torrent file visibility.
  7. Polish
    • Basic auth or IP allowlist if desired.
    • Guardrails against batch torrent downloads.
    • Minimal styling for the UI.

Open Questions for Product Owner

  • What poll interval do you consider acceptable by default (e.g. 5, 10, or 15 minutes)?
  • Do you want any basic auth in front of the UI out of the box, or will this live behind an existing reverse proxy?
Reference in New Issue View Git Blame Copy Permalink