Jason Stedwell
33 lines
2.8 KiB
Markdown
33 lines
2.8 KiB
Markdown
# UI Stock Tracker — Design Notes & Development History
|
|
|
|
Supplementary notes that go beyond the [README](./README.md): the reasoning behind the trickier parts of the scraper and the notable fixes made during development. For setup, architecture, API, and schema, see the README; for deployment see [UNRAID.md](./UNRAID.md).
|
|
|
|
## Stock Detection Rationale
|
|
|
|
The scraper classifies each product page as `in_stock`, `sold_out`, or `unknown` using case- and whitespace-insensitive substring matching against the page's buttons. Two non-obvious details drove the matching rules:
|
|
|
|
**Auth-state difference.** The scraper runs logged out. On a sold-out item, Ubiquiti shows `"Login for Notifications"` to logged-out visitors instead of the logged-in `"Notify me when available"`. Both strings must be in the sold-out list, or logged-out checks would fall through to `unknown`.
|
|
|
|
**The `label` attribute exists only on the in-stock button.** The production DOM carries `label="Add to Cart"` on the primary Add-to-Cart button (confirmed in DevTools), but the sold-out "Login for Notifications" button has no `label` attribute. So in-stock has two redundant signals (the attribute *and* the button text) while sold-out relies on text alone — this asymmetry is by design, not an oversight.
|
|
|
|
When neither signal matches, the scraper logs a `[Scraper] UNKNOWN debug` payload (page title, button labels, button texts) so a misclassification can be diagnosed from container logs without reproducing locally.
|
|
|
|
To avoid reading the page before React finishes rendering, the scraper waits 2.5s plus an active wait for a `<button>` to appear (up to 8s) before evaluating.
|
|
|
|
## Fixes Applied During Development
|
|
|
|
### 1. `npm ci` → `npm install` (Dockerfile)
|
|
`npm ci` requires a committed `package-lock.json`. Lockfiles were never committed, so the build failed immediately. Both the frontend and backend build stages use `npm install` instead. (This is why no lockfiles are present in the repo.)
|
|
|
|
### 2. TypeScript DOM lib missing (backend)
|
|
`scraper.ts` uses `page.evaluate()`, whose callback runs in the browser context. TypeScript flagged `document`, `navigator`, and `HTMLMetaElement` as unknown because the backend `tsconfig.json` only included `"lib": ["ES2020"]`. Fixed by adding the DOM libs:
|
|
|
|
```json
|
|
"lib": ["ES2020", "DOM", "DOM.Iterable"]
|
|
```
|
|
|
|
`DOM.Iterable` was specifically required to allow `for...of` iteration over `NodeListOf<HTMLSpanElement>`.
|
|
|
|
### 3. Two containers → one container
|
|
The original design used separate `frontend` and `backend` Docker services in docker-compose. These were consolidated into a single container running nginx and Node side by side under **supervisord**, with nginx proxying `/api/` to `localhost:3001`. Removed `backend/Dockerfile`, `frontend/Dockerfile`, and `frontend/nginx.conf`; the root-level `Dockerfile`, `nginx.conf`, and `supervisord.conf` replaced them.
|