mirror of
https://github.com/mistergibson/radio.git
synced 2026-11-03 19:31:41 -08:00
bug fixes
This commit is contained in:
parent
5ee5f19255
commit
369910f561
13 changed files with 662 additions and 1533 deletions
451
README.md
451
README.md
|
|
@ -1,311 +1,294 @@
|
||||||
[](https://firstdonoharm.dev/version/3/0/full.html)
|
[](https://firstdonoharm.dev/version/3/0/full.html)
|
||||||
|
# JRuby Radio Automation
|
||||||
|
|
||||||
# Radio Automation Stack
|
Self-hosted internet radio automation: **JRuby** scripts drive a **Liquidsoap 2.2.4** stream into **Icecast**, on a schedule written in cron syntax. One script talks to podcast feeds and SQLite, one script is the clock, one Liquidsoap config plays what the clock queues. The Python implementation that used to live alongside this one has been removed; this stack is JRuby-only.
|
||||||
|
|
||||||
A self-hosted internet radio automation system for Linux Mint 22.3 (Ubuntu 24.04 Noble). It continuously plays background music, interrupts it with scheduled podcast shows and external streams, downloads new podcast episodes from RSS feeds (or stores them as live-stream references), and manages playback history — all fed to an Icecast server via liquidsoap.
|
This README documents the **current architecture**: a single queue file (`current_queue.txt`) read by a single `request.dynamic` in Liquidsoap, with live-stream slots captured locally by ffmpeg before air.
|
||||||
|
|
||||||
Two complete, interchangeable implementations are provided:
|
---
|
||||||
|
|
||||||
- **Python** — `install_for_python`, `fetch_podcasts.py`, `update_playlists.py`
|
## Architecture
|
||||||
- **Ruby (JRuby)** — `install_for_jruby`, `fetch_podcasts.rb`, `update_playlists.rb`
|
|
||||||
|
|
||||||
Both read the same `config.json`, `schedule.txt`, and directory layout, and produce identical output. Pick one stack and stick with it; they share state, so don't run both simultaneously.
|
|
||||||
|
|
||||||
## Architecture at a glance
|
|
||||||
|
|
||||||
```
|
```
|
||||||
┌──────────────┐
|
cron (every minute, liquidsoap user)
|
||||||
schedule.txt ─▶│ │
|
|
|
||||||
config.json ──▶│ liquidsoap ├──▶ Icecast /audio.mp3 ──▶ listeners
|
v
|
||||||
station.liq ──▶│ (<svc>.service)│
|
schedule.txt ---> scheduler.rb --fire markers (state/fire_markers/)
|
||||||
│ │
|
config.json ---/ |
|
||||||
playlists/*.txt ◀── update_playlists.{py,rb}
|
| writes ONE bare URI + TTL sidecar
|
||||||
▲ │
|
v
|
||||||
│ selects unplayed ▼
|
<storage>/queues/current_queue.txt <storage>/queues/current_queue.meta
|
||||||
podcasts/<slug>/ state/played.db (episode records + played flag)
|
|
|
||||||
▲
|
"show" -> episode local path or media URL (resolved from SQLite, marked played)
|
||||||
│ downloads archived episodes
|
"stream"-> LOCAL capture path: ffmpeg records the live URL into
|
||||||
fetch_podcasts.{py,rb} ──▶ state/subscriptions.db (show registry)
|
<storage>/captures/*.mp3 and that growing file is queued
|
||||||
▲
|
|
|
||||||
OPML file / gpodder.net sync
|
v
|
||||||
|
station.liq (liquidsoap, started by station.rb)
|
||||||
|
request.dynamic(prefetch=0) reads the queue file every tick;
|
||||||
|
a recurrent poller re-invokes fetch() only when the URI changes;
|
||||||
|
empty queue -> resolving silence file -> source goes to background
|
||||||
|
|
|
||||||
|
fallback(track_sensitive=false) --- blank() safety (mksafe)
|
||||||
|
| \
|
||||||
|
v -> rotating background music
|
||||||
|
output.icecast (music/jingles/announcements/
|
||||||
|
%mp3 128k podcasts, 0.5s crossfade)
|
||||||
|
|
|
||||||
|
v
|
||||||
|
Icecast (host:port/mount)
|
||||||
|
|
||||||
|
cron (hourly) ---> fetch_podcasts.rb --fetch-all
|
||||||
|
downloads new episodes of archived shows into
|
||||||
|
<storage>/podcasts/<slug>/ and records them in SQLite
|
||||||
```
|
```
|
||||||
|
|
||||||
The moving parts:
|
The key design decision: **liquidsoap never relays a live Icecast mount directly.** A scheduled `stream` slot is recorded by ffmpeg into a finite, growing MP3 in `captures/`, and that *local path* is queued. Liquidsoap opens the growing file and plays it as an open-ended stream; at the writer's end-of-file the audio simply ends and the queue expires normally. The re-invocation poller in `station.liq` exists precisely for this case: a growing capture never fires request-finished the way a finite file does, so a wall-clock thread re-invokes the source's `fetch()` when the queue file names a new URI, and the next scheduled item cuts in on time regardless.
|
||||||
|
|
||||||
- **liquidsoap** (`station.liq`) runs as a systemd service, streams MP3 to Icecast, and switches between continuous background music and scheduled content.
|
## Components
|
||||||
- **fetch_podcasts** (cron, hourly at :00) registers shows, pulls new episode metadata from RSS, and either downloads audio into `podcasts/<slug>/` (archived shows) or records only the enclosure URL for live streaming (non-archived shows).
|
|
||||||
- **update_playlists** (cron, hourly at :30) picks one unplayed episode per show from the database, writes an annotated URI line, and records it as played.
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `station.rb` | 3-line entry point: `cd`s to its own directory (which exports `RADIO_DIR` for the silence fallback) and `exec`s liquidsoap on `station.liq`. This is what the systemd unit runs. |
|
||||||
|
| `station.liq` | The Liquidsoap config: queue-reading `request.dynamic`, poller, fallback mux, background music, Icecast output. |
|
||||||
|
| `scheduler.rb` | The clock. Run from cron every minute; matches `schedule.txt` against the current time and writes the due URI into the queue file. Includes two dry-fire dev hooks. |
|
||||||
|
| `fetch_podcasts.rb` | Podcast ingestion: add/remove/archive shows, import OPML, download new episodes, sync with a gPodder server. |
|
||||||
|
| `update_playlists.rb` | **Legacy.** Writes per-show playlist files under `storage/queue/`. Superseded by `scheduler.rb`'s direct queue writes; kept for reference and its `--json` inventory report. Not in the air path. |
|
||||||
|
| `run_radio.sh` | Launcher for every JRuby script: sets the gem environment and JVM heap, then `exec`s jruby. |
|
||||||
|
| `console.sh` | Interactive `jirb` console with the same gem environment, running as the `liquidsoap` user. |
|
||||||
|
| `install_for_jruby` | One-shot installer: gems, databases, `config.json`, systemd unit, cron entries. |
|
||||||
|
| `config.json` | Storage root, Icecast connection, gPodder credentials. |
|
||||||
|
| `schedule.txt` | The schedule, in cron style. |
|
||||||
|
|
||||||
## Directory layout
|
## Directory layout
|
||||||
|
|
||||||
| Path | Purpose | Tracked in git? |
|
Everything lives under the storage root chosen at install time (default `/mnt/storage/radio`); the install directory (e.g. `/srv/radio`) holds the scripts, `config.json`, `schedule.txt`, and the co-located `.gems/` directory.
|
||||||
|------|---------|-----------------|
|
|
||||||
| `<storage>/music/` | Background music library (scanned recursively) | No |
|
|
||||||
| `<storage>/podcasts/` | Downloaded episodes, one subfolder per show slug (archived shows only) | No |
|
|
||||||
| `<storage>/jingles/` | Jingle audio | No |
|
|
||||||
| `<storage>/announcements/` | Announcement audio | No |
|
|
||||||
| `<storage>/playlists/` | Generated `.txt` queue files, one per show | No |
|
|
||||||
| `<storage>/state/` | SQLite databases (`subscriptions.db`, `played.db`) | No |
|
|
||||||
| `<storage>/logs/` | Rotated logs for each component | No |
|
|
||||||
| `config.json` | Credentials and connection settings | No (secret) |
|
|
||||||
| `schedule.txt` | Human-edited cron schedule of shows/streams | Yes |
|
|
||||||
| `station.liq` | liquidsoap configuration | Yes |
|
|
||||||
|
|
||||||
Everything that grows over time lives under a single **storage path** chosen at install time and recorded in `config.json`. The code itself stays in the install directory.
|
```
|
||||||
|
<storage>/
|
||||||
|
music/ background music (read recursively by playlist(dir))
|
||||||
|
podcasts/ downloaded episodes, one sub-directory per show slug
|
||||||
|
jingles/ station jingles
|
||||||
|
announcements/ PSA / announcement audio
|
||||||
|
state/
|
||||||
|
subscriptions.db shows table (what we subscribe to)
|
||||||
|
played.db episodes table (what exists and what has aired)
|
||||||
|
fire_markers/ scheduler idempotency markers (pruned after 1 day)
|
||||||
|
logs/
|
||||||
|
fetch_podcasts.log, scheduler_cron.log, fetch_cron.log, update_cron.log, ...
|
||||||
|
queues/
|
||||||
|
current_queue.txt THE queue: exactly one bare URI, nothing else
|
||||||
|
current_queue.meta THE sidecar: one epoch integer, the second the item stops airing
|
||||||
|
captures/
|
||||||
|
<host>_<YYYYMMDD_HHMMSS>.mp3 ffmpeg stream captures (pruned after 2 days)
|
||||||
|
*.fflog each capture's ffmpeg stderr
|
||||||
|
```
|
||||||
|
|
||||||
## Configuration: `config.json`
|
`queues/` and `captures/` are created by `scheduler.rb` on first run if absent; `fire_markers/` likewise.
|
||||||
|
|
||||||
Generated by the installer, edited by hand afterward. Contains three sections:
|
---
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
Run the installer as root from the directory that will become the service name (a directory named `radio` yields `radio.service`):
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo ./install_for_jruby
|
||||||
|
```
|
||||||
|
|
||||||
|
It detects JRuby (PATH or `/opt/jruby/bin/jruby`), prompts for a storage root and Icecast/gPodder settings, creates the media/state/log directories, drops and recreates the two SQLite databases with their full schemas, installs the gems `sequel`, `jdbc-sqlite3`, `json`, and `nokogiri` into the co-located `.gems/`, writes `config.json` (mode 600), generates and enables the systemd unit (`User=liquidsoap`, `ExecStart=<jruby> station.rb`, `Restart=on-failure`), and adds cron entries to the `liquidsoap` user's crontab for the hourly fetch and the legacy `update_playlists.rb`.
|
||||||
|
|
||||||
|
Two things the installer does not do, by design of the current architecture:
|
||||||
|
|
||||||
|
1. **The scheduler cron entry.** `scheduler.rb` must run every minute. Add it to the liquidsoap user's crontab:
|
||||||
|
```
|
||||||
|
* * * * * cd /srv/radio && ./run_radio.sh scheduler.rb >> /mnt/storage/radio/logs/scheduler_cron.log 2>&1
|
||||||
|
```
|
||||||
|
(adjust paths to your install and storage roots)
|
||||||
|
2. **`silence.mp3`.** The empty-queue fallback in `station.liq` resolves a real silent MP3 living beside the scripts (`<install dir>/silence.mp3`). Generate one, e.g.:
|
||||||
|
```
|
||||||
|
ffmpeg -f lavfi -i anullsrc=r=44100:cl=stereo -t 60 -codec:a libmp3lame -b:a 128k silence.mp3
|
||||||
|
```
|
||||||
|
The path is *not* hard-coded: `station.rb` exports `RADIO_DIR` (its own resolved directory) before exec'ing liquidsoap, and `station.liq` reads it with `environment.get`. The service works wherever it is installed. A WARNING is logged at startup if `RADIO_DIR` is unset.
|
||||||
|
|
||||||
|
Then review `config.json` and `systemctl start radio`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## config.json
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"storage": "/mnt/storage/radio",
|
"storage": "/mnt/storage/radio",
|
||||||
"icecast": {
|
"icecast": {
|
||||||
"host": "localhost",
|
"host": "192.168.0.200",
|
||||||
"port": 7777,
|
"port": 7777,
|
||||||
"mount": "/audio.mp3",
|
"mount": "/data",
|
||||||
"username": "source",
|
"source_username": "source",
|
||||||
"password": "SECRET"
|
"source_password": "..."
|
||||||
},
|
},
|
||||||
"gpodder": {
|
"gpodder": {
|
||||||
"enable": false,
|
"enabled": true,
|
||||||
"host": "https://gpodder.net",
|
"host": "gpodder.net",
|
||||||
"username": "",
|
"username": "...",
|
||||||
"password": ""
|
"password": "...",
|
||||||
|
"device_id": "radio-..."
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
The `storage` block is the base path for all media, state, and log directories. The `icecast` block holds the source credentials liquidsoap uses to push the stream (the *source* password from your `icecast.xml`, not the admin password). The `gpodder` block enables optional automatic subscription syncing from gpodder.net; when `enable` is `false`, the fetcher works purely from locally registered shows or imported OPML.
|
`fetch_podcasts.rb` treats gPodder sync as active only when `gpodder.enabled` is exactly `true` **and** `username`, `password`, and `device_id` are all non-empty. The OPML subscription list is read from `https://<host>/subscriptions/<username>/<device_id>.opml`.
|
||||||
|
|
||||||
Keep this file out of version control — it holds passwords. A `config.example.json` with placeholders can be committed instead if desired.
|
Note: the bundled installer writes this block with the key `"enable"` rather than `"enabled"`. The code reads `enabled`. If gPodder sync silently does nothing after an install, check which key your `config.json` actually has and set `"enabled": true`.
|
||||||
|
|
||||||
## The schedule: `schedule.txt`
|
|
||||||
|
|
||||||
Each non-comment line defines one recurring trigger. Fields are whitespace-separated:
|
|
||||||
|
|
||||||
```
|
|
||||||
min hour dom mon dow TYPE TARGET [RUNLENGTH_SECONDS]
|
|
||||||
```
|
|
||||||
|
|
||||||
- The first five fields are a standard cron expression.
|
|
||||||
- `TYPE` is either `show` or `stream`.
|
|
||||||
- `TARGET` is a show slug (for `show`) or a URL (for `stream`).
|
|
||||||
- `RUNLENGTH_SECONDS` is required for `stream` entries (how long to play before returning to music) and ignored for `show` entries, whose duration comes from the selected episode's stored runlength.
|
|
||||||
|
|
||||||
Example:
|
|
||||||
|
|
||||||
```
|
|
||||||
# Weekday morning news, Tuesday 08:00
|
|
||||||
0 8 * * 2 show hardcore_history
|
|
||||||
# Live remote stream, Monday 06:00 for one hour
|
|
||||||
0 6 * * 1 stream http://example.org:8000/live.mp3 3600
|
|
||||||
```
|
|
||||||
|
|
||||||
Lines beginning with `#` and blank lines are skipped. Malformed lines are logged and ignored rather than aborting the load.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Installation
|
## schedule.txt
|
||||||
|
|
||||||
There are two installers. Each is idempotent — safe to re-run after changes. Both must be run as root and derive the service name from the basename of their containing directory.
|
Whitespace-separated fields, cron style:
|
||||||
|
|
||||||
## Choosing a stack
|
```
|
||||||
|
# min hour dom mon dow TYPE TARGET [RUNLENGTH_SECONDS]
|
||||||
Pick based on what you want to maintain. The Python stack uses CPython libraries (`feedparser`, `requests`); the Ruby stack uses JRuby with the `jdbc-sqlite3` gem for database access. Functionally they are equivalent.
|
0 7 * * 1-5 stream https://streams.kpfa.org:8443/kpfa 3600
|
||||||
|
30 12 * * * show the_history_of_rome
|
||||||
## `install_for_python`
|
|
||||||
|
|
||||||
Creates the storage directory tree, generates `config.json`, writes the systemd unit, and sets up the cron jobs. Run it with:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo ./install_for_python
|
|
||||||
```
|
```
|
||||||
|
|
||||||
It walks through interactive prompts for the storage path, the Icecast host/port/mount/source username/password, and the gPodder sync settings, pre-filled from any existing `config.json` so re-runs preserve current values. It confirms a summary before making changes and exits cleanly if you decline.
|
Lines starting with `#` and blank lines are ignored. The five time fields support `*`, a single number `N`, a range `A-B`, a step `*/S`, and comma lists of any of those (`0,30`, `9-17/2`). `dow` is 0-6 with Sunday = 0.
|
||||||
|
|
||||||
## `install_for_jruby`
|
`TYPE` is `show` or `stream`.
|
||||||
|
|
||||||
The JRuby equivalent, with additional provisioning logic:
|
**`show`**: `TARGET` is a show slug (or feed URL). At fire time the scheduler looks the slug up in `subscriptions.db`, picks the most recent unplayed episode from `played.db`, marks it played, and queues its `local_path` (falling back to its `url`). `RUNLENGTH`, when given, is the number of seconds the item stays on air; without one the slot is capped at 4 hours so background music always resumes. An entry with no playable episode is *deferred*, not marked fired, and retries the next minute.
|
||||||
|
|
||||||
1. Installs base packages (liquidsoap, icecast2, jq, curl, unzip).
|
**`stream`**: `TARGET` is a live media URL. The scheduler starts ffmpeg recording it to `captures/<host>_<timestamp>.mp3` for `RUNLENGTH` seconds (default 3600), waits about 5 seconds, and queues the *local capture path* once the file has real bytes in it. The queue TTL is the capture length plus 300 seconds of slack. If ffmpeg produced nothing within 5 seconds, the queue is cleared instead, so background music carries the slot rather than a stale URI. This is how a finite-length slot of a continuous stream gets played: the capture is a growing MP3 that liquidsoap reads as an open-ended stream until the writer stops.
|
||||||
2. Detects Java; if a suitable JVM is already present it is reused, otherwise OpenJDK headless is installed.
|
|
||||||
3. Detects JRuby; only if none exists does it download a pinned release into `/opt`.
|
|
||||||
4. Integrates via `update-alternatives`, registering `/usr/local/bin/jruby` (plus `gem`, `bundle`, etc.) so the interpreter resolves consistently for cron, systemd, and shells alike.
|
|
||||||
5. **Installs gems one at a time** (`jdbc-sqlite3`, then `json`) with a raised JVM heap to avoid out-of-memory failures during multi-gem resolution on low-RAM hosts. Note: the native `sqlite3` gem is *not* used — it requires a C extension that won't build on JRuby. `jdbc-sqlite3` ships the SQLite JDBC driver as a JAR instead.
|
|
||||||
6. Creates directories, generates `config.json`, writes the systemd unit, and sets up cron — same as the Python installer.
|
|
||||||
|
|
||||||
Run it with:
|
**Expiry.** Every queued item is stamped with an absolute end time (epoch seconds) in `current_queue.meta`. Each scheduler tick clears the queue file once that time has passed, so a finished episode or capture never replays and background music resumes on its own. Worst-case overshoot is one cron minute.
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo ./install_for_jruby
|
|
||||||
```
|
|
||||||
|
|
||||||
If you later swap JRuby versions, `update-alternatives --config jruby` switches them without touching any generated config, cron, or service files.
|
|
||||||
|
|
||||||
### Installer options and environment
|
|
||||||
|
|
||||||
Neither installer takes command-line flags; configuration is collected interactively. Behavior is driven by:
|
|
||||||
|
|
||||||
- **Existing `config.json`** — provides defaults for all prompts.
|
|
||||||
- **Target directory** — determines the resulting service name.
|
|
||||||
- **System state** — for the JRuby installer, existing Java/JRuby installations are detected and reused rather than overwritten.
|
|
||||||
|
|
||||||
After either installer completes, follow the printed next steps: drop music into `music/`, edit `schedule.txt`, review `station.liq`, then start the services.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Tools
|
## The scripts and their options
|
||||||
|
|
||||||
## `fetch_podcasts` (`.py` / `.rb`)
|
All JRuby scripts are invoked through the launcher (see `run_radio.sh` below), e.g. `./run_radio.sh scheduler.rb --dryfire-show the_history_of_rome`.
|
||||||
|
|
||||||
Fetches podcast episodes, manages the show registry, and downloads audio. Runs automatically via cron every hour, but supports manual invocation for administration.
|
### scheduler.rb
|
||||||
|
|
||||||
Common prefix (adjust per stack):
|
Runs every minute from cron. On each tick, in order: expires the queue if its sidecar time has passed, then (unless a dry-fire hook was given) prunes old fire markers and captures, loads `schedule.txt`, and fires every entry matching the current minute. Idempotency: each entry fires at most once per calendar minute, tracked by a marker file named `<YYYYMMDDHHMM>_<entry-index>` in `state/fire_markers/`; markers older than one day are pruned.
|
||||||
|
|
||||||
```bash
|
Queue writes are atomic (temp file + rename), and the queue file is kept to exactly one bare URI with no trailing annotation, because `station.liq` hands the file's contents straight to `request.create`.
|
||||||
# Python
|
|
||||||
python3 fetch_podcasts.py [options]
|
|
||||||
# Ruby
|
|
||||||
jruby fetch_podcasts.rb [options]
|
|
||||||
```
|
|
||||||
|
|
||||||
### Options
|
Options:
|
||||||
|
|
||||||
| Option | Argument | Effect |
|
| Option | Effect |
|
||||||
|--------|----------|--------|
|
|---|---|
|
||||||
| *(none)* | — | Default mode. If gpodder sync is enabled, syncs subscriptions first, then fetches new episodes for all registered shows. |
|
| *(no arguments)* | Normal cron tick: expire check, then match `schedule.txt` against the current time and fire. |
|
||||||
| `--list-shows` | — | Prints a table of registered shows (slug, archived flag, source, name). |
|
| `--dryfire-stream <url> [seconds]` | Fire one stream capture immediately, bypassing the clock match. Records `<url>` with ffmpeg for `[seconds]` (default 3600) and queues the local capture exactly as a scheduled `stream` entry would. For testing the capture path without waiting for a scheduled minute. |
|
||||||
| `--detail` | — | With `--list-shows`, additionally prints each show's feed URL. |
|
| `--dryfire-show <slug>` | Run the exact `show` resolution path a scheduled entry uses (lookup, pick most recent unplayed episode, mark it played, queue it), immediately. Queues nothing (background continues) if the slug has no playable episode. |
|
||||||
| `--add-show` | `<feed-url>` | Fetches a single feed, registers it, and immediately processes its episodes. |
|
|
||||||
| `--delete-show` | `<slug>` | Removes a show from the registry, deletes its downloaded files, and removes its queue file. |
|
|
||||||
| `--archive` | `<slug>` | Marks a show as archived — subsequent fetches download episodes to disk. |
|
|
||||||
| `--unarchive` | `<slug>` | Marks a show as non-archived — subsequent fetches store only the enclosure URL for live streaming. |
|
|
||||||
| `--import-opml` | `<file.opml>` | Imports show records from an OPML file (e.g. exported from gpodder). Imported shows are marked protected so they aren't pruned by gpodder sync. |
|
|
||||||
|
|
||||||
### Show registration and protection
|
Example log lines worth knowing: `RESOLVED:` (episode chosen), `QUEUED show:` / `QUEUED stream capture:` (URI written), `EXPIRED:` (slot over, queue cleared), `SKIP (already fired this minute):`, `DEFERRED: no playable episode yet`, and `STREAM CAPTURE started:` with the ffmpeg pid.
|
||||||
|
|
||||||
Shows enter the registry three ways: manual `--add-show`, `--import-opml`, or gpodder.net sync. Shows added via OPML import or manual add carry an `opml_import` protection flag. When gpodder sync runs, it only prunes shows that were *originally* synced from gpodder.net and have since disappeared from the account — protected shows are never auto-deleted, even if absent from the subscription list.
|
### fetch_podcasts.rb
|
||||||
|
|
||||||
Every show is assigned a stable `guid` on import: taken from the OPML outline's `guid` attribute when present, otherwise a generated UUID v4. Episodes likewise get a `guid` from the RSS item (falling back to a hash of the entry), giving each record a unique key independent of filename.
|
Podcast ingestion and the show registry. Backed by two SQLite databases (`state/subscriptions.db`, `state/played.db`), opened through Sequel's JDBC adapter (`jdbc:sqlite:`) with WAL and a 5-second busy timeout. Episode GUIDs are SHA-1 digests; slugs are lowercased with non-alphanumeric characters mapped to `_`. Requires `nokogiri` (the script aborts deliberately if it fails to load).
|
||||||
|
|
||||||
### Archived vs. live shows
|
Every invocation first attempts the gPodder OPML sync (see `config.json` above); if gPodder is not enabled or credentials are incomplete, this is a no-op, and the requested command then runs.
|
||||||
|
|
||||||
The `archived` integer flag controls how a show's episodes are handled:
|
Usage:
|
||||||
|
|
||||||
- **Archived (`archived = 1`, the default):** each new episode is downloaded into `podcasts/<slug>/` and its local `file_path` is stored. Playback reads from disk.
|
|
||||||
- **Non-archived (`archived = 0`):** no download occurs; only the remote `enclosure_url` is stored. Playback streams live from that URL.
|
|
||||||
|
|
||||||
Toggle at any time with `--archive` / `--unarchive`. Flipping the flag affects only episodes fetched *after* the change; previously downloaded files remain valid playable content.
|
|
||||||
|
|
||||||
Video-only feeds are filtered out at registration: a show is rejected if its latest enclosure has a `video/*` MIME type or a video file extension. Mixed-content feeds are also handled per-episode, skipping any individual entry whose enclosure is video.
|
|
||||||
|
|
||||||
### Playback duration
|
|
||||||
|
|
||||||
Episode durations are extracted from the RSS `media:duration` element (preferred) or estimated from the `enclosure length` byte count, and stored in seconds as `runlength`. This value is carried into the liquidsoap annotation so the scheduler can clamp each show's playback to its slot.
|
|
||||||
|
|
||||||
## `update_playlists` (`.py` / `.rb`)
|
|
||||||
|
|
||||||
Selects the next unplayed episode per show and writes an annotated URI line for `station.liq` to consume. Runs automatically via cron every hour at :30 (offset from the fetcher so fresh episode records are available).
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Python
|
|
||||||
python3 update_playlists.py [options]
|
|
||||||
# Ruby
|
|
||||||
jruby update_playlists.rb [options]
|
|
||||||
```
|
|
||||||
|
|
||||||
### Options
|
|
||||||
|
|
||||||
| Option | Argument | Effect |
|
|
||||||
|--------|----------|--------|
|
|
||||||
| *(none)* | — | For each show, selects an unplayed episode, writes `playlists/<slug>.txt`, and records it as played. |
|
|
||||||
| `--json` | — | Emits a JSON summary of each show's total episode count, played count, and unplayed count, then exits. Useful for monitoring. |
|
|
||||||
|
|
||||||
### Selection logic
|
|
||||||
|
|
||||||
For each show, the updater queries `state/played.db` for the earliest episode with `played = 0`, ordered by insertion. This works identically for archived shows (local `file_path`) and live shows (remote `enclosure_url`), because both store their episodes in the table. The selected episode is written as a single annotated URI line:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
annotate:liq_runlength="<seconds>",liq_title="<title>":<uri>
|
fetch_podcasts.rb [--list|--detail|--add-show <url>|--remove <slug>|--archive <slug>|--unarchive <slug>|--import-opml <file>|--fetch-all]
|
||||||
```
|
```
|
||||||
|
|
||||||
where `<uri>` is the local file path for archived episodes or the enclosure URL for live ones. The episode is then marked `played = 1` with a timestamp, so the next cycle advances to a different episode and a restarted liquidsoap won't replay one already committed to the queue.
|
| Option | Effect |
|
||||||
|
|---|---|
|
||||||
|
| `--list` | Print all registered shows ordered by slug: slug, title, archive flag. |
|
||||||
|
| `--list --detail` | `--detail` is a modifier on `--list`, not a standalone command. Adds per-show episode counts: episodes total and unplayed. |
|
||||||
|
| `--add-show <url>` | Register a podcast feed. Probes the feed (to `/tmp/radio_feed_probe.xml`) to derive its title, then inserts the show with a fresh slug. |
|
||||||
|
| `--remove <slug>` | Remove a show, matched by case-insensitive substring against slugs or by exact feed URL. Deletes its episodes from `played.db`, then the show itself. |
|
||||||
|
| `--archive <slug>` | Mark a show archived. Only archived shows are fetched by `--fetch-all` and only archived shows are eligible for scheduled `show` slots' reset cycle. |
|
||||||
|
| `--unarchive <slug>` | Clear the archive flag. |
|
||||||
|
| `--import-opml <file>` | Import subscriptions from a local OPML file. Each imported show is marked archived and flagged as OPML-imported. |
|
||||||
|
| `--fetch-all` | Fetch every **archived** show: read its feed, download new episodes into `<storage>/podcasts/<slug>/`, and record each in `played.db` with duration, size, and local path. Episodes already known (by GUID) are skipped. This is the cron-installed hourly command. |
|
||||||
|
| *(no arguments)* | Prints the usage line (after the gPodder sync attempt). |
|
||||||
|
|
||||||
## `station.liq`
|
### update_playlists.rb (legacy)
|
||||||
|
|
||||||
The liquidsoap program itself. Not invoked directly — it runs under the systemd service. Key behaviors:
|
Superseded by `scheduler.rb`, which now reads `schedule.txt` and resolves episodes directly. This script implements the older per-show-playlist design: on each run it reads `schedule.txt`, and for each entry matching the current tick writes an annotated URI (`annotate:liq_runlength="...",liq_title="...":<uri>`) into `<storage>/queue/<name>.txt`, marking the episode played and resetting a show's episodes to unplayed once exhausted. The current `station.liq` does not read these files, so the script is out of the air path.
|
||||||
|
|
||||||
- Resolves all paths relative to its own location, so the whole tree can be relocated without editing the file.
|
| Option | Effect |
|
||||||
- Loads Icecast credentials and the storage path from `config.json`.
|
|---|---|
|
||||||
- Maintains a continuous random background-music playlist drawn from `music/`.
|
| *(no arguments)* | Process the current tick and write per-show playlist files. |
|
||||||
- Uses a `request.dynamic` source with a callback that reads the pre-selected `<slug>.txt` queue file when a scheduled show is due, applies the runlength clamp via a watcher thread, and falls through to background music when the show's slot expires.
|
| `--json` | Additionally print a JSON summary of every registered show: slug, title, total/played/unplayed episode counts. Useful for monitoring scripts. |
|
||||||
|
|
||||||
To reload after editing, restart the service:
|
### station.rb
|
||||||
|
|
||||||
```bash
|
The systemd entry point. Three lines: change directory to the script's own location (this is what makes `RADIO_DIR` resolve for the silence fallback), then `exec liquidsoap station.liq`. No options.
|
||||||
sudo systemctl restart <servicename>
|
|
||||||
|
### run_radio.sh
|
||||||
|
|
||||||
|
The launcher every JRuby script goes through. It exists because mutating `GEM_HOME`/`GEM_PATH` from inside a running JRuby process does not reliably affect gem resolution (JRuby #5269); exporting them *before* jruby boots is the reliable path. It points the gem environment at the co-located `.gems/` directory and gives the JVM a bounded heap.
|
||||||
|
|
||||||
|
```
|
||||||
|
Usage: ./run_radio.sh <script.rb> [args...]
|
||||||
```
|
```
|
||||||
|
|
||||||
where `<servicename>` is the basename of your install directory.
|
The JVM is started with `-J-Xms256m -J-Xmx2g`. Note this JRuby build ignores `JRUBY_OPTS`, so any JVM flags must go on the command line via `-J`, not through the environment. Typical invocations:
|
||||||
|
|
||||||
|
```
|
||||||
|
sudo -u liquidsoap ./run_radio.sh fetch_podcasts.rb --fetch-all
|
||||||
|
sudo -u liquidsoap ./run_radio.sh scheduler.rb --dryfire-stream "https://streams.kpfa.org:8443/kpfa" 300
|
||||||
|
```
|
||||||
|
|
||||||
|
### console.sh
|
||||||
|
|
||||||
|
Opens an interactive `jirb` console with the same gem environment (`GEM_HOME`/`GEM_PATH` pointed at `.gems/`), running as the `liquidsoap` user via sudo, with native-access JVM options set. An optional argument names a Ruby file to pre-load with `-r`:
|
||||||
|
|
||||||
|
```
|
||||||
|
./console.sh # plain console
|
||||||
|
./console.sh helper.rb # console with helper.rb pre-loaded
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Operations
|
## station.liq, in brief
|
||||||
|
|
||||||
## Starting and stopping
|
The Liquidsoap config is extensively commented in the file itself; the load-bearing design points:
|
||||||
|
|
||||||
```bash
|
**One queue, one feeder.** A single `request.dynamic(prefetch=0, read_queue)` reads `queues/current_queue.txt` each tick and hands out the URI it names. `prefetch=0` means the poller is the *only* thing that feeds the source, so the queue cannot be filled by two mechanisms at once (prefetch plus a poller was the played-twice bug). The callback remembers the last URI it handed out in a string `ref` and the poller calls `fetch()` only when the queue file names something different; on a duplicate poll it does nothing, so nothing accumulates. (`fetch()` *appends* to the request queue; calling it on a timer without a consumer once grew the queue to fifty silence requests.)
|
||||||
sudo systemctl start icecast2
|
|
||||||
sudo systemctl start <servicename>
|
**Empty queue resolves, never fails.** When the queue file is empty or absent, the callback returns a request for `<RADIO_DIR>/silence.mp3` instead of a failing request. A callback that keeps returning unresolvable requests drives a `request.dynamic` permanently DOWN and it stops polling, which is how a queued show once went unseen until a restart. The silent file keeps the source UP and polling, is inaudible, and `fallback` still prefers a real show the moment one appears.
|
||||||
sudo systemctl status <servicename>
|
|
||||||
tail -f <storage>/logs/liquidsoap.log
|
**Cut-in timing.** `fallback(track_sensitive=false, [show_source, background_music])` is what makes a scheduled show cut in live at its appointed time rather than waiting for the current background track to finish. `mksafe` wraps the mux so the encoder never starves.
|
||||||
|
|
||||||
|
**Background music.** Four `playlist(dir)` sources (music, jingles, announcements, podcasts) under `rotate`, wrapped in one `crossfade(fade_in=0.5, fade_out=0.5, duration=0.5, ...)`. `duration` is how much audio crossfade buffers *in advance* per track, not the fade length; sizing it to the fade means a track shorter than the buffer cannot empty it (the buffer-emptied collapse). `minimum` is deliberately omitted: on this build it is a threshold on the cross's own duration, and any minimum above 0.5 vetoes every handoff with "Not enough data for crossing". `rotate` carries no `transition_length` and no `transitions`, so it hands off with a hard cut and crossfade is the only operator composing transitions. There is deliberately no crossfade above the `fallback`: layer handoffs (a show expiring mid-frame) are hard cuts, so the encoder only ever sees complete, valid frames.
|
||||||
|
|
||||||
|
**Output.** `output.icecast(%mp3(bitrate=128), ...)` to the host/port/mount from `config.json`, connection parameters loaded via `let json.parse` with an explicit type annotation (without it, fields used only in string interpolations default to null).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Operations
|
||||||
|
|
||||||
|
**Crontab (liquidsoap user)**, for the current architecture:
|
||||||
|
|
||||||
|
```
|
||||||
|
* * * * * cd /srv/radio && ./run_radio.sh scheduler.rb >> /mnt/storage/radio/logs/scheduler_cron.log 2>&1
|
||||||
|
0 * * * * cd /srv/radio && ./run_radio.sh fetch_podcasts.rb --fetch-all >> /mnt/storage/radio/logs/fetch_cron.log 2>&1
|
||||||
```
|
```
|
||||||
|
|
||||||
Enable both at boot:
|
**Monitoring.** The service journal is the primary window: `journalctl -u radio -f`. Useful signatures:
|
||||||
|
|
||||||
```bash
|
- A queued item starting: `[requests:3] Prepared "<path or uri>" (RID N).`
|
||||||
sudo systemctl enable icecast2 <servicename>
|
- A show ending naturally: `[decoder:2] Decoding "<path>" ended: Ffmpeg_decoder.End_of_file` followed within a second by the next request's `Prepared`. That pair is the healthy end-of-slot signature.
|
||||||
```
|
- Queue handoff: `[mksafe:3] Switch to ...` lines around a fire.
|
||||||
|
- Scheduler activity: `RESOLVED` / `QUEUED` / `EXPIRED` / `SKIP` lines in `scheduler_cron.log`.
|
||||||
|
|
||||||
## Managing shows
|
**Sanity checks.**
|
||||||
|
|
||||||
```bash
|
- Is anything queued? `cat <storage>/queues/current_queue.txt` (empty means background music is on air) and `cat <storage>/queues/current_queue.meta` (epoch second the slot ends).
|
||||||
# List all registered shows with details
|
- Did a slot fire? `ls <storage>/state/fire_markers/ | tail` and the scheduler log.
|
||||||
... fetch_podcasts --list-shows --detail
|
- Test a stream slot without waiting for its clock time: `./run_radio.sh scheduler.rb --dryfire-stream "<url>" 300`.
|
||||||
|
- Test a show slot: `./run_radio.sh scheduler.rb --dryfire-show <slug>`.
|
||||||
|
- Show inventory: `./run_radio.sh fetch_podcasts.rb --list --detail`.
|
||||||
|
- Episode inventory as JSON: `./run_radio.sh update_playlists.rb --json`.
|
||||||
|
|
||||||
# Add a show from its feed URL
|
**Housekeeping is automatic.** Fire markers older than one day and captures older than two days are pruned each scheduler tick. A capture's `.fflog` is ffmpeg's stderr for that capture, next to the MP3.
|
||||||
... fetch_podcasts --add-show https://example.com/feed.xml
|
|
||||||
|
|
||||||
# Import a gpodder OPML export
|
|
||||||
... fetch_podcasts --import-opml ~/gpodder-subscriptions.opml
|
|
||||||
|
|
||||||
# Switch a show between archived and live
|
|
||||||
... fetch_podcasts --archive some_show_slug
|
|
||||||
... fetch_podcasts --unarchive some_show_slug
|
|
||||||
|
|
||||||
# Remove a show and all its data
|
|
||||||
... fetch_podcasts --delete-show some_show_slug
|
|
||||||
```
|
|
||||||
|
|
||||||
(Replace `...` with the appropriate interpreter prefix shown in the Tools section.)
|
|
||||||
|
|
||||||
## Monitoring
|
|
||||||
|
|
||||||
Check the cron logs for fetch/update activity:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
tail -f <storage>/logs/fetch.log
|
|
||||||
tail -f <storage>/logs/update.log
|
|
||||||
```
|
|
||||||
|
|
||||||
Query episode state as JSON:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
... update_playlists --json
|
|
||||||
```
|
|
||||||
|
|
||||||
## Re-running an installer
|
|
||||||
|
|
||||||
Safe at any time. Existing `config.json` supplies prompt defaults, existing Java/JRuby (Ruby stack) or venv (Python stack) are detected and reused, and cron entries are deduplicated so re-runs don't create duplicates.
|
|
||||||
|
|
||||||
|
**Restarting.** `systemctl restart radio`. On start, `station.liq` prints the loaded config line; if `RADIO_DIR` is unset you will see an explicit WARNING that the silence fallback will not resolve, which means the service was not started via `station.rb`/the systemd unit.
|
||||||
|
|
|
||||||
|
|
@ -1,674 +0,0 @@
|
||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
fetch_podcasts.py - Podcast subscription management and episode fetching
|
|
||||||
for the liquidsoap radio automation stack.
|
|
||||||
|
|
||||||
All data (podcasts, state DBs, logs, playlists) lives under the storage path
|
|
||||||
defined in config.json ("storage" key), keeping the boot drive clean.
|
|
||||||
|
|
||||||
Concurrency model:
|
|
||||||
- An exclusive, non-blocking lockfile (state/radio.lock) guarantees this
|
|
||||||
process and update_playlists.py never touch the databases simultaneously.
|
|
||||||
If the updater holds the lock, this run logs a skip and exits 0.
|
|
||||||
- Both databases run in WAL mode with a busy timeout as a second safety net.
|
|
||||||
|
|
||||||
Politeness / efficiency:
|
|
||||||
- The audio-vs-video verdict for each show is cached in subscriptions.db
|
|
||||||
(column media_class). Repeat runs reuse it instead of re-fetching every
|
|
||||||
feed just to re-confirm it is audio.
|
|
||||||
- The gpodder OPML pull uses bounded retry with exponential backoff + jitter
|
|
||||||
so transient failures or a 429 degrade gracefully rather than hammering.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import fcntl
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import random
|
|
||||||
import re
|
|
||||||
import shutil
|
|
||||||
import socket
|
|
||||||
import sqlite3
|
|
||||||
import sys
|
|
||||||
import time
|
|
||||||
import uuid
|
|
||||||
import xml.etree.ElementTree as ET
|
|
||||||
from pathlib import Path
|
|
||||||
from urllib.parse import quote
|
|
||||||
|
|
||||||
# --- Force IPv4-only resolution ---------------------------------------------
|
|
||||||
# Box has no usable IPv6 route; gpodder.net publishes both A and AAAA records,
|
|
||||||
# so dual-stack getaddrinfo returns AAAA first and stalls. Constrain every
|
|
||||||
# AF_UNSPEC lookup to AF_INET before it reaches the resolver.
|
|
||||||
_original_getaddrinfo = socket.getaddrinfo
|
|
||||||
|
|
||||||
def _ipv4_only_getaddrinfo(host, port, family=0, type=0, proto=0, flags=0):
|
|
||||||
if family in (0, socket.AF_UNSPEC):
|
|
||||||
family = socket.AF_INET
|
|
||||||
return _original_getaddrinfo(host, port, family, type, proto, flags)
|
|
||||||
|
|
||||||
socket.getaddrinfo = _ipv4_only_getaddrinfo
|
|
||||||
|
|
||||||
try:
|
|
||||||
import requests.packages.urllib3.util.connection as _urllib3_conn
|
|
||||||
_urllib3_conn.HAS_IPV6 = False
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
import feedparser
|
|
||||||
import requests
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parent
|
|
||||||
CONFIG_PATH = ROOT / "config.json"
|
|
||||||
|
|
||||||
AUDIO_EXTS = {".mp3", ".m4a"}
|
|
||||||
VIDEO_EXTS = {".mp4", ".mov", ".avi", ".webm", ".mkv"}
|
|
||||||
|
|
||||||
STATE_DIR = None
|
|
||||||
SUBS_DB = None
|
|
||||||
PLAYED_DB = None
|
|
||||||
PODCASTS_DIR = None
|
|
||||||
LOGS_DIR = None
|
|
||||||
PLAYLISTS_DIR = None
|
|
||||||
LOCK_FILE = None
|
|
||||||
|
|
||||||
def load_config():
|
|
||||||
with open(CONFIG_PATH) as f:
|
|
||||||
return json.load(f)
|
|
||||||
|
|
||||||
def init_paths():
|
|
||||||
global STATE_DIR, SUBS_DB, PLAYED_DB, PODCASTS_DIR, LOGS_DIR, PLAYLISTS_DIR, LOCK_FILE
|
|
||||||
cfg = load_config()
|
|
||||||
storage = Path(cfg["storage"]).expanduser().resolve()
|
|
||||||
STATE_DIR = storage / "state"
|
|
||||||
SUBS_DB = STATE_DIR / "subscriptions.db"
|
|
||||||
PLAYED_DB = STATE_DIR / "played.db"
|
|
||||||
PODCASTS_DIR = storage / "podcasts"
|
|
||||||
LOGS_DIR = storage / "logs"
|
|
||||||
PLAYLISTS_DIR = storage / "playlists"
|
|
||||||
LOCK_FILE = STATE_DIR / "radio.lock"
|
|
||||||
|
|
||||||
logging.basicConfig(
|
|
||||||
level=logging.INFO,
|
|
||||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
|
||||||
handlers=[logging.StreamHandler(sys.stdout)],
|
|
||||||
)
|
|
||||||
log = logging.getLogger("fetch_podcasts")
|
|
||||||
|
|
||||||
def log_error(msg):
|
|
||||||
log.error(msg)
|
|
||||||
|
|
||||||
def _setup_logging():
|
|
||||||
fh = logging.FileHandler(LOGS_DIR / "fetch.log")
|
|
||||||
fh.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(message)s"))
|
|
||||||
log.addHandler(fh)
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Mutual exclusion: one writer across fetch + update at any moment.
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
_lock_fd = None
|
|
||||||
|
|
||||||
def acquire_lock():
|
|
||||||
global _lock_fd
|
|
||||||
STATE_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
fd = os.open(str(LOCK_FILE), os.O_CREAT | os.O_RDWR, 0o644)
|
|
||||||
try:
|
|
||||||
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
|
||||||
except OSError:
|
|
||||||
os.close(fd)
|
|
||||||
return False
|
|
||||||
os.ftruncate(fd, 0)
|
|
||||||
os.write(fd, str(os.getpid()).encode())
|
|
||||||
_lock_fd = fd
|
|
||||||
return True
|
|
||||||
|
|
||||||
def release_lock():
|
|
||||||
global _lock_fd
|
|
||||||
if _lock_fd is not None:
|
|
||||||
try:
|
|
||||||
fcntl.flock(_lock_fd, fcntl.LOCK_UN)
|
|
||||||
os.close(_lock_fd)
|
|
||||||
finally:
|
|
||||||
_lock_fd = None
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Schema: single source of truth per database, applied idempotently.
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
SHOWS_COLUMNS = {
|
|
||||||
"slug": "TEXT PRIMARY KEY",
|
|
||||||
"guid": "TEXT NOT NULL UNIQUE",
|
|
||||||
"name": "TEXT NOT NULL",
|
|
||||||
"feed_url": "TEXT NOT NULL UNIQUE",
|
|
||||||
"source": "TEXT DEFAULT 'manual'",
|
|
||||||
"opml_import": "INTEGER DEFAULT 0",
|
|
||||||
"archived": "INTEGER DEFAULT 1",
|
|
||||||
"media_class": "TEXT",
|
|
||||||
"created_at": "TEXT DEFAULT (datetime('now'))",
|
|
||||||
}
|
|
||||||
|
|
||||||
EPISODES_COLUMNS = {
|
|
||||||
"id": "INTEGER PRIMARY KEY AUTOINCREMENT",
|
|
||||||
"show_slug": "TEXT NOT NULL",
|
|
||||||
"guid": "TEXT NOT NULL",
|
|
||||||
"title": "TEXT",
|
|
||||||
"file_path": "TEXT",
|
|
||||||
"enclosure_url": "TEXT",
|
|
||||||
"runlength": "INTEGER",
|
|
||||||
"played": "INTEGER DEFAULT 0",
|
|
||||||
"played_at": "TEXT",
|
|
||||||
}
|
|
||||||
|
|
||||||
def _connect(db_path):
|
|
||||||
conn = sqlite3.connect(str(db_path), timeout=5.0)
|
|
||||||
conn.row_factory = sqlite3.Row
|
|
||||||
conn.execute("PRAGMA journal_mode=WAL")
|
|
||||||
conn.execute("PRAGMA busy_timeout=5000")
|
|
||||||
return conn
|
|
||||||
|
|
||||||
def _ensure_columns(conn, table, columns):
|
|
||||||
exists = conn.execute(
|
|
||||||
"SELECT name FROM sqlite_master WHERE type='table' AND name=?", (table,)
|
|
||||||
).fetchone()
|
|
||||||
if exists is None:
|
|
||||||
defs = ",\n ".join(f"{name} {spec}" for name, spec in columns.items())
|
|
||||||
extra = "\n ,UNIQUE(show_slug, guid)" if table == "episodes" else ""
|
|
||||||
conn.execute(f"CREATE TABLE {table} (\n {defs}{extra}\n )")
|
|
||||||
else:
|
|
||||||
existing = {row[1] for row in conn.execute(f"PRAGMA table_info({table})")}
|
|
||||||
for name, spec in columns.items():
|
|
||||||
if name not in existing:
|
|
||||||
col_default = spec.split("DEFAULT", 1)[1].strip() if "DEFAULT" in spec else "NULL"
|
|
||||||
conn.execute(f"ALTER TABLE {table} ADD COLUMN {name} {col_default}")
|
|
||||||
conn.commit()
|
|
||||||
|
|
||||||
def open_subs_db():
|
|
||||||
conn = _connect(SUBS_DB)
|
|
||||||
_ensure_columns(conn, "shows", SHOWS_COLUMNS)
|
|
||||||
return conn
|
|
||||||
|
|
||||||
def open_played_db():
|
|
||||||
conn = _connect(PLAYED_DB)
|
|
||||||
_ensure_columns(conn, "episodes", EPISODES_COLUMNS)
|
|
||||||
conn.execute(
|
|
||||||
"CREATE INDEX IF NOT EXISTS idx_episodes_show_played ON episodes (show_slug, played)"
|
|
||||||
)
|
|
||||||
conn.commit()
|
|
||||||
return conn
|
|
||||||
|
|
||||||
def slugify(name):
|
|
||||||
s = re.sub(r"[^a-z0-9]+", "_", name.lower()).strip("_")
|
|
||||||
return s[:60] or "show"
|
|
||||||
|
|
||||||
def gen_uuid():
|
|
||||||
return str(uuid.uuid4())
|
|
||||||
|
|
||||||
def classify_feed(parsed):
|
|
||||||
"""Return 'audio', 'video', or 'unknown' based on the latest enclosure."""
|
|
||||||
if not parsed.entries:
|
|
||||||
return "unknown"
|
|
||||||
entry = parsed.entries[0]
|
|
||||||
enclosures = entry.get("enclosures") or []
|
|
||||||
if not enclosures:
|
|
||||||
return "unknown"
|
|
||||||
mime_type = (enclosures[0].get("type") or "").lower()
|
|
||||||
if mime_type.startswith("audio/"):
|
|
||||||
return "audio"
|
|
||||||
if mime_type.startswith("video/"):
|
|
||||||
return "video"
|
|
||||||
url = (enclosures[0].get("href") or "").lower()
|
|
||||||
if any(url.endswith(ext) for ext in AUDIO_EXTS):
|
|
||||||
return "audio"
|
|
||||||
if any(url.endswith(ext) for ext in VIDEO_EXTS):
|
|
||||||
return "video"
|
|
||||||
return "unknown"
|
|
||||||
|
|
||||||
def get_media_class(subs_db, slug):
|
|
||||||
row = subs_db.execute("SELECT media_class FROM shows WHERE slug = ?", (slug,)).fetchone()
|
|
||||||
return row["media_class"] if row else None
|
|
||||||
|
|
||||||
def set_media_class(subs_db, slug, cls):
|
|
||||||
subs_db.execute("UPDATE shows SET media_class = ? WHERE slug = ?", (cls, slug))
|
|
||||||
subs_db.commit()
|
|
||||||
|
|
||||||
def fetch_feed(feed_url):
|
|
||||||
try:
|
|
||||||
parsed = feedparser.parse(feed_url)
|
|
||||||
except Exception as e:
|
|
||||||
log_error(f"Feed parse error for {feed_url}: {e}")
|
|
||||||
return None
|
|
||||||
if parsed.bozo and not parsed.entries:
|
|
||||||
log_error(f"Bozo feed (no entries) for {feed_url}: {parsed.bozo_exception}")
|
|
||||||
return None
|
|
||||||
return parsed
|
|
||||||
|
|
||||||
def gpodder_sync(cfg):
|
|
||||||
g = cfg["gpodder"]
|
|
||||||
base = g["host"].rstrip("/")
|
|
||||||
username = g["username"]
|
|
||||||
password = g["password"]
|
|
||||||
url = f"{base}/subscriptions/{quote(username, safe='')}.opml"
|
|
||||||
|
|
||||||
print(f"--- Syncing subscriptions from {base} ---")
|
|
||||||
print(f"Fetching subscriptions for '{username}'...")
|
|
||||||
|
|
||||||
# Bounded retry with exponential backoff + jitter. Retries on network
|
|
||||||
# errors and on 429/5xx; does not retry on 401/404 (auth/not-found).
|
|
||||||
max_attempts = 4
|
|
||||||
backoff_base = 2.0
|
|
||||||
resp = None
|
|
||||||
for attempt in range(1, max_attempts + 1):
|
|
||||||
try:
|
|
||||||
resp = requests.get(
|
|
||||||
url,
|
|
||||||
auth=(username, password),
|
|
||||||
headers={"User-Agent": "radio-automation/1.0"},
|
|
||||||
timeout=60,
|
|
||||||
)
|
|
||||||
break
|
|
||||||
except requests.RequestException as e:
|
|
||||||
if attempt == max_attempts:
|
|
||||||
log_error(f"gPodder sync failed after {attempt} attempts: {e}")
|
|
||||||
return []
|
|
||||||
delay = backoff_base ** attempt + random.uniform(0, 1)
|
|
||||||
log.warning("gPodder request error (%s); retrying in %.1fs", e.__class__.__name__, delay)
|
|
||||||
time.sleep(delay)
|
|
||||||
|
|
||||||
if resp is None:
|
|
||||||
return []
|
|
||||||
|
|
||||||
if resp.status_code == 200:
|
|
||||||
body = resp.text
|
|
||||||
if not body.strip():
|
|
||||||
log_error("gPodder sync returned an empty body.")
|
|
||||||
return []
|
|
||||||
return parse_opml(body)
|
|
||||||
elif resp.status_code == 401:
|
|
||||||
log_error("gPodder sync failed: 401 Unauthorized. Check username/password in config.json.")
|
|
||||||
elif resp.status_code == 404:
|
|
||||||
log_error("gPodder sync failed: 404 Not Found. User may not exist or has no subscriptions.")
|
|
||||||
elif resp.status_code == 429:
|
|
||||||
log_error("gPodder sync throttled (429). Will retry next cycle.")
|
|
||||||
elif resp.status_code >= 500:
|
|
||||||
log_error(f"gPodder sync server error ({resp.status_code}). Will retry next cycle.")
|
|
||||||
else:
|
|
||||||
log_error(f"gPodder sync failed: unexpected response {resp.status_code}: {resp.text[:200]}")
|
|
||||||
return []
|
|
||||||
|
|
||||||
def parse_opml(xml_string):
|
|
||||||
shows = []
|
|
||||||
try:
|
|
||||||
root = ET.fromstring(xml_string)
|
|
||||||
except ET.ParseError as e:
|
|
||||||
log_error(f"Failed to parse OPML XML: {e}")
|
|
||||||
return []
|
|
||||||
for outline in root.iter("outline"):
|
|
||||||
feed_url = (outline.attrib.get("xmlUrl") or "").strip()
|
|
||||||
name = (outline.attrib.get("text") or "").strip()
|
|
||||||
guid = (outline.attrib.get("guid") or "").strip()
|
|
||||||
if not feed_url or not re.match(r"^https?://", feed_url):
|
|
||||||
continue
|
|
||||||
shows.append({"name": name, "feed_url": feed_url, "guid": guid if guid else None})
|
|
||||||
return shows
|
|
||||||
|
|
||||||
def register_remote_shows(remote_shows):
|
|
||||||
db = open_subs_db()
|
|
||||||
added = 0
|
|
||||||
skipped_video = 0
|
|
||||||
for show in remote_shows:
|
|
||||||
slug = slugify(show["name"])
|
|
||||||
existing = db.execute("SELECT slug FROM shows WHERE slug = ?", (slug,)).fetchone()
|
|
||||||
if existing is not None:
|
|
||||||
continue
|
|
||||||
# Reuse a cached classification if we somehow already know it; otherwise
|
|
||||||
# fetch once to classify. New shows are fetched here regardless.
|
|
||||||
parsed = fetch_feed(show["feed_url"])
|
|
||||||
if parsed is None:
|
|
||||||
log.warning("Skipping '%s': could not fetch feed.", show["name"])
|
|
||||||
continue
|
|
||||||
cls = classify_feed(parsed)
|
|
||||||
if cls == "video":
|
|
||||||
log.info("Skipping '%s' (%s): video podcast, not audio.", show["name"], slug)
|
|
||||||
skipped_video += 1
|
|
||||||
continue
|
|
||||||
guid = show["guid"] or gen_uuid()
|
|
||||||
db.execute(
|
|
||||||
"INSERT INTO shows (slug, guid, name, feed_url, source, opml_import, archived, media_class) "
|
|
||||||
"VALUES (?, ?, ?, ?, 'gpodder', 0, 1, ?)",
|
|
||||||
(slug, guid, show["name"], show["feed_url"], cls),
|
|
||||||
)
|
|
||||||
log.info("Registered new show: %s (%s) [%s]", show["name"], slug, cls)
|
|
||||||
added += 1
|
|
||||||
db.commit()
|
|
||||||
db.close()
|
|
||||||
if skipped_video:
|
|
||||||
log.info("Filtered out %d video podcast(s).", skipped_video)
|
|
||||||
return added
|
|
||||||
|
|
||||||
def prune_stale_shows(remote_shows):
|
|
||||||
db = open_subs_db()
|
|
||||||
remote_slugs = {slugify(s["name"]) for s in remote_shows}
|
|
||||||
stale = db.execute(
|
|
||||||
"SELECT slug, name FROM shows WHERE source = 'gpodder' AND opml_import = 0"
|
|
||||||
).fetchall()
|
|
||||||
removed = 0
|
|
||||||
for row in stale:
|
|
||||||
if row["slug"] not in remote_slugs:
|
|
||||||
remove_show_data(row["slug"])
|
|
||||||
db.execute("DELETE FROM shows WHERE slug = ?", (row["slug"],))
|
|
||||||
log.info("Pruned stale show: %s (%s)", row["name"], row["slug"])
|
|
||||||
removed += 1
|
|
||||||
db.commit()
|
|
||||||
db.close()
|
|
||||||
return removed
|
|
||||||
|
|
||||||
def extract_duration(entry):
|
|
||||||
dur = entry.get("media_duration") or entry.get("duration")
|
|
||||||
if dur:
|
|
||||||
try:
|
|
||||||
return int(float(dur))
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
pass
|
|
||||||
iso = entry.get("iso_8601_duration")
|
|
||||||
if iso:
|
|
||||||
m = re.match(r"PT(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?", iso)
|
|
||||||
if m:
|
|
||||||
h, mn, s = (int(g) if g else 0 for g in m.groups())
|
|
||||||
return h * 3600 + mn * 60 + s
|
|
||||||
enclosures = entry.get("enclosures") or []
|
|
||||||
if enclosures:
|
|
||||||
length = enclosures[0].get("length")
|
|
||||||
if length:
|
|
||||||
try:
|
|
||||||
return int(int(length) * 8 / 128000)
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
return None
|
|
||||||
|
|
||||||
def download_episode(url, dest_dir, filename):
|
|
||||||
dest = dest_dir / filename
|
|
||||||
if dest.exists():
|
|
||||||
return str(dest)
|
|
||||||
try:
|
|
||||||
with requests.get(url, stream=True, timeout=120) as r:
|
|
||||||
r.raise_for_status()
|
|
||||||
tmp = dest.with_suffix(dest.suffix + ".part")
|
|
||||||
with open(tmp, "wb") as f:
|
|
||||||
for chunk in r.iter_content(chunk_size=8192):
|
|
||||||
f.write(chunk)
|
|
||||||
tmp.rename(dest)
|
|
||||||
return str(dest)
|
|
||||||
except requests.RequestException as e:
|
|
||||||
log_error(f"Download failed for {url}: {e}")
|
|
||||||
return None
|
|
||||||
|
|
||||||
def safe_filename(title, fallback):
|
|
||||||
name = re.sub(r"[^\w\s.-]", "", title or "").strip().replace(" ", "_")
|
|
||||||
return (name[:120] or fallback) + ".mp3"
|
|
||||||
|
|
||||||
def show_archived(subs_db, slug):
|
|
||||||
row = subs_db.execute("SELECT archived FROM shows WHERE slug = ?", (slug,)).fetchone()
|
|
||||||
if row is None:
|
|
||||||
return True
|
|
||||||
return row["archived"] == 1
|
|
||||||
|
|
||||||
def fetch_show_episodes(slug, name, feed_url):
|
|
||||||
dest_dir = PODCASTS_DIR / slug
|
|
||||||
dest_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
subs_db = open_subs_db()
|
|
||||||
|
|
||||||
# Respect a cached video classification without re-fetching.
|
|
||||||
cached_cls = get_media_class(subs_db, slug)
|
|
||||||
if cached_cls == "video":
|
|
||||||
subs_db.close()
|
|
||||||
return 0
|
|
||||||
|
|
||||||
parsed = fetch_feed(feed_url)
|
|
||||||
if parsed is None:
|
|
||||||
subs_db.close()
|
|
||||||
return 0
|
|
||||||
|
|
||||||
# Classify and cache if we don't already have a verdict.
|
|
||||||
if cached_cls is None:
|
|
||||||
cls = classify_feed(parsed)
|
|
||||||
set_media_class(subs_db, slug, cls)
|
|
||||||
if cls == "video":
|
|
||||||
log.info("'%s' (%s) classified as video; skipping.", name, slug)
|
|
||||||
subs_db.close()
|
|
||||||
return 0
|
|
||||||
|
|
||||||
played_db = open_played_db()
|
|
||||||
seen = {
|
|
||||||
row["guid"]
|
|
||||||
for row in played_db.execute("SELECT guid FROM episodes WHERE show_slug = ?", (slug,))
|
|
||||||
}
|
|
||||||
archived = show_archived(subs_db, slug)
|
|
||||||
new_count = 0
|
|
||||||
for entry in parsed.entries:
|
|
||||||
guid = entry.get("id") or entry.get("link") or entry.get("title", "")
|
|
||||||
if guid in seen:
|
|
||||||
continue
|
|
||||||
enclosures = entry.get("enclosures") or []
|
|
||||||
if not enclosures:
|
|
||||||
continue
|
|
||||||
mime = (enclosures[0].get("type") or "").lower()
|
|
||||||
if mime.startswith("video/"):
|
|
||||||
continue
|
|
||||||
audio_url = enclosures[0].get("href")
|
|
||||||
if not audio_url:
|
|
||||||
continue
|
|
||||||
title = entry.get("title", "untitled")
|
|
||||||
duration = extract_duration(entry)
|
|
||||||
|
|
||||||
if archived:
|
|
||||||
filename = safe_filename(title, guid[-20:])
|
|
||||||
file_path = download_episode(audio_url, dest_dir, filename)
|
|
||||||
if file_path is None:
|
|
||||||
continue
|
|
||||||
played_db.execute(
|
|
||||||
"INSERT OR IGNORE INTO episodes (show_slug, guid, title, file_path, enclosure_url, runlength, played) "
|
|
||||||
"VALUES (?, ?, ?, ?, ?, ?, 0)",
|
|
||||||
(slug, guid, title, file_path, audio_url, duration),
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
played_db.execute(
|
|
||||||
"INSERT OR IGNORE INTO episodes (show_slug, guid, title, file_path, enclosure_url, runlength, played) "
|
|
||||||
"VALUES (?, ?, ?, NULL, ?, ?, 0)",
|
|
||||||
(slug, guid, title, audio_url, duration),
|
|
||||||
)
|
|
||||||
new_count += 1
|
|
||||||
kind = "downloaded" if archived else "live"
|
|
||||||
log.info(" New episode: %s [%s]", title, kind)
|
|
||||||
played_db.commit()
|
|
||||||
played_db.close()
|
|
||||||
subs_db.close()
|
|
||||||
return new_count
|
|
||||||
|
|
||||||
def fetch_all_episodes():
|
|
||||||
db = open_subs_db()
|
|
||||||
shows = db.execute("SELECT slug, name, feed_url FROM shows ORDER BY name").fetchall()
|
|
||||||
db.close()
|
|
||||||
total_new = 0
|
|
||||||
for show in shows:
|
|
||||||
log.info("--- Fetching: %s (%s) ---", show["name"], show["slug"])
|
|
||||||
try:
|
|
||||||
n = fetch_show_episodes(show["slug"], show["name"], show["feed_url"])
|
|
||||||
total_new += n
|
|
||||||
except Exception as e:
|
|
||||||
log_error(f"Unexpected error fetching {show['slug']}: {e}")
|
|
||||||
log.info("=== Fetch complete: %d new episode(s) ===", total_new)
|
|
||||||
|
|
||||||
def list_shows(detail=False):
|
|
||||||
db = open_subs_db()
|
|
||||||
rows = db.execute(
|
|
||||||
"SELECT slug, name, feed_url, source, opml_import, archived, media_class FROM shows ORDER BY name"
|
|
||||||
).fetchall()
|
|
||||||
db.close()
|
|
||||||
if not rows:
|
|
||||||
print("No shows registered.")
|
|
||||||
return
|
|
||||||
print(f"{'SLUG':<30} {'ARCHIVED':<10} {'MEDIA':<8} {'SOURCE':<10} NAME")
|
|
||||||
for r in rows:
|
|
||||||
arch = "yes" if r["archived"] == 1 else "no"
|
|
||||||
media = r["media_class"] or "?"
|
|
||||||
line = f"{r['slug']:<30} {arch:<10} {media:<8} {r['source']:<10} {r['name']}"
|
|
||||||
if detail:
|
|
||||||
line += f"\n{'':<50} {r['feed_url']}"
|
|
||||||
print(line)
|
|
||||||
|
|
||||||
def add_show(feed_url):
|
|
||||||
parsed = fetch_feed(feed_url)
|
|
||||||
if parsed is None or not parsed.feed.get("title"):
|
|
||||||
log_error(f"Could not determine show title from {feed_url}")
|
|
||||||
return
|
|
||||||
cls = classify_feed(parsed)
|
|
||||||
if cls == "video":
|
|
||||||
log_error(f"Refusing to add '{parsed.feed['title']}': video podcast detected.")
|
|
||||||
return
|
|
||||||
name = parsed.feed["title"]
|
|
||||||
slug = slugify(name)
|
|
||||||
guid = gen_uuid()
|
|
||||||
db = open_subs_db()
|
|
||||||
db.execute(
|
|
||||||
"INSERT OR IGNORE INTO shows (slug, guid, name, feed_url, source, opml_import, archived, media_class) "
|
|
||||||
"VALUES (?, ?, ?, ?, 'manual', 0, 1, ?)",
|
|
||||||
(slug, guid, name, feed_url, cls),
|
|
||||||
)
|
|
||||||
db.commit()
|
|
||||||
db.close()
|
|
||||||
log.info("Added show: %s (%s) [%s]", name, slug, cls)
|
|
||||||
fetch_show_episodes(slug, name, feed_url)
|
|
||||||
|
|
||||||
def set_archive(slug, value):
|
|
||||||
db = open_subs_db()
|
|
||||||
row = db.execute("SELECT name FROM shows WHERE slug = ?", (slug,)).fetchone()
|
|
||||||
if row is None:
|
|
||||||
log_error(f"No show found with slug '{slug}'.")
|
|
||||||
db.close()
|
|
||||||
return
|
|
||||||
db.execute("UPDATE shows SET archived = ? WHERE slug = ?", (value, slug))
|
|
||||||
db.commit()
|
|
||||||
db.close()
|
|
||||||
state = "archived" if value == 1 else "non-archived (live)"
|
|
||||||
log.info("Show '%s' (%s) is now %s.", row["name"], slug, state)
|
|
||||||
|
|
||||||
def remove_show_data(slug):
|
|
||||||
pod_dir = PODCASTS_DIR / slug
|
|
||||||
if pod_dir.exists():
|
|
||||||
shutil.rmtree(pod_dir)
|
|
||||||
txt = PLAYLISTS_DIR / f"{slug}.txt"
|
|
||||||
if txt.exists():
|
|
||||||
txt.unlink()
|
|
||||||
|
|
||||||
def delete_show(slug):
|
|
||||||
db = open_subs_db()
|
|
||||||
row = db.execute("SELECT name FROM shows WHERE slug = ?", (slug,)).fetchone()
|
|
||||||
if row is None:
|
|
||||||
log_error(f"No show found with slug '{slug}'.")
|
|
||||||
db.close()
|
|
||||||
return
|
|
||||||
remove_show_data(slug)
|
|
||||||
db.execute("DELETE FROM shows WHERE slug = ?", (slug,))
|
|
||||||
db.commit()
|
|
||||||
db.close()
|
|
||||||
played_db = open_played_db()
|
|
||||||
played_db.execute("DELETE FROM episodes WHERE show_slug = ?", (slug,))
|
|
||||||
played_db.commit()
|
|
||||||
played_db.close()
|
|
||||||
log.info("Deleted show: %s (%s)", row["name"], slug)
|
|
||||||
|
|
||||||
def import_opml(path):
|
|
||||||
try:
|
|
||||||
with open(path) as f:
|
|
||||||
content = f.read()
|
|
||||||
except OSError as e:
|
|
||||||
log_error(f"Cannot read OPML file: {e}")
|
|
||||||
return
|
|
||||||
shows = parse_opml(content)
|
|
||||||
db = open_subs_db()
|
|
||||||
added = 0
|
|
||||||
skipped_video = 0
|
|
||||||
for show in shows:
|
|
||||||
slug = slugify(show["name"])
|
|
||||||
existing = db.execute("SELECT slug FROM shows WHERE slug = ?", (slug,)).fetchone()
|
|
||||||
if existing is not None:
|
|
||||||
continue
|
|
||||||
parsed = fetch_feed(show["feed_url"])
|
|
||||||
if parsed is None:
|
|
||||||
log.warning("OPML import: skipping '%s', could not fetch feed.", show["name"])
|
|
||||||
continue
|
|
||||||
cls = classify_feed(parsed)
|
|
||||||
if cls == "video":
|
|
||||||
log.info("OPML import: skipping '%s' (%s): video podcast.", show["name"], slug)
|
|
||||||
skipped_video += 1
|
|
||||||
continue
|
|
||||||
guid = show["guid"] or gen_uuid()
|
|
||||||
db.execute(
|
|
||||||
"INSERT INTO shows (slug, guid, name, feed_url, source, opml_import, archived, media_class) "
|
|
||||||
"VALUES (?, ?, ?, ?, 'opml', 1, 1, ?)",
|
|
||||||
(slug, guid, show["name"], show["feed_url"], cls),
|
|
||||||
)
|
|
||||||
added += 1
|
|
||||||
db.commit()
|
|
||||||
db.close()
|
|
||||||
log.info("OPML import: %d added, %d video shows filtered out.", added, skipped_video)
|
|
||||||
|
|
||||||
def run_fetch(config):
|
|
||||||
g = config["gpodder"]
|
|
||||||
if g.get("enable") is True:
|
|
||||||
remote = gpodder_sync(config)
|
|
||||||
if not remote:
|
|
||||||
log.warning("No subscriptions retrieved from gPodder; using local registry only.")
|
|
||||||
else:
|
|
||||||
added = register_remote_shows(remote)
|
|
||||||
pruned = prune_stale_shows(remote)
|
|
||||||
log.info("Sync: %d added, %d pruned.", added, pruned)
|
|
||||||
fetch_all_episodes()
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description="Podcast fetcher for radio automation")
|
|
||||||
parser.add_argument("--list-shows", action="store_true", help="List registered shows")
|
|
||||||
parser.add_argument("--detail", action="store_true", help="With --list-shows, show feed URLs")
|
|
||||||
parser.add_argument("--add-show", metavar="FEED_URL", help="Add a show from a feed URL")
|
|
||||||
parser.add_argument("--delete-show", metavar="SLUG", help="Delete a show and its data")
|
|
||||||
parser.add_argument("--archive", metavar="SLUG", help="Mark a show as archived (download episodes)")
|
|
||||||
parser.add_argument("--unarchive", metavar="SLUG", help="Mark a show as non-archived (stream live)")
|
|
||||||
parser.add_argument("--import-opml", metavar="FILE", help="Import shows from an OPML file")
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
init_paths()
|
|
||||||
STATE_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
LOGS_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
PODCASTS_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
PLAYLISTS_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
_setup_logging()
|
|
||||||
|
|
||||||
config = load_config()
|
|
||||||
|
|
||||||
needs_lock = not args.list_shows
|
|
||||||
|
|
||||||
if needs_lock and not acquire_lock():
|
|
||||||
log.info("Another radio process holds the lock; skipping this run.")
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
if args.archive:
|
|
||||||
set_archive(args.archive, 1)
|
|
||||||
elif args.unarchive:
|
|
||||||
set_archive(args.unarchive, 0)
|
|
||||||
elif args.list_shows:
|
|
||||||
list_shows(detail=args.detail)
|
|
||||||
elif args.add_show:
|
|
||||||
add_show(args.add_show)
|
|
||||||
elif args.delete_show:
|
|
||||||
delete_show(args.delete_show)
|
|
||||||
elif args.import_opml:
|
|
||||||
import_opml(args.import_opml)
|
|
||||||
else:
|
|
||||||
run_fetch(config)
|
|
||||||
finally:
|
|
||||||
if needs_lock:
|
|
||||||
release_lock()
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
|
|
@ -49,6 +49,13 @@ $db_s.execute("PRAGMA busy_timeout=5000;")
|
||||||
$db_p.execute("PRAGMA journal_mode=WAL;")
|
$db_p.execute("PRAGMA journal_mode=WAL;")
|
||||||
$db_p.execute("PRAGMA busy_timeout=5000;")
|
$db_p.execute("PRAGMA busy_timeout=5000;")
|
||||||
|
|
||||||
|
# JRuby/Sequel quirk: the JDBC connection binds lazily, and datasets built on
|
||||||
|
# $db_p can resolve against the wrong database file ("no such table: main.shows"
|
||||||
|
# when inserting episodes). Force each connection to bind to its own file now,
|
||||||
|
# while we still have an explicit handle to verify it.
|
||||||
|
$db_s.from(:shows).count
|
||||||
|
$db_p.from(:episodes).count
|
||||||
|
|
||||||
def log(level, msg)
|
def log(level, msg)
|
||||||
ts = Time.now.strftime("%Y-%m-%d %H:%M:%S")
|
ts = Time.now.strftime("%Y-%m-%d %H:%M:%S")
|
||||||
line = "[#{ts}] [#{level}] #{msg}"
|
line = "[#{ts}] [#{level}] #{msg}"
|
||||||
|
|
@ -139,11 +146,10 @@ def http_stream_to_file(url, dest_path, user = nil, pass = nil)
|
||||||
unless res.is_a?(Net::HTTPSuccess)
|
unless res.is_a?(Net::HTTPSuccess)
|
||||||
raise "HTTP #{res.code}: #{res.message}"
|
raise "HTTP #{res.code}: #{res.message}"
|
||||||
end
|
end
|
||||||
buffer = ::StringIO.new (res.read_body)
|
body = res.read_body
|
||||||
buffer.rewind
|
# JRuby bug: StringIO#read with a block does not reliably yield chunks,
|
||||||
File.open(tmp_dest, "wb") do |f|
|
# resulting in a 0-byte file. Write the body directly instead.
|
||||||
buffer.read() { |chunk| f.write(chunk) }
|
File.open(tmp_dest, "wb") { |f| f.write(body) }
|
||||||
end
|
|
||||||
File.rename(tmp_dest, dest_path)
|
File.rename(tmp_dest, dest_path)
|
||||||
success = true
|
success = true
|
||||||
rescue Exception => error
|
rescue Exception => error
|
||||||
|
|
@ -166,7 +172,9 @@ def parse_feed(opts)
|
||||||
channel = doc.at_xpath("//channel")
|
channel = doc.at_xpath("//channel")
|
||||||
if channel
|
if channel
|
||||||
items = []
|
items = []
|
||||||
channel.elements("item").each do |item|
|
# JRuby bug: node.elements("tagname") raises ArgumentError under JRuby/Nokogiri.
|
||||||
|
# XPath is the verified-working form (the raw-feed probe used it successfully).
|
||||||
|
channel.xpath("./item").each do |item|
|
||||||
enc = item.at_xpath("./enclosure")
|
enc = item.at_xpath("./enclosure")
|
||||||
next unless enc
|
next unless enc
|
||||||
next unless enc.attributes["type"].value =~ /audio/i
|
next unless enc.attributes["type"].value =~ /audio/i
|
||||||
|
|
@ -194,7 +202,11 @@ def parse_feed(opts)
|
||||||
est_dur = (enc_len / (128 * 1024)).round if enc_len > 0
|
est_dur = (enc_len / (128 * 1024)).round if enc_len > 0
|
||||||
|
|
||||||
items << {
|
items << {
|
||||||
guid: item.at_xpath("./guid")&.text.presence || gen_guid(enc.attributes["url"].value),
|
# JRuby has no ActiveSupport; String#presence is unavailable. Use strip + empty? check.
|
||||||
|
guid: begin
|
||||||
|
g = item.at_xpath("./guid")&.text.to_s.strip
|
||||||
|
g.empty? ? nil : g
|
||||||
|
end || gen_guid(enc.attributes["url"].value),
|
||||||
title: item.at_xpath("./title")&.text || "Untitled",
|
title: item.at_xpath("./title")&.text || "Untitled",
|
||||||
url: enc.attributes["url"].value,
|
url: enc.attributes["url"].value,
|
||||||
duration_seconds: dur_sec > 0 ? dur_sec : (est_dur || 0),
|
duration_seconds: dur_sec > 0 ? dur_sec : (est_dur || 0),
|
||||||
|
|
@ -453,36 +465,49 @@ def cmd_fetch_all
|
||||||
|
|
||||||
next unless parsed
|
next unless parsed
|
||||||
|
|
||||||
show_dir = File.join(PODCAST_DIR, show[:slug])
|
# Option A: record URL-only episode rows; do NOT pre-download full MP3s.
|
||||||
ensure_dir(show_dir)
|
# The old code called http_stream_to_file(ep[:url], dest) for every item
|
||||||
|
# before inserting the row. For BradCast (3028 eps x ~55 MB ~= 167 GB) that
|
||||||
|
# hung/filled disk on the first pull, so zero rows ever landed and
|
||||||
|
# resolve_show had nothing to queue. We now insert immediately with url set
|
||||||
|
# and local_path NULL; liquidsoap streams the remote URL at air time
|
||||||
|
# (resolve_show already falls back to ep[:url] when local_path is absent).
|
||||||
new_eps = 0
|
new_eps = 0
|
||||||
|
skipped_existing = 0
|
||||||
parsed[:items].each do |ep|
|
parsed[:items].each do |ep|
|
||||||
|
# Dedup by guid: skip anything we've already recorded for this feed.
|
||||||
existing = $db_p[:episodes].where(guid: ep[:guid]).first
|
existing = $db_p[:episodes].where(guid: ep[:guid]).first
|
||||||
if existing
|
if existing
|
||||||
|
skipped_existing += 1
|
||||||
next
|
next
|
||||||
end
|
end
|
||||||
|
|
||||||
filename = "#{ep[:guid]}.mp3"
|
# Defensive: require a non-empty playback target. parse_feed only yields
|
||||||
dest = File.join(show_dir, filename)
|
# audio enclosures, but guard against a malformed item anyway.
|
||||||
dl_ok = http_stream_to_file(ep[:url], dest)
|
url = ep[:url].to_s.strip
|
||||||
size = dl_ok ? File.size(dest) : 0
|
next if url.empty?
|
||||||
|
|
||||||
now = Time.now.utc.strftime("%Y-%m-%d %H:%M:%S")
|
now = Time.now.utc.strftime("%Y-%m-%d %H:%M:%S")
|
||||||
|
begin
|
||||||
$db_p[:episodes].insert(
|
$db_p[:episodes].insert(
|
||||||
guid: ep[:guid],
|
guid: ep[:guid],
|
||||||
show_guid: show[:guid],
|
show_guid: show[:guid],
|
||||||
title: ep[:title],
|
title: ep[:title],
|
||||||
url: ep[:url],
|
url: url,
|
||||||
duration_seconds: ep[:duration_seconds],
|
duration_seconds: ep[:duration_seconds] || 0,
|
||||||
played: 0,
|
played: 0,
|
||||||
local_path: dest,
|
local_path: nil, # Option A: stream remotely; no local file
|
||||||
file_size_bytes: size,
|
file_size_bytes: ep[:file_size_bytes] || 0,
|
||||||
created_at: now
|
created_at: now
|
||||||
)
|
)
|
||||||
new_eps += 1
|
new_eps += 1
|
||||||
|
rescue StandardError => e
|
||||||
|
# Log per-item failures instead of letting one bad row abort the whole
|
||||||
|
# show (and silently lose every remaining episode).
|
||||||
|
log("WARN", "Insert failed for #{show[:slug]} ep #{ep[:guid]}: #{e.message}")
|
||||||
end
|
end
|
||||||
log("INFO", "#{show[:slug]}: #{new_eps} new episodes downloaded")
|
end
|
||||||
|
log("INFO", "#{show[:slug]}: #{new_eps} new episodes recorded (#{skipped_existing} already known)")
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -193,15 +193,18 @@ chmod 664 "$STORAGE_ROOT/state/"*.db 2>/dev/null || true
|
||||||
# --- Cron entries (liquidsoap user's crontab already exists from package) -----
|
# --- Cron entries (liquidsoap user's crontab already exists from package) -----
|
||||||
CRON_FETCH="0 * * * * cd ${INSTALL_DIR} && ${INSTALL_DIR}/run_radio.sh fetch_podcasts.rb --fetch-all >> ${STORAGE_ROOT}/logs/fetch_cron.log 2>&1"
|
CRON_FETCH="0 * * * * cd ${INSTALL_DIR} && ${INSTALL_DIR}/run_radio.sh fetch_podcasts.rb --fetch-all >> ${STORAGE_ROOT}/logs/fetch_cron.log 2>&1"
|
||||||
CRON_UPDATE="30 * * * * cd ${INSTALL_DIR} && ${INSTALL_DIR}/run_radio.sh update_playlists.rb >> ${STORAGE_ROOT}/logs/update_cron.log 2>&1"
|
CRON_UPDATE="30 * * * * cd ${INSTALL_DIR} && ${INSTALL_DIR}/run_radio.sh update_playlists.rb >> ${STORAGE_ROOT}/logs/update_cron.log 2>&1"
|
||||||
|
CRON_SCHED="* * * * * cd ${INSTALL_DIR} && ${INSTALL_DIR}/run_radio.sh scheduler.rb >> ${STORAGE_ROOT}/logs/scheduler_cron.log 2>&1"
|
||||||
CURRENT_CRONTAB=$(crontab -u liquidsoap -l 2>/dev/null || true)
|
CURRENT_CRONTAB=$(crontab -u liquidsoap -l 2>/dev/null || true)
|
||||||
NEW_CRONTAB="${CURRENT_CRONTAB}"
|
NEW_CRONTAB="${CURRENT_CRONTAB}"
|
||||||
if ! echo "${NEW_CRONTAB}" | grep -qF "fetch_podcasts.rb --fetch-all"; then
|
if ! echo "${NEW_CRONTAB}" | grep -qF "fetch_podcasts.rb --fetch-all"; then
|
||||||
NEW_CRONTAB="${NEW_CRONTAB}${NEW_CRONTAB:+$'\n'}${CRON_FETCH}"
|
NEW_CRONTAB="${NEW_CRONTAB}${NEW_CRONTAB:+$'\n'}${CRON_FETCH}"
|
||||||
fi
|
fi
|
||||||
if ! echo "${NEW_CRONTAB}" | grep -qF "update_playlist.rb"; then
|
if ! echo "${NEW_CRONTAB}" | grep -qF "update_playlists.rb"; then
|
||||||
NEW_CRONTAB="${NEW_CRONTAB}${NEW_CRONTAB:+$'\n'}${CRON_UPDATE}"
|
NEW_CRONTAB="${NEW_CRONTAB}${NEW_CRONTAB:+$'\n'}${CRON_UPDATE}"
|
||||||
fi
|
fi
|
||||||
|
if ! echo "${NEW_CRONTAB}" | grep -qF "scheduler.rb"; then
|
||||||
|
NEW_CRONTAB="${NEW_CRONTAB}${NEW_CRONTAB:+$'\n'}${CRON_SCHED}"
|
||||||
|
fi
|
||||||
echo "${NEW_CRONTAB}" | crontab -u liquidsoap -
|
echo "${NEW_CRONTAB}" | crontab -u liquidsoap -
|
||||||
echo "==> Cron entries installed for user liquidsoap"
|
echo "==> Cron entries installed for user liquidsoap"
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,313 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
# install_for_python - Radio automation installer for Linux Mint 22.3
|
|
||||||
#
|
|
||||||
# Installs the liquidsoap-based radio automation stack:
|
|
||||||
# - Creates the 'liquidsoap' system user
|
|
||||||
# - Prompts for storage path, Icecast credentials, gpodder.net credentials
|
|
||||||
# - Generates config.json from the answers
|
|
||||||
# - Sets up directory structure under <storage>/
|
|
||||||
# - Ensures SQLite databases exist with correct schema
|
|
||||||
# - Installs JRuby gems (sequel, jdbc-sqlite3, nokogiri, json) one at a time
|
|
||||||
# - Writes the self-locating run_radio.sh launcher
|
|
||||||
# - Installs crontab entries (as liquidsoap) for periodic fetch + update
|
|
||||||
# - Optionally enables the systemd service
|
|
||||||
#
|
|
||||||
# Usage: sudo ./install_for_python
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Derive service name from containing directory name
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
SERVICE_NAME="$(basename "$SCRIPT_DIR")"
|
|
||||||
SYSTEMD_UNIT="/etc/systemd/system/${SERVICE_NAME}.service"
|
|
||||||
|
|
||||||
echo "=== ${SERVICE_NAME} Installer ==="
|
|
||||||
echo
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Root check
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
if [[ $EUID -ne 0 ]]; then
|
|
||||||
echo "ERROR: Please run as root (sudo)." >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Create liquidsoap system user if missing
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
if ! id -u liquidsoap >/dev/null 2>&1; then
|
|
||||||
echo "Creating system user 'liquidsoap'..."
|
|
||||||
useradd --system --create-home --shell /usr/sbin/nologin liquidsoap
|
|
||||||
echo "Done."
|
|
||||||
else
|
|
||||||
echo "User 'liquidsoap' already exists."
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Prompt for configuration values
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
echo
|
|
||||||
read -rp "Station data directory [/mnt/storage/radio]: " STORAGE
|
|
||||||
STORAGE="${STORAGE:-/mnt/storage/radio}"
|
|
||||||
|
|
||||||
read -rp "Icecast host [192.168.0.200]: " ICECAST_HOST
|
|
||||||
ICECAST_HOST="${ICECAST_HOST:-192.168.0.200}"
|
|
||||||
|
|
||||||
read -rp "Icecast source port [7777]: " ICECAST_PORT
|
|
||||||
ICECAST_PORT="${ICECAST_PORT:-7777}"
|
|
||||||
|
|
||||||
read -rp "Icecast mount point [/data]: " ICECAST_MOUNT
|
|
||||||
ICECAST_MOUNT="${ICECAST_MOUNT:-/data}"
|
|
||||||
|
|
||||||
read -rp "Icecast source username [source]: " SOURCE_USER
|
|
||||||
SOURCE_USER="${SOURCE_USER:-source}"
|
|
||||||
|
|
||||||
read -rsp "Icecast source password: " SOURCE_PASS
|
|
||||||
echo
|
|
||||||
|
|
||||||
GPODDER_ENABLE=""
|
|
||||||
until [[ "$GPODDER_ENABLE" =~ ^[YyNn]$ ]]; do
|
|
||||||
read -rp "Enable gPodder.net sync? [y/N]: " GPODDER_ENABLE
|
|
||||||
done
|
|
||||||
GPODDER_ENABLED=false
|
|
||||||
[[ "$GPODDER_ENABLE" =~ ^[Yy]$ ]] && GPODDER_ENABLED=true
|
|
||||||
|
|
||||||
GPODDER_HOST="gpodder.net"
|
|
||||||
GPODDER_USER=""
|
|
||||||
GPODDER_PASS=""
|
|
||||||
if [[ "$GPODDER_ENABLED" == true ]]; then
|
|
||||||
read -rp "gPodder.net host [gpodder.net]: " GPODDER_HOST_IN
|
|
||||||
GPODDER_HOST="${GPODDER_HOST_IN:-gpodder.net}"
|
|
||||||
read -rp "gPodder.net username: " GPODDER_USER
|
|
||||||
read -rsp "gPodder.net password: " GPODDER_PASS
|
|
||||||
echo
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Create directory structure
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
echo
|
|
||||||
echo "Creating directory structure under ${STORAGE} ..."
|
|
||||||
mkdir -p "${STORAGE}"/{state,podcasts,music,jingles,announcements,playlists,logs}
|
|
||||||
chown -R liquidsoap:liquidsoap "${STORAGE}"
|
|
||||||
chmod -R u+rwX,g+rwX,o+rX "${STORAGE}"
|
|
||||||
echo "Done."
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Generate config.json
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
CONFIG_FILE="${SCRIPT_DIR}/config.json"
|
|
||||||
cat > "$CONFIG_FILE" <<EOF
|
|
||||||
{
|
|
||||||
"storage": "${STORAGE}",
|
|
||||||
"icecast": {
|
|
||||||
"host": "${ICECAST_HOST}",
|
|
||||||
"port": ${ICECAST_PORT},
|
|
||||||
"mount": "${ICECAST_MOUNT}",
|
|
||||||
"source_username": "${SOURCE_USER}",
|
|
||||||
"source_password": "${SOURCE_PASS}"
|
|
||||||
},
|
|
||||||
"gpodder": {
|
|
||||||
"enable": ${GPODDER_ENABLED},
|
|
||||||
"host": "https://${GPODDER_HOST}",
|
|
||||||
"username": "${GPODDER_USER}",
|
|
||||||
"password": "${GPODDER_PASS}"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
EOF
|
|
||||||
chown liquidsoap:liquidsoap "$CONFIG_FILE"
|
|
||||||
chmod 640 "$CONFIG_FILE"
|
|
||||||
echo "Wrote ${CONFIG_FILE}"
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Ensure SQLite databases exist with correct schema
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
STATE_DIR="${STORAGE}/state"
|
|
||||||
SUBS_DB="${STATE_DIR}/subscriptions.db"
|
|
||||||
PLAYED_DB="${STATE_DIR}/played.db"
|
|
||||||
|
|
||||||
ensure_schema() {
|
|
||||||
local db="$1" table_sql="$2" index_sql="$3"
|
|
||||||
sqlite3 "$db" <<<"$table_sql"
|
|
||||||
[[ -n "$index_sql" ]] && sqlite3 "$db" <<<"$index_sql"
|
|
||||||
sqlite3 "$db" "PRAGMA journal_mode=WAL;"
|
|
||||||
}
|
|
||||||
|
|
||||||
SHOWS_SQL='CREATE TABLE IF NOT EXISTS shows (
|
|
||||||
slug TEXT PRIMARY KEY,
|
|
||||||
guid TEXT NOT NULL UNIQUE,
|
|
||||||
name TEXT NOT NULL,
|
|
||||||
feed_url TEXT NOT NULL UNIQUE,
|
|
||||||
source TEXT DEFAULT '"'"'manual'"'"',
|
|
||||||
opml_import INTEGER DEFAULT 0,
|
|
||||||
archived INTEGER DEFAULT 1,
|
|
||||||
media_class TEXT,
|
|
||||||
created_at TEXT
|
|
||||||
);'
|
|
||||||
|
|
||||||
EPISODES_SQL='CREATE TABLE IF NOT EXISTS episodes (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
show_slug TEXT NOT NULL,
|
|
||||||
guid TEXT NOT NULL,
|
|
||||||
title TEXT,
|
|
||||||
file_path TEXT,
|
|
||||||
enclosure_url TEXT,
|
|
||||||
runlength INTEGER,
|
|
||||||
played INTEGER DEFAULT 0,
|
|
||||||
played_at TEXT,
|
|
||||||
UNIQUE (show_slug, guid)
|
|
||||||
);'
|
|
||||||
|
|
||||||
INDEX_EPISODES_SQL='CREATE INDEX IF NOT EXISTS idx_episodes_show_played ON episodes (show_slug, played);'
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "Initializing SQLite databases..."
|
|
||||||
ensure_schema "$SUBS_DB" "$SHOWS_SQL" ""
|
|
||||||
ensure_schema "$PLAYED_DB" "$EPISODES_SQL" "$INDEX_EPISODES_SQL"
|
|
||||||
chown liquidsoap:liquidsoap "$SUBS_DB" "$PLAYED_DB"
|
|
||||||
echo "Done."
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Install JRuby runtime dependencies (one gem per command to bound memory)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
JRUBY_BIN="/opt/jruby/bin/jruby"
|
|
||||||
GEM_BIN="/opt/jruby/bin/gem"
|
|
||||||
GEMS_HOME="${SCRIPT_DIR}/.gems"
|
|
||||||
|
|
||||||
if [[ -x "$JRUBY_BIN" ]]; then
|
|
||||||
echo
|
|
||||||
echo "Installing JRuby gem dependencies into ${GEMS_HOME} ..."
|
|
||||||
mkdir -p "$GEMS_HOME"
|
|
||||||
chown liquidsoap:liquidsoap "$GEMS_HOME"
|
|
||||||
|
|
||||||
install_gem() {
|
|
||||||
local gem_name="$1"
|
|
||||||
shift
|
|
||||||
local extra_args=("$@")
|
|
||||||
echo " -> gem install ${gem_name} ${extra_args[*]}"
|
|
||||||
sudo -u liquidsoap bash -c "
|
|
||||||
export GEM_HOME=${GEMS_HOME}
|
|
||||||
export GEM_PATH=${GEMS_HOME}:/opt/jruby/lib/ruby/gems/shared
|
|
||||||
${GEM_BIN} install ${gem_name} ${extra_args[*]} --no-document
|
|
||||||
" || echo " WARNING: failed to install ${gem_name}"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Core persistence
|
|
||||||
install_gem "sequel"
|
|
||||||
install_gem "jdbc-sqlite3"
|
|
||||||
|
|
||||||
# XML parsing. Pure-Java build under JRuby (--platform java) bundles
|
|
||||||
# Xerces/NekoHTML/Xalan as JARs; no libxml2-dev or compiler required.
|
|
||||||
install_gem "nokogiri" "--platform" "java"
|
|
||||||
|
|
||||||
# Convenience JSON (usually stdlib, but ensure availability)
|
|
||||||
install_gem "json"
|
|
||||||
|
|
||||||
echo "JRUBY DEPENDENCIES COMPLETE"
|
|
||||||
else
|
|
||||||
echo
|
|
||||||
echo "WARNING: ${JRUBY_BIN} not found; skipping JRuby gem installation."
|
|
||||||
echo " Install JRuby manually, then re-run this installer or run:"
|
|
||||||
echo " sudo -u liquidsoap ${GEM_BIN} install sequel jdbc-sqlite3 json"
|
|
||||||
echo " sudo -u liquidsoap ${GEM_BIN} install nokogiri --platform java"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Self-locating launcher (sets GEM_HOME/GEM_PATH before JRuby boots)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
LAUNCHER="${SCRIPT_DIR}/run_radio.sh"
|
|
||||||
cat > "$LAUNCHER" <<'LAUNCH_EOF'
|
|
||||||
#!/usr/bin/env bash
|
|
||||||
# run_radio.sh - self-locating JRuby launcher for the radio automation scripts.
|
|
||||||
# Exports GEM_HOME/GEM_PATH BEFORE jruby boots, because mutating them inside a
|
|
||||||
# running JRuby process does not reliably affect gem resolution (JRuby #5269).
|
|
||||||
#
|
|
||||||
# Usage: ./run_radio.sh <script.rb> [args...]
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
SELF_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
GEMS_HOME="${SELF_DIR}/.gems"
|
|
||||||
JRUBY_HOME="/opt/jruby"
|
|
||||||
|
|
||||||
export GEM_HOME="${GEMS_HOME}"
|
|
||||||
export GEM_PATH="${GEMS_HOME}:${JRUBY_HOME}/lib/ruby/gems/shared"
|
|
||||||
|
|
||||||
exec "${JRUBY_HOME}/bin/jruby" "${SELF_DIR}/$@"
|
|
||||||
LAUNCH_EOF
|
|
||||||
chmod +x "$LAUNCHER"
|
|
||||||
chown liquidsoap:liquidsoap "$LAUNCHER"
|
|
||||||
echo "Wrote ${LAUNCHER}"
|
|
||||||
|
|
||||||
# Make the Ruby scripts executable
|
|
||||||
for rb in "${SCRIPT_DIR}"/*.rb; do
|
|
||||||
[[ -f "$rb" ]] && chmod +x "$rb" && chown liquidsoap:liquidsoap "$rb"
|
|
||||||
done
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Crontab entries (installed into liquidsoap's crontab, not root's)
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
CRON_FETCH="*/30 * * * * cd ${SCRIPT_DIR} && ./run_radio.sh fetch_podcasts.rb >> ${STORAGE}/logs/cron_fetch.log 2>&1"
|
|
||||||
CRON_UPDATE="15 * * * * cd ${SCRIPT_DIR} && ./run_radio.sh update_playlists.rb >> ${STORAGE}/logs/cron_update.log 2>&1"
|
|
||||||
|
|
||||||
echo
|
|
||||||
echo "Installing crontab entries for user 'liquidsoap'..."
|
|
||||||
( crontab -l -u liquidsoap 2>/dev/null || true ) | grep -vF "run_radio.sh" | \
|
|
||||||
{ cat; echo "$CRON_FETCH"; echo "$CRON_UPDATE"; } | crontab -u liquidsoap -
|
|
||||||
echo "Crontab updated."
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Optional systemd service
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
ENABLE_SERVICE=""
|
|
||||||
until [[ "$ENABLE_SERVICE" =~ ^[YyNn]$ ]]; do
|
|
||||||
read -rp "Enable ${SERVICE_NAME} systemd service now? [y/N]: " ENABLE_SERVICE
|
|
||||||
done
|
|
||||||
|
|
||||||
if [[ "$ENABLE_SERVICE" =~ ^[Yy]$ ]]; then
|
|
||||||
echo
|
|
||||||
echo "Writing ${SYSTEMD_UNIT} ..."
|
|
||||||
cat > "$SYSTEMD_UNIT" <<UNIT_EOF
|
|
||||||
[Unit]
|
|
||||||
Description=${SERVICE_NAME} radio automation (liquidsoap)
|
|
||||||
After=network-online.target icecast2.service
|
|
||||||
Wants=network-online.target
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=simple
|
|
||||||
User=liquidsoap
|
|
||||||
WorkingDirectory=${SCRIPT_DIR}
|
|
||||||
ExecStart=${SCRIPT_DIR}/station.liq
|
|
||||||
Restart=on-failure
|
|
||||||
RestartSec=10
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=multi-user.target
|
|
||||||
UNIT_EOF
|
|
||||||
|
|
||||||
systemctl daemon-reload
|
|
||||||
systemctl enable --now "${SERVICE_NAME}"
|
|
||||||
echo "Service ${SERVICE_NAME} enabled and started."
|
|
||||||
else
|
|
||||||
echo "Skipping systemd service (you can enable it later with:"
|
|
||||||
echo " sudo cp ${SCRIPT_DIR}/${SERVICE_NAME}.service /etc/systemd/system/ && sudo systemctl enable --now ${SERVICE_NAME})"
|
|
||||||
fi
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# Summary
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
echo
|
|
||||||
echo "=============================================="
|
|
||||||
echo " Installation complete."
|
|
||||||
echo "----------------------------------------------"
|
|
||||||
echo " Storage: ${STORAGE}"
|
|
||||||
echo " Config: ${CONFIG_FILE}"
|
|
||||||
echo " Launcher: ${LAUNCHER}"
|
|
||||||
echo " Service: ${SERVICE_NAME}"
|
|
||||||
echo "----------------------------------------------"
|
|
||||||
echo " Quick start:"
|
|
||||||
echo " cd ${SCRIPT_DIR}"
|
|
||||||
echo " sudo -u liquidsoap ./run_radio.sh fetch_podcasts.rb --list-shows"
|
|
||||||
echo " sudo -u liquidsoap ./run_radio.sh fetch_podcasts.rb --add-show <rss-url>"
|
|
||||||
echo " sudo -u liquidsoap ./run_radio.sh update_playlists.rb"
|
|
||||||
echo "=============================================="
|
|
||||||
|
|
@ -25,6 +25,9 @@ JRUBY_BIN="/usr/bin/jruby"
|
||||||
export GEM_HOME="${GEMS_DIR}"
|
export GEM_HOME="${GEMS_DIR}"
|
||||||
export GEM_PATH="${GEMS_DIR}"
|
export GEM_PATH="${GEMS_DIR}"
|
||||||
|
|
||||||
|
RADIO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
export RADIO_DIR
|
||||||
|
|
||||||
if [[ $# -lt 1 ]]; then
|
if [[ $# -lt 1 ]]; then
|
||||||
echo "Usage: $(basename "$0") <script.rb> [args...]" >&2
|
echo "Usage: $(basename "$0") <script.rb> [args...]" >&2
|
||||||
exit 1
|
exit 1
|
||||||
|
|
|
||||||
|
|
@ -22,3 +22,6 @@
|
||||||
0 17 * * 1-5 stream https://streams.kpfa.org:8443/kpfa 3600
|
0 17 * * 1-5 stream https://streams.kpfa.org:8443/kpfa 3600
|
||||||
# Evening News KPFA
|
# Evening News KPFA
|
||||||
0 18 * * 1-5 stream https://streams.kpfa.org:8443/kpfa 3600
|
0 18 * * 1-5 stream https://streams.kpfa.org:8443/kpfa 3600
|
||||||
|
# test
|
||||||
|
# 50 15 * * 1-7 stream https://streams.kpfa.org:8443/kpfa 3600
|
||||||
|
|
||||||
|
|
|
||||||
278
scheduler.rb
278
scheduler.rb
|
|
@ -1,4 +1,4 @@
|
||||||
#!/usr/bin/env ruby
|
#!/usr/bin/env jruby
|
||||||
# scheduler.rb - Clock-driven show/stream scheduler for the radio automation.
|
# scheduler.rb - Clock-driven show/stream scheduler for the radio automation.
|
||||||
#
|
#
|
||||||
# Designed to run every minute from cron as the liquidsoap user:
|
# Designed to run every minute from cron as the liquidsoap user:
|
||||||
|
|
@ -9,13 +9,17 @@
|
||||||
# <storage>/queues/current_queue.txt so station.liq picks it up within ~2s.
|
# <storage>/queues/current_queue.txt so station.liq picks it up within ~2s.
|
||||||
#
|
#
|
||||||
# Resolution rules:
|
# Resolution rules:
|
||||||
# stream -> the URL is written verbatim. RUNLENGTH (seconds) is carried in a
|
# stream -> ffmpeg captures the live URL to a finite MP3 in captures/ for
|
||||||
# comment line; liquidsoap plays the live stream until the next
|
# RUNLENGTH seconds (default 3600) and that LOCAL path is queued.
|
||||||
# scheduled event overwrites the file.
|
|
||||||
# show -> look up the slug in subscriptions.db, find the next unplayed
|
# show -> look up the slug in subscriptions.db, find the next unplayed
|
||||||
# episode in played.db (prefer local_path, fall back to url),
|
# episode in played.db (prefer local_path, fall back to url),
|
||||||
# mark it played, and write that path/url.
|
# mark it played, and write that path/url.
|
||||||
#
|
#
|
||||||
|
# Expiry: each queued item gets an end time in queues/current_queue.meta
|
||||||
|
# (RUNLENGTH for shows, capped at 4h without one; RUNLENGTH+300s for stream
|
||||||
|
# captures). Every cron tick clears current_queue.txt once its item has
|
||||||
|
# expired, so background music resumes and nothing loops.
|
||||||
|
#
|
||||||
# Idempotent: a fire marker prevents re-firing the same entry twice in one
|
# Idempotent: a fire marker prevents re-firing the same entry twice in one
|
||||||
# minute if cron double-runs or the script is invoked manually.
|
# minute if cron double-runs or the script is invoked manually.
|
||||||
|
|
||||||
|
|
@ -23,13 +27,11 @@ require 'json'
|
||||||
require 'sequel'
|
require 'sequel'
|
||||||
require 'fileutils'
|
require 'fileutils'
|
||||||
require 'time'
|
require 'time'
|
||||||
|
require 'uri'
|
||||||
|
|
||||||
SCRIPT_DIR = File.expand_path(File.dirname(__FILE__))
|
SCRIPT_DIR = File.expand_path(File.dirname(__FILE__))
|
||||||
CONFIG_PATH = File.join(SCRIPT_DIR, "config.json")
|
CONFIG_PATH = File.join(SCRIPT_DIR, "config.json")
|
||||||
SCHEDULE_FILE = File.join(SCRIPT_DIR, "schedule.txt")
|
SCHEDULE_FILE = File.join(SCRIPT_DIR, "schedule.txt")
|
||||||
QUEUE_DIR = nil # derived below
|
|
||||||
QUEUE_FILE = nil # derived below
|
|
||||||
FIRE_MARKER_DIR = nil
|
|
||||||
|
|
||||||
CFG = JSON.parse(File.read(CONFIG_PATH))
|
CFG = JSON.parse(File.read(CONFIG_PATH))
|
||||||
STORAGE = CFG["storage"]
|
STORAGE = CFG["storage"]
|
||||||
|
|
@ -39,11 +41,32 @@ PLAYED_DB = File.join(STATE_DIR, "played.db")
|
||||||
LOG_DIR = File.join(STORAGE, "logs")
|
LOG_DIR = File.join(STORAGE, "logs")
|
||||||
|
|
||||||
QUEUE_DIR = File.join(STORAGE, "queues")
|
QUEUE_DIR = File.join(STORAGE, "queues")
|
||||||
|
# Single queue file, consumed by one request.dynamic in station.liq. Both show
|
||||||
|
# and stream entries write their URI here; whichever is due overwrites it.
|
||||||
QUEUE_FILE = File.join(QUEUE_DIR, "current_queue.txt")
|
QUEUE_FILE = File.join(QUEUE_DIR, "current_queue.txt")
|
||||||
|
# Sidecar for QUEUE_FILE: a single epoch integer = the wall-clock second at
|
||||||
|
# which the queued item stops airing. Written whenever a URI is queued with a
|
||||||
|
# TTL, deleted whenever the queue is cleared. Each cron tick checks it and
|
||||||
|
# clears the queue once expired, so a finished episode never replays and
|
||||||
|
# background music resumes on its own.
|
||||||
|
QUEUE_META = File.join(QUEUE_DIR, "current_queue.meta")
|
||||||
FIRE_MARKER_DIR = File.join(STATE_DIR, "fire_markers")
|
FIRE_MARKER_DIR = File.join(STATE_DIR, "fire_markers")
|
||||||
|
# Live-capture output dir: ffmpeg writes scheduled stream slots here as finite
|
||||||
|
# MP3s that station.liq plays like any other local file.
|
||||||
|
CAPTURE_DIR = File.join(STORAGE, "captures")
|
||||||
|
|
||||||
|
# Create dirs and make them world-writable-ish (0775) so the cron-run process
|
||||||
|
# can write into them even though the installer's chown ran before queues/ existed.
|
||||||
FileUtils.mkdir_p(QUEUE_DIR)
|
FileUtils.mkdir_p(QUEUE_DIR)
|
||||||
FileUtils.mkdir_p(FIRE_MARKER_DIR)
|
FileUtils.mkdir_p(FIRE_MARKER_DIR)
|
||||||
|
FileUtils.mkdir_p(CAPTURE_DIR)
|
||||||
|
begin
|
||||||
|
FileUtils.chmod(0o775, QUEUE_DIR)
|
||||||
|
FileUtils.chmod(0o775, FIRE_MARKER_DIR)
|
||||||
|
FileUtils.chmod(0o775, CAPTURE_DIR)
|
||||||
|
rescue => e
|
||||||
|
log("WARN: could not chmod queue/fire-marker/capture dirs: #{e.message}")
|
||||||
|
end
|
||||||
|
|
||||||
def log(msg)
|
def log(msg)
|
||||||
puts "#{Time.now.strftime('%Y-%m-%d %H:%M:%S')} #{msg}"
|
puts "#{Time.now.strftime('%Y-%m-%d %H:%M:%S')} #{msg}"
|
||||||
|
|
@ -142,12 +165,23 @@ end
|
||||||
|
|
||||||
# --- Database helpers ----------------------------------------------------------
|
# --- Database helpers ----------------------------------------------------------
|
||||||
|
|
||||||
|
# JRuby: Sequel.sqlite() loads the CRuby-only 'sqlite3' gem and raises
|
||||||
|
# AdapterNotFound. Use the JDBC form exactly as fetch_podcasts.rb does
|
||||||
|
# (verified-working there), and set PRAGMAs via Database#execute.
|
||||||
def open_subs_db
|
def open_subs_db
|
||||||
Sequel.sqlite(SUBS_DB, database_options: { timeout: 5 })
|
db = Sequel.connect("jdbc:sqlite:" + SUBS_DB)
|
||||||
|
db.execute("PRAGMA journal_mode=WAL;")
|
||||||
|
db.execute("PRAGMA busy_timeout=5000;")
|
||||||
|
db.execute("PRAGMA foreign_keys=OFF;")
|
||||||
|
db
|
||||||
end
|
end
|
||||||
|
|
||||||
def open_played_db
|
def open_played_db
|
||||||
Sequel.sqlite(PLAYED_DB, database_options: { timeout: 5 })
|
db = Sequel.connect("jdbc:sqlite:" + PLAYED_DB)
|
||||||
|
db.execute("PRAGMA journal_mode=WAL;")
|
||||||
|
db.execute("PRAGMA busy_timeout=5000;")
|
||||||
|
db.execute("PRAGMA foreign_keys=OFF;")
|
||||||
|
db
|
||||||
end
|
end
|
||||||
|
|
||||||
# Resolve a show slug to the next unplayed episode's playable path/url.
|
# Resolve a show slug to the next unplayed episode's playable path/url.
|
||||||
|
|
@ -162,16 +196,26 @@ def resolve_show(slug)
|
||||||
return nil
|
return nil
|
||||||
end
|
end
|
||||||
|
|
||||||
ep = played_db[:episodes]
|
total_eps = played_db[:episodes].where(show_guid: show[:guid]).count
|
||||||
|
unplayed_ep = played_db[:episodes]
|
||||||
.where(show_guid: show[:guid], played: 0)
|
.where(show_guid: show[:guid], played: 0)
|
||||||
.order(Sequel.desc(:created_at))
|
.order(Sequel.desc(:created_at))
|
||||||
.first
|
.first
|
||||||
|
|
||||||
if ep.nil?
|
if unplayed_ep.nil?
|
||||||
log("INFO: no unplayed episodes for show '#{slug}' (#{show[:title]})")
|
# Distinguish the two failure modes so a silent "no audio" is diagnosable:
|
||||||
|
# * zero episodes at all -> fetch never populated this show's feed
|
||||||
|
# * episodes but all played -> archive-cycle reset isn't running
|
||||||
|
if total_eps.zero?
|
||||||
|
log("WARN: show '#{slug}' (#{show[:title]}) has ZERO episodes in played.db; run fetch_podcasts.rb --fetch-all to populate it")
|
||||||
|
else
|
||||||
|
log("WARN: show '#{slug}' (#{show[:title]}) has #{total_eps} episode(s) but ALL are marked played; nothing to queue until new episodes arrive or flags reset")
|
||||||
|
end
|
||||||
return nil
|
return nil
|
||||||
end
|
end
|
||||||
|
|
||||||
|
ep = unplayed_ep
|
||||||
|
|
||||||
playable = ep[:local_path] || ep[:url]
|
playable = ep[:local_path] || ep[:url]
|
||||||
if playable.nil? || playable.to_s.strip.empty?
|
if playable.nil? || playable.to_s.strip.empty?
|
||||||
log("WARN: episode #{ep[:guid]} for '#{slug}' has neither local_path nor url")
|
log("WARN: episode #{ep[:guid]} for '#{slug}' has neither local_path nor url")
|
||||||
|
|
@ -188,26 +232,207 @@ ensure
|
||||||
played_db&.disconnect
|
played_db&.disconnect
|
||||||
end
|
end
|
||||||
|
|
||||||
# --- Queue writer --------------------------------------------------------------
|
# --- Queue writer ------------------------------------------------------------
|
||||||
# Writes the URI (and optionally a runlength hint) atomically: write to a temp
|
# The single queue file must contain ONLY the bare URI, nothing else. station.liq
|
||||||
# file in the same directory, then rename over the target. This prevents
|
# trims whitespace before handing it to request.create, but a second line would
|
||||||
# station.liq from reading a partially-written file.
|
# still corrupt the URI, so we keep it to exactly one token.
|
||||||
|
# Atomic write: temp file in the same dir, then rename over the target so
|
||||||
|
# station.liq never reads a partially-written file.
|
||||||
|
|
||||||
def write_queue(uri, runlength_sec = nil)
|
# ttl_sec: when > 0, stamp QUEUE_META with an expiry time (now + ttl_sec) so a
|
||||||
tmp = "#{QUEUE_FILE}.tmp"
|
# later cron tick clears the queue once the slot is over. When nil/0, leave any
|
||||||
content = uri
|
# existing meta untouched (the queue keeps whatever expiry it already had).
|
||||||
if runlength_sec && runlength_sec > 0
|
def write_uri_file(path, uri, ttl_sec = nil)
|
||||||
content += "\n# runlength=#{runlength_sec}"
|
tmp = "#{path}.tmp"
|
||||||
|
begin
|
||||||
|
File.write(tmp, uri)
|
||||||
|
File.rename(tmp, path)
|
||||||
|
rescue Errno::EACCES => e
|
||||||
|
log("WARN: atomic write to #{path} blocked (#{e.message}); falling back to direct write")
|
||||||
|
File.write(path, uri)
|
||||||
|
File.chmod(0o664, path) if File.exist?(path)
|
||||||
|
ensure
|
||||||
|
File.delete(tmp) if File.exist?(tmp)
|
||||||
|
end
|
||||||
|
if ttl_sec && ttl_sec.to_i > 0
|
||||||
|
write_queue_meta(Time.now.to_i + ttl_sec.to_i)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# Best-effort: expiry bookkeeping must never block a fire.
|
||||||
|
def write_queue_meta(epoch)
|
||||||
|
tmp = "#{QUEUE_META}.tmp"
|
||||||
|
begin
|
||||||
|
File.write(tmp, epoch.to_i.to_s)
|
||||||
|
File.rename(tmp, QUEUE_META)
|
||||||
|
rescue => e
|
||||||
|
log("WARN: could not write queue expiry meta: #{e.message}; queue will persist until next overwrite")
|
||||||
|
ensure
|
||||||
|
begin
|
||||||
|
File.delete(tmp) if File.exist?(tmp)
|
||||||
|
rescue StandardError
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# Returns the queue expiry as a Time, or nil when no (valid) meta exists.
|
||||||
|
def queue_expiry
|
||||||
|
return nil unless File.exist?(QUEUE_META)
|
||||||
|
raw = File.read(QUEUE_META).strip
|
||||||
|
if raw.match?(/\A\d+\z/)
|
||||||
|
Time.at(raw.to_i)
|
||||||
|
else
|
||||||
|
Time.parse(raw)
|
||||||
|
end
|
||||||
|
rescue ArgumentError
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
|
||||||
|
# Lazy expiry check, run at the top of every cron tick: if the queued item is
|
||||||
|
# past its end time, clear the queue. station.liq's request.dynamic then goes
|
||||||
|
# fallible and the fallback drops back to background music within ~2s.
|
||||||
|
# Worst-case overshoot is one cron minute.
|
||||||
|
def expire_queue_if_due(now)
|
||||||
|
exp = queue_expiry
|
||||||
|
return if exp.nil?
|
||||||
|
return if now.to_i < exp.to_i
|
||||||
|
log("EXPIRED: queued item past its end time (#{exp.strftime('%Y-%m-%d %H:%M:%S')}); clearing queue -> background music")
|
||||||
|
clear_uri_file(QUEUE_FILE)
|
||||||
|
end
|
||||||
|
|
||||||
|
def clear_uri_file(path)
|
||||||
|
begin
|
||||||
|
File.delete(QUEUE_META) if File.exist?(QUEUE_META)
|
||||||
|
rescue => e
|
||||||
|
log("WARN: could not delete #{QUEUE_META}: #{e.message}")
|
||||||
|
end
|
||||||
|
return unless File.exist?(path)
|
||||||
|
begin
|
||||||
|
File.write("#{path}.empty", "")
|
||||||
|
File.rename("#{path}.empty", path)
|
||||||
|
rescue => e
|
||||||
|
log("WARN: could not clear #{path}: #{e.message}")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# Fire a SHOW entry: put the episode path/url into the single queue file.
|
||||||
|
# runlength_sec (from schedule.txt, when given) becomes the queue TTL so the
|
||||||
|
# entry self-expires at the end of its slot; without one we cap at 4 hours so
|
||||||
|
# even an unbounded show eventually hands back to background music.
|
||||||
|
def fire_show(playable, runlength_sec = nil)
|
||||||
|
ttl = (runlength_sec && runlength_sec > 0) ? runlength_sec : 4 * 3600
|
||||||
|
write_uri_file(QUEUE_FILE, playable, ttl)
|
||||||
|
log("QUEUED show: #{playable} (expires in #{ttl}s)")
|
||||||
|
end
|
||||||
|
|
||||||
|
# Build a deterministic capture path from the stream URL + fire time so we can
|
||||||
|
# tell captures apart and avoid clobbering a still-playing one.
|
||||||
|
def capture_path_for(url, now)
|
||||||
|
safe_host = URI.parse(url).host.to_s.gsub(/[^a-zA-Z0-9.-]/, "_")
|
||||||
|
ts = now.strftime("%Y%m%d_%H%M%S")
|
||||||
|
File.join(CAPTURE_DIR, "#{safe_host}_#{ts}.mp3")
|
||||||
|
end
|
||||||
|
|
||||||
|
# Prune captured MP3s older than 2 days so the dir doesn't grow unbounded.
|
||||||
|
def prune_old_captures(now)
|
||||||
|
cutoff = now - 172_800 # 2 days
|
||||||
|
Dir.glob(File.join(CAPTURE_DIR, "*.mp3")).each do |f|
|
||||||
|
begin
|
||||||
|
File.delete(f) if File.mtime(f) < cutoff
|
||||||
|
rescue => e
|
||||||
|
log("WARN: could not prune capture #{f}: #{e.message}")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# Fire a STREAM entry by capturing it locally with ffmpeg, then queuing the
|
||||||
|
# LOCAL PATH (not the live URL). This sidesteps liquidsoap's inability to relay
|
||||||
|
# an infinite Icecast mount: station.liq just plays a finite MP3 like any show.
|
||||||
|
# * ffmpeg runs in the background, stops after RUNLENGTH seconds (-t).
|
||||||
|
# * If no runlength is given, default to 3600s (1h) so the capture is bounded.
|
||||||
|
# * We wait ~5s for ffmpeg to actually start writing before queuing, so
|
||||||
|
# station.liq picks up a real (non-empty) file. If ffmpeg fails to start,
|
||||||
|
# we fall back to queuing nothing (background music carries the slot).
|
||||||
|
def fire_stream(url, runlength_sec = nil)
|
||||||
|
dur = (runlength_sec && runlength_sec > 0) ? runlength_sec : 3600
|
||||||
|
cap = capture_path_for(url, Time.now)
|
||||||
|
|
||||||
|
# JRuby's native spawn launcher does NOT accept an Array command combined
|
||||||
|
# with a redirect hash (raises "wrong first argument"), and it does not
|
||||||
|
# reliably accept open IO objects as redirect values ("wrong exec redirect").
|
||||||
|
# The documented-working form on JRuby is: a SINGLE shell String for the
|
||||||
|
# command, plus plain FILENAME strings for :out/:err. We quote every path
|
||||||
|
# and the URL so shell word-splitting can't break them.
|
||||||
|
sh_cmd = %(ffmpeg -hide_banner -loglevel error -i "#{url}" -t #{dur} -acodec libmp3lame -b:a 128k -y "#{cap}")
|
||||||
|
fflog = "#{cap}.fflog"
|
||||||
|
pid = spawn(sh_cmd, out: fflog, err: fflog)
|
||||||
|
log("STREAM CAPTURE started: #{url} -> #{cap} (pid=#{pid}, dur=#{dur}s)")
|
||||||
|
|
||||||
|
# Give ffmpeg a moment to open the input and start writing. We judge success
|
||||||
|
# purely by whether the capture file has real bytes in it -- that's the actual
|
||||||
|
# signal station.liq needs. (Process.waitpid2 with WNOHANG returns nil on
|
||||||
|
# JRuby for a freshly spawned child, so we don't use it for liveness.)
|
||||||
|
sleep 5
|
||||||
|
size = File.exist?(cap) ? File.size(cap) : 0
|
||||||
|
|
||||||
|
if size > 0
|
||||||
|
# TTL = capture length + 5 min slack: ffmpeg stops writing after dur, and
|
||||||
|
# the queue self-clears shortly after the capture has finished playing.
|
||||||
|
write_uri_file(QUEUE_FILE, cap, dur + 300)
|
||||||
|
log("QUEUED stream capture: #{cap} (size=#{size} bytes so far, expires in #{dur + 300}s)")
|
||||||
|
else
|
||||||
|
# ffmpeg wrote nothing within 5s -- clear the queue so background music
|
||||||
|
# carries the slot instead of liquidsoap replaying a stale/broken URI.
|
||||||
|
log("WARN: capture not ready (size=#{size}); clearing queue -> background music")
|
||||||
|
clear_uri_file(QUEUE_FILE)
|
||||||
|
ff_err = File.read("#{cap}.fflog") rescue "<no log>"
|
||||||
|
log("ffmpeg stderr: #{ff_err.strip[0, 400]}")
|
||||||
end
|
end
|
||||||
File.write(tmp, content)
|
|
||||||
File.rename(tmp, QUEUE_FILE)
|
|
||||||
log("QUEUED: #{uri}#{runlength_sec ? " (runlength=#{runlength_sec}s)" : ''}")
|
|
||||||
end
|
end
|
||||||
|
|
||||||
# --- Main ----------------------------------------------------------------------
|
# --- Main ----------------------------------------------------------------------
|
||||||
|
|
||||||
def main
|
def main
|
||||||
now = Time.now
|
now = Time.now
|
||||||
|
|
||||||
|
# Lazy expiry first: whatever was queued earlier stops airing at its slot
|
||||||
|
# end (or the 4h show cap / capture+5min stream cap) without needing a new
|
||||||
|
# scheduled event to overwrite it.
|
||||||
|
expire_queue_if_due(now)
|
||||||
|
|
||||||
|
# Dev hook: --dryfire-stream URL [SECS] fires one stream capture immediately,
|
||||||
|
# bypassing the clock match. Lets us test the ffmpeg-capture path on demand
|
||||||
|
# instead of waiting for a scheduled minute. Example:
|
||||||
|
# ./run_radio.sh scheduler.rb --dryfire-stream "https://streams.kpfa.org:8443/kpfa" 300
|
||||||
|
if ARGV[0] == "--dryfire-stream"
|
||||||
|
url = ARGV[1]
|
||||||
|
secs = ARGV[2]&.to_i
|
||||||
|
raise ArgumentError, "--dryfire-stream requires a URL" if url.nil? || url.empty?
|
||||||
|
log("[dryfire] triggering stream capture now: #{url} (#{secs || 3600}s)")
|
||||||
|
fire_stream(url, secs)
|
||||||
|
log("[dryfire] done.")
|
||||||
|
return
|
||||||
|
end
|
||||||
|
|
||||||
|
# Dev hook: --dryfire-show SLUG runs the exact same resolve_show -> fire_show
|
||||||
|
# path a scheduled "show" entry uses, but immediately, without waiting for the
|
||||||
|
# clock match. Marks the chosen episode played=1 just like a real fire.
|
||||||
|
# ./run_radio.sh scheduler.rb --dryfire-show the_history_of_rome
|
||||||
|
if ARGV[0] == "--dryfire-show"
|
||||||
|
slug = ARGV[1]
|
||||||
|
raise ArgumentError, "--dryfire-show requires a slug" if slug.nil? || slug.empty?
|
||||||
|
log("[dryfire] resolving show now: #{slug}")
|
||||||
|
uri = resolve_show(slug)
|
||||||
|
if uri
|
||||||
|
fire_show(uri)
|
||||||
|
log("[dryfire] queued -- station.liq should pick it up on its next poll")
|
||||||
|
else
|
||||||
|
log("[dryfire] nothing queued (see WARN above); background music continues")
|
||||||
|
end
|
||||||
|
return
|
||||||
|
end
|
||||||
|
|
||||||
log("--- scheduler tick #{now.strftime('%Y-%m-%d %H:%M:%S')} ---")
|
log("--- scheduler tick #{now.strftime('%Y-%m-%d %H:%M:%S')} ---")
|
||||||
|
|
||||||
unless File.exist?(SCHEDULE_FILE)
|
unless File.exist?(SCHEDULE_FILE)
|
||||||
|
|
@ -216,6 +441,7 @@ def main
|
||||||
end
|
end
|
||||||
|
|
||||||
prune_old_markers(now)
|
prune_old_markers(now)
|
||||||
|
prune_old_captures(now)
|
||||||
entries = parse_schedule(SCHEDULE_FILE)
|
entries = parse_schedule(SCHEDULE_FILE)
|
||||||
log("Loaded #{entries.length} schedule entries")
|
log("Loaded #{entries.length} schedule entries")
|
||||||
|
|
||||||
|
|
@ -235,14 +461,14 @@ def main
|
||||||
|
|
||||||
case entry[:type]
|
case entry[:type]
|
||||||
when "stream"
|
when "stream"
|
||||||
write_queue(entry[:target], entry[:runlength])
|
fire_stream(entry[:target], entry[:runlength])
|
||||||
mark_fired(key)
|
mark_fired(key)
|
||||||
fired_any = true
|
fired_any = true
|
||||||
|
|
||||||
when "show"
|
when "show"
|
||||||
uri = resolve_show(entry[:target])
|
uri = resolve_show(entry[:target])
|
||||||
if uri
|
if uri
|
||||||
write_queue(uri)
|
fire_show(uri, entry[:runlength])
|
||||||
mark_fired(key)
|
mark_fired(key)
|
||||||
fired_any = true
|
fired_any = true
|
||||||
else
|
else
|
||||||
|
|
|
||||||
BIN
silence.mp3
Normal file
BIN
silence.mp3
Normal file
Binary file not shown.
157
station.liq
157
station.liq
|
|
@ -1,8 +1,20 @@
|
||||||
# station.liq - Cascadia Digital Collective Radio Automation
|
# station.liq - Cascadia Digital Collective Radio Automation
|
||||||
# Liquidsoap 2.2.4-1+dev compatible
|
# Liquidsoap 2.2.4-1+dev compatible
|
||||||
# Run from /srv/radio so relative paths resolve correctly
|
# Run from /srv/radio so relative paths resolve correctly
|
||||||
|
#
|
||||||
|
# One queue, one primitive. Both kinds of scheduled content go through a single
|
||||||
|
# request.dynamic reading current_queue.txt:
|
||||||
|
# * "show" entries -> a finite episode file (local path or media URL)
|
||||||
|
# * "stream" entries -> a continuous live Icecast/Shoutcast mount URL
|
||||||
|
# The JRuby scheduler writes whichever URI is due into current_queue.txt at its
|
||||||
|
# scheduled clock time; liquidsoap polls the file each tick and plays it. When
|
||||||
|
# the file is empty the source goes DOWN and the mux falls through to the
|
||||||
|
# looping background music.
|
||||||
|
|
||||||
settings.log.file.set(false)
|
settings.log.file.set(false)
|
||||||
|
# Level 3: normal verbosity. (Level 4 was used while hunting the transition
|
||||||
|
# bug and reverted once the crossfade fix was confirmed live on 15 Oct.)
|
||||||
|
settings.log.level.set(3)
|
||||||
|
|
||||||
# --- Load config ---
|
# --- Load config ---
|
||||||
# Explicit type annotation required: without it, fields used only in #{...}
|
# Explicit type annotation required: without it, fields used only in #{...}
|
||||||
|
|
@ -20,6 +32,15 @@ announcements_dir = "#{storage}/announcements"
|
||||||
podcasts_dir = "#{storage}/podcasts"
|
podcasts_dir = "#{storage}/podcasts"
|
||||||
queue_dir = "#{storage}/queues"
|
queue_dir = "#{storage}/queues"
|
||||||
|
|
||||||
|
# --- Silence fallback resource ------------------------------------------------
|
||||||
|
# The empty-queue branch of read_queue returns a request that RESOLVES (a real
|
||||||
|
# silent file) instead of one that fails. The file lives beside this script;
|
||||||
|
# station.rb exports RADIO_DIR (its own resolved directory) before exec'ing
|
||||||
|
# liquidsoap, so no install path is baked in here and the service works
|
||||||
|
# wherever it is installed.
|
||||||
|
radio_dir = environment.get(default="", "RADIO_DIR")
|
||||||
|
silence_uri = "file://#{radio_dir}/silence.mp3"
|
||||||
|
|
||||||
icecast_host = host
|
icecast_host = host
|
||||||
icecast_port = port
|
icecast_port = port
|
||||||
icecast_mount = mount
|
icecast_mount = mount
|
||||||
|
|
@ -36,31 +57,137 @@ jingles_pl = playlist(jingles_dir)
|
||||||
announcements_pl = playlist(announcements_dir)
|
announcements_pl = playlist(announcements_dir)
|
||||||
podcasts_pl = playlist(podcasts_dir)
|
podcasts_pl = playlist(podcasts_dir)
|
||||||
|
|
||||||
# Rotate among the four directories; transition_length bounds each handoff.
|
# Rotate among the four directories. NO transition_length and NO transitions:
|
||||||
rotating_bg = rotate(transition_length=0.5, [music_pl, jingles_pl, announcements_pl, podcasts_pl])
|
# both default to a plain hard cut (transition_length defaults to 5.0, which is
|
||||||
|
# what the outgoing track's "4.94585034014s remaining" was reporting -- rotate
|
||||||
|
# was composing a ~5s layer transition while the crossfade above it composed a
|
||||||
|
# 0.5s blend over the SAME boundary, so the two got out of sync and the
|
||||||
|
# crossfade's buffer went stale). rotate now hands off with a hard cut;
|
||||||
|
# crossfade is the only operator that blends.
|
||||||
|
rotating_bg = rotate([music_pl, jingles_pl, announcements_pl, podcasts_pl])
|
||||||
|
|
||||||
# Crossfade 0.5 s between consecutive tracks of the rotated stream.
|
# Crossfade 0.5 s between consecutive tracks of the rotated stream. This is the
|
||||||
background_music = crossfade(fade_in=0.5, fade_out=0.5, rotating_bg)
|
# ONLY operator composing transitions; see the note above.
|
||||||
|
#
|
||||||
|
# duration=0.5 is the fix for the buffer-emptied collapse. `duration` is NOT the
|
||||||
|
# fade length -- it is how much audio crossfade buffers IN ADVANCE from each
|
||||||
|
# track to compute the blend, and it defaults to 5.0. We were fading for 0.5s
|
||||||
|
# but buffering 5.0s, and the 2.2.4 release notes say plainly that a track
|
||||||
|
# shorter than the buffered duration is a programming error: all its data gets
|
||||||
|
# consumed computing the fade-in and none is left, which is exactly the
|
||||||
|
# "Buffer emptied, buffering needed" collapse in the journal. Sizing the buffer
|
||||||
|
# to the fade (0.5) means the crossfade only ever holds half a second of each
|
||||||
|
# track, so it cannot outrun the data.
|
||||||
|
# `minimum` is deliberately OMITTED. Per the build's own `-h crossfade`,
|
||||||
|
# `minimum` is a threshold on the CROSS's own duration ("Minimum duration for a
|
||||||
|
# cross"), and a cross can never exceed `duration` -- so any minimum above 0.5
|
||||||
|
# here is a bar nothing can clear, and it vetoed EVERY handoff with
|
||||||
|
# "Not enough data for crossing". The default (-1.) always tries, which is what
|
||||||
|
# we want: short-track protection is already delivered by duration=0.5 itself,
|
||||||
|
# since the fade only ever consumes half a second of each track.
|
||||||
|
background_music = crossfade(fade_in=0.5, fade_out=0.5, duration=0.5, rotating_bg)
|
||||||
|
|
||||||
|
# --- Scheduled-content source (reads current_queue.txt each tick) -------------
|
||||||
|
# Both kinds of scheduled content go through this single request.dynamic:
|
||||||
|
# * "show" entries -> a finite episode file (local path or media URL)
|
||||||
|
# * "stream" entries -> a continuous live Icecast/Shoutcast mount URL
|
||||||
|
# The scheduler writes whichever URI is due into current_queue.txt; an empty
|
||||||
|
# queue makes this source go DOWN so the mux falls through to background.
|
||||||
|
#
|
||||||
|
# last_uri remembers the URI this callback last HANDED OUT, so a poll that
|
||||||
|
# finds the queue file unchanged does not re-queue the same request. It is a
|
||||||
|
# string ref on purpose: ref : ('a) -> () -> 'a takes its type from the
|
||||||
|
# initial value, so ref(null) pins to a bare nullable that == cannot compare
|
||||||
|
# (orderable-type error). ref("") pins to string and compares cleanly.
|
||||||
|
last_uri = ref("")
|
||||||
|
|
||||||
# --- Show queue source (reads current_queue.txt each tick) ---
|
|
||||||
# In 2.2.x the callback must return a request (type ()->request), not null.
|
|
||||||
# When the queue is empty we return a NON-resolving URI ("noshow:"). Liquidsoap
|
|
||||||
# fails to resolve it, so this source goes DOWN -- which is exactly what lets
|
|
||||||
# fallback() drop through to the rotating background music below. (A resolvable
|
|
||||||
# sentinel like synth: would keep the show source up forever, masking the music.)
|
|
||||||
def read_queue()
|
def read_queue()
|
||||||
qf = "#{queue_dir}/current_queue.txt"
|
qf = "#{queue_dir}/current_queue.txt"
|
||||||
if file.exists(qf) and file.contents(qf) != "" then
|
if file.exists(qf) and file.contents(qf) != "" then
|
||||||
request.create(file.contents(qf))
|
let u = string.trim(file.contents(qf))
|
||||||
|
# Always hand out the URI the queue file currently names, and remember it.
|
||||||
|
# A duplicate is NOT re-queued here: the poller below decides whether to
|
||||||
|
# call fetch() at all, and it only does so when this URI changes. Feeding a
|
||||||
|
# request on a duplicate poll was the bug -- fetch() APPENDS to the queue
|
||||||
|
# (doc: "Try feeding the queue with a new request"), so a poller that fired
|
||||||
|
# every 5s on an unchanged file added one silence request per poll, with
|
||||||
|
# nothing draining them (prefetch=0 has no consumer). The source stayed UP
|
||||||
|
# behind a pile of silence, so fallback never handed off to background
|
||||||
|
# music, and the queued show never played.
|
||||||
|
last_uri.set(u)
|
||||||
|
request.create(u)
|
||||||
else
|
else
|
||||||
request.create("noshow:")
|
# Empty queue: return a request that RESOLVES, not one that fails.
|
||||||
|
# "noshow:" is an unregistered protocol, so request.create("noshow:")
|
||||||
|
# never resolves; a request.dynamic whose callback keeps returning
|
||||||
|
# failing requests goes permanently DOWN and stops polling the file --
|
||||||
|
# which is why the URI written at 12:00:10 was never seen until a
|
||||||
|
# restart. A real silent file resolves through the static "file"
|
||||||
|
# protocol, keeps the source UP and polling, and is inaudible, so
|
||||||
|
# fallback still prefers a real show the moment one appears.
|
||||||
|
if radio_dir == "" then
|
||||||
|
print("WARNING: RADIO_DIR is unset; start via station.rb or the silence fallback will not resolve")
|
||||||
|
end
|
||||||
|
request.create(silence_uri)
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
show_source = request.dynamic(prefetch=1, read_queue)
|
# prefetch=0: the poller below is the ONLY thing that feeds this source, so
|
||||||
|
# every callback invocation has exactly one cause and the queue cannot be
|
||||||
|
# filled by two mechanisms at once (prefetch=1 plus a poller is the "played
|
||||||
|
# twice" bug).
|
||||||
|
show_source = request.dynamic(prefetch=0, read_queue)
|
||||||
|
|
||||||
# --- Mux: show takes priority, fall back to background music ---
|
# --- Re-invocation poller -----------------------------------------------------
|
||||||
main_out = mksafe(fallback([show_source, background_music]))
|
# Why this exists: a "stream" entry is captured by ffmpeg and the LOCAL capture
|
||||||
|
# path is queued while ffmpeg is still writing it, so liquidsoap opens a
|
||||||
|
# growing file and reads it as an open-ended stream. At the writer's EOF it
|
||||||
|
# wraps back to byte 0 instead of firing request-finished -- and the dynamic
|
||||||
|
# callback is only re-invoked on request end, so the NEXT queued URI is never
|
||||||
|
# pulled. That is the 16:00 capture looping instead of handing off to 17:00.
|
||||||
|
#
|
||||||
|
# fetch() re-invokes the callback on demand (confirmed on this build), so a
|
||||||
|
# recurrent thread calls it on a wall clock and the next queued URI cuts in
|
||||||
|
# regardless of whether the current capture ever ends. fast=false keeps this
|
||||||
|
# below request resolution in priority, since fetch() can block.
|
||||||
|
#
|
||||||
|
# The poller -- not the callback -- owns the fetch decision. It reads the queue
|
||||||
|
# file itself and calls fetch() ONLY when the file names a URI different from
|
||||||
|
# the one already handed out. On a duplicate it does nothing, so no request is
|
||||||
|
# appended and the queue stays bounded. (fetch() appends; calling it on a timer
|
||||||
|
# without a consumer is what grew the queue to 50 silence requests.)
|
||||||
|
poll_interval = 5.
|
||||||
|
def poll_queue()
|
||||||
|
qf = "#{queue_dir}/current_queue.txt"
|
||||||
|
if file.exists(qf) and file.contents(qf) != "" then
|
||||||
|
let u = string.trim(file.contents(qf))
|
||||||
|
if u != last_uri() then
|
||||||
|
# ignore(): fetch() returns bool, but an if with no else must be unit.
|
||||||
|
# Without this the branch's bool becomes the if's value and the type
|
||||||
|
# checker rejects it (Error 5: unit vs bool). ignore converts it.
|
||||||
|
ignore(show_source.fetch())
|
||||||
|
end
|
||||||
|
end
|
||||||
|
poll_interval
|
||||||
|
end
|
||||||
|
thread.run.recurrent(fast=false, delay=0., poll_queue)
|
||||||
|
|
||||||
|
# --- Mux: scheduled content first, then background music ----------------------
|
||||||
|
# fallback picks the first child that is UP. With the queue empty, show_source
|
||||||
|
# is down and background_music carries the broadcast. track_sensitive=false is
|
||||||
|
# what makes a queued show cut in live at its appointed time instead of
|
||||||
|
# waiting for the current background track to finish.
|
||||||
|
mux = fallback(track_sensitive=false, [show_source, background_music])
|
||||||
|
|
||||||
|
# NO crossfade here. The blender lives INSIDE background_music (above), where
|
||||||
|
# every transition it composes is a track end it controls. A crossfade placed
|
||||||
|
# above the fallback was asked to blend layer handoffs it could not control:
|
||||||
|
# expiry kills the running show mid-frame (a skip, not a track end), and the
|
||||||
|
# docs warn a crossfade whose track "ends without any warning" may not have
|
||||||
|
# enough data to compose with. Those half-composed frames are the "bad data"
|
||||||
|
# that stopped clients. Layer handoffs are hard cuts now, which is exactly how
|
||||||
|
# live radio beds work; the encoder only ever sees complete, valid frames.
|
||||||
|
main_out = mksafe(mux)
|
||||||
|
|
||||||
# --- Output to Icecast ---
|
# --- Output to Icecast ---
|
||||||
output.icecast(
|
output.icecast(
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,4 @@
|
||||||
#!/usr/bin/env jruby
|
#!/usr/bin/env jruby
|
||||||
Dir.chdir(File.expand_path(__dir__))
|
Dir.chdir(File.expand_path(__dir__))
|
||||||
|
ENV["RADIO_DIR"] = File.expand_path(File.dirname(__FILE__))
|
||||||
exec("liquidsoap", File.expand_path("station.liq"))
|
exec("liquidsoap", File.expand_path("station.liq"))
|
||||||
|
|
|
||||||
|
|
@ -1,4 +0,0 @@
|
||||||
import os, subprocess, sys
|
|
||||||
install_dir = os.path.dirname(os.path.abspath(__file__))
|
|
||||||
os.chdir(install_dir)
|
|
||||||
sys.exit(subprocess.call(["liquidsoap", os.path.join(install_dir, "station.liq")]))
|
|
||||||
|
|
@ -1,251 +0,0 @@
|
||||||
#!/usr/bin/env python3
|
|
||||||
"""
|
|
||||||
update_playlists.py - Select the next unplayed episode per show and write an
|
|
||||||
annotated URI queue file for station.liq to consume.
|
|
||||||
|
|
||||||
Concurrency model mirrors fetch_podcasts.py: shares the same exclusive
|
|
||||||
lockfile (state/radio.lock); skips cleanly if the fetcher holds it. Databases
|
|
||||||
run in WAL mode with a busy timeout as a second safety net.
|
|
||||||
|
|
||||||
Selection: for each show, pick the earliest episode with played=0, write
|
|
||||||
playlists/<slug>.txt as a single annotated URI line, then mark it played.
|
|
||||||
Works identically for archived (local file_path) and live (enclosure_url)
|
|
||||||
shows because both store their episodes in the same table.
|
|
||||||
"""
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import fcntl
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import sqlite3
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parent
|
|
||||||
CONFIG_PATH = ROOT / "config.json"
|
|
||||||
|
|
||||||
STATE_DIR = None
|
|
||||||
SUBS_DB = None
|
|
||||||
PLAYED_DB = None
|
|
||||||
PODCASTS_DIR = None
|
|
||||||
LOGS_DIR = None
|
|
||||||
PLAYLISTS_DIR = None
|
|
||||||
LOCK_FILE = None
|
|
||||||
|
|
||||||
def load_config():
|
|
||||||
with open(CONFIG_PATH) as f:
|
|
||||||
return json.load(f)
|
|
||||||
|
|
||||||
def init_paths():
|
|
||||||
global STATE_DIR, SUBS_DB, PLAYED_DB, PODCASTS_DIR, LOGS_DIR, PLAYLISTS_DIR, LOCK_FILE
|
|
||||||
cfg = load_config()
|
|
||||||
storage = Path(cfg["storage"]).expanduser().resolve()
|
|
||||||
STATE_DIR = storage / "state"
|
|
||||||
SUBS_DB = STATE_DIR / "subscriptions.db"
|
|
||||||
PLAYED_DB = STATE_DIR / "played.db"
|
|
||||||
PODCASTS_DIR = storage / "podcasts"
|
|
||||||
LOGS_DIR = storage / "logs"
|
|
||||||
PLAYLISTS_DIR = storage / "playlists"
|
|
||||||
LOCK_FILE = STATE_DIR / "radio.lock"
|
|
||||||
|
|
||||||
logging.basicConfig(
|
|
||||||
level=logging.INFO,
|
|
||||||
format="%(asctime)s [%(levelname)s] %(message)s",
|
|
||||||
handlers=[logging.StreamHandler(sys.stdout)],
|
|
||||||
)
|
|
||||||
log = logging.getLogger("update_playlists")
|
|
||||||
|
|
||||||
def log_error(msg):
|
|
||||||
log.error(msg)
|
|
||||||
|
|
||||||
def _setup_logging():
|
|
||||||
fh = logging.FileHandler(LOGS_DIR / "update.log")
|
|
||||||
fh.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(message)s"))
|
|
||||||
log.addHandler(fh)
|
|
||||||
|
|
||||||
# Shared schema definitions (must match fetch_podcasts.py).
|
|
||||||
SHOWS_COLUMNS = {
|
|
||||||
"slug": "TEXT PRIMARY KEY",
|
|
||||||
"guid": "TEXT NOT NULL UNIQUE",
|
|
||||||
"name": "TEXT NOT NULL",
|
|
||||||
"feed_url": "TEXT NOT NULL UNIQUE",
|
|
||||||
"source": "TEXT DEFAULT 'manual'",
|
|
||||||
"opml_import": "INTEGER DEFAULT 0",
|
|
||||||
"archived": "INTEGER DEFAULT 1",
|
|
||||||
"media_class": "TEXT",
|
|
||||||
"created_at": "TEXT DEFAULT (datetime('now'))",
|
|
||||||
}
|
|
||||||
|
|
||||||
EPISODES_COLUMNS = {
|
|
||||||
"id": "INTEGER PRIMARY KEY AUTOINCREMENT",
|
|
||||||
"show_slug": "TEXT NOT NULL",
|
|
||||||
"guid": "TEXT NOT NULL",
|
|
||||||
"title": "TEXT",
|
|
||||||
"file_path": "TEXT",
|
|
||||||
"enclosure_url": "TEXT",
|
|
||||||
"runlength": "INTEGER",
|
|
||||||
"played": "INTEGER DEFAULT 0",
|
|
||||||
"played_at": "TEXT",
|
|
||||||
}
|
|
||||||
|
|
||||||
def _connect(db_path):
|
|
||||||
conn = sqlite3.connect(str(db_path), timeout=5.0)
|
|
||||||
conn.row_factory = sqlite3.Row
|
|
||||||
conn.execute("PRAGMA journal_mode=WAL")
|
|
||||||
conn.execute("PRAGMA busy_timeout=5000")
|
|
||||||
return conn
|
|
||||||
|
|
||||||
def _ensure_columns(conn, table, columns):
|
|
||||||
exists = conn.execute(
|
|
||||||
"SELECT name FROM sqlite_master WHERE type='table' AND name=?", (table,)
|
|
||||||
).fetchone()
|
|
||||||
if exists is None:
|
|
||||||
defs = ",\n ".join(f"{name} {spec}" for name, spec in columns.items())
|
|
||||||
extra = "\n ,UNIQUE(show_slug, guid)" if table == "episodes" else ""
|
|
||||||
conn.execute(f"CREATE TABLE {table} (\n {defs}{extra}\n )")
|
|
||||||
else:
|
|
||||||
existing = {row[1] for row in conn.execute(f"PRAGMA table_info({table})")}
|
|
||||||
for name, spec in columns.items():
|
|
||||||
if name not in existing:
|
|
||||||
col_default = spec.split("DEFAULT", 1)[1].strip() if "DEFAULT" in spec else "NULL"
|
|
||||||
conn.execute(f"ALTER TABLE {table} ADD COLUMN {name} {col_default}")
|
|
||||||
conn.commit()
|
|
||||||
|
|
||||||
def open_subs_db():
|
|
||||||
conn = _connect(SUBS_DB)
|
|
||||||
_ensure_columns(conn, "shows", SHOWS_COLUMNS)
|
|
||||||
return conn
|
|
||||||
|
|
||||||
def open_played_db():
|
|
||||||
conn = _connect(PLAYED_DB)
|
|
||||||
_ensure_columns(conn, "episodes", EPISODES_COLUMNS)
|
|
||||||
conn.execute(
|
|
||||||
"CREATE INDEX IF NOT EXISTS idx_episodes_show_played ON episodes (show_slug, played)"
|
|
||||||
)
|
|
||||||
conn.commit()
|
|
||||||
return conn
|
|
||||||
|
|
||||||
_lock_fd = None
|
|
||||||
|
|
||||||
def acquire_lock():
|
|
||||||
global _lock_fd
|
|
||||||
STATE_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
fd = os.open(str(LOCK_FILE), os.O_CREAT | os.O_RDWR, 0o644)
|
|
||||||
try:
|
|
||||||
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
|
||||||
except OSError:
|
|
||||||
os.close(fd)
|
|
||||||
return False
|
|
||||||
os.ftruncate(fd, 0)
|
|
||||||
os.write(fd, str(os.getpid()).encode())
|
|
||||||
_lock_fd = fd
|
|
||||||
return True
|
|
||||||
|
|
||||||
def release_lock():
|
|
||||||
global _lock_fd
|
|
||||||
if _lock_fd is not None:
|
|
||||||
try:
|
|
||||||
fcntl.flock(_lock_fd, fcntl.LOCK_UN)
|
|
||||||
os.close(_lock_fd)
|
|
||||||
finally:
|
|
||||||
_lock_fd = None
|
|
||||||
|
|
||||||
def _annotate_uri(runlength, title, uri):
|
|
||||||
def q(v):
|
|
||||||
return '"' + str(v).replace('"', '\\"') + '"'
|
|
||||||
ann = f"annotate:liq_runlength={q(runlength)},liq_title={q(title)}:"
|
|
||||||
return ann + uri
|
|
||||||
|
|
||||||
def select_next_episode(slug, played_db):
|
|
||||||
return played_db.execute(
|
|
||||||
"SELECT guid, title, file_path, enclosure_url, runlength "
|
|
||||||
"FROM episodes WHERE show_slug=? AND played=0 ORDER BY id ASC LIMIT 1",
|
|
||||||
(slug,),
|
|
||||||
).fetchone()
|
|
||||||
|
|
||||||
def mark_as_played(slug, guid, played_db):
|
|
||||||
played_db.execute(
|
|
||||||
"UPDATE episodes SET played=1, played_at=datetime('now') WHERE show_slug=? AND guid=?",
|
|
||||||
(slug, guid),
|
|
||||||
)
|
|
||||||
played_db.commit()
|
|
||||||
|
|
||||||
def write_queue_file(slug, ep, archived):
|
|
||||||
uri = ep["file_path"] if archived else ep["enclosure_url"]
|
|
||||||
if not uri:
|
|
||||||
return None
|
|
||||||
line = _annotate_uri(ep["runlength"], ep["title"], uri)
|
|
||||||
out = PLAYLISTS_DIR / f"{slug}.txt"
|
|
||||||
out.write_text(line + "\n")
|
|
||||||
return out
|
|
||||||
|
|
||||||
def update_all():
|
|
||||||
subs_db = open_subs_db()
|
|
||||||
played_db = open_played_db()
|
|
||||||
shows = subs_db.execute("SELECT slug, name, archived FROM shows ORDER BY name").fetchall()
|
|
||||||
updated = 0
|
|
||||||
for show in shows:
|
|
||||||
slug = show["slug"]
|
|
||||||
archived = show["archived"] == 1
|
|
||||||
ep = select_next_episode(slug, played_db)
|
|
||||||
if ep is None:
|
|
||||||
continue
|
|
||||||
out = write_queue_file(slug, ep, archived)
|
|
||||||
if out is None:
|
|
||||||
log.warning("No playable URI for %s (%s); skipping.", show["name"], slug)
|
|
||||||
continue
|
|
||||||
mark_as_played(slug, ep["guid"], played_db)
|
|
||||||
updated += 1
|
|
||||||
log.info("Queued %s: %s -> %s", slug, ep["title"], out.name)
|
|
||||||
subs_db.close()
|
|
||||||
played_db.close()
|
|
||||||
log.info("=== Update complete: %d show(s) queued ===", updated)
|
|
||||||
|
|
||||||
def json_summary():
|
|
||||||
subs_db = open_subs_db()
|
|
||||||
played_db = open_played_db()
|
|
||||||
shows = subs_db.execute("SELECT slug, name FROM shows ORDER BY name").fetchall()
|
|
||||||
result = {}
|
|
||||||
for show in shows:
|
|
||||||
slug = show["slug"]
|
|
||||||
counts = played_db.execute(
|
|
||||||
"SELECT COUNT(*) AS total, SUM(CASE WHEN played=0 THEN 1 ELSE 0 END) AS unplayed "
|
|
||||||
"FROM episodes WHERE show_slug=?",
|
|
||||||
(slug,),
|
|
||||||
).fetchone()
|
|
||||||
total = counts["total"] or 0
|
|
||||||
unplayed = counts["unplayed"] or 0
|
|
||||||
result[slug] = {"name": show["name"], "total": total, "unplayed": unplayed, "played": total - unplayed}
|
|
||||||
subs_db.close()
|
|
||||||
played_db.close()
|
|
||||||
print(json.dumps(result, indent=2))
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description="Playlist updater for radio automation")
|
|
||||||
parser.add_argument("--json", action="store_true", help="Emit JSON summary and exit")
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
init_paths()
|
|
||||||
STATE_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
LOGS_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
PLAYLISTS_DIR.mkdir(parents=True, exist_ok=True)
|
|
||||||
_setup_logging()
|
|
||||||
|
|
||||||
needs_lock = not args.json
|
|
||||||
|
|
||||||
if needs_lock and not acquire_lock():
|
|
||||||
log.info("Another radio process holds the lock; skipping this run.")
|
|
||||||
return
|
|
||||||
|
|
||||||
try:
|
|
||||||
if args.json:
|
|
||||||
json_summary()
|
|
||||||
else:
|
|
||||||
update_all()
|
|
||||||
finally:
|
|
||||||
if needs_lock:
|
|
||||||
release_lock()
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
Loading…
Reference in a new issue