dickshow-index/README.md

Dick Show Index

A complete pipeline to download, transcribe, analyze, and search all episodes of The Dick Show podcast.

What This Does

1. Downloads all episodes from the official RSS feed (Libsyn)
2. Transcribes audio locally using Whisper (faster-whisper, CPU/GPU)
3. Analyzes each episode with an LLM (via opencode) to extract:
   • Talking points with timestamps
   • Arguments, ideas, and notable quotes
   • Transgression Index (TI) 0-10 scoring
   • Speaker attribution and content warnings
4. Fact-checks analyses against a verified research dossier
5. Indexes everything in SQLite FTS5 for instant search
6. Exports a master JSON (data/dickshow_analysis.json) for web apps

Quick Start

# 1. Clone and install
git clone https://kode.hannesahlin.se/n3wk/dickshow-index.git
cd dickshow-index
./install.sh

# 2. (Optional) Authenticate opencode for higher rate limits
opencode auth login

# 3. Run the Rekieta arc (28 episodes):
python pipeline_all.py --eps 84 87 89 93 96 121 124 152 166 176 198 209 213 275 285 299 321 344 352 368 376 412 416 424 425 435 439 455

# Or run the full backlog (520 episodes):
python pipeline_all.py

What You Get

After running, you'll have:

data/
├── episodes.jsonl          # Episode metadata (title, date, audio URL, duration)
├── index.db                # SQLite FTS5 search index
├── dickshow_analysis.json  # Master analysis JSON (for web apps)
└── analysis/
    ├── ep0084.json         # Per-episode analysis + fact-check
    └── ...

transcripts/
├── ep0084.json             # Raw transcript with timestamps
└── ...

Search Examples

# Simple search
python search.py "isom trademark"

# Filter by tag
python search.py maddox --tag lolsuit

# Phonetic search (Whisper mangles names)
python search.py "raketa OR riketa OR rakita" --tag nick_rekieta

# Date range + tag
python search.py maddox --from 2017-01-01 --to 2018-12-31 --tag lolsuit

# Context around matches
python search.py "pull up" --tag eric_july --context 2

Output includes timestamps for deep-linking:

[Ep 373 | Mon, 28 Aug 2023] 00:25:32-00:25:35 #nick_rekieta
    "I was on Nick's show, Busting Balls, Nick Riketa."
    -> https://dick.show/episode-373/
    mp3: https://traffic.libsyn.com/.../The_Dick_Show_373.mp3

Configuration

Set via environment variables or edit config.py:

Variable	Default	Description
OPENCODE_ANALYZE_MODEL	opencode/nemotron-3-ultra-free	LLM for analysis
DS_ANALYZE_PAUSE	45	Seconds between analysis calls
DS_ANALYZE_TIMEOUT	600	Analysis timeout (seconds)
DS_MIN_FREE_RAM	1073741824 (1GB)	Pause if RAM below this
DS_MODEL	small	Whisper model: tiny, base, small, medium, large-v3
DS_DEVICE	cpu	cpu or cuda
DS_COMPUTE	int8	int8, float16, float32
DS_KEEP_AUDIO	0	Keep MP3s after transcription (1 to keep)
DS_ENCODING_CHECK	50	Episodes between UTF-8 verification

Encoding & Character Handling

All JSON files are written as UTF-8 with ensure_ascii=False. The RSS feed contains Unicode characters (em/en dashes, accented characters) which are preserved correctly in the JSON files.

If you see mojibake like â€" or é:

• The JSON files are correctly stored as UTF-8
• The issue is the consumer (text editor, web browser, terminal) reading the file with wrong encoding (Latin-1/Windows-1252)
• Fix: Ensure your tools/editor/browser use UTF-8 encoding

Verification & Repair:

# Verify all JSON files are valid UTF-8
python verify_encoding.py

# Fix common mojibake patterns (run if data was corrupted)
python fix_mojibake.py --dry-run   # preview fixes
python fix_mojibake.py              # apply fixes

# Check progress including encoding status
python check_progress.py

For web apps displaying this data:

• Set Content-Type: application/json; charset=utf-8
• Include <meta charset="utf-8"> in HTML
• Ensure database connections use UTF-8

Resuming / Sharing Work

The pipeline is fully resumable:

• Re-run the same command → skips completed episodes
• Only missing transcripts/analyses are processed
• Share transcripts/, data/analysis/, and data/index.db to collaborate

Requirements

• Python 3.10+
• ffmpeg (for audio processing - usually auto-installed with faster-whisper)
• opencode CLI (installed by ./install.sh)
• ~50 GB free disk for full backlog (MP3s auto-deleted after transcription by default)

Architecture

download.py     → fetches MP3s from Libsyn RSS (resumable, size-verified)
preprocess.py   → converts MP3 → clean 16kHz mono WAV (handles Libsyn quirks)
transcribe.py   → faster-whisper, chunked (10-min windows, no OOM)
analyze.py      → opencode subprocess with timeout + retries
verify.py       → second-pass fact-check against research dossier
index.py        → SQLite FTS5 index with hashtags
search.py       → CLI search with timestamps + context
pipeline_all.py → orchestrates everything with resume logic

For Contributors

The pipeline is designed for distributed workloads:

1. Each user runs a subset of episodes (--eps 84 87 89 ...)
2. Results are merged via data/analysis/ JSON files and index.db
3. Master JSON (data/dickshow_analysis.json) aggregates everything

License

Private research tool over public broadcasts. Content is adult/extreme — use responsibly.

Support

• Issues: https://kode.hannesahlin.se/n3wk/dickshow-index/issues
• Pipeline logs: rekieta_run.log, backlog_run.log
• Analysis logs: analyze_run.log, analyze_err.log