dickshow-index/README.md

# Dick Show Index

A complete pipeline to download, transcribe, analyze, and search all episodes of **The Dick Show** podcast.

## What This Does

1. **Downloads** all episodes from the official RSS feed (Libsyn)
2. **Transcribes** audio locally using Whisper (faster-whisper, CPU/GPU)
3. **Analyzes** each episode with an LLM (via opencode) to extract:
   - Talking points with timestamps
   - Arguments, ideas, and notable quotes
   - Transgression Index (TI) 0-10 scoring
   - Speaker attribution and content warnings
4. **Fact-checks** analyses against a verified research dossier
5. **Indexes** everything in SQLite FTS5 for instant search
5. **Exports** a master JSON (`data/dickshow_analysis.json`) for web apps

## Quick Start

```bash
# 1. Clone and install
git clone https://kode.hannesahlin.se/n3wk/dickshow-index.git
cd dickshow-index
./install.sh

# 2. (Optional) Authenticate opencode for higher rate limits
opencode auth login

# 3. Run the Rekieta arc (28 episodes):
python pipeline_all.py --eps 84 87 89 93 96 121 124 152 166 176 198 209 213 275 285 299 321 344 352 368 376 412 416 424 425 435 439 455

# Or run the full backlog (520 episodes):
python pipeline_all.py
```

## What You Get

After running, you'll have:

```
data/
├── episodes.jsonl          # Episode metadata (title, date, audio URL, duration)
├── index.db                # SQLite FTS5 search index
├── dickshow_analysis.json  # Master analysis JSON (for web apps)
└── analysis/
    ├── ep0084.json         # Per-episode analysis + fact-check
    └── ...

transcripts/
├── ep0084.json             # Raw transcript with timestamps
└── ...
```

## Search Examples

```bash
# Simple search
python search.py "isom trademark"

# Filter by tag
python search.py maddox --tag lolsuit

# Phonetic search (Whisper mangles names)
python search.py "raketa OR riketa OR rakita" --tag nick_rekieta

# Date range + tag
python search.py maddox --from 2017-01-01 --to 2018-12-31 --tag lolsuit

# Context around matches
python search.py "pull up" --tag eric_july --context 2
```

Output includes **timestamps** for deep-linking:
```
[Ep 373 | Mon, 28 Aug 2023] 00:25:32-00:25:35 #nick_rekieta
    "I was on Nick's show, Busting Balls, Nick Riketa."
    -> https://dick.show/episode-373/
    mp3: https://traffic.libsyn.com/.../The_Dick_Show_373.mp3
```

## Configuration

Set via environment variables or edit `config.py`:

| Variable | Default | Description |
|----------|---------|-------------|
| `OPENCODE_ANALYZE_MODEL` | `opencode/nemotron-3-ultra-free` | LLM for analysis |
| `DS_ANALYZE_PAUSE` | `45` | Seconds between analysis calls |
| `DS_ANALYZE_TIMEOUT` | `600` | Analysis timeout (seconds) |
| `DS_MIN_FREE_RAM` | `1073741824` (1GB) | Pause if RAM below this |
| `DS_MODEL` | `small` | Whisper model: `tiny`, `base`, `small`, `medium`, `large-v3` |
| `DS_DEVICE` | `cpu` | `cpu` or `cuda` |
| `DS_COMPUTE` | `int8` | `int8`, `float16`, `float32` |
| `DS_KEEP_AUDIO` | `0` | Keep MP3s after transcription (`1` to keep) |
| `DS_ENCODING_CHECK` | `50` | Episodes between UTF-8 verification |

## Encoding & Character Handling

All JSON files are written as **UTF-8** with `ensure_ascii=False`. The RSS feed contains Unicode characters (em/en dashes, accented characters) which are preserved correctly in the JSON files.

**If you see mojibake like `â€"` or `é`:**
- The JSON files are correctly stored as UTF-8
- The issue is the *consumer* (text editor, web browser, terminal) reading the file with wrong encoding (Latin-1/Windows-1252)
- **Fix**: Ensure your tools/editor/browser use UTF-8 encoding

**Verification & Repair:**
```bash
# Verify all JSON files are valid UTF-8
python verify_encoding.py

# Fix common mojibake patterns (run if data was corrupted)
python fix_mojibake.py --dry-run   # preview fixes
python fix_mojibake.py              # apply fixes

# Check progress including encoding status
python check_progress.py
```

**For web apps displaying this data:**
- Set `Content-Type: application/json; charset=utf-8`
- Include `<meta charset="utf-8">` in HTML
- Ensure database connections use UTF-8

## Resuming / Sharing Work

The pipeline is **fully resumable**:
- Re-run the same command → skips completed episodes
- Only missing transcripts/analyses are processed
- Share `transcripts/`, `data/analysis/`, and `data/index.db` to collaborate

## Requirements

- **Python 3.10+**
- **ffmpeg** (for audio processing - usually auto-installed with faster-whisper)
- **opencode CLI** (installed by `./install.sh`)
- **~50 GB free disk** for full backlog (MP3s auto-deleted after transcription by default)

## Architecture

```
download.py     → fetches MP3s from Libsyn RSS (resumable, size-verified)
preprocess.py   → converts MP3 → clean 16kHz mono WAV (handles Libsyn quirks)
transcribe.py   → faster-whisper, chunked (10-min windows, no OOM)
analyze.py      → opencode subprocess with timeout + retries
verify.py       → second-pass fact-check against research dossier
index.py        → SQLite FTS5 index with hashtags
search.py       → CLI search with timestamps + context
pipeline_all.py → orchestrates everything with resume logic
```

## For Contributors

The pipeline is designed for distributed workloads:
1. Each user runs a subset of episodes (`--eps 84 87 89 ...`)
2. Results are merged via `data/analysis/` JSON files and `index.db`
3. Master JSON (`data/dickshow_analysis.json`) aggregates everything

## License

Private research tool over public broadcasts. Content is adult/extreme — use responsibly.

## Support

- Issues: https://kode.hannesahlin.se/n3wk/dickshow-index/issues
- Pipeline logs: `rekieta_run.log`, `backlog_run.log`
- Analysis logs: `analyze_run.log`, `analyze_err.log`