.--._.--.
____ ___ ___ ___ ___ ___ _ __ ___ _____ ( o, o )
| _ \ _ \ __/ __/ _ \/ _ \| '_ \/ __|_ _| `=^=`'--'
| |_) | / _| (_| (_) | (_) | | | (__ | | v0.7.3-beta · public-rev d4f9c1a
| _ <|_|\\_\\___\\___\\___/ \\___/|_| |_|\\___| |_| © 2024–2026 Trash Barge Labs
|_| \_\ licensed under the WTFPL
trash-panda fleet control · operate compute clusters built from
reclaimed thinkpads, e-waste GPUs, and one (1) marine deep-cycle battery.
────[ session ]──────────────────────────────────────────────────────────────────────────────────F1=help────
$ whoami
ryan // operator, gremlin-class
$ raccoonctl --help
raccoonctl — control the trash-barge fleet from one terminal.
USAGE
raccoonctl [--cluster <name>] <command> [args...]
COMMANDS
status print fleet health, one node per line
top interactive TUI dashboard (this screen)
forage schedule a pretraining run on idle nodes
groom run scheduled maintenance · clear caches · de-fur fans
heist opportunistically pull weights from a peer cluster
nap soft-suspend a node (keeps RAM, drops VRAM)
trash hard-evict a job and quarantine its checkpoint
paw ssh into a node by short-name, e.g. paw bandit-03
FLAGS
-c, --cluster <name> default: $RACCOON_CLUSTER = trash-barge-01
-j, --json emit ndjson, no ANSI
-q, --quiet errors only
-v, --verbose include peer-gossip and DHT noise
--no-scanlines you monster
ENVIRONMENT
RACCOON_CLUSTER which barge to talk to
RACCOON_TOKEN 40-char hex, see ~/.config/raccoon/auth
NO_COLOR respected · also disables phosphor
$ raccoonctl top --cluster trash-barge-01
[ OK ] auth · token valid through 2026-07-12T00:00:00Z
[ OK ] dht · 14 peers, 3 hops to nearest seed
[ WARN] ntp · drift 312ms, resyncing
[ OK ] tui · entering ncurses mode█
────[ fleet // trash-barge-01 ]───────────────────────────────────────────────────17 nodes · 3 racks────
.--._.--. ryan@trash-barge-01
( o, o ) ─────────────────────────────
`=^=`'--' cluster: trash-barge (mk.iv)
/ \\ os: NixOS 24.05 (raccoon)
/ _ _ \\ kernel: 6.6.31-trashpanda
| (o) (o) | uptime: 42 days, 6 h, 19 m
| ^ | shell: zsh 5.9 · prezto
\\ '---' / wm: sway 1.9 · waybar
\\ / term: wezterm + tmux 3.4
'-.___.-' cpu: 14× Threadripper 7960X
_/ \\_ gpu: 23× 3090 · 4× MI50 · 1× P40
/ _ _ \\ mem: 2.4 TiB / 3.0 TiB DDR5 ECC
'_/ \\_' vram: 524 GiB / 568 GiB
disk: 94 TiB ZFS (raidz2, 3 vdevs)
net: 40 GbE infiniband, 1 GbE wan
power: 3.41 kW drawn · 12.6 kWh/24h
cooling: 4 box fans · 1 oscillating
vermin: 3 confirmed · tolerated
▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮
colors: phosphor / amber / fault
╭─ nodes ──────────────────────────────── sort: load ↓ ────╮
│ name role gpus load mem netio st │
│ bandit-01 trainer 8/8 0.94 214G 38Gb RUN │
│ bandit-02 trainer 8/8 0.91 209G 37Gb RUN │
│ bandit-03 trainer 7/8 0.71 178G 31Gb RUN │
│ bandit-04 trainer 0/8 0.02 12G 0Gb NAP │
│ scrounge-01 eval 4/4 0.55 102G 8Gb RUN │
│ scrounge-02 eval 4/4 0.49 97G 7Gb RUN │
│ midden-01 storage — 0.18 256G 12Gb RUN │
│ midden-02 storage — 0.21 248G 11Gb RUN │
│ gremlin-01 jupyter 2/2 0.34 48G 1Gb RUN │
│ gremlin-02 jupyter 1/2 0.12 22G 0.4Gb RUN │
│ marshmallow coord — 0.08 6G 0.2Gb RUN │
│ tanuki-01 forage 4/4 0.88 188G 9Gb RUN │
│ tanuki-02 forage 4/4 0.86 185G 9Gb RUN │
│ tanuki-03 forage 0/4 — — — DWN │
│ possum-01 archive — 0.04 12G 0.1Gb RUN │
│ possum-02 archive — 0.05 13G 0.1Gb RUN │
│ opossum spare — 0.00 1G 0Gb SLP │
├──────────────────────────────────────────────────────────┤
│ 17 nodes · 14 RUN · 1 NAP · 1 SLP · 1 DWN │
╰──────────────────────────────────────────────────────────╯
────[ telemetry // last 60 min ]──────────────────────────────────────────────refresh 2s────
╭─ gpu utilization · per rack ─────────────────────────────╮
│ │
│ rack-A █████████████████████████████▓▓▓▓░░ 82% │
│ rack-B █████████████████████████▓▓▓▓░░░░░ 71% │
│ rack-C ██████████████████████████████████ 96% │
│ forage ███████████████████████████████░░░ 88% │
│ spare ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 0% │
│ │
│ scale: each block = ~3 % of theoretical FP16 peak │
│ │
│ aggregate: 524 GiB VRAM live · 2.18 PF/s sustained │
│ headroom: ~12 % before thermal cap on rack-C │
╰──────────────────────────────────────────────────────────╯
╭─ power & thermals · 60-min sparkline ────────────────────╮
│ │
│ draw kW ▁▂▂▃▃▄▅▅▆▆▇▇▇▇▆▆▆▇▇▇█▇▆▆▆▅▅▄▄▃▃▂▂▂▃▄ 3.41 │
│ temp °C ▂▂▃▃▃▄▄▅▅▅▆▆▇▇▇▆▆▆▆▆▇▇█▇▇▆▆▆▅▅▄▄▄▄▅▅ 72.1 │
│ fans rpm ▃▃▃▄▄▅▅▆▆▆▇▇▇▇▇▆▆▆▆▆▇▇█▇▇▇▆▆▆▆▅▅▅▅▅▆ 2840 │
│ net Gb/s ▂▃▃▄▄▅▆▆▇▇▇█▇▆▅▄▄▃▃▄▅▆▇▇▇▇▆▆▅▅▅▆▆▇▇█ 38.4 │
│ disk MB/s ▁▁▂▂▂▃▃▃▄▄▄▅▅▅▆▆▇█▇▇▆▅▄▄▃▃▃▂▂▂▂▂▂▃▃▃ 1410 │
│ │
│ battery ▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▱▱▱▱▱ 74 % · on shore power │
│ solar in ▰▰▰▰▰▰▰▰▰▰▰▰▱▱▱▱▱▱▱▱ 0.62 kW │
│ grid out ▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰ 2.79 kW │
│ │
│ cap @ 4.0 kW · breaker trips at 4.4 · stay frosty │
╰──────────────────────────────────────────────────────────╯
────[ jobs & event log ]───────────────────────────────────────────────────────tail -f────
╭─ forage queue · pretraining ──────────────────────────────────────────────────╮
│ id owner model step / total eta progress │
│ f-9821 ryan racoon-7b-v3 68420 / 80000 02:14:08 ▰▰▰▰▰▰▰▰▰▰▰▰▰▰▱▱▱▱ 85% │
│ f-9822 ryan racoon-7b-v3 220 / 30000 11:52:30 ▰░░░░░░░░░░░░░░░░░ 1% │
│ f-9817 aria possum-1.3b 8120 / 8200 00:01:42 ▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰ 99% │
│ f-9819 aria possum-1.3b-rl 412 / 2000 04:18:00 ▰▰▰▰░░░░░░░░░░░░░░ 21% │
│ f-9810 sam tanuki-mod-mix 14000 / 14000 00:00:00 ▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰▰ 100% DONE │
│ e-3340 sam tanuki-eval-h6 31 / 64 00:08:11 ▰▰▰▰▰▰▰▰░░░░░░░░░░ 48% │
│ f-9805 bree gremlin-coder 200 / 4000 06:40:00 ▰░░░░░░░░░░░░░░░░░ 5% │
│ h-0009 ryan heist-pull-llama-2 — / — queued ▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱ — │
│ g-0042 cron groom-fans-weekly — / — queued ▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱▱ — │
├────────────────────────────────────────────────────────────────────────────────┤
│ selected: f-9821 racoon-7b-v3 enter=detail · k=kill · n=nap · t=top │
│ │
│ loss ▁▂▂▃▃▃▄▄▄▄▅▅▆▆▆▆▇▇▇▇▇▇▇▇▆▆▆▅▅▅▄▄▃▃▃▂ 1.842 ↓ grad-norm 0.41 │
│ tok/s ▆▇▇█▇▇▇▆▇▇█▇▇▇▆▆▇▇█▇▇▆▆▆▇▇█▇▇▇▆▆▇▇▇▇ 412 814 peer p99 401k │
╰────────────────────────────────────────────────────────────────────────────────╯
╭─ log · trash-barge-01 ─────────────────────────╮
│ 17:42:01 [OK] bandit-01 ckpt 68000 ok │
│ 17:42:03 [OK] tanuki-02 shard rebal │
│ 17:42:08 [WARN] rack-C temp 78°C │
│ 17:42:11 [OK] midden-01 scrub 14% done │
│ 17:42:14 [OK] bandit-02 step 68420 │
│ 17:42:18 [OK] gremlin-01 nb saved │
│ 17:42:21 [FAIL] tanuki-03 link down eth1 │
│ 17:42:22 [OK] ↳ rerouted via ib0 │
│ 17:42:25 [OK] scrounge-01 eval batch 31 │
│ 17:42:29 [OK] forage queue +1 (f-9822) │
│ 17:42:31 [WARN] ntp drift 312ms · resync │
│ 17:42:34 [OK] bandit-01 loss 1.844 ↓ │
│ 17:42:36 [OK] peer gossip · 14 alive │
│ 17:42:38 [OK] midden-02 prune 412 MiB │
│ 17:42:41 [OK] bandit-03 step 68420 │
│ 17:42:44 [note] a raccoon was here. │
│ 17:42:46 [OK] bandit-02 loss 1.842 ↓ │
│ 17:42:49 [OK] tanuki-02 ckpt rotated │
│ │ 17:42:52 [OK] bandit-01 step 68421 │
╰─────────────────────────────────────────────╯
────[ training loss · racoon-7b-v3 · last 7 days ]──────────────────────────────────log y────
loss
3.0 ┤█·
│█▆·
2.6 ┤██▅▄·
│█████▄▃·
2.2 ┤████████▄▃▂·
│████████████▆▅▄▃▃▂▂▂·
1.9 ┤████████████████████▇▆▅▅▄▄▃▃▃·
│████████████████████████████████▆▆▅▅▄▄▃▃▃▃▂▂▂▂▂▂·
1.7 ┤█████████████████████████████████████████████████▇▆▆▅▅▅▄▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁█
└────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬─────
d-7 d-6 d-5 d-4 d-3 d-2 d-1 12h 6h now
▲ restart ▼ eval pause ● manual sched bump × tanuki-03 dropout
d-6 17:14 d-4 02:00 d-2 11:08 d-0 17:42
────[ keybinds & quick recipes ]──────────────────────────────────────────────────────────────
╭─ global keys ─────────────────╮
│ q quit │
│ ? help / cheat sheet │
│ r refresh now │
│ / fuzzy filter │
│ g goto node by name │
│ : command palette │
│ tab cycle pane │
│ ^c cancel selected job │
│ ^l redraw screen │
╰────────────────────────────────╯
╭─ job actions ─────────────────╮
│ ent open detail view │
│ n nap node │
│ t top inside job │
│ k kill (confirm) │
│ p pin to top │
│ y yank id to clipboard │
│ e edit hyperparams │
│ l follow log stream │
│ ! shell on host │
╰────────────────────────────────╯
╭─ recipes ─────────────────────╮
│ $ raccoonctl forage \\ │
│ --model racoon-7b-v3 \\ │
│ --ckpt latest \\ │
│ --on rack-A,rack-B │
│ │
│ $ raccoonctl heist \\ │
│ peer://midden.bog/llama-2 │
│ │
│ $ raccoonctl groom --fans │
│ │
│ $ raccoonctl nap bandit-04 │
╰────────────────────────────────╯
────[ peer map · trash-net DHT ]──────────────────────────────────────────────────14 peers────
· · · · · ·
· ●━━━━━━━━━━━━● midden.bog ·
· ┃ ·
●━━━━━━━━━━━━● ●━━━━━━━━● ·
· ┃ ┃ ┃ ·
●━━━━━━━━━●━━━━━[ TRASH-BARGE-01 ]━━━●━━━━━━━━● ·
· ┃ ┃ ⤓ you are here ┃
●━━━━━● ●━━━━━━━━● ●━━━━━● ·
· ┃ ┃ ┃ ·
● ●━━━ ✗ ━━━ tanuki-03 ●
stoop.dump gutter.farm attic.eu1
legend: ● reachable ● degraded ● unreachable ━ link ━ ✗ ━ backoff
consensus: 3/4 quorum on shard k=7 last gossip: 17:42:36 (now-16s)
────[ command palette ]──────────────────────────────────────────────────────: open────
╭──────────────────────────────────────────────────────────────────────────────────────────╮
│ : forage█ │
├──────────────────────────────────────────────────────────────────────────────────────────┤
│ ▶ forage schedule a pretraining run on idle nodes │
│ forage --resume resume the most recent forage from its last checkpoint │
│ forage --rl attach a small RL head and forage on synthetic prefs │
│ forage --offline build training shards from local middens, no network │
│ ──────────────────────────────────────────────────────────────────────────────── │
│ groom run scheduled maintenance · clear caches · de-fur fans │
│ heist opportunistically pull weights from a peer cluster │
│ nap soft-suspend a node │
│ ──────────────────────────────────────────────────────────────────────────────── │
│ 9 results · ↑ ↓ to move · enter to run · ^g to abort │
╰──────────────────────────────────────────────────────────────────────────────────────────╯
$ raccoonctl :forage --model racoon-7b-v3 --resume█