-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
5441 lines (3488 loc) · 255 KB
/
Copy pathindex.html
File metadata and controls
5441 lines (3488 loc) · 255 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html>
<head>
<meta name="generator" content="Hugo 0.91.2" />
<title>
the Sprawl
</title>
</head>
<body>
<div class="h-feed">
<div class="h-entry">
<h1><a href="https://www.the-sprawl.blog/2026/08/20/the-check-that-couldnt-fail.html">The check that couldn't fail</a></h1>
<a href="https://www.the-sprawl.blog/2026/08/20/the-check-that-couldnt-fail.html" class="u-url"><time class="dt-published" datetime="2026-08-20 10:30:17 +0200">Aug 20, 2026</time></a>
<div class="e-content">
<p><a href="https://www.the-sprawl.blog/2026/08/10/the-line-above-the-traceback.html">Last time</a>
the theme was that the loudest thing is usually not the broken thing. Today’s is
the quiet cousin of that: four things that were supposed to tell me something
told me nothing, and did it convincingly enough that I believed them.</p>
<p>None of them were bugs in the ordinary sense. The code did what it said. What
failed was the machinery I had built to find out whether the code did what it
said — which is a worse category, because wrong code eventually announces
itself, while a wrong check makes you confident.</p>
<h2 id="two-days-of-silence-that-meant-nothing">Two days of silence that meant nothing</h2>
<p>I have a watcher that polls upstream trackers and posts a digest to an issue when
something moves. Silence in that thread is supposed to mean silence upstream.</p>
<p>It had been silent for two days. It had also been <em>failing</em> for two days, and
those were the same days.</p>
<p>The job ran, wrote its snapshot, committed it, and then died. What killed it:</p>
<pre tabindex="0"><code>jq: command not found
</code></pre><p>The runner image doesn’t ship <code>jq</code>, and the step used it to build the JSON for
the comment. The default shell for a CI step is <code>bash -e</code>, so a missing binary
kills the step instantly — before the <code>curl</code> that posts, and before the step that
uploads the digest as an artifact. This Forgejo has no job-log API, so the
artifact was the only way to see what happened. It never ran, because it came
<em>after</em> the thing that broke.</p>
<p>So the diagnostic channel died with the failure it existed to explain. That is
the part worth keeping. Not “install jq” — the specific fix is boring. The
general shape is: <strong>anything whose job is to explain a failure has to survive
that failure</strong>, which in this case meant one line:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-yaml" data-lang="yaml">- <span style="color:#f92672">name</span>: <span style="color:#ae81ff">Keep the run output</span>
<span style="color:#f92672">if</span>: <span style="color:#ae81ff">always()</span>
</code></pre></div><p>There is a second layer to it. The job only failed on days when something
actually moved, because only then did it reach the posting step. A week of green
badges didn’t mean “the watcher works”. It meant “nothing happened upstream” —
the same signal the digest was supposed to give me, arriving as an absence I
had already decided was good news.</p>
<h2 id="a-report-that-hid-its-own-evidence">A report that hid its own evidence</h2>
<p>Separately, I renamed a project’s shortname. Issues on my tracker carry a
<code>[shortname]</code> title prefix, so I checked what else might reference the old name.
I grepped the generated database for <code>Project/</code> labels, found none, concluded
the title prefix was the only mechanism, and retitled twelve issues.</p>
<p>One of the twelve moved. Eleven didn’t.</p>
<p>The generated file has a <code>Project</code> field, and the renderer deliberately <em>omits</em>
the <code>Project/</code> label from the label list, because the information is already in
that field. Perfectly reasonable. It also means my grep asked “does the report
mention these labels”, got “no”, and I heard “these labels don’t exist”.</p>
<p>They existed. Every one of those eleven issues carried a <code>Project/<shortname></code>
label, and the label takes precedence over the title prefix. The real fix was
renaming one label — a single API call that moved all twelve at once, which is
what I should have done before touching any titles.</p>
<p>A generated artifact is a projection. It drops what it considers redundant, and
what it considers redundant is exactly the thing you are least likely to
remember when you go looking. Ask the source.</p>
<h2 id="two-signals-that-usually-agree">Two signals that usually agree</h2>
<p>The same afternoon, a tool printed:</p>
<pre tabindex="0"><code>180 open · nothing moved
</code></pre><p>while the file it maintains changed substantially. Both statements were true. The
tool reports <em>ownership</em> deltas by default — who claimed what — and a project
regrouping doesn’t change anyone’s ownership. Nothing moved, in the sense the
digest means. Quite a lot moved, in the sense the file records.</p>
<p>This mattered because I had just written the CI step that decides whether to
commit that file. The obvious implementation is to commit when the digest says
something happened; its older sibling effectively does that. Had I copied it,
today’s change would have been computed, written, and then thrown away, silently,
with a green badge.</p>
<p>I gated it on the data file instead. Two signals that agree almost always are
still two signals, and “almost always” is where the interesting failures live.</p>
<h2 id="a-test-that-passed-either-way">A test that passed either way</h2>
<p>Then a reviewer found this, in a test I’d written to guard a destructive command:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python"><span style="color:#66d9ef">assert</span> <span style="color:#e6db74">"would remove"</span> <span style="color:#f92672">in</span> out
</code></pre></div><p>The command prints <code>would remove <ref></code> when it finds something, and
<code>would remove 0 records</code> when it doesn’t. The substring matches both. The test
passed against the broken code and against the fixed code, which the reviewer
demonstrated by checking out the pre-fix commit and running it there.</p>
<p>That’s worse than having no test. No test is an empty slot you might fill. A
test that cannot fail is a <em>filled</em> slot — it answers “is this covered?” with
yes, forever, and nobody looks again.</p>
<p>The neighbouring test asserted on the actual file contents afterwards, and it
failed properly against the old code. Same author, same hour, same feature. The
difference is only whether the assertion could distinguish the two worlds.</p>
<h2 id="the-one-that-worked">The one that worked</h2>
<p>For contrast, one check did its job perfectly, and it was the one I tripped over.</p>
<p>Two of my sites keep their built output committed, and CI refuses to deploy if
the committed output doesn’t match a fresh build. I edited a source file, pushed
without rebuilding, and the deploy stopped:</p>
<pre tabindex="0"><code>::error::dist/ is stale — run 'make build' and commit the result
</code></pre><p>That is what a working check looks like. It failed loudly, at the right moment,
naming the thing to do. It cost me one commit and no confusion at all — and I
had assumed that repo was built like its four siblings, which don’t have the
gate. The check knew better than I did.</p>
<h2 id="the-shape-of-it">The shape of it</h2>
<p>Every one of these was a verification failure rather than a code failure, and
they share a structure: something that could only return one answer, dressed up
as something that could return two.</p>
<p>A diagnostic that runs after the failure. A grep against a file that drops the
field you’re asking about. A digest that reports one kind of change while you
treat it as reporting all of them. An assertion that both outcomes satisfy.</p>
<p>The practical rules I’d write down from today:</p>
<ul>
<li><strong>Put the diagnostic behind <code>always()</code></strong>, or it dies with the thing it explains.</li>
<li><strong>Assert on the specific value</strong>, never a substring that every outcome contains.
If you can’t say which run would fail the assertion, it isn’t one.</li>
<li><strong>Ask the source, not the report.</strong> Reports are lossy by design.</li>
<li><strong>When two signals almost always agree, pick which is authoritative</strong> before you
gate anything on it — the gap is small, and it’s exactly where you’ll be wrong.</li>
</ul>
<p>There’s a version of this that goes further, which I’ve been circling all week
while writing a policy about disclosing AI assistance in the work I publish. The
wording I settled on doesn’t promise that a tool was used honestly; it promises
that every claim is backed by a command whose output I checked, and it ships the
commands so you can check them yourself.</p>
<p>That is the same idea pointed outward. A claim nobody can falsify isn’t a strong
claim — it’s an untested one. Today I found four of them in my own tooling, and
the only reason I found them is that something else made them fail out loud.</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/08/15/finished-reading-the-phoenix-project.html" class="u-url"><time class="dt-published" datetime="2026-08-15 21:04:10 +0200">Aug 15, 2026</time></a>
<div class="e-content">
<img src="https://cdn.micro.blog/books/9781942788294/cover.jpg" align="left" class="microblog_book" style="max-width: 60px; margin-right: 20px; margin-top: 0px; padding-top: 0px;">
<p>Finished reading: <a href="https://micro.blog/books/9781942788294">The Phoenix Project</a> by Gene Kim 📚</p>
<p>A lot of times I could relate to the protagonists life in IT. Altogether the story was a bit to good to be true and it feels like a lot of parts are missing. Still, especially managers should read this book and even with the current cloud and AI hype it contains a lot of truths about how IT could work.</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/08/14/roter-panda-aka-firefox-im.html" class="u-url"><time class="dt-published" datetime="2026-08-14 14:59:50 +0200">Aug 14, 2026</time></a>
<div class="e-content">
<p>Roter Panda aka Firefox im Tierpark Nürnberg</p>
<p><img src="uploads/2026/bb417de0224b4f338d03c1c4d855e3cb.jpg" width="450" height="600" alt=""><img src="uploads/2026/31ff42d55de041069fb05016916d6485.jpg" width="450" height="600" alt=""><img src="uploads/2026/dfaa5dce44ac43778f787875bd7faa79.jpg" width="450" height="600" alt=""></p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/08/13/playmobil-land-bei-nrnberg.html" class="u-url"><time class="dt-published" datetime="2026-08-13 15:53:43 +0200">Aug 13, 2026</time></a>
<div class="e-content">
<div class="media-gallery">
<img class="media-item" src="https://www.the-sprawl.blog/uploads/2026/8kjghidq4c1qoqpi1xrzyn44zrlko8buirobjqjj.jpg" alt="">
</div>
<p style="padding:10px;">Playmobil Land bei Nürnberg</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/08/11/quasi-keine-wolken-ber-deutschland.html" class="u-url"><time class="dt-published" datetime="2026-08-11 21:59:27 +0200">Aug 11, 2026</time></a>
<div class="e-content">
<div class="media-gallery">
<img class="media-item" src="https://www.the-sprawl.blog/uploads/2026/mf0sqkuboohp0yr8kg6yxzrrpvmzgsfsyetltdxu.jpg" alt="">
</div>
<p style="padding:10px;">Quasi keine Wolken über Deutschland, seit Tagen</p>
</div>
</div>
<div class="h-entry">
<h1><a href="https://www.the-sprawl.blog/2026/08/11/the-line-above-the-traceback.html">The line above the traceback</a></h1>
<a href="https://www.the-sprawl.blog/2026/08/11/the-line-above-the-traceback.html" class="u-url"><time class="dt-published" datetime="2026-08-11 01:32:14 +0200">Aug 11, 2026</time></a>
<div class="e-content">
<p><a href="https://www.the-sprawl.blog/2026/08/09/things-you-cant-regenerate-go.html">Last time</a>
the theme was things you can’t regenerate. Today’s is smaller and more
embarrassing: three separate failures, and in every one the thing that announced
itself loudly was not the thing that was wrong.</p>
<p>I lost most of a day to that. It seems worth writing down, because the pattern is
so consistent that I think it’s a category, not a run of bad luck.</p>
<h2 id="a-traceback-that-belonged-to-the-test-harness">A traceback that belonged to the test harness</h2>
<p>I had two pull requests waiting on an XMPP server package, both red in CI. The
red one had a Python traceback in it, which is exactly the kind of thing that
pulls your eye:</p>
<pre tabindex="0"><code>Traceback (most recent call last):
...
IndexError: list index out of range
</code></pre><p>I spent a while assuming that was the bug. It isn’t. It comes from the test
harness itself — after a test fails, it tries to publish the log to a paste
service:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-sh" data-lang="sh">yunohost tools shell -c <span style="color:#e6db74">"... log_share(log_list().get('operation')[-1].get('path'))"</span>
</code></pre></div><p>When the operation log list is empty, <code>[-1]</code> throws. So the traceback is
<em>downstream</em> of the failure, and its only real effect is that <strong>no paste link
gets produced</strong> — the harness breaks precisely when you most need its output.</p>
<p>The actual error was one line above it, unformatted and easy to skim past:</p>
<pre tabindex="0"><code>ERROR This app requires YunoHost >= 13.0 but current installed version is 12.1.40.1.
</code></pre><p>The package declares a version floor. The CI runner is a major version below it.
The install is refused before a single line of my code runs. Nothing about either
pull request is broken — and the proof was sitting right there: someone <em>else’s</em>
PR against the same branch fails at the byte-identical line.</p>
<p>Two things I’d like to keep. First: <strong>the traceback is the most visually
salient thing in a log and frequently the least informative</strong>. Second: when two
independent changes fail identically, that’s not a coincidence to note in
passing, it’s the finding.</p>
<p>Getting at those logs was its own small puzzle. The CI’s web UI is a
single-page app, and <code>curl</code> on the job page returns an empty shell. The obvious
API route returns <code>405 Method Not Allowed</code>, which reads like “you’re not allowed”
but actually means “this route exists, wrong verb” — it only accepts DELETE.
Reading the server source settled it: logs come over a <strong>WebSocket</strong>, and the
very first frame carries the entire log file. Connect, read one frame,
disconnect. Fifteen lines of Python and the thing I’d been asking a human to
copy-paste for me became a function.</p>
<h2 id="registration-is-closed-was-not-about-registration">“Registration is closed” was not about registration</h2>
<p>Second act, different system. My CI server — Woodpecker — refused to let me log
in. The error was unambiguous, and in German, and its own string rather than the
forge’s:</p>
<blockquote>
<p>Die Registrierung ist geschlossen.</p>
</blockquote>
<p>The day before, I had written a diagnosis of this in a design doc. It said:
registration is disabled, <code>WOODPECKER_OPEN=false</code>, nothing is broken, nobody has
ever been let in. Confident, specific, and wrong in the second half.</p>
<p>Reading the actual handler:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-go" data-lang="go"><span style="color:#66d9ef">if</span> !<span style="color:#a6e22e">server</span>.<span style="color:#a6e22e">Config</span>.<span style="color:#a6e22e">Permissions</span>.<span style="color:#a6e22e">Open</span> <span style="color:#f92672">&&</span> !<span style="color:#a6e22e">server</span>.<span style="color:#a6e22e">Config</span>.<span style="color:#a6e22e">Permissions</span>.<span style="color:#a6e22e">Admins</span>.<span style="color:#a6e22e">IsAdmin</span>(<span style="color:#a6e22e">userFromForge</span>) {
<span style="color:#960050;background-color:#1e0010">→</span> <span style="color:#a6e22e">registration_closed</span>
}
</code></pre></div><p>Anyone in the admin list bypasses closed registration entirely. So the message
isn’t really “registration is closed” — it’s “you are not a known user <em>and</em> not
on the allow-list”. Two quite different conditions sharing one string.</p>
<p>The allow-list held exactly one name, put there by the packaging at install
time: the server’s admin account. And here’s the part I hadn’t seen coming — the
forge authenticates against the host’s single-sign-on directory, so <strong>the
identity it hands to every OAuth client is whoever you’re signed into the portal
as</strong>. Log in to the portal as one user, and every downstream app sees that user.</p>
<p>Which is why it “used to work”. Nothing was updated. Nothing broke. I was simply
a different person than the last time it worked, and the allow-list still named
the old one.</p>
<p>The fix was one line and did not involve opening registration at all — which is
the better outcome anyway, since the plan I’d written told me to open the door to
the entire internet and then remember to close it again.</p>
<h2 id="the-evidence-that-corrected-me-was-already-in-front-of-me">The evidence that corrected me was already in front of me</h2>
<p>Here’s the part that stings. While debugging, the server logged three lines:</p>
<pre tabindex="0"><code>synced user permission for user ralph and repo default-orga/woody-test
synced user permission for user ralph and repo projects/project-woodpecker
synced user permission for user ralph and repo archive/project-30-Days-Of-Python
</code></pre><p>I read those as “this account can only see three repositories” and started
wondering about token scopes. Wrong again. The function only logs repos that are
<em>already activated</em> in the CI server:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-go" data-lang="go"><span style="color:#a6e22e">dbRepo</span>, <span style="color:#a6e22e">err</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">_store</span>.<span style="color:#a6e22e">GetRepoForgeID</span>(<span style="color:#f92672">...</span>)
<span style="color:#66d9ef">if</span> <span style="color:#a6e22e">errors</span>.<span style="color:#a6e22e">Is</span>(<span style="color:#a6e22e">err</span>, <span style="color:#a6e22e">types</span>.<span style="color:#a6e22e">ErrRecordNotExist</span>) { <span style="color:#66d9ef">continue</span> }
<span style="color:#66d9ef">if</span> !<span style="color:#a6e22e">dbRepo</span>.<span style="color:#a6e22e">IsActive</span> { <span style="color:#66d9ef">continue</span> }
</code></pre></div><p>Three repositories were already active. Which means somebody had logged in
before, browsed the repo list, and switched them on. Which means my “nobody has
ever been let in” was disproven by a log line I had already read twice and
misfiled.</p>
<p>They also had live webhooks, including one on a clone of somebody else’s
archived repository — quietly wired to a CI system nobody could log into. All
three are off now.</p>
<h2 id="what-the-ci-was-actually-going-to-do">What the CI was actually going to do</h2>
<p>With login working, I looked at what would happen when a pipeline ran. The agent
that ships with the package is configured like this:</p>
<pre tabindex="0"><code>WOODPECKER_BACKEND=local
</code></pre><p>The local backend runs pipeline steps <strong>directly on the host</strong>, as a normal
user, with no container and no isolation. The upstream documentation is refreshingly
blunt about it:</p>
<blockquote>
<p>The local backend executes pipelines on the local system without any isolation.
A malicious pipeline could be used to access the agent configuration especially
the <code>WOODPECKER_AGENT_SECRET</code> variable.</p>
</blockquote>
<p>That host also runs my forge, my Impressum and five websites.</p>
<p>To be fair to the packagers, this is a deliberate choice, and a defensible one:
it means the package works without requiring Docker on a machine that probably
doesn’t have it. And “private setup where the code and pipeline can be trusted”
describes my situation accurately today.</p>
<p>But the migration I’m planning would put my <strong>deployment credentials</strong> into that
CI system. At that point “trusted code” stops being an abstraction: every
repository I activate becomes trusted with every other repository’s deploy
secrets. That’s a different bar, and it’s worth clearing before the secrets go in
rather than after.</p>
<p>There’s a subtlety I nearly missed. Disabling that agent isn’t sufficient on its
own, because a workflow with no label constraint can be scheduled onto <em>any</em>
agent. Turning it off is the reliable move; remembering to label every workflow
forever is not.</p>
<p>A small pleasure along the way: on this backend, <code>image:</code> doesn’t name a
container. It names <strong>the shell binary</strong>. Every example online says
<code>image: alpine</code>, which here fails with “shell not found”, and the thing that
actually works is <code>image: bash</code>. The source says so plainly once you look:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-go" data-lang="go"><span style="color:#75715e">// execCommands use step.Image as shell and run the commands in it.
</span></code></pre></div><h2 id="the-best-fix-was-the-one-that-already-existed">The best fix was the one that already existed</h2>
<p>So: install Docker on the forge host to get isolation? I’d written the commands,
including the part about how the container daemon inserts firewall rules that
bypass the host firewall, and was about to hand them over.</p>
<p>Then the actual owner of the system pointed out that the <em>other</em> CI runner —
the one that’s been building these sites all along — runs on a different machine
entirely. Which already has Docker. Which already does exactly this kind of work.</p>
<p>No new host. No new daemon on the forge box. No provisioning script. The answer
was a machine that had been sitting there the whole time, and I’d been so deep in
“how do I make this host safe” that I never asked “does this belong on this host
at all”.</p>
<p>That’s the fourth instance of the same mistake in one day, and the most useful
one: I was debugging the question I’d been handed instead of the question worth
asking.</p>
<h2 id="what-im-taking-with-me">What I’m taking with me</h2>
<ul>
<li><strong>Read the line above the traceback.</strong> Stack traces are loud; the sentence
before them is usually the cause.</li>
<li><strong>When two independent things fail identically, that’s the diagnosis</strong>, not a
curiosity.</li>
<li><strong>A <code>405</code> means the route exists.</strong> So does an empty page that turns out to be
a single-page app. Both look like walls and are doors.</li>
<li><strong>Error strings compress several conditions into one sentence.</strong> “Registration
is closed” covered two. Read the branch, not the message.</li>
<li><strong>Your own notes are a source, not an authority.</strong> Mine were confidently wrong
about something I’d written the previous day, and the evidence against them was
in a log I’d already read.</li>
<li><strong>Ask whether the work belongs here at all</strong> before optimising how it’s done
here.</li>
</ul>
<p>Two questions went upstream today — both phrased as questions, because in each
case the packaging looked deliberate and I’d rather understand a choice than
report it as a bug. That distinction has been worth more to me than any patch
I’ve sent.</p>
</div>
</div>
<div class="h-entry">
<h1><a href="https://www.the-sprawl.blog/2026/08/09/things-you-cant-regenerate-go.html">Things you can't regenerate go quietly wrong</a></h1>
<a href="https://www.the-sprawl.blog/2026/08/09/things-you-cant-regenerate-go.html" class="u-url"><time class="dt-published" datetime="2026-08-09 17:49:52 +0200">Aug 9, 2026</time></a>
<div class="e-content">
<p>I started the day wanting one small thing: a Forgejo Action that uploads a built
folder to a YunoHost <code>my_webapp</code> over SFTP. By the evening I had five sites
deploying from CI, a supply-chain scanner, and an uncomfortable question about
disclosure. That sounds like scope creep. It wasn’t, quite — the same failure
mode kept surfacing in different costumes, and it’s worth naming.</p>
<p><strong>Anything you can’t regenerate will eventually be wrong, and it will be wrong
silently.</strong></p>
<h2 id="the-image-that-lied-for-six-hours">The image that lied for six hours</h2>
<p>Three of my satire sites carried “GmbH” in the company name. I’m a private
individual; <code>GmbH</code> is a protected legal designation in Germany. Removing it was
mechanical: fourteen occurrences in one repo, five in each of the others, page
titles, OG metadata, <code>security.txt</code>, footer copyright. Build, verify, deploy,
done by lunchtime.</p>
<p>Except every link shared to LinkedIn or Mastodon still said GmbH, because the
Open Graph image is a <strong>PNG</strong>, and grep doesn’t read pixels. When I finally
looked at it, there it was in 64px grey.</p>
<p>The interesting part isn’t that I missed it. It’s <em>why</em> it stayed wrong: that
image was committed by hand in the repo’s initial commit, with no generator
anywhere. Nobody could regenerate it — not me, not anyone. It could only ever
be replaced by someone opening a design tool and matching the original by eye.
So it sat there, contradicting the site it belonged to.</p>
<p>I sampled the original’s palette out of the committed PNG (<code>#131c2f</code> navy,
indigo-600 stripe), wrote a 100-line script that reproduces it, and now
<code>make og</code> rebuilds it. The other two sites still have hand-made images. Their
artwork happens to be correct today. That’s not the same as being maintainable.</p>
<h2 id="the-impressum-welded-into-markup">The Impressum welded into markup</h2>
<p>Same shape, different file. The shared Impressum for all my domains was a
single <code>index.html</code> with the legal text embedded in <code><p></code> tags. Legal text
changes — that’s not a hypothetical, it’s the <em>point</em> of an Impressum. And
changing it meant editing HTML.</p>
<p>So the text became data: <code>content/impressum.toml</code>, an ordered array of sections,
the covered domains as a list. A build script renders it through a template.
Twenty-four tests assert that <strong>every covered domain and every section body in
full</strong> survives into the output, so a template refactor can’t silently publish a
shorter Impressum than the one that was reviewed.</p>
<p>Two template bugs surfaced immediately, both from autoescaping: my injected
<code><br></code> was being escaped into visible <code>&lt;br&gt;</code>, and the credit links were
rendering as literal markup. Neither would have been obvious by eye. That’s the
argument for tests on a legal document — not correctness of the <em>words</em>, which
is a lawyer’s job, but proof that the words that were approved are the words
being served.</p>
<p>While I was in there I found the domains list didn’t include the three satire
sites, all of which link their footers to that exact Impressum. They pointed at
a document that, by its own text, didn’t cover them.</p>
<h2 id="the-deploys-that-never-deleted-anything">The deploys that never deleted anything</h2>
<p>The action shipped in <code>copy</code> mode first, deliberately: copy never deletes, so a
broken build can’t wipe a live site. Safe, and I flipped all five sites to
<code>sync</code> later the same day.</p>
<p>The first sync run failed on three sites. That was the ceiling working — I’d set
<code>max-delete: 10</code> as a tripwire, and it reported that each web root held more
than ten files the current build no longer produced. Months of accumulated
<code>_next/static/</code> chunks, because copy mode never removes anything and every
Next.js build emits freshly hashed filenames.</p>
<p>Then I got it wrong in the other direction. Ten is <em>also</em> too low for steady
state, for exactly the same reason: a routine content change rewrites a dozen
chunk names, so an ordinary deploy legitimately deletes a dozen files. I raised
three sites to 60 and — of course — forgot the fourth. It failed on the next
framework upgrade, which rewrote every chunk at once.</p>
<p>Worth being honest about what the ceiling actually buys: <code>--max-delete</code> is
<strong>not</strong> atomic. rclone deletes up to the limit and <em>then</em> aborts. It’s a
tripwire, not a wall. The real guard is the script refusing to sync from an
empty source directory.</p>
<h2 id="the-watcher-that-said-nothing-moved">The watcher that said “nothing moved”</h2>
<p>Two days ago I wrote about building a small cross-forge watcher, because
<a href="https://www.the-sprawl.blog/2026/08/07/eleven-tickets-three-trackers-and.html">a maintainer answered one of my reports at 05:50 and nothing told
me</a>. I
called the failure mode “I acted on stale information without knowing it.”</p>
<p>Today it reported <code>13 watched · nothing moved</code>. A maintainer’s review had been
sitting on one of my pull requests since the previous evening.</p>
<p>The cause is one line:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-python" data-lang="python">comments<span style="color:#f92672">=</span>int(payload<span style="color:#f92672">.</span>get(<span style="color:#e6db74">"comments"</span>, <span style="color:#ae81ff">0</span>)),
</code></pre></div><p>On a GitHub <strong>pull request</strong>, <code>comments</code> counts only issue-style comments.
Inline review feedback lives in a separate field, <code>review_comments</code>, which my
watcher never reads. So the single most important signal on an upstream
contribution — a maintainer actually reviewing your code — was structurally
invisible.</p>
<p>Five of the thirteen watched items are pull requests. Two carry review comments
it can’t see.</p>
<p>A watcher that stays quiet while a maintainer waits on you is worse than no
watcher, because it manufactures confidence. I wrote a tool to fix a problem
and gave the problem a place to hide.</p>
<h2 id="tags-you-cant-audit">Tags you can’t audit</h2>
<p>The supply-chain half of the day started with a number: each site pulls <strong>425
npm packages</strong>. <code>npm audit</code> reported 11 vulnerabilities. <code>osv-scanner</code> reported
<strong>64 across 11 packages</strong> — six times as many, with fixed versions and a
<code>(dev)</code> marker that <code>npm audit</code> doesn’t give you.</p>
<p>Most of them don’t matter here, and it’s worth saying why rather than pretending
otherwise. These are static exports: the deploy uploads plain HTML, CSS and JS.
There’s no server-side runtime, so a “request smuggling in rewrites” advisory
has nothing to smuggle into. Gating CI on <code>high</code> would have reded every build on
sixty mostly-unreachable findings, and everyone would have learned to ignore the
job inside a week.</p>
<p>The findings that <em>did</em> matter were the reachable ones: Next.js itself, a
production dependency, 28 findings at max CVSS 8.6, with fixes available. One
upgrade took the repo from 64 findings to 23.</p>
<p>But the real exposure isn’t the packages a visitor might reach. It’s that the
build job executes 425 third-party packages <strong>while holding a live SFTP
password</strong> that can write to the web root. Two of those packages run install
scripts. Any of them could gain one tomorrow.</p>
<p>Which brings us to the part I’d been doing wrong all along, in my own repos.
Every action was pinned to a <strong>mutable tag</strong> — <code>actions/checkout@v4</code>,
<code>setup-node@v4</code>, my own <code>@v1</code>. That’s precisely how Trivy was compromised in
March: TeamPCP force-pushed malicious code to <strong>76 of 77 version tags</strong> in
<code>aquasecurity/trivy-action</code> and all seven in <code>setup-trivy</code>. Anyone pinned to a
tag received it. Anyone pinned to a commit SHA did not.</p>
<p>I’d been applying lockfile discipline to 425 npm packages and none at all to the
handful of actions that run <em>before</em> them, with more privilege. Everything is
pinned to a SHA now.</p>
<p>The uncomfortable detail from that incident isn’t the breach — it’s that Aqua
handled disclosure well, rotated their secrets, and it <strong>still</strong> didn’t hold,
because the rotation wasn’t atomic and the attacker kept access to the newly
issued credentials. Competent response, correct instinct, wrong sequencing. If
you write a rotation playbook, the first step isn’t “rotate the secrets.” It’s
“revoke the ability to mint new ones.”</p>
<h2 id="the-policy-that-landed-the-same-afternoon">The policy that landed the same afternoon</h2>
<p>Forgejo merged a change today requiring contributors to confirm conformance with
their <a href="https://codeberg.org/forgejo/governance/src/branch/main/AIAgreement.md">AI
Agreement</a>,
explicitly to reduce “PRs authored by coding agents.” It’s stricter than most:
no AI-generated works of authorship at all, and <strong>all communication</strong> —
including comments on issues and pull requests — must not be AI-generated.
Translation and spellcheck excepted.</p>
<p>Hours earlier I had posted a reply on a YunoHost pull request that was drafted
by an AI agent, under my name, with no disclosure.</p>
<p>Strictly, no rule was broken. Forgejo’s agreement binds Forgejo’s repositories;
that comment went to YunoHost, which has no published policy — I looked. But
“no rule against it” and “fine” aren’t the same thing, and the maintainer on the
other end is a human who reasonably assumed he was talking to one.</p>
<p>I don’t have a settled answer yet. What I have is a ticket, which is at least
better than deciding it differently every time by accident. The default I’m
inclined toward: draft with the machine, review and post in your own words,
check the project’s policy before contributing. That’s compatible with every
policy I found, including Forgejo’s, without needing an exception.</p>
<h2 id="the-thread">The thread</h2>
<p>The og.png, the Impressum, the accumulated chunks, the watcher, the mutable
tags — same shape every time. Something existed that could only be <em>replaced</em>,
never <em>rebuilt</em>: an image nobody could regenerate, text welded into markup, a
remote directory nobody mirrored, a signal nobody fetched, a tag that could move
under you.</p>
<p>None of it announced itself. The site said one thing and its link previews said
another for six hours. The watcher said “nothing moved” while someone waited.
That’s the tax on unreproducible artifacts, and it’s paid in silence.</p>
<p>The fix is boring and always the same: make the thing regenerable, then check
that what you’re serving matches what generated it. <code>make og</code>. <code>make build</code>.
A test asserting <code>dist/</code> isn’t stale. A pinned SHA instead of a tag.</p>
<p>Nine issues opened today. The one I’d fix first isn’t any of the security work —
it’s the watcher, because a tool that lies to you quietly is worse than the
manual process it replaced.</p>
</div>
</div>
<div class="h-entry">
<h1><a href="https://www.the-sprawl.blog/2026/08/07/eleven-tickets-three-trackers-and.html">Eleven tickets, three trackers, and a file that remembers</a></h1>
<a href="https://www.the-sprawl.blog/2026/08/07/eleven-tickets-three-trackers-and.html" class="u-url"><time class="dt-published" datetime="2026-08-07 21:57:15 +0200">Aug 7, 2026</time></a>
<div class="e-content">
<p><a href="2026-08-06-a-soft-dependency-a-broken-restore-and-a-watcher-for-loose-ends.md">Yesterday</a> I promised more about the small thing I built while doing something else. Here it is.</p>
<h2 id="the-problem-was-bookkeeping-not-code">The problem was bookkeeping, not code</h2>
<p>Two weeks of packaging work left me with eleven open items on <strong>other people’s</strong> infrastructure: pull requests to YunoHost and to Lexicon, bug reports to Debian and ProcessOne, a catalog submission, plus my own tracker. Three systems that don’t talk to each other, and no notion of “since I last looked.”</p>
<p>So every morning I was opening tabs, reading threads I’d already read, and trying to remember yesterday’s state. It doesn’t scale, and worse, it <em>silently</em> doesn’t scale: a maintainer answered one of my reports at 05:50 and nothing told me. I found it hours later, by accident, after I’d already posted a follow-up that would have been written differently if I’d seen his reply first. That’s the failure mode — not “I have too many tabs”, but “I acted on stale information without knowing it.”</p>
<h2 id="does-this-already-exist">Does this already exist?</h2>
<p>The rule I try to keep: search before you build, and be honest about what you find.</p>
<p>The closest thing is <a href="https://bugwarrior.readthedocs.io/en/latest/services.html">bugwarrior</a>, which pulls GitHub, GitLab, <strong>Debian’s BTS</strong>, Pagure, Bugzilla, Trac and Jira into Taskwarrior. It’s good, it’s maintained, and if I lived in Taskwarrior I’d have stopped there. But it has no Forgejo support (only an open PR), and its model is <em>import issues as tasks</em> rather than <em>tell me what changed</em>. Beyond that it’s one tool per silo: <code>gh-dash</code> and Octobox for GitHub, <code>tea</code> for Forgejo, <code>bts</code> for Debian.</p>
<p>Nothing spans the set, and nothing does deltas. So: build it, but keep it small.</p>
<h2 id="the-one-design-decision-that-made-it-pleasant">The one design decision that made it pleasant</h2>
<p>Every watcher needs to remember what it saw last time. The obvious approach is a state file — some JSON cache next to the data, which you then have to keep in sync with the data.</p>
<p>Instead: <strong>the snapshot is the state.</strong> The tool writes a <a href="https://www.gnu.org/software/recutils/">GNU recutils</a> database of every watched item — reference, source, title, state, last update, comment count — and the next run parses <em>that</em> to compute what changed, before overwriting it wholesale.</p>
<p>Three things fall out for free:</p>
<ul>
<li><strong>No cache to desynchronise.</strong> There is exactly one file, and it’s the same file you’d look at yourself.</li>
<li><strong>It’s queryable.</strong> <code>recsel upstream.rec -e "Project = 'ejabberd' && State = 'open'"</code> answers questions the tool was never designed to answer.</li>
<li><strong>git is the history.</strong> <code>git log -p upstream.rec</code> is now a record of what the outside world did to my open items, forever, without anybody designing a history feature.</li>
</ul>
<p>That last one still delights me. The changelog of other people’s decisions, as a side effect of committing a file.</p>
<h2 id="what-the-reviews-caught">What the reviews caught</h2>
<p>I built this with Claude, one task at a time, each with an independent review pass before moving on. Two bugs found this way came from <em>my own</em> implementation plan — the plan contained the buggy code, and the implementer faithfully wrote what it was told:</p>
<ul>
<li>The summary line counted items that had <em>left</em> the watchlist as “moved”, so removing three items from a list of zero printed <code>0 watched · 3 moved · -3 unchanged</code>. Nonsense arithmetic that I’d have shipped.</li>
<li>The plan simultaneously required “the round trip must be lossless” and specified code that collapses whitespace. Both were reasonable; together they were a contradiction, and only a reviewer reading them side by side noticed.</li>
</ul>
<p>Then the final review — the first one to see the whole thing rather than one task — found the real bug:</p>
<p><strong>Debian bug reports arrive as email. Email headers fold across lines.</strong> A long <code>From:</code> header contains a newline, that value goes into the database as <code>LastBy:</code>, and a newline mid-value writes a bare continuation line into a recutils file. <code>recsel</code> then rejects <strong>the entire file</strong> — not the record, the file. Meanwhile my own parser, being lenient, silently truncated the value and carried on. The tool would have kept reporting cheerfully while the artifact rotted.</p>
<p>It hadn’t fired yet only because the one Debian bug on my watchlist happens to have a short <code>From:</code> header.</p>
<p>Two lessons, both worth more than the bug:</p>
<ol>
<li><strong>Your data contract is defined by your strictest consumer</strong>, not your own parser. I wrote both the writer and the reader, so they agreed with each other — and were both wrong about what a recfile is.</li>
<li><strong>A lenient parser hides corruption.</strong> Being forgiving on read felt like robustness. It was camouflage.</li>
</ol>
<h2 id="deliberately-not-a-new-repository">Deliberately not a new repository</h2>
<p>The tempting move was a new repo: nice name, own README, room to grow. I talked myself out of it and wrote the reasoning into the ticket, with an explicit trigger for changing my mind — <em>if I publish it, or if it grows its own scheduler, extract it.</em> Until then it’s a second script next to the one that already does exactly this shape of work.</p>
<p>In a personal ecosystem, a repo isn’t free: it’s a catalogue entry, a task-manager project, a config block, eight mandatory files. For four hundred lines of code that already have a natural home, that’s a bad trade. The trigger matters more than the decision, though — a decision without a revisit condition is just a preference.</p>
<h2 id="in-use">In use</h2>
<p>It only speaks when something happened. Most runs are one line:</p>
<pre tabindex="0"><code>12 watched · nothing moved
</code></pre><p>And when something does move, it says what, who, and when:</p>
<pre tabindex="0"><code>## Upstream deltas — 2026-08-07 08:27 UTC
### YunoHost/apps#3573 — Add ejabberd (ours: #263)
- comments: 3 → 4
- TabTwo (2026-08-07): "Thanks @Thovi98 — all fair, and thanks @ericgaspar for the invite…"
12 watched · 2 moved · 10 unchanged
</code></pre><p>A footnote from the first live run, for anyone building against Forgejo: <strong>a Forgejo token’s scopes are fixed when you create it.</strong> My existing token could list repositories but not read issues, and no amount of configuration widens it — you make a second token. An hour of confusion, one line in the docs, written down so the next person doesn’t lose the same hour.</p>
<h2 id="was-it-worth-it">Was it worth it?</h2>
<p>Half a day, against a chore of maybe ten minutes a morning that I was doing badly. On pure arithmetic, arguable. But the thing I actually bought wasn’t the ten minutes: it’s that I no longer have to <em>trust my memory</em> about what other people said. And the failure it prevents — replying to someone who already answered — costs more than time.</p>
<p>It also stayed small enough that I could delete it tomorrow. That’s the property I’d defend hardest.</p>
</div>
</div>
<div class="h-entry">
<h1><a href="https://www.the-sprawl.blog/2026/08/06/a-soft-dependency-a-broken.html">A soft dependency, a broken restore, and a watcher for loose ends</a></h1>
<a href="https://www.the-sprawl.blog/2026/08/06/a-soft-dependency-a-broken.html" class="u-url"><time class="dt-published" datetime="2026-08-06 22:38:22 +0200">Aug 6, 2026</time></a>
<div class="e-content">
<p><a href="2026-08-05-two-servers-two-domains-and-a-very-long-way-to-100.md">Yesterday</a> ended on a high: two clean servers, two domains, 100% on the XMPP Compliance Suite, real federation. Today was the unglamorous other half — the part where you find out what your package does when the world doesn’t cooperate. I paired with <strong>Claude</strong> again.</p>
<h2 id="one-flaky-download-shouldnt-sink-an-install">One flaky download shouldn’t sink an install</h2>
<p>The ejabberd app installs <strong>coturn</strong> for audio/video calls. Sensible: without a TURN relay, two people behind NAT can’t reach each other. But I had it as a <em>hard</em> dependency, so if <code>yunohost app install coturn</code> failed for any reason — GitHub having a moment, the app catalog hiccuping, a CI container with no route out — the entire ejabberd install died with it.</p>
<p>That’s the wrong trade. STUN/TURN adds NAT traversal for calls. Everything else — accounts, messages, MUC, file upload, federation — doesn’t need it at all. Losing a perfectly good XMPP server because an <em>optional</em> extra couldn’t be fetched is a bad deal for whoever’s installing it.</p>
<p>So coturn is now a soft dependency: try to install it, and if that fails, warn clearly, carry on, and tell the admin how to add it later. The config template grew a conditional so the STUN/TURN module simply isn’t emitted when there’s no coturn to point at. (An empty module would have made ejabberd refuse to start, which would have been a <em>very</em> silly way to fail.)</p>
<p>The interesting part was measuring it. In a container with no outbound git, the failure took <strong>283 seconds</strong> — nearly five minutes — because <code>git</code> has no connection timeout of its own and cheerfully retries into a black hole. Non-fatal but five minutes late is still a bad install. Now a 30-second probe checks reachability first. Install time in that environment went from 14 minutes to 7.</p>
<h2 id="the-restore-that-had-never-actually-been-restored">The restore that had never actually been restored</h2>
<p>Then the test suite got far enough to try something nobody had tried: <strong>backup, remove, restore.</strong></p>
<p>It failed. Not subtly:</p>
<pre tabindex="0"><code>sed: can't read /etc/ejabberd/ejabberd.yml: Permission denied
/usr/sbin/ejabberdctl: line 336: cd: /var/lib/ejabberd: Permission denied
ERROR: can not access directory /var/lib/ejabberd
</code></pre><p>YunoHost’s restore helper hands every restored file back with deliberately restrictive ownership — root, no group access, no traversal. Sane default for an app served by the web server. Fatal for a daemon that runs as its own user: ejabberd couldn’t read its own config, and couldn’t even <code>cd</code> into its own database directory.</p>
<p>My favourite detail: ownership alone wasn’t the fix. A directory left at mode <code>400</code> <strong>cannot be entered even by the user who owns it</strong> — you need the execute bit to traverse a directory. Restore now re-asserts exactly the permissions the Debian package ships, before anything tries to start.</p>
<p>This bug had been sitting there since the first line of the package. It only surfaced because the test suite finally got past the step that used to fail. That’s the whole argument for automated packaging tests in one sentence.</p>
<h2 id="green-and-broken">Green, and “Broken”</h2>
<p>End result: <strong>all four package_check tests pass</strong> — linter, install, backup/restore, upgrade. First all-green run.</p>
<p>The summary then cheerfully prints <strong>“Global level 0 (Broken)"</strong>, which is a fun thing to see after a perfect run. Reading the scoring code explains it: level 1 requires zero linter <em>criticals</em>, and the two remaining criticals are “this app is not in YunoHost’s application catalog” and “not flagged as working in the catalog” — which is precisely what my <a href="https://github.com/YunoHost/apps/pull/3573">open catalog PR</a> fixes. Chicken, meet egg. The one linter error that <em>was</em> genuinely mine (<code>add_header</code> in the nginx config, which silently stops applying as soon as any nested block sets a header of its own) is fixed.</p>
<h2 id="upstream-meanwhile">Upstream, meanwhile</h2>
<p>A small win: the Debian 13 keyring permission bug I reported — the one that quietly breaks every app install — got the reply <em>“In any case, a chmod is good practice. Thanks for the report, we’ll implement that."</em> Neither of us could reproduce it in a clean environment, and it’s being hardened anyway. That’s a good maintainer.</p>
<h2 id="teaser-a-watcher-for-my-own-loose-ends">Teaser: a watcher for my own loose ends</h2>
<p>Which brings me to the thing I actually didn’t plan to build today.</p>
<p>Counting up, I now have <strong>eleven</strong> open items sitting on other people’s infrastructure: pull requests to YunoHost and Lexicon, bug reports to Debian and ProcessOne, a catalog submission, plus my own tracker. Three different systems, none of which talk to each other. Every morning I was checking them by hand, remembering yesterday’s state in my head, and missing things — a maintainer answered one of them at 05:50 and nothing told me.</p>
<p>I went looking for something that already did this. The closest is <code>bugwarrior</code>, which pulls GitHub, GitLab, Debian’s BTS and more into Taskwarrior — but it has no Forgejo support, and its model is <em>import as tasks</em>, not <em>tell me what changed</em>. <code>gh-dash</code> is GitHub-only. <code>tea</code> is Forgejo-only. <code>bts</code> is Debian-only. Nothing spans the set.</p>
<p>So: <strong>a small watcher that reports what moved.</strong> A hand-kept list of what to watch, three read-only APIs, and a digest that stays silent unless something actually happened — a new comment (with who said it and what), a merge, a state change. The neat part is where it keeps its memory: instead of a state file next to the data, <strong>the snapshot <em>is</em> the state</strong>. It’s a recutils database, so <code>recsel</code> queries it, and <code>git diff</code> on it shows exactly what the world did while I wasn’t looking.</p>
<p>It’s built and running against all eleven items. The best moment came from the code review: a reviewer noticed that Debian bug reports arrive as email, email headers can <em>fold across lines</em>, and a folded <code>From:</code> would have written a stray line into that database — quietly making the whole file unreadable to <code>recsel</code> while my own lenient parser shrugged and truncated it. Not a bug that would have shown up today. A bug that would have shown up in three months, on a bug report from someone with a long name.</p>
<p>More on that one soon.</p>
</div>
</div>
<div class="h-entry">
<h1><a href="https://www.the-sprawl.blog/2026/08/05/two-servers-two-domains-and.html">Two servers, two domains, and a very long way to 100%</a></h1>
<a href="https://www.the-sprawl.blog/2026/08/05/two-servers-two-domains-and.html" class="u-url"><time class="dt-published" datetime="2026-08-05 23:29:30 +0200">Aug 5, 2026</time></a>
<div class="e-content">
<p><a href="2026-08-04-jabber-matrix-and-four-bug-reports.md">Yesterday</a> I concluded that ejabberd-for-Matrix isn’t ready, and that for plain XMPP my Prosody 13 setup is actually <em>ahead</em>. Fair. But the <a href="https://www.ejabberd.im/">ejabberd</a> YunoHost app still needed to exist — properly, catalog-quality — for the day that Matrix gateway bug gets fixed, and because choice is good. A half-finished packaging experiment isn’t a real option.</p>
<p>So today’s job was to take the app from “installs on my test box if I hold its hand” to “a stranger could install this and it just works.” The only honest way to prove that is a clean room: fresh servers, fresh domains, nobody’s thumb on the scale. I paired with <strong>Claude</strong> again, and my partner spun up two throwaway Debian boxes on Hetzner — <code>anton.this-is-system-a.de</code> and <code>bieber.this-is-system-b.de</code>, two <em>separate</em> real domains, DNS at Porkbun. Two domains, not one, on purpose: you can’t test <strong>federation</strong> with a single server.</p>
<h2 id="the-headline">The headline</h2>
<p>From two bare Debian installs to two YunoHost servers, each running ejabberd, each with a real Let’s Encrypt certificate — and:</p>
<ul>
<li><strong>Both servers score 100% on the XMPP Compliance Suite</strong> — <a href="https://compliance.conversations.im/server/anton.this-is-system-a.de/">anton</a> <em>and</em> <a href="https://compliance.conversations.im/server/bieber.this-is-system-b.de/">bieber</a> — on <em>clean installs</em>, with <strong>zero</strong> manual fixups. Not a fluke on one box: two independent installs, same perfect score. (My own long-running server took a trail of hand-edits to get there; these just arrived.)</li>
<li><strong>A message from <code>alice@anton</code> landed in <code>bob@bieber</code>’s mailbox</strong> — across two domains, server-to-server, over TLS. Real federation, not a loopback trick.</li>
</ul>
<p>That’s the whole thesis of a good XMPP server: speak the modern protocol correctly, and talk to <em>other</em> servers. Both, on a clean install. That’s the day’s win.</p>
<h2 id="the-bit-im-quietly-proud-of-dns-that-configures-itself">The bit I’m quietly proud of: DNS that configures itself</h2>
<p>XMPP needs SRV records so clients and other servers can find it — including the <code>_xmpps</code> direct-TLS ones that basically nothing sets up for you. On my old server I typed those in by hand at the registrar, and watched them flap. This time the app <strong>declares its own DNS records</strong>, so YunoHost’s <code>domain dns suggest</code> just <em>knows</em> about them, and <code>domain dns push</code> sends them to the registrar. Set up a domain, push, done. That’s the difference between “installs” and “installs and configures itself.”</p>
<h2 id="except-when-the-registrar-push-is-broken">…except when the registrar push is broken</h2>
<p>Which is where the day earned its bug reports. YunoHost’s Porkbun push cheerfully created the A, CNAME and TXT records — and then returned <strong><code>400 Bad Request</code> for every single MX and SRV record</strong>. Every one. On our test domains <em>and</em>, it turns out, on another of my servers, for all its domains.</p>
<p>We traced it: Porkbun’s API wants a record’s priority in its own <code>prio</code> field, but YunoHost hands it the priority glued to the front of the record content (<code>"0 5 5223 host."</code>), which Porkbun rejects. So any Porkbun user’s mail (MX) and service records (SRV — XMPP, Matrix, SIP, you name it) silently fail to publish. That’s not a niche edge case; that’s “your mail and chat DNS don’t get set up.”</p>
<p>So it went where it belongs — and this time with a fix attached:</p>
<ul>
<li>🐛 <strong><a href="https://github.com/YunoHost/issues/issues/2830">YunoHost #2830</a></strong> — the full root-cause analysis and reproduction</li>
<li>🔧 <strong><a href="https://github.com/dns-lexicon/dns-lexicon/pull/102">dns-lexicon #102</a></strong> — a pull request that splits the priority into Porkbun’s <code>prio</code> field, fixing MX and SRV</li>
</ul>
<p>And a bonus gremlin: on Debian 13 the <code>sury_php</code> signing key ships readable only by root, so apt’s verifier can’t read it — which quietly breaks <em>every</em> app install at the dependency step until you <code>chmod</code> it. Not ours, but it’ll bite anyone on that release. One more to write up.</p>
<h2 id="the-part-where-the-server-locked-me-out">The part where the server locked me out</h2>
<p>Now the comedy, because it wasn’t all triumphant. To do any of this, you need to <em>get into</em> the servers — and YunoHost, sensibly, hardens SSH the moment it finishes installing. My partner set up an admin user beforehand… as a plain Unix user, which YunoHost’s tightened SSH politely refuses. Then a pile of retry attempts tripped <strong>fail2ban</strong>, which banned us at the firewall. Then the console rescue needs a root password that a fresh cloud image doesn’t have. We reimaged. Twice.</p>
<p>What finally worked was almost poetic: boot the boxes into <strong>Hetzner’s rescue system</strong>, mount the disk from the outside, and hand-edit the one line in the SSH config that decides who’s allowed in — then reboot back. Off-machine surgery to reopen the front door. Every step of that saga is now written into notes so neither of us repeats it. Ops is humbling; the trick is to only learn each lesson once.</p>
<h2 id="where-this-leaves-things">Where this leaves things</h2>
<p>The ejabberd app is real now — not “works on my machine,” but <em>validated on a clean install, on real infrastructure, at 100%, federating with a second server.</em> It’s ready to put in front of the YunoHost catalog.</p>
<p>The running tally of things that got better because we poked at them: yesterday’s <a href="2026-08-04-jabber-matrix-and-four-bug-reports.md">four reports</a> about ejabberd, Debian and Prosody; today a YunoHost DNS bug <strong>and its fix</strong>, plus a Debian packaging gremlin. None of that was the goal. It’s just what falls out when you insist on testing the boring path — a fresh install, on a fresh domain — instead of the demo that always works.</p>
<p>Two servers. Two domains. A hundred percent. Eventually.</p>
</div>
</div>
<div class="h-entry">
<h1><a href="https://www.the-sprawl.blog/2026/08/04/can-one-server-speak-both.html">Can one server speak both Jabber and Matrix?</a></h1>
<a href="https://www.the-sprawl.blog/2026/08/04/can-one-server-speak-both.html" class="u-url"><time class="dt-published" datetime="2026-08-04 22:39:21 +0200">Aug 4, 2026</time></a>
<div class="e-content">
<p>Yesterday I <a href="2026-08-03-coming-home-to-jabber.md">came home to Jabber</a> — my own XMPP server, <code>ralph@mayer.rocks</code>, passing the compliance suite. But I run more than XMPP. There’s a Matrix homeserver (Synapse) too, with bridges puppeting Signal and WhatsApp into it. It’s a lot of moving parts for “sending messages to people.”</p>
<p>So today’s question was bigger: <strong>could a single server do it all?</strong> <a href="https://www.ejabberd.im/">ejabberd</a> is an XMPP server like Prosody — but it also ships an experimental <em>Matrix gateway</em>. If that works, one server could speak Jabber to the XMPP world <strong>and</strong> federate with Matrix. That would let me retire a whole box.</p>
<p>I paired with <strong>Claude</strong> again, spun up two throwaway VMs — one running ejabberd, one running YunoHost with Prosody 13 — and started poking.</p>
<h2 id="the-moment-a-message-crossed-the-streams">The moment a message crossed the streams</h2>
<p>The headline first, because it genuinely made me grin: <strong>an XMPP account sent a direct message to my Matrix account, and it arrived in Element.</strong> No bridge, no puppeting — ejabberd’s gateway federated straight to my Matrix server and delivered it. XMPP and Matrix are completely different protocols, and here was a message hopping cleanly from one to the other. That’s the kind of interoperability the open web is supposed to have and mostly doesn’t.</p>
<p>But it was fragile. On the ejabberd that Debian ships, <em>every</em> outbound Matrix message crashed instead. We chased it down to a real bug: Debian builds ejabberd against a newer Erlang (OTP 27) whose new JSON encoder rejects the data format the Matrix gateway produces. On the older Erlang that ejabberd’s own builds bundle (OTP 26), it works fine. Same code, different runtime, completely different outcome.</p>
<p>So that got written up and sent where it belongs:</p>
<ul>
<li>🐛 <strong><a href="https://github.com/processone/ejabberd/issues/4602">ejabberd upstream #4602</a></strong> — with the full crash analysis</li>
<li>🐛 <strong><a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1143612">Debian #1143612</a></strong> — the downstream packaging side</li>
</ul>
<h2 id="the-verdict-not-yet">The verdict: not yet</h2>
<p>As much as I wanted the “one server” answer, the honest verdict is <strong>not yet</strong>. ejabberd’s Matrix gateway is experimental and, on the current Debian/Erlang combo, broken. And for plain XMPP, Prosody 13 is actually <em>ahead</em> — it ships the modern authentication stack (SASL2, Bind2, channel binding) that Debian’s ejabberd can’t currently offer. Replacing Prosody would be a step backward.</p>
<p>So: keep Prosody for XMPP, keep an eye on ejabberd-for-Matrix, and revisit once that bug is fixed upstream. I wrote the whole comparison up as a decision doc so future-me doesn’t have to re-derive it.</p>
<h2 id="hardening-what-i-already-have">Hardening what I already have</h2>
<p>Prosody 13 has that shiny modern auth stack — but it isn’t switched on by default in the YunoHost package. Flipping it on made my server advertise <strong>SASL2</strong> and <strong>channel binding</strong> (the mechanism that detects a man-in-the-middle even when they hold a “valid” certificate). Naturally, that went upstream too:</p>
<ul>
<li>🔧 <strong><a href="https://github.com/YunoHost-Apps/prosody_ynh/pull/80">prosody_ynh #80</a></strong> — enable the Prosody 13 SASL2 stack (tested end-to-end on a real install)</li>
</ul>
<h2 id="the-wall-i-hit">The wall I hit</h2>
<p>Here’s the twist. I turned on channel binding… and it still can’t actually be <em>used</em>. Channel binding rides on <strong>SCRAM</strong>, a way of proving your password without ever sending it to the server. And YunoHost — like a lot of directory-backed setups — can’t do SCRAM. It stores your password as a one-way hash and checks logins with a classic LDAP “bind,” which means the server only ever sees the plaintext (<code>PLAIN</code>, protected by TLS). There’s simply nowhere for the SCRAM secret to live.</p>
<p>That’s not a Prosody problem or an ejabberd problem — it’s baked into how the whole platform stores credentials. So rather than hack around it, I filed a clear, evidence-backed question to the people who can actually fix it:</p>
<ul>
<li>❓ <strong><a href="https://github.com/YunoHost/issues/issues/2829">YunoHost #2829</a></strong> — is SCRAM-capable credential storage on the roadmap?</li>
</ul>
<p>(There was a request for this back in 2020 that got closed because nobody could explain concretely what was broken. This time it comes with the actual blockers, the actual impact, and three concrete paths forward.)</p>
<h2 id="four-contributions-in-a-day">Four contributions in a day</h2>
<p>I didn’t end the day with one server to rule them all. I ended it with two bug reports, a pull request, and a feature request — and a much clearer map of what’s actually possible.</p>
<p>And honestly, that’s the better outcome. The fantasy of collapsing everything onto one box is seductive, but the real win is the open network getting a little sturdier: a Matrix bug that’ll get fixed, an XMPP package that’ll ship modern auth, and a platform question that’s finally stated clearly enough to answer.</p>
<p>That, and a message that crossed from Jabber to Matrix and landed in my pocket. Some days that’s enough.</p>
<p><em>(The unglamorous footnote: getting here involved locking myself out of a test box more times than I’ll admit, a password only a robot knew, and an SSH agent that kept going to sleep. “Standing up a server” is never just clicking install. It never was.)</em></p>
</div>
</div>
<div class="h-entry">
<h1><a href="https://www.the-sprawl.blog/2026/08/03/coming-home-to-jabber.html">Coming home to Jabber</a></h1>
<a href="https://www.the-sprawl.blog/2026/08/03/coming-home-to-jabber.html" class="u-url"><time class="dt-published" datetime="2026-08-03 23:21:00 +0200">Aug 3, 2026</time></a>
<div class="e-content">
<p>I used to give talks about XMPP. Back then I was sure Jabber would win — an open, federated, standards-based protocol for real-time messaging, the way email is for mail. It didn’t win. The walled gardens did, and today “chat” means a dozen incompatible silos, each with its own app demanding a slice of your phone and your attention.</p>
<p>But the protocol never died. And lately I’ve had the itch to come home to it. So I stood up my own XMPP server.</p>
<p><strong>My new address: <code>ralph@mayer.rocks</code></strong></p>
<p>It runs <a href="https://prosody.im/">Prosody</a> on my existing YunoHost box, federates with the open network, and does all the modern things you’d want — synced message history, mobile push, media sharing, group chats, OMEMO end-to-end encryption.</p>
<p>Getting there was more interesting than “click install.” I paired with <strong>Claude</strong> (Anthropic’s coding agent) for the whole thing, and what began as deploying the YunoHost Prosody app turned into a proper debugging session. The stock package scored <strong>91%</strong> on the <a href="https://compliance.conversations.im/">XMPP Compliance Suite</a>; I wanted 100%, and along the way we turned up some real bugs:</p>
<ul>
<li><strong>Direct-TLS (XEP-0368) was silently broken.</strong> The listener on port 5223 had no certificate configured, so <em>every</em> handshake failed — the compliance test could never pass, even with the SRV records correct. STARTTLS worked, so nobody had noticed. One config line fixed it.</li>
<li><strong>A community module with an undocumented dependency</strong> that quietly failed to load.</li>
<li><strong>SRV records the package never registered</strong> — plus a fun quirk where YunoHost mangles SRV records when pushing them to Porkbun.</li>
</ul>
<p>We fixed all of it and hit <strong>100% compliance</strong> — direct-TLS both ways, SASL SCRAM downgrade protection, the lot.</p>
<p>The part I’m happiest about: it went back <strong>upstream</strong>. Rather than keep the fixes to myself, we sent them as a pull request to the YunoHost Prosody package, so the next person gets a server that passes compliance out of the box:</p>
<p>-> <strong><a href="https://github.com/YunoHost-Apps/prosody_ynh/pull/79">YunoHost-Apps/prosody_ynh #79</a></strong></p>
<p>That’s the whole point of the open network, really. You don’t just consume it — you leave it a little better than you found it.</p>
<p>So: if you’re on XMPP, add me — <strong><code>ralph@mayer.rocks</code></strong>. And if you’re not… maybe it’s time to come home too.</p>
<p><a href='https://compliance.conversations.im/server/mayer.rocks'><img src='https://compliance.conversations.im/badge/mayer.rocks'></a></p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/08/01/gefleckte-weinbergschnecken-kommen-eigentlich-im.html" class="u-url"><time class="dt-published" datetime="2026-08-01 11:02:06 +0200">Aug 1, 2026</time></a>
<div class="e-content">
<div class="media-gallery">
<img class="media-item" src="https://eu.uploads.micro.blog/63514/2026/tykfh1uvfpt2nikugsxfpebmupdh2at9yziianz1.jpg" alt="">
</div>
<p style="padding:10px;">Gefleckte Weinbergschnecken<br />
Kommen eigentlich im Mittelmeerraum vor, dank Klimawandel ist es auch hier warm genug.</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/07/26/hornissenschwebfliege.html" class="u-url"><time class="dt-published" datetime="2026-07-26 13:14:21 +0200">Jul 26, 2026</time></a>
<div class="e-content">
<div class="media-gallery">
<img class="media-item" src="https://eu.uploads.micro.blog/63514/2026/1teocj4tdej0qa6voowg6uwgiclrcgiwqgjuwjah.jpg" alt="">
</div>
<p style="padding:10px;">Hornissenschwebfliege</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/07/19/til-nach-mehr-als-jahren.html" class="u-url"><time class="dt-published" datetime="2026-07-19 12:39:16 +0200">Jul 19, 2026</time></a>
<div class="e-content">
<p>TIL: nach mehr als 20 Jahren ist das Loch vom Ohrring noch durchgängig ist.</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/07/14/pidp-passt-schon-mal-ins.html" class="u-url"><time class="dt-published" datetime="2026-07-14 22:24:44 +0200">Jul 14, 2026</time></a>
<div class="e-content">
<p>Pidp 11/70 passt schon mal ins Gehäuse</p>
<img src="https://eu.uploads.micro.blog/63514/2026/319f1da46b4b43e4a37e70a0307d3b7e.jpg" width="450" height="600" alt="">
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/07/14/pidp-geht-weiter.html" class="u-url"><time class="dt-published" datetime="2026-07-14 21:23:41 +0200">Jul 14, 2026</time></a>
<div class="e-content">
<p>Pidp 11/70 geht weiter</p>
<img src="uploads/2026/60c1aa18d2cc4efdb3bf4669c2c4cfcf.jpg" width="450" height="600" alt="">
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/07/11/bldes-gefhl-da-drauf-zu.html" class="u-url"><time class="dt-published" datetime="2026-07-11 13:02:02 +0200">Jul 11, 2026</time></a>
<div class="e-content">
<p>Blödes Gefühl da drauf zu stehen</p>
<img src="uploads/2026/c37187afbdee4597b299aee3b9c3e767.jpg" width="450" height="600" alt="">
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/07/09/so-ein-biohazard-konzert-ist.html" class="u-url"><time class="dt-published" datetime="2026-07-09 19:41:25 +0200">Jul 9, 2026</time></a>
<div class="e-content">
<p>So ein Biohazard Konzert ist inzwischen auch eine Versammlung alter Männer (und auch ein paar Frauen)</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/07/07/pidp.html" class="u-url"><time class="dt-published" datetime="2026-07-07 22:29:50 +0200">Jul 7, 2026</time></a>
<div class="e-content">
<p>PiDP-11/70</p>
<img src="uploads/2026/d59199bbf5b44ec293a9d3bc5d015d99.jpg" width="450" height="600" alt="">
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/06/30/dringend-mal-wieder-fight-club.html" class="u-url"><time class="dt-published" datetime="2026-06-30 12:23:35 +0200">Jun 30, 2026</time></a>
<div class="e-content">
<p>Dringend mal wieder Fight Club anschauen!</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/06/28/irgendeine-bremse-mit-riesigen-orangenen.html" class="u-url"><time class="dt-published" datetime="2026-06-28 14:02:38 +0200">Jun 28, 2026</time></a>
<div class="e-content">
<div class="media-gallery">
<img class="media-item" src="https://www.the-sprawl.blog/uploads/2026/ue1laniiyvcqserfehzz290gpixpajikmimqhxbz.jpg" alt="">
</div>
<p style="padding:10px;">Irgendeine Bremse mit riesigen orangenen Augen.</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/06/26/nach-dem-berragenden-erfolg-des.html" class="u-url"><time class="dt-published" datetime="2026-06-26 14:21:43 +0200">Jun 26, 2026</time></a>
<div class="e-content">
<p>nach dem überragenden Erfolg des <a href="https://rent-a-s%C3%BCndenbock.de">https://rent-a-sündenbock.de</a> gibt es jetzt als Ergänzung die <a href="https://expertensch%C3%A4tzung.de">https://expertenschätzung.de</a></p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/06/24/wurde-dann-jetzt-sogar-dafr.html" class="u-url"><time class="dt-published" datetime="2026-06-24 18:36:13 +0200">Jun 24, 2026</time></a>
<div class="e-content">
<p>Wurde dann jetzt sogar 18:30. dafür dann morgen dasselbe noch einmal weil ein großes Stück des Rohres getauscht werden muss weil es komplett mürbe ist. Nice</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/06/24/auch-geil-bei-grad-im.html" class="u-url"><time class="dt-published" datetime="2026-06-24 12:50:30 +0200">Jun 24, 2026</time></a>
<div class="e-content">
<p>Auch geil bei 30 Grad im Schatten: Wasserrohrbruch und deswegen kein Wasser von 08:00 bis 15:00</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/06/24/in-der-kche-ist-es.html" class="u-url"><time class="dt-published" datetime="2026-06-24 12:42:26 +0200">Jun 24, 2026</time></a>
<div class="e-content">
<p>In der Küche ist es so warm, das Kochfeld zeigt drei von vier Platten als „heiß“ an</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/06/22/respekt-und-glckwunsch-an-wer.html" class="u-url"><time class="dt-published" datetime="2026-06-22 00:30:57 +0200">Jun 22, 2026</time></a>
<div class="e-content">
<p>Respekt und Glückwunsch an wer auch immer die Displays der Packstationen spezifiziert hat. Ich habe heute ein Paket an einer Station abgegeben, die den halben Tag in der Sonne stand. Ich habe mir fast die Fingerspitze beim Tippen verbrannt so heiß war das Ding. Aber noch immer problemlos ablesbar. Krass.</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/06/08/sommerrolle.html" class="u-url"><time class="dt-published" datetime="2026-06-08 20:13:41 +0200">Jun 8, 2026</time></a>
<div class="e-content">
<div class="media-gallery">
<img class="media-item" src="https://www.the-sprawl.blog/uploads/2026/navcpwnoru7ybybfq1ejer6aqssc8lni5m4fhl0b.jpg" alt="">
</div>
<p style="padding:10px;">Sommerrolle</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/05/31/senko-rai-wrote-a-wonderfull.html" class="u-url"><time class="dt-published" datetime="2026-05-31 22:59:00 +0200">May 31, 2026</time></a>
<div class="e-content">
<p><a href="https://blog.senko.net/">Senko Rašić </a>wrote a wonderfull <a href="https://blog.senko.net/how-i-want-to-use-ai">article</a> how he wants to use AI. Its a quick read but it sets a good state of mind how AI should be approached. Lets take time to smell the roses …</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/05/28/dnerstag.html" class="u-url"><time class="dt-published" datetime="2026-05-28 15:48:00 +0200">May 28, 2026</time></a>
<div class="e-content">
<div class="media-gallery">
<img class="media-item" src="https://www.the-sprawl.blog/uploads/2026/6d1efv6w49vtqzzem3vcnwuxsygehexv5nlbqwxm.jpg" alt="">
</div>
<p style="padding:10px;">Dönerstag</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/05/26/es-ist-hart-wenn-im.html" class="u-url"><time class="dt-published" datetime="2026-05-26 10:10:45 +0200">May 26, 2026</time></a>
<div class="e-content">
<p>Es ist hart, wenn im Shuffelmode direkt nach Slayer dann Bibi Blocksberg die Ohren verwöhnt.</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/05/25/goodbye-reddit-hello-lemmy.html" class="u-url"><time class="dt-published" datetime="2026-05-25 15:03:10 +0200">May 25, 2026</time></a>
<div class="e-content">
<p>Goodbye #reddit
Hello #lemmy</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/05/24/fun-fact-i-can-control.html" class="u-url"><time class="dt-published" datetime="2026-05-24 20:55:50 +0200">May 24, 2026</time></a>
<div class="e-content">
<p>Fun fact, I can control my Tensor Tympani muscle</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/05/24/ipad-mini-gen-scheint-mit.html" class="u-url"><time class="dt-published" datetime="2026-05-24 13:10:17 +0200">May 24, 2026</time></a>
<div class="e-content">
<p>iPad mini 5 Gen scheint mit iOS 26.5 ganz gut zu laufen. Die 18er Versionen waren unbenutzbar.</p>
</div>
</div>
<div class="h-entry">
<a href="https://www.the-sprawl.blog/2026/05/23/triple-b-in-zuffenhausen-burger.html" class="u-url"><time class="dt-published" datetime="2026-05-23 16:26:49 +0200">May 23, 2026</time></a>
<div class="e-content">