 |
Post AwIQbxdbfQpPdSvYyP by [email protected] |
 |
More posts by [email protected] |
|
Post #AwIQbt3eh9h5Qo4nNg by [email protected] |
|
0 likes, 0 repeats |
|
|
|
🧵 I broke my mastodon :psyduck_confuse:I haven't been able to see all th… |
|
|
|
Post #AwIQbtAOI757lhEAsa by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@domo just checking that it wasn't me? i deleted one or two posts |
|
|
|
Post #AwIQbtZYmXDN1l0H6u by [email protected] |
|
0 likes, 0 repeats |
|
|
|
what's weird is that I'm on the latest mastodon (4.4.1) and I ran all t… |
|
|
|
Post #AwIQbu5oqb1Edo62OO by [email protected] |
|
0 likes, 0 repeats |
|
|
|
ok, so it might be this issue:https://github.com/mastodon/mastodon/issues/35315… |
|
|
|
Post #AwIQbucms1OGI3WMmO by [email protected] |
|
0 likes, 0 repeats |
|
|
|
oh dear, the replicas are most unhappy:1:S 18 Jul 2025 08:49:05.858 # Bad file … |
|
|
|
Post #AwIQbvAopUc1zbRXpA by [email protected] |
|
0 likes, 0 repeats |
|
|
|
tangential aside: sure am happy that databases have been dropping the master/sl… |
|
|
|
Post #AwIQbvCaiu1w56GxaS by [email protected] |
|
0 likes, 0 repeats |
|
|
|
here's the full log btw:1:S 18 Jul 2025 08:49:04.754 * oO0OoO0OoO0Oo Valkey… |
|
|
|
Post #AwIQbvjCle7NiFX0QC by [email protected] |
|
0 likes, 0 repeats |
|
|
|
I might say fuck it and rebuild the whole valkey cluster. Provided mastodon doe… |
|
|
|
Post #AwIQbwBZ4Cnr8Cnecq by [email protected] |
|
0 likes, 0 repeats |
|
|
|
elevator music plays |
|
|
|
Post #AwIQbwouhuHL6FD4xU by [email protected] |
|
0 likes, 0 repeats |
|
|
|
oh no....oh dear.... oh fuck...The problem is much deeper than I would have exp… |
|
|
|
Post #AwIQbxdbfQpPdSvYyO by [email protected] |
|
0 likes, 0 repeats |
|
|
|
ok, so seaweedfs has an unrelated livesnessprobe issue that can safely be ignor… |
|
|
|
Post #AwIQbxdbfQpPdSvYyP by [email protected] |
|
0 likes, 0 repeats |
|
|
|
arrrrrgh, and we didn't finish setting up the new loki infra, so processing… |
|
|
|
Post #AwIQby8RolUxB7MC2q by [email protected] |
|
0 likes, 0 repeats |
|
|
|
ok, it's kinda better. We're still having backup issues though, so I ca… |
|
|
|
Post #AwIQbyixd0hn0MRLxQ by [email protected] |
|
0 likes, 0 repeats |
|
|
|
so it looks like backups broke because restic shit the bed and wrote it's c… |
|
|
|
Post #AwIQbzMJGiBGyOqmI4 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
This was fixed by added the env vars stated by the bot in this thread:https://g… |
|
|
|
Post #AwIQbzMfFOSqzV13qK by [email protected] |
|
0 likes, 0 repeats |
|
|
|
If you're not aware, restic is literally just... fancy rsync. It's sort… |
|
|
|
Post #AwIQc03Yfum98X5JhY by [email protected] |
|
0 likes, 0 repeats |
|
|
|
OK, all backups, valkey, and postgresql stuff are fixed, this just leaves the d… |
|
|
|
Post #AwIQc03ueb3j9dFbFo by [email protected] |
|
0 likes, 0 repeats |
|
|
|
ok, yesterday was pain because my day job also paged me out a lot so I couldn&#… |
|
|
|
Post #AwIQc0diVThOwg0C3s by [email protected] |
|
0 likes, 0 repeats |
|
|
|
oh dear, something has been messed up about where our postgresql PVCs are store… |
|
|
|
Post #AwIQc5x2l385Q45CNc by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@cxxvii I don't think so. It seems to happen more with people I don't h… |
