 |
Post Av5iSwEWkJlYzc38Xw by [email protected] |
 |
More posts by [email protected] |
|
Post #AeU0vo4sqz8FBJN64O by [email protected] |
|
0 likes, 4 repeats |
|
|
|
in case there are other nerds out there who haven’t yet read this classic, be… |
|
|
|
Post #AeVH1sRFLXMOrshxFA by [email protected] |
|
0 likes, 3 repeats |
|
|
|
@shortridge While working tech support, I got a call on a Monday. Some VPNs whi… |
|
|
|
Post #AeVHGXVxQKEyJiu3Hs by [email protected] |
|
0 likes, 1 repeats |
|
|
|
@bob_zim @shortridge @xssfoxreminds me of a new-deployment airgapped site where… |
|
|
|
Post #AeWW7q94ThYjxpT4iW by [email protected] |
|
0 likes, 1 repeats |
|
|
|
@shortridge Actually, I take it back, this is my favorite bug story of all time… |
|
|
|
Post #AuwSCjKRbsnmeCQYro by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge a can hear the cries of "how did any of this ever wor… |
|
|
|
Post #AuwSCjSb7ZK93UF4Zk by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@tmcfarlane @bob_zim @shortridge I am a molecular biologist and I'm sorry t… |
|
|
|
Post #AuwSCjZKiWiBONOS4e by [email protected] |
|
0 likes, 1 repeats |
|
|
|
@stevegis_ssg @tmcfarlane @shortridge Vision is my favorite example of this. Th… |
|
|
|
Post #Auz7bzeMY9PwwXpSgi by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge What year did this happen? |
|
|
|
Post #Auz7bzlS7n5ZIX97js by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@rsalz Pretty sure it was late 2010. |
|
|
|
Post #Auz7cnLHnLrp3rHvtY by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge Are you able to reveal what NTP client we should be avoidi… |
|
|
|
Post #Auz7dI67Rn9eRnQl3g by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@shortridge Some time later, I was no longer working tech support. I got hired … |
|
|
|
Post #Auz7dIDuynOQpz4zDM by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge Last year, we had an interesting problem. Some VMs, in one… |
|
|
|
Post #Auz7dIsgXE0EsQ9Xl2 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge After a while, I noticed that the VMs that dropped out did… |
|
|
|
Post #Auz7dJJGwNGoCsamCO by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge So, I have a kernel core and the tooling to look at it, ne… |
|
|
|
Post #Auz7dLtrJJiyEvj1VI by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@me @shortridge It wasn’t quite homebrew. It was an NTP appliance sold by som… |
|
|
|
Post #Auz7dMVn2I488ZTJcu by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@me @shortridge The popularity of this story has me thinking about it some more… |
|
|
|
Post #Auz7g4jlZt4HsZIqjg by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge @vaurora Okay, that’s a weirdly annoying combination of … |
|
|
|
Post #Auz7iYiL5xQlQjostM by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim this is an exquisite tragicomystery, thank you for sharing it, I’m i… |
|
|
|
Post #Auz7jQWV2C2uJehBw0 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim Thanks for sharing :) |
|
|
|
Post #Auz7lGn6Bp3oi9L3dQ by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge is it wrong that I want to see one of these circular depen… |
|
|
|
Post #Auz7ldJoQsIgZyoUgC by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge A great example of why debugging SUCKS! |
|
|
|
Post #Auz7ldQC39P8tlnacq by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mildbeard @shortridge Debugging complex, distributed systems like this certain… |
|
|
|
Post #Auz7lq0nEtoLwl0zL6 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim Ahh, so probably the reference NTP client, but used incorrectly (`ntpd… |
|
|
|
Post #Auz7mDFSqWDG0v3Wts by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@stuartl It was basically a cronjob running ntpdate, like the docs explicitly t… |
|
|
|
Post #Auz7mWx3afN97cb6zA by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@rfc6919 @shortridge @xssfox Oof. I’m so glad my current company’s radio cl… |
|
|
|
Post #Auz7mgSwDVDod6aCQa by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge I kept waiting for it to be DNS. |
|
|
|
Post #Auz7nEPE1oSDtR6iMS by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge on the plus side, the VPN itself apparently didn’t rely … |
|
|
|
Post #Auz7nNoN3gur41wQIy by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge "if it was not tested, it doesnt work" |
|
|
|
Post #Auz7oR26ZBWXLRroEi by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge So cool to know all our infrastructure is rock solid like … |
|
|
|
Post #Auz7pXiJDHggM28TSa by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge @voided thanks for sharing! That was a fun ride. |
|
|
|
Post #Auz7q0U8F4ZvaSj3tA by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim Wonderful!I maintained CA software for a few years and I've seen m… |
|
|
|
Post #Auz7sNlaeWt8NB1a0e by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@rfc6919 @bob_zim @shortridge @xssfox nothing as elaborate as that, but as a gr… |
|
|
|
Post #Auz7sOWjpEbOjP5EUy by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@rfc6919 @bob_zim @shortridge @xssfox Ah, and #3, that time some rando from Flo… |
|
|
|
Post #Auz7soxvWlncjG0JUG by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge Nicely written incident investigation, I enjoyed reading. |
|
|
|
Post #Auz7tTkts6jzEjDzNI by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @stevegis_ssg @shortridge anyone that has done a bit of life drawing i… |
|
|
|
Post #Auz7tlQeA9Ee37t4Iy by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge That is absolutele wild and I wish I was remotely surprise… |
|
|
|
Post #Auz7ulmmJxmHRiicPQ by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@stuartl It was actually *only* the ntpdate-like binary (might have actually be… |
|
|
|
Post #Auz7yYLyCixywcz1kW by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridgeWow, that was a bit of a roller-coaster ride |
|
|
|
Post #Auz80K1CTLiF446zeC by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge nice workthorough! I would never have tracked all that dow… |
|
|
|
Post #Auz84uRF0PI5B4TnfM by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@zetasyanthis With embedded boards, you kind of don't expect much. But when… |
|
|
|
Post #Auz84uXGe06xTlIc3k by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@vjon Oh these were really expensive FPGA boards. We ended up using NTP for the… |
|
|
|
Post #Auz86BSjCEQ8GYmbS4 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge is it not also a bug to be revoking certificates that are … |
|
|
|
Post #Auz86meMvuiVbNVvEW by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge Sooo… It actually was not the DNS?! :-) BTW, very nice c… |
|
|
|
Post #Auz86w2S1kKOifqmWG by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim aaaaaaaaaa :blobfoxdizzy: |
|
|
|
Post #Auz87EcuuuC0Mz39rE by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim I‘ve seen private CAs with a CRL in the multi megabyte range. They h… |
|
|
|
Post #Auz87kOUo3fetWY5p2 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@shortridge Right? It was such a weird sequence of failures.Recovery was very p… |
|
|
|
Post #Auz87ttfTWxAMoCc9w by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@letoams @shortridge Management traffic (like traffic from the VPN endpoints to… |
|
|
|
Post #Auz88ToFx80zkRwniq by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@dannotdaniel @shortridge At the time, I was the last level of support on a tea… |
|
|
|
Post #Auz88sbLmFXudf6LZo by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @shortridge I love problems like this because there’s no instruction… |
|
|
|
Post #Auz8CGUtOD9tcRh9fM by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@asg @shortridge I don’t remember if I even looked at the CRL itself in great… |
|
|
|
Post #Auz8CYvmqxMon4Pt5M by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@Forbearance It happens quite a lot. Tons of mid-size startups don’t survive … |
|
|
|
Post #Auz8CkaxUDpuxwlQHo by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim @toba @shortridge I thoroughly enjoyed this — wow! |
|
|
|
Post #Av5iSvtc052HwkGQwi by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim>> More significant tests are more likely to notice the problem.I… |
|
|
|
Post #Av5iSw1PX5H4Kvuf6O by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@devnull @shortridge Sure, but “monitoring” isn’t well-defined. Ping is o… |
|
|
|
Post #Av5iSw7n9MNWeitl32 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim By monitoring, I mean a set of tests that actually tests whatever serv… |
|
|
|
Post #Av5iSwEWkJlYzc38Xw by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zimTo me Ping is NOT monitoring. It actually doesn't always confirm ne… |
|
|
|
Post #Av5iSwLGLH9bKVCW2q by [email protected] |
|
0 likes, 1 repeats |
|
|
|
@devnull @shortridge > To me Ping is NOT monitoring.Tell it to literally any… |
|
|
|
Post #Av5iSxED2z6e4uuOgq by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@bob_zim The things is, in such situation, you have no idea what responds to Pi… |
