 |
Post Asuc3CtvKABh5oAZjU by [email protected] |
 |
More posts by [email protected] |
|
Post #AsrXFVTozWV6wc7tPk by [email protected] |
|
0 likes, 0 repeats |
|
|
|
apparently loading 750MB of JSON into Python is enough to make a machine with 1… |
|
|
|
Post #AsrXFVcKTtJ3N06gfw by [email protected] |
|
0 likes, 4 repeats |
|
|
|
i'm sorry they did WHAT |
|
|
|
Post #AsrXwaOAcG2ecZ1avw by [email protected] |
|
0 likes, 1 repeats |
|
|
|
@mei big android 'zipalign' energy right there (renaming extensions so … |
|
|
|
Post #AssMz3YWEy8zK1nhXE by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei That is absolutely brilliant. A+ hack right there. |
|
|
|
Post #AssN18rnSvDKNAoPzc by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei oh i see you have come across the horror that is serving files via GitHub … |
|
|
|
Post #AssN18yX3sbMi3xnUW by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@anya but- but-is compressing the damn file yourself so damn hard? you could us… |
|
|
|
Post #AssN195GepzP2x7AzQ by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei @anya also like... shouldn't the HTTP compression cover it anyway? |
|
|
|
Post #AssN19C0FnNRNqGYUK by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@Lunaphied @mei one would hope so, but unfortunately, GitHub... |
|
|
|
Post #AssN4SOVJqBTnygvw0 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei iirc there was some amount of reason to this. maybe the json has to be loa… |
|
|
|
Post #AssNBR0irTRTXbtYTw by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@Lunaphied @mei @anya Oh dear … I was about to ask why they aren't compre… |
|
|
|
Post #AssNBfjY7bcNCgyYIS by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei holly s... |
|
|
|
Post #AssNCtrf4LGTJrM6Ua by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei even using pandas? |
|
|
|
Post #AssNCtyOfIeVekVTzU by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@nitinkhanna huh? you can try tinkering with this script if you'd like: htt… |
|
|
|
Post #AssNHJoyVLvCysCpLU by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei this year I solved a comparable problem by using gz-compressed files and p… |
|
|
|
Post #AssNKvk5yF7DSbe5q4 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei @anya FAT is a bitmap. maybe that's a clue as to why they did that. |
|
|
|
Post #AssNP14O6cfx8o5N5s by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@[email protected] yeah, you have to handle it differently in this case :neoca… |
|
|
|
Post #AssNREQ5RSR57mI8R6 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@[email protected] that sounds like... efficient... processing |
|
|
|
Post #AssNSgJpCQu3wQ3PAe by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei lol, epic :D |
|
|
|
Post #AssNUpIgmE1MNPXsMi by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@anya @mei browsers handle gzip and brotli decompression completely transparent… |
|
|
|
Post #AssNVobaqYO6VhrKng by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei That's actually pretty brilliant lol |
|
|
|
Post #AssNeeIOXydv8r5jxQ by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei @fanf this is incredible. the json file has to be plain text because its .… |
|
|
|
Post #AssNeePq6Ib7VwZgYq by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@beka_valentine @mei @fanf Or even just split up the data into multiple files, … |
|
|
|
Post #AssNfpo9EMPx8TVd5M by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@[email protected] @[email protected] perfect is the enemy of the good it wor… |
|
|
|
Post #AssNgwztz9oniuXUUC by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei Gott im himmel... |
|
|
|
Post #AssNhiTzR6oQzpygPQ by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei @foone Rekt |
|
|
|
Post #AssNq97kGimBwIanqa by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei the proliferation of JSON and it’s consequences |
|
|
|
Post #AssNr81Tqd0goX8A3E by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei I wonder if https://github.com/ijl/orjson would handle it meaningfully bet… |
|
|
|
Post #AssNtMhmEMqAvfyt8a by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@Lunaphied @mei @anya It should and it does. As far as I can tell, automatic gz… |
|
|
|
Post #AssO22jTrDsDakuqUC by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei my api accepts unlimited json so i think i should do something about it �… |
|
|
|
Post #AssO4VpocwqoQi3Z6O by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei @gsuberland loooooooool |
|
|
|
Post #AssO4kqMpBCkylQiXo by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@xot @Lunaphied @mei maybe that changed recently then? because last time i chec… |
|
|
|
Post #AssO5hLvI0uQRQdjma by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei That's unholy. |
|
|
|
Post #AssO8CKB6kbz4u125A by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei @gsuberland this has got real "we're using an int as a bitfield &… |
|
|
|
Post #AssO8mm5aSKQ7tLpaK by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei What they needed to do. |
|
|
|
Post #AssO9TuPDPWxu8Za1A by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei Smol HACK |
|
|
|
Post #AssOIiENKrNQviJbYe by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei oh. wow. |
|
|
|
Post #AssOWpzRHfvv3442tc by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei hahaha, I love that! |
|
|
|
Post #AssOfAlpuFFDoafyyG by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@burnitdown @mei sorry, you mean FAT as in filesystems? this is a single json f… |
|
|
|
Post #AssOqjXQyskkpYS6ZE by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@anya @mei @Lunaphied Could be. I first needed it for JSON in February and was … |
|
|
|
Post #AssOtPKOZwxR1Bd2I4 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei personally, I don't see the problem |
|
|
|
Post #AssOtnBhruGf1BkR4y by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei @gsuberland chaotic neutral |
|
|
|
Post #AssOwKjU7XZddPl81g by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@munin @mei @gsuberland or “we turned an int into 64 Booleans because they mi… |
|
|
|
Post #AssOxj7Cx5eLXtWHwW by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@KimmoAhokas @mei true! but only if the server decided to send the file with co… |
|
|
|
Post #AssOz9Ht86Tt0VC3o8 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei I have a 50 MB test PDF that runs a lot of parsers out of memory, on accou… |
|
|
|
Post #AssP0eFzTJDpLY1MGG by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei this is so cursed |
|
|
|
Post #AssP66d3PeVKmxPY8W by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei okay so its not just me |
|
|
|
Post #AssP6iKbEiJzij4LeC by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei haha, what the hell...Don't most modern http servers (have the option … |
|
|
|
Post #AssritJtcuEvWPjUjw by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei Compressing/decompressing in the script ❌ Waiting until GH decides to de… |
|
|
|
Post #Asuc13pk48ogOKBrFY by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei Who actually thinks that they converted the file to a bitmap image hence .… |
|
|
|
Post #Asuc13xBcSlslPfnqy by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@frassmith @mei Why don't you just ask, who understands the post? |
|
|
|
Post #Asuc144HC6RV7OzSu8 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@khleedril @mei because, I suspect that everyone who took the time to respond, … |
|
|
|
Post #Asuc14BMlk77TOJ7xI by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@frassmith @khleedril @mei Just changed extensions (I'm so outraged) |
|
|
|
Post #Asuc14JWHQdTsg7dfE by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@jleedev @frassmith @mei 😂 You would have built OCR into the app... |
|
|
|
Post #Asuc38nkZAsANVFaYS by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@Musicaloris @mei I'd guesstimate that a JSON file would compress slightly … |
|
|
|
Post #Asuc38vC7UpMkajX9s by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@saua @mei That's sound reasoning, but you may be underestimating the redun… |
|
|
|
Post #Asuc391viSDP5Tsuem by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@Musicaloris @mei Sure, but if you were to put that same data in JSON, you'… |
|
|
|
Post #Asuc3CtvKABh5oAZjU by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei ... my brain hurts. |
|
|
|
Post #Asuc5deuoGu6ezJDns by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei should have used jpeg for better compression |
|
|
|
Post #Asuc9aNKxZPxdPy8v2 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei haha good one |
|
|
|
Post #AsucC01YZN7flIvrnM by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei :BlobCatGoogly: Okay, but what does it look like if I open it in Paint? |
|
|
|
Post #AsucHIDGN5NXcavqa0 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@m4lvin I don’t think {"ωCPO.str":{ is a valid BMP header :neocat_… |
|
|
|
Post #AsucHT4nys9JJY3kQK by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@vitaut i don't think that's how it works |
|
|
|
Post #AsucIRamz5Nt07U8bg by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei I don't even know if this is a good trick or a nasty one. |
|
|
|
Post #AsucItcWmDPbvcVvCi by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@anya @mei yes, as in maybe it's like a fs image? |
|
|
|
Post #AsucPT5JplH5em7A5Q by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei does github not use content-encoding: zlib |
|
|
|
Post #AsucPx8u4zy2sa1Rq4 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@[email protected]:neocat_drake_dislike: filename.json.gz:neocat_drake_l… |
|
|
|
Post #AsucTOmHmz1xSP5PIO by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei Yeh because browsers use 15.5GiB... |
|
|
|
Post #AsucW6n97LXcXimSxc by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei import ujson as json |
|
|
|
Post #AsucjjM53CyXuTcHaa by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@frassmith @mei Guarantee they just changed the extension. Standard method for … |
|
|
|
Post #AsuckQEnc9hWuLXZi4 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@saua @Musicaloris yeah, depending on the compression format, I'd expect th… |
|
|
|
Post #AsucofaAVO43emYr20 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei I'm torn between "WTAF" and "I should remember that for… |
|
|
|
Post #AsuctB8B9O2wf3cSHI by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei @mcc Everyone in this situation is part of the problem |
|
|
|
Post #AsucySiJCoFEdo2QUa by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei It seems to take a long time to start downloading and I guess that's t… |
|
|
|
Post #AsucySp2nldGyhBnzU by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@geospacedman holy shit that's a ridiculously high compression ratio |
|
|
|
Post #Asud6wNanCDIxsOJEG by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@naught101 @mei Bold of you to assume GitHub uses a professional web server of … |
|
|
|
Post #Asud6wUKO9bLIlXgjA by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@aeveltstra @mei yeah... but they bought it from someone else.. |
|
|
|
Post #Asud8YidjUT4A70Wv2 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei @anya Especially since the GitHub Pages Server only supports gzip... (gzip… |
|
|
|
Post #AsudAGSgaGgo7KMHlg by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@meiEveryone makes fun of Java's memory efficiency while Python sits there … |
|
|
|
Post #AsudLh4PzhYLpNSPsO by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei Small hack, not many dead |
|
|
|
Post #AsudOm8FdB6kOZifrs by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei Ok, that's pretty ingenious |
|
|
|
Post #AsudQU9JSgwNCZ21Eu by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei @anya And this is python! There's multiple file compression formats in… |
|
|
|
Post #AsudZHssQsDVlhXbQO by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@mei On the one hand that's kinda clever, on the other hand, why does GitHu… |
|
|
|
Post #AsudobuUG5yiDnbIpM by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@[email protected] FIY, it's probably the garbage collector doing that, it pe… |
