Subj : Re: read json from file
To : Mortifis
From : echicken
Date : Sat Jun 20 2020 10:57 pm
Re: Re: read json from file
By: Mortifis to echicken on Sat Jun 20 2020 21:43:56
Mo> }, .... that's the 1st three records of 209,578 records ... gotta read all
Mo> 209,578 records (1 million, 8 hundred 86 thousand, 2 hundred 13 lines) ? ...
Mo> bleh ... lol
Not necessarily, but you'll need something more than what we have on hand.
Streaming JSON parsers for handling really large files are a thing, but I don't know if there's one that can be readily ported to our environment.
JSON.parse() only wants to parse a complete JSON string. You'd need to be able to pre-process what you're reading from the file to be sure that JSON.parse() won't choke on it.
That's tricky to do in a generic way that could handle any old JSON you throw at it.
Easier if you do it as a custom job for this particular file, and if this file is just a flat array of objects, all with the same keys and types of values. It's either going to be a bit complicated but fairly solid, or simple and hacky and maybe not super reliable.
