 |
Post AwmEgS4mjPFv2A9FOS by [email protected] |
 |
More posts by [email protected] |
|
Post #AwlSfZvfFm4ZIeOqzg by [email protected] |
|
0 likes, 0 repeats |
|
|
|
So I need to run some software across several machines: I basically have to pro… |
|
|
|
Post #AwlSrCOGaylNlrJrXM by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone would a batch scheduler be a good fit? You could add as many jobs as you… |
|
|
|
Post #AwlStyup9R0OqdsIAC by [email protected] |
|
0 likes, 0 repeats |
|
|
|
basically I want to be able to define 10,000 processing setups, and that gets … |
|
|
|
Post #AwlT6x7N96lX2NvfLU by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone sounds a lot like a job for mpi. Years ago I’ve written software in py… |
|
|
|
Post #AwlTIpZr0Wrn7qunvE by [email protected] |
|
0 likes, 1 repeats |
|
|
|
@foone GNU Parallel, hands down. Its lightweight distributed-work syntax is pre… |
|
|
|
Post #AwlTLmZqcOb4gpJsoq by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone ok, as reluctant as I am to push my own software, actually a project of … |
|
|
|
Post #AwlTPKLFXTdkHGBKbo by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@fooneGNU Parallel can handle this - it SSHes into each machine and runs, by de… |
|
|
|
Post #AwlTrSZ2YsvRt42yES by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone for a small set of heterogeneous machines and jobs that run just for a f… |
|
|
|
Post #AwlU0hhLfB2PQlLCEa by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone for embarrassingly parallel jobs, some kind of basic task queue like das… |
|
|
|
Post #AwlUCmMg8eXxNYQlVI by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@fooneThank you for asking this, because today I learned about GNU Parallel and… |
|
|
|
Post #AwlUJz1tRwWIgX8yfI by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone My first thought is using a queue system (Rabbit, Celery, etc), put the … |
|
|
|
Post #AwlULY5Sam0bhgYhBw by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone all those I know of need quite a lot of setup, is doing the partitioning… |
|
|
|
Post #AwlUNMTmii0yHYnhei by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone I believe GNU Parallel can do this quite effectively if you have passwor… |
|
|
|
Post #AwlVA9hB2V84p5Kc0O by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone probably not the best solution but I've used celery queues and pytho… |
|
|
|
Post #AwlVCVMfRdTSbkrIJ6 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone For this I'd either use GNU Parallel or if I couldn't get it to … |
|
|
|
Post #AwlVapOEduktfJBq76 by [email protected] |
|
0 likes, 1 repeats |
|
|
|
@foone I recently did something like this: convert 100,000s of DNGs into JPEG b… |
|
|
|
Post #AwlVpt7B4m4rYsOPCK by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone Hadoop MapReduce is probably overkill for this, but would allow you to a… |
|
|
|
Post #AwlWY0aqdFtrUuJDeq by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone Torque or Slurm (https://slurm.schedmd.com/quickstart.html) maybe? |
|
|
|
Post #AwlX8BMC5fvhIzhgiu by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone https://gearman.org sounds like a good fit with a little bit of work and… |
|
|
|
Post #AwlX9RRtkKq3VGyLRY by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone I'll second GNU Parallel for one off or occasional batches.Also wort… |
|
|
|
Post #AwlX9bYeA6B8r6CsLo by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone I've used this in the past to make scalable processing systems. Just… |
|
|
|
Post #AwlXCEQ6dzCNxoSrAW by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone For Benchmarks I have used ReBench (https://github.com/smarr/ReBench/)Bu… |
|
|
|
Post #AwlXCshsrJBcuX5cQq by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone sorry, no packages yet (just u-build and a linux-64 binary), but if this… |
|
|
|
Post #AwlXEaoGN3Gxz0tD5k by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone a thing that immediately comes to my mind is pueue: https://github.com/N… |
|
|
|
Post #AwlXJo1CgkXqOgQbbc by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone You're comfortable with Python; celery is good for these types of ta… |
|
|
|
Post #AwlY7gVwrYLlbQaonw by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone Would it be possible to break the items into batches to process sequenti… |
|
|
|
Post #Awlf0Xn1bM38CD7eBU by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone Depending on scale:Shell for loop with fork and ssh.Clush.Ray. |
|
|
|
Post #AwlnQ47vJF3zMPZtqK by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone would something like Celery work for this? Your description sounds like … |
|
|
|
Post #AwlnlI5Gc0smjGSMPQ by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone (That'd be my first impulse, anyway.) |
|
|
|
Post #AwlnnybxT3fPCj1oPY by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@foone Maybe something like an ansible script that calls a python script that d… |
|
|
|
Post #AwmEgS4mjPFv2A9FOS by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@tychotithonus @foone https://github.com/aaronriekenberg/rust-parallel if you d… |
|
|
|
Post #AwmEgSBWKMdxN3IctM by [email protected] |
|
0 likes, 1 repeats |
|
|
|
@mxshift Wow, I had no idea! Thanks for taking the time to link to this, this i… |
