About a year ago I tried to reanimate one pull-request to Apache
Spark on GitHub which was proposed by one guy more than a year before
that. I hoped then that the PR would help me with my experiments with
LightFM which generated extra large models (4GB+) which were
unconsummable by Python earlier than 3.4. When I tried the PR on our
cluster at work, I understood that it was not a solution. And the PR
itself didn't want to reanimate at all --- instead of directly
resolving merge conflicts there happened to be some unexpected
unit-test failures which I didn't know how to fix. Today they
contacted me and wrote that they prepare for the Spark 3 release, and
it would be nice to have my PR finally merged since it considers
upgrading the version of pickle protocol which is useful per se. The
good news was that they had added a utility to run a single PySpark
unit-test which simplified debugging tremendously. Finally, I
suggested a fix in some old sacred code which was written at the
beginning of 2015 when I didn't even know about the existence of
Apache Spark. Now my reviewers and I will have some fun deciding
whether it was a good idea or not.
