* * * * *

  “99 failing tests in the queue! 99 failing tests! Check one out, grind it
                     out, 98 failing tests in the queue!”

So I'm facing this nearly twenty-hour long regression test [1] and I had this
idea—instead of querying the mocked end point if it did its thing or not,
have the mocked endpoint check to see if it did its thing or not.

I now send the mocked endpoint the testcase itself (along with a query flag
of true or false). The mocked endpoint will save this information, and set a
queried flag for this testcase to false. If it is queried, it updates the
queried flag for the given request. At the end of the regression test (and I
pause for a few extra seconds to let any pending requests to hopefully
finish), the mocked endpoint will then go through the list of all the
testcases it was told about, and check to see if the query flag and queried
flag match—if not, it logs an error.

Sure, now we have two error logs to check, but I think that's better than
waiting nearly twenty hours for results.

I got a baseline time for the subset of the regression test [2] without the
mock checks—35 seconds. I'm trying to beat a time of 4 hours, 30 minutes with
the mock checks.

The new method ran the subset in 40 seconds. The entirety of the regression
test, 15,852 tests, took just a few minutes—about the same as before.

I can live with that.

Now all that's left is to write the validation logic—I still don't have it
down yet.

[1] gopher://gopher.conman.org/0Phlog:2021/06/09.1
[2] gopher://gopher.conman.org/0Phlog:2021/06/09.2

Email author at [email protected]

You are viewing proxied material from gopher.conman.org. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.