One of the optimizer's tests randomly fails sometimes

[pavelkomarov]

I think it's a matter of how the Nelder-Mead converges. I haven't seriously gotten to the optimizer code yet (because it lies at the end of a path running through #68, #77, and a couple follow-on PRs), but I'll try to make these more rock-solid when I do, because random failures are annoying.

[pavelkomarov]

I'm not positive the unit tests for the optimizer make sense. It's both difficult and trivial to verify the optimizations have done the right thing. That is, they take a long time to run, and at the end we get an answer that is presumably the lowest cost among a bunch of possibilities, according to some search process, but we can't be totally sure because "closing the loop" would require perfect knowledge of the (hyper)parameter search space, which is rather complicated and opaque to us without exhaustive search. If we're using Nelder-Mead out of scipy, then testing it is their business, and we can assume it works. So what does that leave for us?

I think these tests maybe need to be cursory to ensure stuff runs, but looking for specific answers should be beyond their scope. That will keep them from being brittle and randomly breaking on us, or breaking by convergence to an alternative answer when we make changes to a differentiation method.

[pavelkomarov]

I've been working on this code finally with #125. Seems like the optimization process for this test usually converges to one number, but every so often it converges to a different number. I've decided to just put an or in the pass condition.

It has been handy to have these tests around with their specific numbers, because they let me know my changes to the optimization code were mostly cosmetic and didn't disrupt any of the existing functionality. After working with them, I think they're cursory enough that they fulfill what I thought this module should maybe have test-wise, so I'm just going to leave them alone at this point, and close this issue. If we change how methods work, yes, these will be brittle and fail on us, but I think it's fine as long as we know not to take those failures super seriously. Just update some numbers. There's no much better way to test this module.