Rewrote parTraverseN and parTraverseN_ for better performance

[djspiewak]

This shifts to a fully bespoke implementation of parTraverseN and such. There are a few things left to clean up, such as a few more tests and running some comparative benchmarks, but early results are very promising. In particular, the failure case from #4434 appears to be around two to three orders of magnitude faster with this implementation (which makes sense, since it handles early abort correctly). Kudos to @SystemFw for the core idea which makes this possible.

One of the things I'm doing here is giving up entirely on universal fairness and merely focusing on in-batch fairness. A simpler way of saying this is that we are hardened against head of line blocking, both for actions and cancelation.

Fixes #4434

[djspiewak]

Pros and cons on performance, though I think it's possible to do better here. It's a little bit slower than the previous implementation in the happy path, but it's several orders of magnitude faster in the error path so I'll call that a win.

Before

[info] Benchmark                             (cpuTokens)  (size)   Mode  Cnt    Score    Error  Units
[info] ParallelBenchmark.parTraverse               10000    1000  thrpt   10  292.924 ±  1.496  ops/s
[info] ParallelBenchmark.parTraverseN              10000    1000  thrpt   10  277.978 ±  1.280  ops/s
[info] ParallelBenchmark.parTraverseNCancel        10000    1000  thrpt   10    0.006 ±  0.001  ops/s
[info] ParallelBenchmark.traverse                  10000    1000  thrpt   10   48.015 ±  0.016  ops/s

After

[info] Benchmark                             (cpuTokens)  (size)   Mode  Cnt    Score   Error  Units
[info] ParallelBenchmark.parTraverse               10000    1000  thrpt   10  293.834 ± 1.152  ops/s
[info] ParallelBenchmark.parTraverseN              10000    1000  thrpt   10  233.868 ± 0.309  ops/s
[info] ParallelBenchmark.parTraverseNCancel        10000    1000  thrpt   10    7.859 ± 0.014  ops/s
[info] ParallelBenchmark.traverse                  10000    1000  thrpt   10   48.059 ± 0.014  ops/s

[djspiewak]

So I haven't golfed the failure down yet, but it really looks like we're hitting a bug in Scala.js, probably stemming from the "null safe" test. @durban you may be amused

I think we could just remove the null safe test now since we're not using an ArrayBuffer internally, but it's kind of a neat surprise.

[durban]

Well, "amused" is one word for it :-) So it's not a bug in Scala.js, as in, it behaves as documented: dereferencing null is undefined behavior in Scala.js (LOL, what? Seriously.), so literally any behavior is "behaving as documented". Apparently scalaJSLinkerConfig could be configured to behave properly for nulls. But removing that very specific test is also fine I think.

[djspiewak]

Well that's fun. I actually thought we had some special checking for when the cur0 action became null in the runloop, but apparently not.

[durban]

That we do check. It's this line: https://github.com/typelevel/cats-effect/blob/series/3.x/core/shared/src/main/scala/cats/effect/IO.scala#L2024 (and fa is null).

[djspiewak]

Ahhhhhhh that makes sense. Okay, by that token, I think it's fair to say that a lot of our combinators just aren't null-safe and that's how it's going to be. :P

[durban]

It's annoying, because in Scala they are null safe. (The test passed before, it just failed on JS.) We'd have to do something like this (everywhere), to make it work on JS:

def combinator(fa: F[A], ...) = {
  if (fa eq null) throw new NullPointerException
}

Which is (1) annoying, (2) very redundant, except on Scala.js, and (3) apparently has performance problems in Scala.js (or maybe that's only the linker setting?).

I don't propose we do this. There is a Scala.js linker setting which fixes the problem. In Scala and Scala Native it works by default.

[durban]

(Just some context about the null test you've removed: I've added that on my old branch, because I've had a bug previously in my implementation, where it didn't handle correctly if f(a) was null. I don't remember exactly, but I think it just ignored it. So I've added the test to make sure we see the NPE. As I've said, I think it's fine to remove it here.)

[mr-git]

Could this fix hit 3.6.4?

[djspiewak]

Could this fix hit 3.6.4?

There are a couple failing tests related to early termination that I'm still trying to track down. Am trying to find the spare time needed to push on it. Help definitely welcome! Otherwise I'll probably get to it within the next few weeks. Sorry :(

[domaspoliakas]

I see that last CI is green, do you mean that you want to reintroduce the tests removed in this commit? 599b790

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[reardonj]

I think I found the issue with the "interrupt never on error" test.

[djspiewak]

@durban I think we might be ready! (again)

[mr-git]

@djspiewak, the #4434 mentions the internal MiniSemaphore, but I wonder, could Semaphore have similar issue too?

[reardonj]

I don't see any other issues. A few nitpicks.

[djspiewak]

Both good calls! Fixing…

[durban]

@djspiewak I've looked at parTraverseN_, left some comments. Nothing critical, though the error handling one would be good to fix. I'll get to parTraverseN soon (for some value of "soon").

[djspiewak]

@durban Btw I thought a bit more about your "what if we just used a Queue?" comment. I actually experimented with that initially and it spiraled into a ridiculous rabbit hole that was worse than this by far (e.g. I had a MiniQueue, it was gross). But revisiting the idea, I think it might actually be simpler.

Given how much of the train is backed up behind this PR, I'd like to push it through, but I'll start another branch experimenting with the queue approach. Maybe in a few years we can land that one… :P

[djspiewak]

Update: I did a quick PoC and it's like, drastically simpler to use a queue explicitly. Also I can share the core machinery between the two functions, which feels absolutely worth it.

[durban]

I've looked at parTraverseN, have one concern about a possible deadlock.

[durban]

Just to check if I'm understanding this after all this time: the false must beTrue always fails, so this checks that the test finishes in at most 500ms, right?

[durban]

@djspiewak Yeah, doing the queue later sounds fine.

[durban]

LGTM.

I have one final question as this PR had multiple different versions: was this final-final version measured to improve performance?

[djspiewak]

It was not. I'll do that test before I merge