Akka-http backend interferes with serving requests under load

[Marcus-Rosti]

Versions used

Akka-grpc: 2.1.3
Akka-http: 10.2.7
Akka version: 2.6.18

Expected Behavior

When using the akka-http backend that serving requests are not affected.

Actual Behavior

Serving requests has seconds longs spikes under heavy load.

Reproducible Test Case

Fairly easy to reproduce, have a server that times out internal requests to a client at 50 ms.

The only difference in these screen shots during a load test is the choice of backend

Using netty backend

Using akka-http backend