...
The above diagram includes the current threading model during inference as I understand it. The most important thing to note here is that when we run inference or read a result we must do it on a single main thread (aka dispatcher thread). It’s therefore very important that we block this thread as little as possible. This is not a problem when we’re submitting predictions, we can call forward on a symbol as many times as we like. Each forward call will quickly kick off a computation graph that will eventually be executed in the engine.
...