VST Connect is not designed for sessions of any kind. The only way it works is to connect one Performer to one recording facility.
In a local setup you’d have to have real wires to provide all parties with simultaneously occuring signals. VST Connect requires delays and buffering in order to connect people remotely and isn’t designed for this scenario, sorry.
In any case, additional buffering is required when connecting over networks. P.A. systems (digital snakes) can provide very low latency, but unless tailored to this particular task, audio interconnections over network will introduce further latencies that will get in the way.