There is always latency with a VST Connect stream. In playback mode, that latency is determined by the “Remote Latency” setting in the Studio VST Connect plugin. You may try to reduce that, depending on the distance and connection quality you may be able to get to 0.5 or so.
Filling buffers means latency. There is no latency-free connection anyway, added to network latency there is the speed of light which does indeed come into play with long distance connections, plus system latency. Not quite sure what you expect?