The latency is compensated so phasing will not be introduced but the roundtrip will prevent low buffer sizes.
Most systems i worked with having UAD needed to be on 512 or even 1024 buffersize, while most native systems can run at 128, 64 or sometimes even 32.