Are you trying to connect locally, means within the same network? That is not possible.
Next I would suggest to entirely de-and then reinstall VST Connect PRO as you tried some things before. Something is strange with your configuration, otherwise there is no reason for Nuendo to show “VST Connect PRO” when it isn’t even installed - I know you have it now, but you shouldn’t have seen it in the first place. That seems highly suspicious.
Finally, the message suggests that you may have some additional firewall or so called “internet security” tools on either end. Make sure those don’t get in the way, try to disable it and see if it works then. But then you said you didn’t. “Internet works fine” isn’t a measure though (it would throw a different message if there was no internet connection), VST Connect requires basic communication which always works unless it is explicitly blocked, however the message says explicitly that it is able to start a connection, but gets no reply at all. Again, both sides should not have any such blocking installed or configured.