Thanks for all the suggestions it is finally resolved.
Feel a bit stupid I didn’t try this before just didn’t think about it. I opened a new project and imported everything from the old project and latency was gone. Still weird though cause nothing is different in the new project.