Sounds like DPC latency. Maybe check using latencymon (free):http://www.resplendence.com/latencymon
In essence it will tell you if there are IRQ conflicts (busses using the same flow) and you can find the culprit. Also might be worth making an OS profile tweaked for audio (like setting CPU as background services, disable wifi adapter and all other usual DAW tweaks).