I would suspect the M-Audio unit's drivers. This could somehow be compounded with the sample rate/# of tracks you are running, but if you can get your hands on an interface with better latency-that would no doubt kill your problem. Any of Steinberg's, RME's, MOTU's, or the Avid Mbox Pro would be beneficial to your situation.
Asus R500A-RS52, Windows 8, 2.6gHz i5 -3230M, 6GB RAM,
Cubase Artist 7.0.6, Wavelab Elements 7, Focusrite Scarlett 8i6, The Grand 3, Melodyne Essential, Superior Drummer, EZDrummer and Pop! EZX, Samson CO1, Yamaha P-35, Mackie MR5mk2 monitors, and more....My tunes on: Soundcloud