I would suspect the M-Audio unit's drivers. This could somehow be compounded with the sample rate/# of tracks you are running, but if you can get your hands on an interface with better latency-that would no doubt kill your problem. Any of Steinberg's, RME's, MOTU's, or the Avid Mbox Pro would be beneficial to your situation.
Asus R500A-RS52, Windows 8, 2.6gHz i5 -3230M, 6GB RAM, 5400RPM HDD
Cubase Artist 7, Wavelab Elements 7, Focusrite Scarlett 8i6, The Grand 3, Melodyne Essential, Korg Nanopad/Nanokontrol, EZDrummer, Samson CO1, Samson HP10, Casio WK-200, Mackie MR5mk2 monitors, and more....
Check out our Forum Knowledgebase
My tunes on: Soundcloud