MIDI notes recorded earlier than they sound

  1. Set your buffer size to something “noticable”, eg 1024 samples

There’s something about that. With the buffer size being so large I’d expect some discrepancy somewhere due to the latency. Although you would expect the drums to be corrected and the audio to arrive late if anything.
Most people do. as you say, record at lower buffer settings and then would maybe mix at a higher level.
Whether that would be noticeable on the drum tracks? Negligible. I’d expect that any effects would be more influenced by the high buffer rate if applied at the mixing stage.
May be a problem but I’m still unsure if we’re looking in the right place although drums would seem obvious.