Don’t know if you were disagreeing or just not understanding what I was trying to say.
Sample buffers provide protection against audio glitches while your computer’s processing audio. A 256-sample buffer provides less protection from these glitches at a 48kHz sample rate than it does at 44.1kHz, both because it represents a shorter amount of audio time at the increased rate, and because your computer has to work harder to deliver audio at that higher rate. So even though you may experience less latency while using a sample buffer of a given size at a higher sample rate, you’re also running a greater risk of getting clicks, pops, and dropouts in your audio if you don’t increase the size of the buffer to scale with the increased sample rate.