It doesn’t scan my plugins every time either, but when it does on occasion, I’m ok with that.
Doesn’t hurt to refresh the list once in a while.
I have a lot too, but it only takes 15 seconds or so - certainly not enough to ruin my creativity at all.