So long as you keep to 24 or 32 bit, you won’t be losing anything. As you say, you should dither to 16-bit only at the very end. Theoretically, you should dither to 24-bit if storing in 24-bit as the intermediate format, but I find it hard to envisage a situation when that would have any effect on a subsequent 16-bit result, and frankly, I don’t bother.
Paul