Small buffers during tracking, large during mixing?

Before I started using a separate machine for running all my VSTi’s on, I tracked with the lowest buffers I could get away with comfortably at 1st, then as the project started getting congested, I had to switch to a much a larger buffer. I can now keep everything to the minimum.