Why are you switching drivers? First you said you switch because you don’t need the interface while “mastering”. Then you say that you can’t record after switching drivers. Why do you need to record while “mastering”?
In any case, check your VST Connections and your track’s input bus assignment after switching drivers. Or, just don’t switch drivers!