I realize that you don’t have 2 computers using the same drive at the same time, but you are still interrupting a drive while it’s connected to one computer when you make the switch. The whole idea of the ‘safely remove hardware’ dialog is to make sure the OS stops all communication with the drive before you pull the plug (or hit the switch in this case, same result).
See here: