Tried your project and could reproduce with “élastique pro ->time” algorithm, however switching to “standard->drums” solved the issue.
Also, it’s not only starting and stopping playback that puts the drums back in correct timing for me, opening the audio editor or toggeling between musical and linear time base while playing works.