In the long run, it would probably sound the most convincing if you locked the vocal/piano to the click as a single unit, and then the drums also. I understand that you want to keep the flow of the piano, but if you do a good editing job, most good musicians will choose the lined up version in a blind test. You could still time warp the click to retain certain pregnant pauses, then have the click follow the piano back to the grid once the piano smooths out.
If that’s not an acceptable solution, (you’d be surprised), then if you don’t want to just edit the drums by ear, it is definitely worth it to take the time to craft a flexible click using time warp. Don’t quit until the click feels smooth and believable against the piano- it’ll never be any easier to make a drum kit sound smooth than it will the click.
If you have a good ear, you’ll probably have to modify the click track beat by beat, rather than measure by measure.
When you finally feel good about the click track, then start cutting the drums. It goes without saying that you’ll edit the kit as a single unit (same as the piano/vocal) to maintain phase.
One useful trick on most rock and jazz music is that they tend to lock up on the snare beat, rather than on the bass drum. Depending on the groove, it’s often best to let the drummer do whatever loosey goosey thing he has in mind, without modification, and just move his entire phrase to cause the snare to fall on the grid.
The same goes for any “pickup” beats in the groove, any time you separate the pickups from the downbeat, it can sound like a stumbling glitch, assuming the drummer had any ability in the first place. Most drummers are good enough to know what speed they’re playing at, and if they pull off their riff adequately, the only problem is they just don’t always land together with the click. Of course if he actually screws up the riff, or he’s a poor drummer, then you’re back to editing eighth note by eighth note. Been there, done that, a thousand times…
BTW, all this may seem ridiculous, but after you’ve done it a thousand times it’ll start seeming more feasible, and you’ll be able to do it MUCH faster. As likely as not, you’ll probably have a thousand edits under your belt by the time you’ve finished this song! It can take a lot of re-doing to get the editing right, if you’re picky. It’s a great job for the anally retentive ADD types.