fx's still heard when group channel is muted

In prefs there’s a setting for muting the channels themselves that are assigned to a group. It sounds like you have effects sends on the individual channels and then are routing the channels to a group. If the channels are not muted then they will still send signal to the fx. So, muting the group doesn’t mute the individual channels in the group unless you select:

Prefs: VST - Group Channels: Mute Sources as well / Mute Pre-Send when mute