There are two consequences of running channels close to saturation:
a) Everything is delayed. This is basic queuing theory. Just look at a bank queue. The more people in it, the average waiting time is longer. Longer data queues require larger buffer settings to ensure that delivery latencies are covered.
b) Not much room for incidental non-audio data, like OS or protocol overheads.