Overhearing one-sided conversations really is more annoying because the brain finds it more taxing and therefore more distracting to listen to only half of a conversation versus the whole thing.
Scientists say much of how humans process language is based on the brain's ability to predict or anticipate what comes next in a sentence.
With half a phone conversation, you're only getting a snapshot, so you can't predict the future which is what the brain has evolved to do.
The study appears in the journal Psychological Science.