It wasn't just the spy he was allegedly shagging