Prime Minister Andy Burnham says UK lost more than it gained after its divorce from the bloc.