It'd be "a grievous harm to the United States of America," Sweeney says