The point isn't that trade is bad per se, the point is that politicians frequently make promises to help out those who are hurt by trade agreements, but then quickly lose interest in those promises once the agreement passes. That inevitably produces public opposition to future trade agreements, and in the end this hurts everyone. That's something worth paying attention to.
But "public opposition to future trade agreement" doesn't hurt "everyone." There are lots of people that have perfectly good reasons to oppose trade agreements. They're the people "who are hurt by trade agreements."
I'm roughly a "free trader" in the sense that removing the blunt protectionism of tariffs and quotas is broadly a worthy if not especially important goal at this point in time, though I'm certainly not in support of a lot of the things which have been thrown under the umbrella of "free trade." But the important point that needs to be made clearly is that even the simple textbook free trade agreement which removes tariffs and quotas is going to negatively impact a significant number of people. Those people are not naive Luddites to be dismissed by the Moustache of Understanding but people who rationally understand that free trade policies are going to hurt their incomes.