Yeah and downvoting a racist will just alienate them and push them further right, and yet we still downvote.

While occasionally someone might realize that they are in the wrong, the value of condemnation is in signaling to the community that such views are not tolerated.