It’s a weird one. “removed” is a technical term, jargon. It became a perjorative term.

Unlike the n-word that was always a pejorative.