Because some people have co-opted the ToE in the past in an attempt to justify their beliefs and behaviours, does not mean that the ToE is wrong. Anyone can abuse an idea or make a strawman out of it.

There is no evidence that black people are inferior to white people. And "survival of the fittest" is a term which describes how organisms with a beneficial mutation reproduce at a greater rate than non-mutated organisms, and eventually take over the gene pool for that species, creating a speciation event. It was never meant to condone the behaviour of amoral people looking for a reason to justify mistreating other people.

Criticise the people, not the theory they misunderstood or misused. You will be hard-pressed to find anyone today who claims that a certain race of people is less evolved than another, because that is a preconceived notion, and the evidence is not there. Even scientists can be guilty of this, if it supports racist beliefs, but the days of that seem to be gone.