Well aside from the fact that blacks and people of color, the lower classes of any color, even the jews in another morbid extension of the same idea, have had to suffer and fight, and suffer and fight still, against an enormous amount of cruel stereoptyping which Darwin's theories supplanted in the minds of people everywhere the ideas flourished as authentic science, I guess there is no reason.

Why do you ask?

Frederick Douglass was actually quite popular. Until Darwin's theories and explanations for the differences between peoples began to be offically taught in the schools, many whites probably would have never conceived such horrid racist ideas themselves, much less expressed them publicly.