“Black sheep of the family” is that racist?

Is it racist to use the phrase “black sheep of the family” ? Well… sort of.