Not to me, because a) right or wrong, there are a lot of books that don't mention various minorities but don't expect us to assume they don't exist and b), we could have killed them off, or chased them off, or made them taboo and unmentionable, or not met them yet. My feeling is, if you're using bits of the real world that your readers would know about but making them AU, at some point you need to get explicit about how the U has been Aed. Doubly so if the alteration you're making is to fictionally remove out of the way of white people a group that we came awfully close to removing in real life.