Obviously, the people who see "I don't see color" as meaning "I see you as a white person" are the bigger racist. I guess the best way to clear it up is to say "I don't see color" means "I see you as a bag of bones and blood of no particular shape form or color".
I prefer the days when everyone didn't have some sort of self-perceived disability that everyone has to acknowledge for the world to be whole.