I have been optimistic in the past and then hit hard by being very incorrect about things that mattered to me
Yes, I think that is one of the potential pitfalls of too much optimism. Pessimists even have a saying for it: "Don't get your hopes up and you'll never be disappointed".
I feel ultimately, that pessimism and optimism are seen as separate to realism specifically because they are detached from reality and are thus always to some extent incorrect. It's no way to sale a ship :)