I understand this line of reasoning but the reason we have a middle-man is to perform the function of organizing things like infrastructure development. How would you organize big projects such as new highways or better public transport? Who would pay for it? Would we only charge the residents that would utilize that infrastructure? How do you even track that? What about healthcare? This is an interesting discussion to be honest! I do hate tax - I pay a lot of it in my country (Australia) BUT it is spent (mostly) for good. Our healthcare is one of the best in the world because of it!
EDIT: I guess crypto and blockchain technologies could help with organization (such as something like what the Aragon project is doing)