But now I find it just creates an unnecessary barrier between the kids and me.
Sometimes barriers are good things.
I went to private boarding school for highschool and the students were forced to wear dress code, which included dress shoes, dress shirt and tie (could substitute a turtle neck), dress pants, dress socks, blazer, no hats. This lead to a more professional mindset in the classroom, and encouraged the students to take school more seriously. If it was up to me this would be enforced at all schools at high school age and certainly college.
If you are preparing students to enter the business world, then they need to be used to getting dressed for business. Teaching them while dressing down lowers standards mentally for all. You need to teach them how to dress and behave as a professional teacher by setting the example.
Do what you want but this is my opinion.