But the obverse is not correct either.
In many ways, being around non-smokers made it harder for me to quit too. I ended up needing to be more alone or in places where I couldn't smoke, like dancing events. I was very conscious of my smell when having to be two inches away from a pretty girl's face.
Take the time you need as I think that you will be better off for it for many more years to come. Three weeks or six, it is a blink of an eye - once it has passed that is :)