My favorite part about vinyl are the jacket and inserts. There is also a trend about vinyl color which is pretty interesting too.
I really enjoy opening and looking at them while I listen to music. It's all part of the experience that a CD or MP3 doesn't offer.
Here's one of my latest buy, a Montreal band I really enjoy,