Most are remnants from our evolutionary past, that without use and necessity have deteriorated over time. Or some such as nipples, are just a common genetic construct between both males and females.
I'm sure most people don't know this (I'm not sure if you do or not, also I could be wrong) but the reason men have nipples is because everyone is born as a female/neutral to begin with. I've never read any scientific proof about it, but I would also assume that this is the same reason why the head of a penis has the same type of skin as a clitoris, mainly because the same area has the same exact type of tissue/nerve endings.
Once the sperm fertilizes the egg, the embryo either has two X chromosomes, or one X and one Y, so if that's what we're considering males and females then it never really happens, you're instantly either a male or a female. However, as for what I remember, the sex of someone is determined by their reproductive organs, so it isn't instant, and the hormones need to help the cells create them. But, the neutral theory explains the nipples and the clitoris theory. I remember reading about "neutral" babies, that then acquire their sex by hormones (this also explains what happens in the case of hermaphrodites).
So, the neutral theory makes sense to me, but at the same time, the chromosome triggers certain hormones (at least I assume they do, so I guess we can consider if you have the Y chromosome then you're going to get male reproductive signals). I'm also not really sure what kind of specific genes the Y chromosome really carries. Having cells that are able to differentiate in the early stages is all what makes this happen as well in my opinion. Maybe some anatomy/biology students or graduates could chime in on this subject (I'm a chemistry major).
