I don't argue with this too much, but it really just depends on tournament success vs overall success. I'm not 100% sure which is more valid for a "title" like this. NCAA tournament success is so fleeting and sometimes arbitrary. As an example: would we call Butler elite? They made it to two championship games in a row. Something Syracuse and UConn for example have never done, although I think we would logically rank both ahead of Butler.
The reason this is a debate at all is because of Syracuse's elite overall win/loss record. Ranking #5 in total wins and #6 in win percentage and having done that while existing in an elite conference for 30+ years of their history is certainly rarified air. It's something UConn, Florida, Michigan State, and others with more championships cannot touch. To put this into perspective, even with a serious downturn in the Syracuse program and an improvement in the UConn program, resulting in them getting 10 wins per season more than us (an almost impossible figure), it would take over 30 years for them to pass us in wins. There is something extraordinarily significant about that type of historical performance - you can't just have a good run of 5 years, or 10 years, or 20 years. You can't go 9-9 in conference and then get hot/sponsored-by-an-angel/take-out-second-mortgage-on-soul/whatever and win a NC and replicate that.
So it's really however you'd like to look at it. I think it is a combo of these things, but Syracuse has the hard part of the equation, the part that takes 50+ years to build. SU gets hot and wins another NC or two and you can start seriously thinking of the elite status.