star ratings

[SUDadx2]

What does a star rating mean?

247 Sports:

89 - 80 = Three-star prospect. One of the top 10% players in the nation. This player will develop into a reliable starter for his college team and is among the best players in his region of the country. Many three-stars have significant pro potential. (247 Sports rated Iffy as a 82-85 three-star prospect.)

Rivals:

5.7-5.5 — 3-star/All Region Selection: considered among the region’s top prospects and generally among the nation’s top 800-850 prospects overall, a potential All-Conference candidate and a player deemed to have mid to low-end pro potential and ability to impact at the college level.

ESPN:

Good prospects: 79-70 [Three stars] These players show flashes of dominance, but not on a consistent basis -- especially when matched up against the top players in the country. Players closer to a 79 rating possess BCS-caliber ability and the potential to be a quality starter or all-conference player. Players closer to a 70 rating are likely non-BCS conference caliber prospects.

Recruiting: What do all those 'stars' mean?

247Sports, Rivals and ESPN each have a unique way of rating and ranking players.

www.cougcenter.com

Of course "stars" matter. But, the ability to develop talent matters more. Syracuse coaches evaluate thousands of kids each year. Coaches watch hundreds of hours of film (digital recordings now) of prospective players every year. Evaluating human athletic performance is not the exact science that Monday morning quarterbacking is, so they will miss some. Imagine watching game film of a 15 or 16 year old kid (most recruiting evaluations are done before a player's senior year) and trying to project how good they will be in 5 or six years. Most people can't do that with their own kids.

A good program needs coaches who can evaluate talent, recruit that talent (sell the program and close the deal), and develop the talent that come to campus.

 

[javadoc]

Two takeaways:

1. SU and Wake at the bottom of the list. Wake is solid and all our conference games have gone down to the wire (and we have been out-gaining everyone). How predictive and valuable are the ratings?
2. I don't think the ratings factor in portal/transfers. Perhaps the "star ratings" by themselves are not reliable predictors, if alternative sources of talent can affect team success significantly.

Bottom line, not arguable, these ratings start as proxies for offers, and are adjusted to reflect business priorities for the authoring service. There is no evaluation component from the "recruiting analysts" at the paid sites. Your trust and dependence on these artificial numbers reflects your trust in the algorithm that maps offers to a score, and then the overrides that are applied to satisfy customer bases.

"But we're last in the ACC according to 247!" You're an imbecile.

 

[qdawgg]

Two takeaways:

1. SU and Wake at the bottom of the list. Wake is solid and all our conference games have gone down to the wire (and we have been out-gaining everyone). How predictive and valuable are the ratings?
2. I don't think the ratings factor in portal/transfers. Perhaps the "star ratings" by themselves are not reliable predictors, if alternative sources of talent can affect team success significantly.

Bottom line, not arguable, these ratings start as proxies for offers, and are adjusted to reflect business priorities for the authoring service. There is no evaluation component from the "recruiting analysts" at the paid sites. Your trust and dependence on these artificial numbers reflects your trust in the algorithm that maps offers to a score, and then the overrides that are applied to satisfy customer bases.

"But we're last in the ACC according to 247!" You're an imbecile.

Making a judgement in the midst of a season on whether the star ratings having any merit is silly. Even if WF and SU both outperform their rating for that year, the statistical evidence is pretty clear historically. If your school is lower in the rankings/ratings you typically finish near the bottom. Schools with the highest rankings/ratings typically finish, most years, closer to the top 25 then schools that don’t. The data is quite clear on this, there’s really nothing to argue. Arguing that SU outperforms their recruiting ranking every once in a great while and so that means the rankings aren’t valid, is pretty silly.

 

[javadoc]

Making a judgement in the midst of a season on whether the star ratings having any merit is silly. Even if WF and SU both outperform their rating for that year, the statistical evidence is pretty clear historically. If your school is lower in the rankings/ratings you typically finish near the bottom. Schools with the highest rankings/ratings typically finish, most years, closer to the top 25 then schools that don’t. The data is quite clear on this, there’s really nothing to argue. Arguing that SU outperforms their recruiting ranking every once in a great while and so that means the rankings aren’t valid, is pretty silly.

It starts as a proxy for offers. To the extent that the coaches of the offering teams know what they're doing, there can be some value there. Questions: does the rating algorithm capture that value accurately? How much is it futzed based on the site's business prejudices?

If I were developing an analytical model for rating prospects, I would include a factor for how successful the recruiting staff was at identifying talent that others couldn't see. I doubt seriously that any of the recruiting sites have that factor built into their models. Which undercuts the entire premise of basing the initial estimate on extant offers.

We're arguing about what happens outside the window of "Alabama and Ohio State offered these kids". I'll bet that if you did a statistical analysis of the reliability of site rankings based on their own numerical rating (i.e. 4.7 instead of "4 star") you would find that the reliability drops off enormously the further you go down the scale.

 

[qdawgg]

It starts as a proxy for offers. To the extent that the coaches of the offering teams know what they're doing, there can be some value there. Questions: does the rating algorithm capture that value accurately? How much is it futzed based on the site's business prejudices?

If I were developing an analytical model for rating prospects, I would include a factor for how successful the recruiting staff was at identifying talent that others couldn't see. I doubt seriously that any of the recruiting sites have that factor built into their models. Which undercuts the entire premise of basing the initial estimate on extant offers.

We're arguing about what happens outside the window of "Alabama and Ohio State offered these kids". I'll bet that if you did a statistical analysis of the reliability of site rankings based on their own numerical rating (i.e. 4.7 instead of "4 star") you would find that the reliability drops off enormously the further you go down the scale.

I don’t disagree with what you’re saying. And maybe I’m arguing about something different than what you’re saying and vice versa. I realize the star system/recruiting rankings are rife with issues at many levels. But I also realize that teams ranked in the top 25 in recruiting, on average, are more likely to be a top 25 team. Teams that are ranked lower, either nationally or within their own league, like SU. On average end the season near the bottom of their conference.

Since I’ve been on this site and the previous site I’ve done 2 different in depth studies of this. So for me, my issue is always with people arguing we’re recruiting better than our yearly recruiting rankings, after winning 1 or 3 games. No, clearly we aren’t. Can we outperform some years despite our recruiting rankings? Absolutely and that largely, in my opinion, comes down to having a good coaching staff. But ending each year near the bottom of the ACC in our recruiting rankings, we more often than not have a record that resembles our recruiting. Whether we like it or not.

Clemson is actually a great example of that this year. Highly ranked team in recruiting and having a very poor season. That doesn’t mean the ratings don’t work, this year hasn’t panned out but on average has Clemsons recruiting ranking matched up with their record. Absolutely.

There are many flaws in the star rating but it’s always teams with poor records that find the biggest issues with them. Even with its flaws, it’s still a great predictor of especially the top ~25 and bottom ~25 teams. Where I would say the ratings can be a toss up and this isn’t a surprise. It’s much harder identifying which 2-3 star players may turn in to 5 star players after 2-4 yrs at college and which ones will underperform. It’s all the middling teams where a coaching staff that can identify talent and coach players up is very important.

 

[SUFaninNJ]

I laugh every time I think about the rutgers take on Duce.

there’s certain areas of the country you know 3 stars are going to produce from.

West Jersey, Philly/DC, Baltimore, Towson and the Peninsula, Florida etc.

Oh come now. This state argues about if there's a Central Jersey or not. You can't just make up regions like West Jersey and toss it about willy nilly!

 

[javadoc]

I don’t disagree with what you’re saying. And maybe I’m arguing about something different than what you’re saying and vice versa. I realize the star system/recruiting rankings are rife with issues at many levels. But I also realize that teams ranked in the top 25 in recruiting, on average, are more likely to be a top 25 team. Teams that are ranked lower, either nationally or within their own league, like SU. On average end the season near the bottom of their conference.

Since I’ve been on this site and the previous site I’ve done 2 different in depth studies of this. So for me, my issue is always with people arguing we’re recruiting better than our yearly recruiting rankings, after winning 1 or 3 games. No, clearly we aren’t. Can we outperform some years despite our recruiting rankings? Absolutely and that largely, in my opinion, comes down to having a good coaching staff. But ending each year near the bottom of the ACC in our recruiting rankings, we more often than not have a record that resembles our recruiting. Whether we like it or not.

Clemson is actually a great example of that this year. Highly ranked team in recruiting and having a very poor season. That doesn’t mean the ratings don’t work, this year hasn’t panned out but on average has Clemsons recruiting ranking matched up with their record. Absolutely.

There are many flaws in the star rating but it’s always teams with poor records that find the biggest issues with them. Even with its flaws, it’s still a great predictor of especially the top ~25 and bottom ~25 teams. Where I would say the ratings can be a toss up and this isn’t a surprise. It’s much harder identifying which 2-3 star players may turn in to 5 star players after 2-4 yrs at college and which ones will underperform. It’s all the middling teams where a coaching staff that can identify talent and coach players up is very important.

Have you shared those studies? I'd be very interested to see them. Have thought about doing the same thing, going back to (embarrassed) about 2003 when I had plans to write a web crawler and keep track of rankings over time, etc etc. And then life gets in the way.

I'm not arguing with the entirety of your premise, either. It's possible (probable?) that an offer from Alabama is a reliable predictor of individual success. Maybe the statistical confidence is 90%. But how valuable is an offer from Wisconsin? NC State? Arizona? So you can look at the team at the top of the ACC and say, yeah, based on offers alone, according to an objective model, that's a good class likely to result in team success. But what about the ACC's median team? It'll quickly become a crapshoot.

And the weight placed on teams' offers should be dynamic. I'll bet it's not. And maybe the ratings should be weighted based on the contributions of individual positions to team success. Maybe you can come up with a complex model that predicts success reliably. I doubt it's there yet, outside of perhaps the top O(10) teams.

And here's where I get irritable and vent. Folks kvetch about "we're last in the ACC according to 678". I'm sick of hearing it, because it's a useless metric. If you put that much weight in what's happening outside the potentially reliable predictor-window in offer-based ratings, then please, put your money where your mouth is (OK, not really possible, pardon the phrase). Let's get a new coaching staff, with a stipulation in the contract that if you don't recruit at XYZ level according to site ABC, then you're fired for cause - even after one year.

 

[qdawgg]

Have you shared those studies? I'd be very interested to see them. Have thought about doing the same thing, going back to (embarrassed) about 2003 when I had plans to write a web crawler and keep track of rankings over time, etc etc. And then life gets in the way.

I'm not arguing with the entirety of your premise, either. It's possible (probable?) that an offer from Alabama is a reliable predictor of individual success. Maybe the statistical confidence is 90%. But how valuable is an offer from Wisconsin? NC State? Arizona? So you can look at the team at the top of the ACC and say, yeah, based on offers alone, according to an objective model, that's a good class likely to result in team success. But what about the ACC's median team? It'll quickly become a crapshoot.

And the weight placed on teams' offers should be dynamic. I'll bet it's not. And maybe the ratings should be weighted based on the contributions of individual positions to team success. Maybe you can come up with a complex model that predicts success reliably. I doubt it's there yet, outside of perhaps the top O(10) teams.

And here's where I get irritable and vent. Folks kvetch about "we're last in the ACC according to 678". I'm sick of hearing it, because it's a useless metric. If you put that much weight in what's happening outside the potentially reliable predictor-window in offer-based ratings, then please, put your money where your mouth is (OK, not really possible, pardon the phrase). Let's get a new coaching staff, with a stipulation in the contract that if you don't recruit at XYZ level according to site ABC, then you're fired for cause - even after one year.

Both times I posted them on here. The last one was around the time that TCU/Boise were making waves and a logical thing to look at with those schools was recruiting and recruiting ranking. So that was quite a few years of. Beginning of Shafer as HC or thereabouts. Don’t have the post anymore and if I saved the data I used for the post I don’t have it anymore. But basics were what I said above. For the top teams and lowest teams, the rankings are extremely reliable predicting success or lack of success. Even between different services. I compared 3 different services. As I said and you pointed out, it’s the middle 50% (probably - maybe closer to 60-70%) of the rest of CFB where rankings can swing a bit.

But here’s a good example. A team that consistently has a national recruiting ranking in the 60-80 range, may end up in the top 25 on occasion. But if you take that team over a 6 year period of time for example. Their overall record is going to reflect being lower in national recruiting and that ranking is probably lower within their conference as well. Again, SU is a perfect example of all this. Our recruiting is near the bottom of the conference every year. Every year we argue our recruits are better than that. In the last 10 years in conference play, our record directly correlates with our recruiting rankings. Even if you’re not happy with how those grades are determined, all the services still arrive at the same consensus, even using different metrics.

Again, I’m not arguing the system is perfect but it is a great barometer for the expected success of a program. Even when colleges have outperformed their rankings, I found no colleges that are able to consistently outperform their rankings year in and year out. Of course it will happen some years but it’s not sustainable. The reality is, if you are perceived to recruit poorly in relation to your conference, your record over a decent period of time will reflect that.