I was going to ask if you were on campus. I, of course, wasn't, but have heard from people who were neighbors, teammates, and classmates of the dissatisfied players (too diplomatic?) that their gripes were legitimate. At best, Ben was a creature of his era who was way too loose with his use of "n-----." At worst, a real bigot. Those guys enjoyed the privileges of being varsity athletes, sure, but they had a rough go of it with Ben and were treated differently than their white teammates.
As you said, there are two sides to these things.