Honestly, I didn't read that as a "negative reaction." Jim Brown basically indicated that it wasn't that big of a deal, but then suggested that he wasn't necessarily an advocate of the idea--not because he didn't want the number unretired, but rather because it would result in having his jersey come down. And as such, him getting that "penalty" wasn't something he preferred. But in the big scheme of things, he wasn't against it or negative about it in general.
That's what I read from his comments.
Easy fix is listed above--do the same thing as the hoops team and retire the jerseys. They should do that this year for Brown / Davis / Little, and the problem is solved.