He demands it of his players. Blind loyalty, no less. It wasn't lost on the Bills.players, nor will it be lost on players for his new team, that he's just passing through, and will throw anybody under the bus to protect his rep. Don't think players will continue to play for a guy like that.
Plus, we still don't know how good a coach he is. He moved two programs to mediocrity, but we can't know whether he would have moved them to, say, good, had he chosen to stay. In fact, you could make the argument (I will) that he KNEW he couldn't make the teams much better and, in fact, might have seen them regress.
In which case, he wouldn't take the next step. Which I think is why he left. Out the door before reality kicks him in the butt.