While I agree that officiating in general is awful, I argue that it is because they let too much go so that when they do make calls, we're left wondering what is a foul. Players foul out. It's always been a part of the game. It's to keep players from turning the game into a rugby match. The players who are prone to fouling out will adjust to a 6 foul rule and just foul more. We saw that in the old Big East. The reason the NBA allows 6 is because the game is 48 minutes (1 foul for every 8 minutes). Accounting for the shorter game, college has the same foul allotment. Call a tighter game and educate the players about not fouling and the game will be much better.
Warning: old man rant ahead. In the 70s and 80s (when I played), you would get whistled many times even if you didn't make contact because you made a play that most often resulted in a foul. For example, if you reached across a dribbler, you got an on the arm foul. If you swatted at the ball from the side when a player went up for a shot, there'd be a foul. We were taught not to do that because it would more than likely get called.
As for players needing to sit, that's a coach's call. There's nothing in the rules that says a player has to sit because he has 3 or 4 fouls. I've often wondered why it's such a knee jerk reaction by coaches to pull a guy out. I wish there was a way to see whether that strategy pays off on average. Is the team better off having that player for the final few minutes at the risk of the game getting out of reach while he sits? It would be an interesting study.