I think in man, the refs see the defensive players constantly hand-checking and poking guys in the back. They are not going to call fouls every 5 seconds on that though it is illegal. When playing zone, you are not guarding a guy and so when the defensive guy rushes the offensive guy and makes contact, not unlike the guys playing man, it seems out of place in contrast and so they call fouls.
I have no issue with guys being physical. The problem to me is that refs allow hand-checking and poking. Guys are getting mugged and it is called "physical defense." Used to be your coach would tell you to play defense with your feet. Now you can hand-check with impunity. It's lazy defense to me and takes a lot away from the game.
With that said, if the refs are not going to call fouls, play man, hand-check, poke and shove people around.
While I am at it, whatever happened to traveling or the 3 second call? They usually call one or two travels per game just for fun and I can't recall the last 3 second call I saw.