My wife & I brought our 4 boys to the SU/BC game this past weekend (Otto's kids club game). Ages are 8 months, 3 yrs, 5.5 yrs and 8.5 yrs old.
Normally I attend games with a bunch of friends & we tailgate beforehand. This time the family joined the rest of us, along with a friend of mine's son (age 9). We all tailgated for a while before going in. The older 3 loved it (Obviously the baby will have zero recollection), eating food in the parking lot, throwing the football around and just the whole atmosphere of everyone outside having a good time. We made it the entire game with really minimal snacks thanks to the pre-game tailgating. Kids never asked to leave, seemed to enjoy watching the game & spending time with my friend's son. Next morning my 5.5 yr old asked if we could go to the SU game the following day. Told him next home game was 4 weeks away, but he was ready!
Typically we'd leave the baby with my parents, but they were unable to watch him so he came along. Full disclosure - my wife knows I'm a maniac at games and am incapable of watching/caring for a small child so she held him for a solid 2/3 of the game, he slept another quarter and I held him the remainder of the time. We sit up in the very top row of the Dome, so we have lots of space to spread out. We were able to put down a blanket, wrap him up & allow him to rest.
If you're really into the game and you think you'll have to spend a fair amount of time entertaining your baby, I'd wait. I don't leave early and I wasn't going to even if my kids wanted to. The older 2 have been to enough sporting events (and like watching) where I knew it was safe to take them, but worried about the other 2. Like I said, only reason mine was there was because my parents were unavailable and my wife was caring for him the majority of the game. I will say, it's good to start them young so they're interested and grow to love the program. I think 3 or 4 years old would be a good test run for a 2 hour bball low key game (exhibition or Cornell type game). Good luck!