Not sure if you heard the same thing I did, but Boeheim specifically referenced this in his post-game on the radio broadcast.
He said when Kadary is out there, teams back off of him because they know he can't shoot. He also said Kadary doesn't make jump shots in practice so Boeheim doesn't think he can make them in a game. When the teams back off, it messes with their spacing for passing and the spacing for our shooters.
Idk, personally I think that trade off (Kadary for Joe) might have been worth it in the closing minutes, but I'm not a HOF college basketball coach.
Good points about a Kadary for Buddy swap. If QG doesn't get in foul trouble, I think we easily walk away with a win.