One other factor is that Kentucky and Duke are bringing in 4-5 recruits every year. Back in the day, before the 1-and-done era, there would be years where Duke would only have 2 scholarships open. Or Kentucky might have a 3-player class.
Now, just about every year, Duke and Kentucky hoard about 35-40% of the class's Top 25 players. That leaves only about fifteen Top 25 guys for the likes of Kansas, UNC, Florida, Ohio State, Michigan, UCLA, Gonzaga, etc. to fend for. We're in there too, but not as much as we used to be.
One reason I slightly respect UNC > Duke is that Roy Boy often times keeps his kids in the program for 3-4 years (Jackson, Berry, Meaks, Maye, Johnson). Meanwhile, Coach K chooses to win the offseason (aka recruiting) and ride that hype. And that is a reason why we've struggled mightily against the Tar Heels and competed quite well with Duke. UNC is wildly talented and mature, while the Devils are uber-talented, but young/raw/unrefined.