So I've wondered this myself before, and I think they could've, but it would be more expensive, and also they probably didn't see the point in releasing a new handheld only a year after the 3ds was released. The 3ds wouldn't have sold as well as it has (almost 300 million as of September 2016!) But I think they could've made the technology because in 2011 the PSVita came out which had console quality graphics on a portable system. Especially if it was as big as the switch is now, I personally think they could've released something similar to the switch. The only thing that I'm not sure about is the joy-cons. I doubt the HD rumble could've existed because that's completely new technology. Also the whole joy-con idea itself might not have worked at that time.
So as for graphical power and processing, I think they could've pulled it off.
But for the whole joy-con system, I'm not so sure.
I hope this gives you some good insight!