This field is required.

My ultimate handheld gaming device... well for starters, it has to be a clamshell. The GBA SP and DS have it right, protecting that screen is priority number one. Next, it would be nice if it had two analog sticks. This isn't a knock against the PSP directly, it's just that modern game development has had access to those style of controls for a good two generations now, and it's long past time for handhelds to catch up. A small form factor is basically a necessity, should be no larger than an iPhone or DSi. The best portable devices are pocket sized or a bit smaller.
 
In terms of media, you still need some sort of solid format to sell in stores. I don't buy into the digital only future, at least as far as handhelds are concerned, because handheld consoles are always going to be most popular with the very young audience who doesn't have easy access to a credit card for buying games online. Something like UMDs would probably be best, in terms of production costs and storage size. That doesn't mean you can't have online integration, preferably with some sort of cheap memory card format. At least one touchscreen is basically a requirement now, with the DS and iPhone making that a standard expected feature.
 
The device should be at least as powerful as last generation's hardware, but this is the least important feature. What's more important is that it's cheap and easy to develop for. Nintendo's learned this lesson well. Wifi multiplayer would be nice, but not really required I don't think. Media functionality is, again, one of those things I don't think necessary, as almost everybody has an MP3 player now, and who's really going to watch movies on a tiny screen.
 
Ultimately, I suppose my ideal handheld is a DSi with a disc format, two analog sticks, and the PSP's horsepower. You start tossing any more features in there, and it just becomes a cluttered mess that can't execute any of it's peripheral angles properly. A handheld gaming device is for gaming, and any other function should be secondary to that nature. There are always going to be better phones, MP3 and PMP devices out there, and trying to compete with them is foolish and unnecessary.