My first handheld was probably this stand-alone handheld (only 1 game, not changeable)... it was an old shooter game similar to an extemely simplified version of Space Invaders (the gameplay was entirely different, but it was in that genre).
The game was cool, rather than being an LCD with a white screen with black graphics (as most stand-alone handhelds were, for example, BlindTyldak's Pac-Man watch), the game was on a black screen with red graphics which lit up (as it was an LED... so the game could even be played in the dark).
My first handheld with changable games was some sort of trivia thing, where different questions were stored on different cartridges.
My first handheld was a pink Gameboy Color. It was the first video game system that was my own, so I still love it very much. I played it all the time, and had the little light accessory thing that lets you see the screen in the dark.