Oh wow, this'll be fun....
Ocarina of Time and Majora's Mask are both must haves. As is Mario 64.
Now looking at less obvious games, there's a good few hidden classics on the N64.
Perfect Dark is basically Goldeneye set in the future, and is a great game. Worth buying for the co-op mode, which is just pure fun. Last great game by the once magnificent Rare.
The original Smash Bros is great, second best in the series behind Melee.
Banjo Kazooie is a great platformer. Buy it's sequel too if you like the original.
1080 Snowboarding is a hidden gem, and is great if you like extreme sport games.
Mario Kart 64 is, in my opinion, the best in the series. Yep, better than the original.
Blast Corps is another hidden gem. You play a giant machine trying to clear paths for a truck full of nukes that has lost control. Not to everyone's tastes, but a very good game.
Conker's Bad Fur Day is great for any teenager. A solid game with some really good humour in there.
Excitebike is like 1080 Snowboarding, a must have if you enjoy extreme sports games.
Fighters Destiny is another hidden gem and is a game for those who are fans of Virtua Fighter, it's more about strategy rather than button mashing.
Killer Instinct Gold is in my opinion, the most underrated fighting game ever. I'm sad that the series never went past 2 games, but this one is pretty damn good. Better than Mortal Kombat 4, anyway.
Ignore the later Mario Party games, the original on the N64 is one of the best multi player experiences I've ever had. And I've had some real good ones.
If you haven't played Resident Evil 2 on any other console, I'd say give it a go. Better than the first one, in my opinion and has a lot more scary moments. Some of the ways those Lickers appear will freak you the fuck out.
Pokemon Stadium is a must have for all Pokemon fans.
Paper Mario is a must have for all Mario and/or RPG fans.
Lylat Wars.... The game that made me an N64 fan. It is absolutely superb and easily the best Starfox game ever made. It's a shame that the game is so underrated because it is an absolute classic from start to finish. Apart from Peppy. He's a douche.
WWF No Mercy is a must have for wrestling fans. While it does show it's age a bit today, it's still a great game. Better than 90% of the Smackdown games.
And lastly, Turok 2. While this is probably the only N64 game worth having that I never played, I've heard nothing but good things about it. It has a great Legacy and a big cult fanbase, so it's worth checking out.
That's it, pretty much every N64 game worth playing that I can think of. But I'm sure Lee might have some I missed.