The toughest I've personally been on was a mountain goat hunt in Southeast Alaska. The terrain and the weather were truly treacherous. Incidentally, the name of that area was Ford's Terror. Quite fitting.
I'm not a fearful person, by nature, but the top 3 moments on my "Scared $#!&less" list, came on that hunt.
I do think there is a distinction between a tough, as in physically demanding, hunt and hunting an animal that is difficult to get a shot on.
For the most elusive animals, I'd have to say a Coues Whitetail would rank at the top. Any sheep or goat is a true trophy with traditional archery. Like Threeunder said, a wolf and cougar without dogs is nigh on impossible. For the common fellow, a coyote can provide as tough of a target as anything.