Cameron from Random Battle has an excellent article on aggro radius on his blog. He correctly points out how little intuitive it is that monsters detect you when you cross an invisible circle, regardless of direction. Which means that you can be killing the monster's friends in plain sight of him, as long as you stand just outside that invisible circle. World of Warcraft made some improvements to aggro radius over previous games: a mob's aggro radius depends on the level difference between you and him, thus if you are higher level you can more easily pass through a group of mobs. Aggro radius in WoW isn't the same for every mob of the same level, some mobs are more aggressive than others, which keeps things interesting. And the aggro radius can vary depending on situation, for example if you open a chest you will aggro a mob that didn't notice you on exactly the same spot when you were just standing there. But in spite these minor improvements, the general concept of aggro radius is still flawed.
A more realistic detection mechanism would include a combination of sight and sound. Sight obviously works in a cone towards the front, although you could include variations, like a wider cone for two-headed ogres, or mobs with no sight at all, or mobs with eyes on their back. Sound (or smell) would work in a radius. Again variations could be included: does a warrior in plate make more noise than a mage in cloth armor? Does the monster hear you open a squeaky door or chest? Does taking an occasional shower makes you less likely to be smelled? Okay, maybe the last one is going too far. :)
If you have played any game from the Metal Gear series, or similar stealth games, you'll know what I'm talking about. It gives rise to completely new strategies and gameplay, like making a noise to get a guard to turn around, so you can pass through the area he was previously watching. In Dungeons & Dragons pen and paper roleplaying, a well-played rogue would use trick like that. In World of Warcraft a rogue *can* make monsters turn around, but that by itself has no effect, it is just a visual clue that the distraction ability succeeded. You can imagine how much more interesting a rogue would be to play if mobs had a more realistic detection system. And of course for all other classes the way we pull and fight and pass through mob-infested areas would change a lot.
So what do you say, Metal Gear Warcraft anyone?