So, what do people look for in MC in general, and for mods specifically? Let's get everyone's opinions on the matter.
To me, MC is mainly about:
1. Personal building - Make your home, your realm, whatever. Everyone has their own ideas, and it's fun to both make your own, and see what others come up with.
2. Group building - The really huge projects that involve the ideas of multiple people and are assembled together can be awesome.
3. Dangerous exploration and challenges, especially in a group. Knowing you're always a couple of creepers away from death, working together to overcome challenges, moving on to the really hard stuff once you think you can handle it. It's a lot of fun.
The problem I have with a lot of mods, is that here's how the game changes regarding those 3:
#1. Massively increased
#2. Neglected - Nothing new is added that matters for this, and people end up just not bothering.
#3. Destroyed - Player power levels increase massively, there's no challenges worthy of that power level, and the result is that with all risk and challenge gone, no one plays together.
If all you're doing is personal building, it's no longer much of a multiplayer game. You can see each other's structures and trade materials and... that's pretty much it.
A good set of mods, IMO, should address all 3 of these:
#1. Provide us new stuff to build and play with. Provide new gear, but nothing that makes us demigods... just stuff we'll want to use for the new challenges.
#2. Have new block types and new mechanical stuff - anything that might make a group of us look and say "Hey, wouldn't it be awesome if we..."
#3. Add new threats, new challenges, new stuff that ensures that while we may be stronger with more tricks up our sleeves, we also face greater challenges. The world should still be dangerous - more dangerous in special areas, in fact. The desire to party should still exist.
Also super important:
-Everything needs to be stable. When a server crashes every few hours, our player count drops to 2-3 within a day or two, and then those people get bored at the lack of players and leave. I don't care how cool the mod is that's crashing it - if we can't stabilize the server, no one will stick with it.
Less important than my big 3, but things I think should at least be looked at are:
-Complexity level, especially regarding newbies. The good mods try to segment things off (new planes, new dungeon/fortress/etc types) as this ensures a newbie can simply play Minecraft, then work with the new stuff when he's learned the basic game. When we have to hit newbies with facts like "Don't use that wood you punched!" and "Don't use ore, you need to build these 7 things so you can powder it first!", it absolutely overwhelms newbies. Remember, when you first start, you're worried about simply learning to survive in the MC world, not pulling out 5th tier constructions. Even regular MC uses segmentation - note how simple it is to get a workbench, furnace, torches, etc, but how you need obsidian, blaze rods and the like before you play with enchanting and brewing. Of course some mods are specifically about adding new uses for old stuff, and that's perfectly fine... but it's best if it doesn't make the normal MC starting guides flat out wrong for the server.
-Installation difficulty. The best mods require the player to do nothing at all (purely server-side), or can be done via installer, although this is of course impractical for many. If we need a sticky explaining what to do, that's fine, but if it's a 20 step process that has to be carefully followed to the letter, expect more people to give up than to actually get it working.
-Complexity for the sake of complexity. Normal MC generally avoids items that do nothing on their own, and serve only as intermediate steps. The book is the only common one I can think of. Sometimes it makes sense to have 1 or 2 of these because there's a clear theme - book = magic in this case - but when you make the player go through 4-5 tiers without actually being able to DO anything until he finishes, it's likely to result in players just giving up.