Ranking Drupal Modules on Drupal.org

One of the biggest dilemmas I faced as a new Drupal developer was figuring out which module was the best to use in any given situation. Only when I finally got around some other Drupal developers did I get a real sense of which were good and which weren’t as good. The best indicator on the drupal.org site before that for me was the number of sites using it. Being a bit of a rebel, this rubs me wrong. Just because a few thousand sites are using it, doesn’t mean that it’s good, right? And what if a newer, better-developed module gets released? It would not pass that test.

I have looked at the issue queues of modules to see what kinds of problems people have had, but there’s no telling which problems are real, which were configuration problems, and who didn’t RTM (this is a PC blog).

I think that Drupal needs a better system. I looked at some other community sites and these were the results: Wordpress.org uses a 5 star rating system. jQuery.org also uses a 5 star rating system. Joomla uses a 5 star rating system. Yes, there’s a pattern alright. Joomla also had reviews available. The only counterpart on drupal.org is the issue queue.

Either of these options would be an improvement over nothing, but I think we could even go one better. A meta-moderated comment/rating system could ensure that the ratings we’re looking at are at least somewhat fair. Think Slashdot for Drupal.

Here’s how I picture it working:

  • Only registered users are able to rank, naturally. A higher threshold could be created using number of commits, number of issues, uses IE, etc.
  • Rankings would be per release so when a new release comes out, it frees the developer to improve his/her module without being dragged down by buggy past releases.
  • It would be great to have reviews. These should also be per release for the same reasons. I’m sure some meta moderation would be needed. Although we all love our community, some of us can be less than kind faced with a problem on a production site. A five star rating would work and could even remove the users ranking from the overall rank if their review is poorly received.
  • A user, logged in or not, should be able to search and sort based on ratings.

These are a few suggestions that could make using drupal.org and developing in general much smoother. Tell me what you think.

Comments

There's no magic solution for this. Both rating and user comments can be somewhat revelent, but as you said, a module lifespan sometime gets reduced when a new one better gets out and take all the place. I think modules should be sorted on fonctional basis, with possibility to navigate between modules. Also, it isn't because two module can do the same thing that they cover it the same way. A module could do a really simple contact form and another one offer you a lot more customization. All depends on the functionality you need. No needs for a bulldozer when a shovel could the job.

Ratings and user comments ARE useful for pointing out issues and tricks. I also look at documentation state, coding style and update frequency. Most of the time I go through all the module code to see how it's working, except for huge modules, which most of the time are used a lot anyway.

What would happen if a module has a good rating and then they release a new version. Would that module then need to start it's rating again??

I like the idea, just not sure about resting the points on every new release.

@jack the rating would be cumulative votes over time. If the support dies down, you would see the rating die down too. ofcourse it would be nice to have a multiple axis rating !

But I will be checking it out now.

Nice to hear that there's already a big discussion going about it. Maybe I'll hop over and see if I can help.

Have you built tools like this before?

It's actually really hard/controversial. Copying what other folks do is unlikely to be enough because most (good) systems use hidden dampers/amplifiers in addition the publicly visible tools.

The book Building Web Reputation Systems is amazingly useful in guiding this kind of work - their site is http://www.buildingreputation.com/

Nice! That book is definitely going on my reading list. I do think a survey of other social coding sites is instructive, even if we can't naively copy them by slapping up fivestar.

To expand the comparison, I surveyed a few other sites today:

  • RubyGems - mainly download stats
  • PEAR - none
  • CPAN - stars and reviews
  • GitHub - counts watchers / forks
  • Python package index - none - actually star ratings were removed! The mailing list is an interesting study of a community that decided star ratings were detrimental.

I was a little surprised by the predominance of the star ratings. Personally, I get excited by the prospect of more stats like percent test coverage and coder review.

Another surprise (though it could be experience bias) was drupal.org project pages seem to overall give more useful signals than most others mentioned here: we have commit stats, issue tracker stats, and were the only site that measured actual project usage, not downloads.

One last note - At what became the Prairie core conversation in Drupalcon Chicago, Leisa memorably pointed out that everything has a reputation system - it's just that sometimes it's only in our heads. :-)

Your suggestions are a really good starting point. For me at least, I think what really matters is implementing a rating system that gives the project, once a certain threshold is achieve, a "Community Approved" status (or something like that). This would definitely help novice users pick modules, since it might be difficult to ascertain good and dependable modules for those users.

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>, <cpp>, <java>, <php>. The supported tag styles are: <foo>, [foo].
  • Web page addresses and email addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Ready for transformation?