Sometimes a reply to a down-voted comment is more insightful than the original. The only comments I down-vote and don't reply to are the "hey this is great" or the one-liner joke comments.

If I think the commenter is wrong or a douche, I'll down-vote and reply with my own side of the debate.