Few comments are plainly "factually wrong." For one, the facts are often a matter of debate. For two, many comments represent judgments, your own being a perfect example: is the statement "thoughtful comments that are factually wrong should be downvoted" itself evaluable as either "factually wrong" or "factually right?" Only if you believe that the "should" in this case exists independently in the world could you answer yes to that. Then it would be a fact to be discovered. Otherwise, (and most people take this position), we can agree on the facts and disagree on the "should."

So it goes with most prescriptive uses of the word "should," actually.