Hah, here's the prof http://news.ycombinator.com/item?id=3196443 (I don't think there's any more unpopular comment you can make here) see it's -4 that shows but realistically I think I had -11 downvotes (it might be -10).

-4 is the lowest score a comment can achieve. Downvotes past -4 won't reduce the post's score, but they will keep draining karma from the user. I don't think there's actually a limit on that.

Actually they will reduce the comment's score too, it just won't show. The difference is that if it is showing -4 and someone upvotes it, it won't neccesarily go to -3, because it's actual score could be much lower.