Opinions seem to vary about whether "friendzoning" actually exists or not; I think this is mainly because people use the term in different ways. (See blog posts by pleated-jeans and legendaryarmor for examples.) Personally, I've been in a few situations where I expressed an interest in someone and they said "I'm sorry, I like you but I just want to be friends." If I assume that this decision is final then I think the term "friend zone" can apply: this relationship now has boundaries, and it won't move beyond that. However, that doesn't mean that the other person has done anything wrong; they're not under any obligation to return my affection.
Anyway, regardless of terminology, I think that a typical rejection involves two statements:
1) "You're a nice guy." (Or words to that effect, e.g. "You're a lovely person.")
2) "I'm not attracted to you."
However, I don't think that this is cause and effect. Looking at the women who've turned me down and subsequently got involved with other people, I wouldn't say that any of those men are arseholes. Similarly, looking at my friends who I know as couples, I honestly can't think of any situations where I think "What does she see in him? He's a complete bastard!"
I'm sure that some women are attracted to unpleasant men; there's apparently been a worrying trend for teenage girls to say that they'd be happy for Chris Brown to hit them. (Source: blog post by The Feminist Hangout and a collection of tweets.) However, I'm not interested in dating any of those women, so this isn't my problem. I'm certainly not going to start going around and kicking puppies in an attempt to seem more appealing!
Adapting one of the tags from that blog, being nice is just basic human decency, so it's not really something to boast about. Borrowing terminology from maths/logic, I'd say that it's necessary but not sufficient for attraction; you need something else going for you as well.
Edit: After I wrote this, I noticed that Cracked had just published an article with some helpful tips - 6 Harsh Truths That Will Make You a Better Person. Good timing there.
As a separate question, does anyone use Tumblr much? The "notes" bit at the bottom of each post seems rather odd, since it mainly consists of a long list saying "foo liked this" or "bar reblogged this". Occasionally there's an actual comment, but it's just buried in the list. Is there any way to eliminate all the liked/reblogged notes and just see actual content?