If you have an unfriendly face, you are going to have a hard time. Some find you indifferent, and some find you cool. People who don't know you both think you are unattainable and thick-skinned. It takes some balls to approach you. Nobody believes that you are capable of feeling hurt or you are actually a friendly person. If you are also an introvert, God help you.
You can be a great listener, and a very kind person, but you are perceived as non-accessible and 'too cool'. This being the story of your life, you have already gotten used to the never ending misunderstandings.