You afraid in social situations and you push people away because of the unconscious beliefs you have about yourself and others. It sounds like you have issues with being vulnerable and exposing yourself to others. I would have similar problems, I would mumble all the time and I would try to run from social situations all the time. Hopefully I can articulate the ideas that helped me improve in these aspects. Letting go of the fear of being vulnrable is a process that takes time but it can be done.
1. Understand where this fear comes from
2. Take action
1. This fear comes from your underlying beliefs. Even though you might not be aware of it, you probably unconsciously believe that you have to be perfect for others to like you, that if you say something wrong people will think that you are stupid and will think less of you, that you are fundamentally imperfect (so you have to be constantly monitoring yourself, doing something so that this doesn't shine through. You don't think that you can simply relax and still be ok)... These beliefs were acquired as you were growing up, in school, through your parents, your peers, ... As a child you misinterpret people's criticism and anger towards you as an indication that you are imperfect just by being yourself. These beliefs are unconscious, like you say, you don't know why you are nervous. They are stored in you in the form of this nervousness/fear you feel and in the feelings of unworthiness you speak of.
2. You can only let go of these beliefs by taking action. You have to receive experiences where you will feel this nervousness, they are a great opportunity to experience the fear and to tell yourself that it is just your body trying to protect you from being rejected by your peers. This might be a challenge if you are harsh on yourself and you beat yourself up for feeling that way. But there is no better recipe for failure than making your problems 10 times heavier by telling yourself that you shouldn't have them. Don't beat yourself up for it. It is there, don't resist it, understand it. It is your body telling you to act so people don't see what you for what you think you are. I recommend that you seek out your opportunities, so that you can rewrite your unconscious opinion of yourself, and to see that you are OK just by being yourself, and that people are welcoming of you as you are. You will come to see things for how they are, and not like your inner fear sees them as.