Birthday confidence knock solution not acceptable to husband

I know this probably sounds like a trivial problem but for me it brings back a lot of issues. I'm 54 and have had counselling for low self esteem which basically goes back to my childhood and my brother I feel as if I can cope with most situations but my birthday is always a total disaster.
I've had some 'bad' birthdays but my 40th was dreadful, we had gone to a member of the family for a birthday party as my cousin was 40 around the same time and my other cousin 30, it was a couple of weeks before my actual day. It was meant to be a joint celebration but although my mum had made arrangements with my aunt to buy something for her nieces she had bought nothing for me, offering instead to take me to the local supermarket to pick out something - I declined. So basically my mum had forgotten my birthday, and with my already low self confidence I was truly shaken - not the gift but the forgetting although she did think to phone me on the day (I never did get a present). My husband was with us that weekend and seemed to understand, but later turned out he had absolutely no idea.
He had been asking me what I wanted and I said I just wanted a pamper day. Nothing fancy or expensive I just wanted him to take the kids to school and come back with a bunch of flowers, take me to town to get a nice hairdo maybe a manicure, then lunch and a nice outfit by which time it would be time to pick the kids up from school and we could do the 'family' bit. So the week before my birthday when I thought he had this all sorted he tells me he has changed his day off to the Saturday so that we can enjoy the day with the kids - it wasn't my birthday and I really wanted a day just for me. So I ended up working my 40th instead, taking my son to cricket practice and he came home about 7 with a takeaway - it just seemed that as I had done everything he wanted on his birthday mine was not worth the effort. We ended up filling up a skip which had been delivered on the Saturday.
My husbands birthday is in April, mine is in May. I always try to make his birthday nice - not over the top - just a family meal and a couple of presents, and his was a couple of days ago. So we are having said family meal with my son and future daughter in law who is absolutely lovely.it is also her birthday Easter Sunday (2 days after his). Her mum lost her husband a couple years back and so she often comes over holiday times so the 3 of them will be coming back here where I will happily make a roast for us all. DIL then asks if her sister and young nephew would be able to come too, I don't have a problem with that but it did occur to me that I make a lot of effort for others while others don't seem to even remember I am there - case in point my son had forgotten my birthday the previous year too. I feel like I am invisible sometimes and I was quiet for a bit (maybe 10 minutes).
When my son and DIL had left my husband then had a go at me saying I had embarrassed him and that it looked as though we had fallen out or something, he was quiet rude (it isn't like him but he does have a dreadful cold).
So I told him exactly how I felt, that I would be spending my next birthday on my own as for the umpteenth year he hadn't bothered to take my birthday off work, would be arriving home at about 9pm and so I was going to have to spend the day alone. Previous years I have had to ask him to take the day off although he nearly always booked a holiday for the week of his birthday, he then said not to bother in future and that he hadn't wanted the fuss on his birthday anyway (I went for a drive at this point)
So the whole thing probably seems very stupid to get upset about, but this happens every year to one extent or another I am reminded that I am not particularly important (my lack of confidence assures me of this) - although I have no doubt whatsoever husband loves me.
So - having got that off my chest - I suggested that we both ignore each others birthdays in future. This isn't petulance, we never celebrate our wedding anniversary, valentines or any other occasion and that doesn't affect my knowledge of his love. It is self preservation, but he cannot see that. He has refused to consider this option because he has said he needs to make an effort to make my day special, but I know that give it a year or two and he will forget again and that whole memory / worthlessness will wash back over me again for weeks and I don't want to feel like that anymore.
How can I make him see that not celebrating our birthdays will make me far happier!