It's not uncommon for people like us to appear selfish to others. My family used to accuse me of being extremely selfish, in addition to other things, just because they couldn't understand how my mind worked.
The thing is, I find that those who are very quick to make assumptions of a person's character like that are guilty of rigid thinking. They think rigidly like that because they have unrealistic and unfair expectations of others, and they are also quick to compare others' thoughts and actions to their own thoughts and actions.
You know what really annoys the crap out of me? When people ask stuff like, "Couldn't you have done this? Couldn't you have thought of that? Didn't it occur to you that ...? Most people would do this, so why couldn't you...?" If people just stopped having a lot of unfair and inappropriate expectations and making assumptions, then we'd stop feeling so disappointed.
There is nothing inherently wrong with being selfish. Is it wrong to put your own needs before others'? No, because why the hell would anyone do that anyway? If you don't care about and look after yourself first, who else would?
When most people hear the word "selfish" they think, "inconsiderate," "rude," overly self-absorbed. Yes, those are common synonyms of selfish, but selfish can also mean that your priority is your needs. Does prioritizing your needs automatically equate to thinking everyone else's needs are far less important than yours? No. But this is how NT's think.
What they see in us as their limited version of selfish is often just lack of awareness of the environment and surroundings, being inside our own heads, trying to cope with various things at once that may be causing us stress.