Put this way: difficulties in social communication, interaction and imagination are at the crux of the Asperger's diagnosis. Without those impairments there would be no diagnosis. All the other stuff - sensory issues, special interests, etc. - are just the icing on the cake. They are not crucial to a diagnosis, but social and communication impairment is.
Asperger's also does not go into 'remission', unlike physical diseases. There aren't times of your life when you have asperger's and other times when you don't. It's always there.
I agree that difficulties in social interaction, etc., are at the crux of the Asperger's diagnosis. That's a matter of record. At least when Asperger's still appeared in the DSM, though, clinicians took both childhood and adult presentation into account when making a diagnosis. This leaves room for an Aspie lucky enough to have become very well adapted the opportunity to still be identified and diagnosed. It was largely my childhood presentation that convinced my doctor that I am an Aspie by clinical standards (like you said, nobody outgrows autism), and the interview process revealed dozens of Aspie traits outside of the clinical specs. A good diagnostician takes all factors into account. And as I said before, it was other Aspies who first identified me, largely based on the long list of traits that don't appear in the diagnostic manual(s). I do believe there is something to "A-dar". Since my diagnosis, I have also been gradually developing an ability to spot other people who are on the spectrum. (I had previously wondered how some Aspie friends of mine can do that.)
I just said this on another thread, to someone who was paraphrasing Tony Attwood:
"There is no neuroscience to support the assertion that people can outgrow autism. High-functioning autism
(New edit: I failed to also include Asperger's) exists on a continuum of severity that ranges from severe impairment to what amounts to just a few relatively minor traits. For those on the "low end" of the severity scale (i.e., who have a mild case), deliberate and sustained effort/therapy has been known to allow some autistics to adapt to the point where they become
functionally indistinguishable from an NT... He's never stated that the differences in actual
neurology of people on the spectrum can ever be cured or outgrown."
So I agree with you completely that Asperger's doesn't go into remission (or disappears).
If I didn't care, it wouldn't matter a jot, but the tragic thing is I do care. I do want to belong. I do want to be part of a group, form a family of my own. That's the limitation of my condition, and it's one which has led to lifelong depression.
I'm sorry to hear you've had such a struggle because of your particular range and severity of traits. Sometimes I feel bad talking about my milder case because I know that others here have a harder road than I do, but nonetheless, the things I say are true of my experience, backed up by my doctor and therapist, and well-researched on my own time. There is a range of experiences of Asperger's. All of them are valid and part of our community culture, here and in general.
I hope over time that you can find a level of functioning you can feel at peace/satisfied with.