Different doctors are ... well different.
I can't quite speak to doctors and ASD, but I have lots of experience with other kinds of doctors and they typically don't know everything, even about the condition they are specialists for. I have Cystic Fibrosis and go to a clinic, which means I see three different doctors who are all specialists, and they are all very different, and it depends on which doctor I see as to what they'll decide is going on and how they will want to handle it. They don't agree on the particulars of CF, they don't agree on what the medicines are for, they don't agree on the best course of action for treatment or when to be worried and when not too. So just because this person was a specialist doesn't mean they know it all or that another doctor would agree with them.
On the other hand ... assuming this person had a valid point, what they might have been pointing too is that simply being obsessed with something doesn't necessarily qualify it as an ASD obsession. It might be an ASD obsession, but it might not. Other conditions can cause obsessions, such as OCD. And even NTs can become obsessed with a topic or a thing, learning everything about it and/or amassing a huge collection.
Not everyone who collects coins, comic books, movies, or music and knows everything about those items has ASD. Not every star trek or star wars nerd knows absolutely everything about that stuff has ASD. Of course, a person with ASD might take it to another level, like one of my friends with ASD who has Star Wars as a special interest and not only knows which light saber combat styles each jedi master uses but has serious opinions about the merits of each one, and he takes that as seriously as NTs take debates about civil rights.
However, because a lot of star wars nerds know a lot, when he starts talking its not immediately obvious his interest is deeper than average. He just sounds super nerdy until you spend more time with him and realize just how deep it goes and how much it matters to him. So if he told that to a doctor trying to diagnose him, the doctor might not be certain right away that his level of interest is really that much more than your average star wars nerd and just say "I'm sorry, that doesn't count". Not because it isn't an ASD obsession, but because it would really take more time than is available during a test to determine if it was at the level of an ASD obsession.
So, instead of focusing on the obsession itself, which can require significant analysis to determine if it really is an ASD type obsession or not, its easier to focus on the subject matter. One thing about NT's and other conditions like OCD, their obsessions tend to be socially acceptable. The same kinds of things other people are interested in ... but people with ASD can develop interests in things no NT would ever be have an obsession with. Like a boy I read about who is obsessed with movies that have horses in them. Not movies in an established genre like sci-fi or movies with a specific actor (which is the kind of interest/collection a NT might have), but movies with horses.
So the doctor probably meant more that your obssession wasn't applicable to his test. It could not be used to determine if you had ASD or not, because it might be an ASD obsession or it might not. If you had an obsession with something unusual that no NT would be obsessed with, then that would be clear and could be marked as a yes.
So really the answer is both.