All of those stars, all of those planets, and we are the 1 and only planet with "intelligent" life? The thought of that is ridiculous.
What more likely is the scenario that physicist Neil DeGrasse Tyson described at one time. He said (paraphrasing), "The difference between a chimpanzee and a human being is roughly 2% of our DNA. Now, imagine some other humanoid-like extraterrestrial that is another 2% different than us, but just as much advanced beyond us as we are to chimpanzees. Do you think they would be able to have the intellect, the understanding, the advancements that would allow them to be interstellar travelers? Would they look at us, in comparison, as we do to chimpanzees? Things that our brains would suggest that are "incomprehensible" would be "simple concepts" to them."
There are too many "origin" stories passed down though the ages, in nearly every culture around the world that, at one time, there were "Gods", "kings", and "leaders" that came from the skies and flew in vehicles. These go back several thousand years and are consistent across the worlds cultures. Too many consistencies to be ignored.
There are too many military and commercial pilots that have documented their sitings of "unexplained aerial phenomena" to be ignored.
Personally, it would be ridiculous to suggest that we are alone. It would be equally ridiculous to suggest that none of those planets in the vast universe could have produced "advanced" intelligences and solved the "impossibilities" of interstellar travel. Now, whether we were colonized, whether we were visited, whether they walk amongst us, we may or may not answer those questions, at least for not a long time. I don't think we, as a species, have the intellect and personality traits, in general, to accept the realities of extraterrestrial visitors even if the "mother ship" was orbiting the Earth right now. Our primate "monkey brain" and fear centers would throw many of us into a panic. It's safer, in our minds, to just say "NO" and not think about it.