One way out would be if we are the first
"advanced" civilization in the galaxy.
Could it be? Well, of the 13(ish) billion
years since the Universe began, and the
roughly 10 billion years since our galaxy formed,
technological humans have existed
on this planet around 150 years or so --
maybe around one fifty-millionth of the
total time.
So if there are any planets at all out there,
some of them are likely to be older than
Earth. Life on Earth apparently arose
as soon after Late Heavy Bombardment as
possible. So any planets out there could
have had life arising on them billions of
years before Earth even formed.
This is similar to applying the Drake Equation
over the history of the galaxy (10 billion
years or so), and asking what is the
probability of an "advanced" civilization
at any given time. The total probability
over the past 10 billion years is likely
to be much, much higher than 1 -- in other
words, advanced civilization(s) would certainly
have arrived.