We Only See What We Know