Many previous studies on facial impression inference have focused on the physical features of the face. However, few have considered the effect of the observer on impression inference. Further, many studies have treated observers’ characteristics with regards to facial impression inference as errors or individual differences and thus largely excluded it from the explanatory variables. The present study, on the other hand, focused on the observers’ characteristics and examined whether the observers’ personality traits and observational behaviors influence the impression inferences of the faces. Experiment 1 examined the relationship between observer characteristics (observational behaviors and personality traits) and facial impression inferences. We found that the observers' personality traits strongly influence observational behaviors, but neither of those influences the impression inferences. Using hierarchical Bayesian models, we found a large proportion of faces’ random effect, confirming many previous studies that facial features indeed affect how the faces are seen. The results of Experiment 1 indicate there might have been an interactive effect of observational behaviors and faces’ physical features on impression inferences. To control the potential interactions, we instructed participants to look at particular areas of faces during impression inference tasks in Experiment 2. Although participants generally looked at the areas instructed to look at, we still found a robust relationship between participants' personality traits and observational behaviors. The results show that participants inferred different impressions, even for the same faces, when they were instructed to look at different areas of faces. We also found significant relationships between participants’ personality traits and impression inference ratings, suggesting people with different personality traits would have different impressions for the very same faces.