The rapid early spread of COVID-19 in the U.S. was experienced very differently by different socioeconomic groups and business industries. In this study, we study aggregate mobility patterns of New York City and Chicago to identify the relationship between the amount of interpersonal contact between people in urban neighborhoods and the disparity in the growth of positive cases among these groups. We introduce an aggregate Contact Exposure Index (CEI) to measure exposure due to this interpersonal contact and combine it with social distancing metrics to show its effect on positive case growth. With the help of structural equations modeling, we find that the effect of exposure on case growth was consistently positive and that it remained consistently higher in lower-income neighborhoods, suggesting a causal path of income on case growth via contact exposure. Using the CEI, schools and restaurants are identified as high-exposure industries, and the estimation suggests that implementing specific mobility restrictions on these point-of-interest categories are most effective. This analysis can be useful in providing insights for government officials targeting specific population groups and businesses to reduce infection spread as reopening efforts continue to expand across the nation.