Poverty deeply impacts society by contributing to unemployment, health disparities, social inequality, and political instability, leading to a series of socio-economic and environmental crises. As climate change alters rainfall patterns, its impact on household poverty becomes increasingly important, yet it remains poorly understood. A major challenge in climate-poverty research is the lack of accurate fine-scale data on the spatial and temporal distribution of community-level wealth assets, particularly in Sub-Saharan Africa (SSA), which is essential for understanding complex societal impacts. Using data from 863,944 households across 34,055 communities in 32 Sub-Saharan African countries between 2010 and 2021, we developed a deep learning model to generate satellite-based, fine-grained wealth data to investigate the temporal and spatial variations in household poverty across SSA. We employ a fixed-effects panel regression model to estimate the relationship between rainfall patterns—such as extreme daily rainfall, the number of wet days, and seasonal variability derived from apportionment entropy—and household poverty across different temperature zones and both rural and urban regions. Our analysis reveals significant variations in how extreme daily rainfall, the number of wet days, and apportionment entropy affect household asset wealth across Sub-Saharan Africa, depending on poverty levels and living conditions. Extremely impoverished households are particularly vulnerable to rainfall variability, while wealthier households can better mitigate these effects. The most significant declines in the International Wealth Index (IWI) occur in low and middle temperature zones, where nearly 84% of the extremely poor households are located and experience high rainfall variability. This work lays the groundwork for future research on the effects of climate change on household poverty of impoverished communities.