Population confinements have been one of the most widely adopted non-pharmaceutical interventions (NPIs) implemented by governments across the globe to help contain the spread of the SARS-CoV-2 virus. While confinement measures have been proven to be effective to reduce the number of infections, they entail significant economic and social costs. Thus, different policy makers and social groups have exhibited varying levels of acceptance of this type of measures. In this context, understanding the factors that determine the willingness of individuals to be confined during a pandemic is of paramount importance, particularly, to policy and decision-makers. In this paper, we study the factors that influence the unwillingness to be confined during the COVID-19 pandemic by means of a large-scale, online population survey deployed in Spain. We apply both quantitative (logistic regression) and qualitative (automatic pattern discovery) methods and consider socio-demographic, economic and psychological factors, together with the 14-day cumulative incidence per 100,000 inhabitants. Our analysis of 109,515 answers to the survey covers data spanning over a 5-month time period to shed light on the impact of the passage of time. We find evidence of pandemic fatigue as the percentage of those who report an unwillingness to be in confinement increases over time; we identify significant gender differences, with women being generally less likely than men to be able to sustain long-term confinement of at least 6 months; we uncover that the psychological impact was the most important factor to determine the willingness to be in confinement at the beginning of the pandemic, to be replaced by the economic impact as the most important variable towards the end of our period of study. Our results highlight the need to design gender and age specific public policies, to implement psychological and economic support programs and to address the evident pandemic fatigue as the success of potential future confinements will depend on the population's willingness to comply with them.