Cognition involves locally segregated and globally integrated processing. This process is hierarchically organized, linking to evidence from hierarchical modules in brain networks. However, it remains a mystery how flexible transitions between these hierarchical processes are associated with human behavior. Here, we used a multisource interference task and measured hierarchical segregation and integration across multiple levels. Our results show more flexible transitions between segregated and integrated brain states in the task state. Crucially, brain flexibility in resting and task states was associated with human behavior. To achieve a better performance, the resting brain is optimized to be more flexible, such that it is prepared to more efficiently switch into a task state where the brain needs less flexibility to stably perform the task. Our hierarchical modular analysis was more effective in detecting the alterations in functional organization and the phenotype of human behavior than graph-based network measures at a single level.