Since sequencing techniques become less expensive, larger sample sizes are applicable for microbiota studies. The aim of this study is to show how, and to what extent, different diversity metrics and different compositions of the microbiota influence the needed sample size to observed dissimilar groups. Empirical 16S rRNA amplicon sequence data obtained from animal experiments, observational human data, and simulated data was used to perform retrospective power calculations. A wide variation of alpha diversity and beta diversity metrics were used to compare the different microbiota data sets and the effect on the sample size. Our data showed that beta diversity metrics are most sensitive to observe differences compared to alpha diversity metrics. The structure of the data influenced which alpha metrics are most sensitive. Regarding beta diversity, the Bray-Curtis metric is in general most sensitive to observe differences between groups, resulting in lower sample size and potential publication bias. We recommend to perform power calculations and to use multiple diversity metrics as an outcome measure. To improve microbiota studies awareness needs to be raised on the sensitivity and bias for microbiota research outcomes created by the used metrics rather than biological differences. We have seen that different alpha and beta diversity metrics lead to different study power: on the basis of this observation, one could be naturally tempted to try all possible metrics until one or more are found that give a statistically significant test result, i.e. p-value < α. This way of proceeding is one of the many forms of the so-called p-value hacking. To this end, in our opinion, the only way to protect ourselves from (the temptation of) p-hacking would be to publish, and we stress here the word publish, a statistical plan before experiments are initiated: this practice is customary for clinical trials where a statistical plan describing the endpoints and the corresponding statistical analyses must be disclosed before the start of the study.