The growing availability of single-cell data has sparked an increased interest in the inference of cell-cell communication from this data. Many tools have been developed for this purpose. Each of them consists of a resource of intercellular interactions prior knowledge and a method to predict potential cell-cell communication events. Yet the impact of the choice of resource and method on the resulting predictions is largely unknown. To shed light on this, we created a framework, available at https://github.com/saezlab/liana/tree/biorxiv-clean, to facilitate a comparative assessment of methods for inferring cell-cell communication from single cell transcriptomics data and then compared 15 resources and 6 methods. We found few unique interactions and a varying degree of overlap among the resources, and observed uneven coverage in terms of pathways and biological categories. We analysed a colorectal cancer single cell RNA-Seq dataset using all possible combinations of methods and resources. We found major differences among the highest ranked intercellular interactions inferred by each method even when using the same resources. The varying predictions lead to fundamentally different biological interpretations, highlighting the need to benchmark resources and methods.