Modelling languages play a central role in developing complex, critical systems. Having a precise, comprehensible, and high-quality modelling language specification is essential to all the different stakeholders using, implementing, or extending the language. Many good practices can be found that improve the understandability or consistency of the languages' semantics. However, designing a modelling language intended for a large audience is still challenging. In this paper, we investigate how to assess the specification of modelling language semantics: (i) how it helps professionals to understand the language (ii) whether it contains errors, contradictions, ambiguities, and (iii) how suitable it is to assess the correctness of the related modelling tools. We identified the various stakeholders' viewpoints on the language and argued why understanding the semantics is crucial for all of them, although on different levels that can be supported by different artefacts. Next, we assessed a state-of-the-art specification for a general-purpose modelling language, the Precise Semantics of UML State Machines (PSSM). We reviewed the text of the specification, analysed and executed PSSM's conformance test suite, and categorized our experiences according to questions that are generally relevant for modelling languages. Finally, we made recommendations for improving the development of future modelling languages by representing the semantic domain and traces more explicitly, applying diverse test design techniques to obtain conformance test suites, and using various tools to support early-phase language design.