This paper introduces a single-stage genomic selection approach which directly integrates environmental covariates within a special factor analytic framework. The factor analytic approach of Smith et al. (2001) is an effective method of analysis for multi-environment trial (MET) datasets, but has limited biological interpretation since the underlying factors are latent so the modelled genotype by environment interaction (GEI) is observable, rather than predictable. The advantage of using known environmental covariates, such as soil moisture and daily temperature, is that the modelled GEI becomes directly interpretable, and thence predictable. This paper develops a model for both predictable and observable GEI in terms of a joint set of known and latent factors, as well as non-genetic sources of variation within trials and environments. This single-stage approach is referred to as the integrated factor analytic linear mixed model (IFA-LMM). The IFA-LMM is demonstrated on a late-stage cotton breeding MET dataset from Bayer Crop Science. The results show that the environmental covariates explain 34.6% of the genetic variance across environments (compared to only 23.3% for a conventional regression model). This represents 92.7% of the crossover GEI. The latent factors then explain 40.7% of the genetic variance, which represents 87.6% of the non-crossover GEI. This demonstrates the ability of the IFA-LMM to model crossover and non-crossover GEI in a manner that is both informative and practical to plant breeding.