Construct Validity of a Dietary Protein Assessment Questionnaire to Explore College Students' Attitudes Towards Dietary Protein

Background: Misconceptions about consuming dietary protein may exist due to unscientic dietary information from the Internet and social media sites, as well as lack of knowledge about evidence-based dietary recommendations. Understanding attitudes towards protein is important for developing effective educational interventions, which may ultimately improve the health of U.S. adults. The objective of this study was to develop a questionnaire to evaluate college students’ attitudes and knowledge towards dietary protein. Methods: The questionnaire had 64 questions, including 8 demographic, 24 knowledge, 14 attitude, and 18 behavior questions. Exploratory factor analysis (EFA) with a principal axis factoring and promax rotation was used to explore the attitude constructs’ dimensionality. Nutrition students’ knowledge responses were compared to non-nutrition students’ responses to assess the knowledge validity. Results: Four hundred seventy participants (87.3% female) provided responses for attitude questions. Fifty-ve nutrition and 51 non-nutrition students provided responses for the knowledge questions. Three factors were retained: animal versus plant protein sources relationship with human and environmental health (Factor 1); organic protein sources (Factor 2); and protein RDA adequacy for weight loss and vegetarian diets (Factor 3). Mean knowledge responses were 66.4 + 11.5% and 47.6 + 16.4% for nutrition and non-nutrition students, respectively (t-test p-value for difference < 0.001). Conclusions: Protein attitudes appear multidimensional and correlated. Further testing is needed to conrm the three-factor model and to estimate test-retest reliability.


Background
Nutrition knowledge is associated with adherence to dietary recommendations. 1 The average protein intake in the US is close to the recommended amounts for all age-sex groups; however, the average intakes of different protein types vary in comparison to recommendations, especially for seafood. 2 This discrepancy may be due to lack of nutrition knowledge due to unreliable sources, and unawareness of evidence-based recommendations. 1 Protein recommendations vary by life cycle phase, disease state, and physical activity. Moreover, dietary protein recommendations vary across professional organizations. [3][4][5] While differences across organizations are consistent in some respect, they vary signi cantly in others which may be a result of antiquated research or inconsistent techniques used to determine needs. [3][4][5] These discrepancies are of consequence for health professionals who provide dietary recommendations for patients and for the public who seek dietary recommendations.
Limited research on protein knowledge and attitudes has been reported, especially among U.S. adults, and no validated instrument exists to accurately assess these constructs. 1 It is crucial to understand protein knowledge and attitudes to design and implement appropriate education tools, increase awareness, and decrease misconceptions. Considering these limitations, the Dietary Protein Assessment Questionnaire (DPAQ) is under development to quantify dietary protein knowledge, attitudes, and sources of information so that researchers can explore the relationships between these constructs and outcomes.
The DPAQ will ultimately help professionals create and provide appropriate educational resources and interventions to improve the health of U.S. adults. This study provides valuable preliminary data on construct validity of the knowledge and attitude questions, which will guide future development of the DPAQ to become the rst validated instrument for dietary protein.

Item Generation
The items for the DPAQ were generated using principles from Don Dillman's book on survey development. 6 The DPAQ consisted of 64 questions on the knowledge, attitudes, and behaviors towards protein, including 8 demographic questions. The knowledge questions consisted of three answer choices (true, false, unsure) and were created to assess respondents' knowledge about dietary protein sources and requirements for various populations, such as physically active individuals and individuals adhering to a vegetarian diet. The attitude questions included a 5-point Likert scale ranging from "strongly disagree = 1" to "strongly agree = 5" with a neutral midpoint to assess respondents' attitudes towards plant and animal protein sources. The behavior questions consisted of multiple-choice answer options to assess respondents' dietary patterns regarding protein.
Nutritional science researchers reviewed the questionnaire for applicability, structure, reading level, and comprehension. The questionnaire was then updated according to feedback. Cognitive interviews were conducted using individuals with no nutrition background to assess information-processing needs of the questionnaire items. 7 Researchers and statisticians reviewed the questionnaire to identify appropriate scaling of answer choices and the questionnaire was updated to create the nal version prior to distribution. The DPAQ was then administered using PsychData (PsychData.com, LLC, State College, PA).

Sample and Recruitment
In the fall 2018, participants were recruited through an open call email sent to students attending Texas Woman's University. The email informed potential participants of the study's purpose, eligibility requirements, and included a link to the DPAQ. Participants were recruited with the help of professors and researchers to voluntarily complete the questionnaire. The online questionnaire link was also posted on social media sites and spread by word of mouth. Eligibility requirements included individuals > 18 years of age with a reliable Internet source.
Data were collected from nutrition students enrolled in a junior-level nutrition class and from non-nutrition students enrolled in a junior-level education class as a comparison group for the knowledge section.
Students were offered extra credit in their respective classes for successful completion of the questionnaire.
Approval of the study was obtained from Texas Woman's University Institutional Review Board. Informed consent was collected from each participant before participation in the questionnaire. Data were deidenti ed except for the nutrition and education students used for the knowledge section.

Validity Measures and Data Analyses
For the attitude questions, responses were randomly partitioned into a dataset to identify possible factor structures and another to re-evaluate the factor structure. The correlation matrix and factor loading scores were examined and items were eliminated according to criteria.
Exploratory factor analysis (EFA) with principal axis factoring and a promax rotation was performed on the 14 attitude questions to identify the dimensionality of the attitude constructs for the subjects. The correlation matrix was examined for items exhibiting multicollinearity. Factor retention criteria included factors > |0.4| and factors comprised of two or more items. Composite scores for the factors were calculated according to their factor loadings. Internal consistency reliability was examined using Cronbach's α. The questionnaire responses were then compared across gender, education, and race/ethnicity using an ANOVA and adjusted for multiple comparisons using the Tukey-Kramer adjustment where necessary.
The knowledge questions were evaluated for construct validity by comparing mean scores between nutrition and non-nutrition majors using independent samples t test. The correct answers were totaled for each student to determine the mean scores. The answers marked "unknown" were given a value of zero and did not contribute to overall mean scores. A p < 0 .05 was considered statistically signi cant for all analyses unless otherwise indicated. All data analysis was performed with SAS/STAT software, Version 9.4. Copyright © 2013 SAS Institute Inc. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc., Cary, NC, USA.

Questionnaire Participants
Four hundred seventy responses were received and 450 provided complete demographic data. The majority of participants were female (87.3%) and the mean age was 28.2 + 11.4y. See Table 1 for complete demographic information.

Exploratory Factor Analysis #1
Two hundred twenty-ve participants were randomized to the rst EFA; 74.2% provided complete responses for the attitude questions. The data exhibited good sampling adequacy (Kaiser-Meyer-Olkin Test = 0.8) and the correlation matrix was suitable for structure detection (Bartlett's Test < .001). There was some evidence of multicollinearity observed within the correlation matrix (determinant = .006). A total of ve items did not meet inclusion criteria (primary factor loading > |0.4|) and were removed for the subsequent EFA. The rst EFA retained four factors and explained 62.3% of the total variance.

Exploratory Factor Analysis #2
Two hundred twenty-ve respondents were randomized to the second EFA; 59.1% provided complete responses for the attitude questions. The data demonstrated good sampling adequacy (Kaiser-Meyer-Olkin Test = 0.76) and the correlation matrix was suitable for structure detection (Bartlett's Test < .001).
The correlation matrix was examined for items exhibiting extreme multicollinearity (determinant = .007).
There was some evidence of multicollinearity observed among the statements of "meat consumption is unhealthy" and "meat should not be consumed" (r = -90). Three factors were retained, which were comprised of the nine items remaining from EFA #1 and explained 73.9% of the total variance. See Table   2 for variance explained by each factor. All items displayed a factor loading > |0.4|.
Factor 1 included ve items related to animal protein sources and their relationship with human and environmental health. Factor 2 included two items pertaining to the healthfulness of organic protein sources. Factor 3 included two items describing the adequacy of the RDA for protein with respect to weight loss and adherence to a vegetarian diet. Factor 1 shared a moderate, inverse relationship with Factor 2 (r = -0.47), and a weak, positive relationship with Factor 3 (r = 0.29). Factor 2 shared a weak, inverse relationship with Factor 3 (r = -0.19). Cronbach's α coe cient for Factor 1 (α = 0.87) and Factor 2 (α = 0.83) displayed evidence of good internal reliability. Satisfactory internal reliability was observed for Factor 3 (α = 0.65).

Knowledge Towards Protein
Fifty-ve nutrition undergraduate students and 51 education undergraduate students' responses were analyzed. The majority of participants were female (95.3%) and the mean age was 27.9 + 11.2y. The nutrition students mean test score was 66.4 + 11.5% with scores ranging from 42 -92%. The education students mean test score was 47.6 + 16.4% with scores ranging from 17 -79%. A signi cant difference in mean test score values was observed between nutrition and education students (18.8 + 14.1; P < .001). Cohen's d indicated a large, standardized difference between nutrition and education mean scores (d = 1.33).

Discussion
Studies that aim to identify knowledge and attitudes towards speci c macronutrients, such as protein, lack validated instruments. [8][9][10][11][12][13]15 Currently, no validated questionnaires exist that attempt to measure the knowledge and attitude constructs of protein among the college student population. 9 Therefore, the aim of this study was to develop a questionnaire to measure protein knowledge and attitudes, as well as to provide preliminary evidence of knowledge construct validity and explore the dimensionality of the attitude constructs, in the DPAQ.
The EFA identi ed a multidimensional structure and the original 14 attitude items could be shortened by ve items without decreasing internal reliability. Five items loaded strongly with human/environmental health (Factor 1). Items contributing positively to the Factor 1 score include 'meat production is harmful to the environment,' 'meat should not be consumed,' and 'the impact of climate change can be reduced by consuming less meat, dairy products, and eggs.' Items contributing negatively include 'meat consumption is unhealthy' and 'egg consumption is harmful to human health.' The inverse contributions of the items 'meat consumption is unhealthy' and 'meat should not be consumed' to the overall Factor 1 score may provide evidence that college students' do not consider health when determining food items that should and should not be consumed.
Two items loaded strongly with organic sources (Factor 2), which suggests that college students agree organic protein sources are healthier and better for the environment. Although the exact extent is unknown, this shows that college students place some value on organic protein sources. Two items also loaded strongly with protein RDA (Factor 3), which shows that college students believe the RDA for protein is adequate in terms of healthy weight loss and people adhering to a vegetarian diet. The factor structure provides evidence that attitude constructs towards protein are multidimensional.
Future development of the DPAQ should further develop the attitude constructs. An analysis of the relationship between nutrition information sources and the attitude constructs would be bene cial to identify strategies to educate college students. Adding more items related to Factor 2 and 3 may help de ne the factors and may strengthen the correlations observed among the protein attitudes measured.
The factors displayed adequate reliability; 14 and good internal reliability was observed among Factor 1 and 2. Factor 3 displayed satisfactory internal reliability. Future development should include increasing internal reliability, such as using more clear and concise wording to assess attitudes towards the protein RDA. Temporal reliability remains unexplored.
The signi cant difference in mean test scores between the undergraduate nutrition and non-nutrition (education) students indicated that the DPAQ instrument had adequate construct validity. The nutrition students' mean test score was greater than the non-nutrition students, which has been observed in previous studies. [15][16][17][18][19] The mean test score of nutrition students in the current study was lower than those in previous studies, which may be due to many factors, such as administering the questionnaire without prior notice or wording of knowledge statements. [10][11][12][13] It is important to note the instruments used in previous studies had content not exclusively on protein, but included content related to general nutrition and salt knowledge among adult and student populations. [9][10][11][12][13] While studies have shown dietary pattern can be in uenced by eating motives and the perceived impacts on human health and the environment, more research is needed. [9][10][11][12][13]20 With further development, the DPAQ may be used to identify knowledge and attitudes towards protein on the topics of human/environmental health, organic sources, and adequacy of the RDA, as well as other topics needed to capture the full nature of protein attitudes.
Due to increased popularity of social media platforms, there has been a commensurate rise in the amount of false nutrition information presented to the general public. [21][22][23] The lack of "media literacy" may contribute to this wide range of false information. Therefore, it is necessary to create validated instruments to assess protein attitudes and knowledge among the general public. Identifying protein knowledge and attitudes will facilitate the design and development of education tools to increase awareness and decrease misconceptions currently associated with protein. Interventions targeting various factors, such as eating motives and reliable nutrition sources, may also lead to improved understanding of evidence-based protein intake.
The strengths of this study include sample sizes, internal consistency of items, and utilizing the evidencebased approach for questionnaire development; however, several limitations exist. Although participants were homogenous in gender, age, and race, results may not be generalizable to other populations. It is important to examine validity in a more diverse population before conducting broader population studies.
Just like any self-reported item, this study is also limited by the truthfulness of participants. Satisfactory internal consistency (α < 0.70) was identi ed for Factor 3, which may provide evidence of inconsistent answers to attitude questions regarding protein RDA. 14 Future studies should focus on increasing internal reliability of the DPAQ by adjusting the number of items, rewording questions, and reformatting the instrument. The instrument's validity should be examined in a more diverse population, as well as provide more complex measurements to explore the attitude constructs multidimensionality.

Conclusions
The results of this study provide preliminary evidence for the attitude and knowledge constructs validity within the DPAQ to be used among the college student population. The instrument, and, in particular, the topic on "protein RDA" requires further development. Attitudes towards protein seem multidimensional and correlated. Additional testing is needed on the DPAQ to con rm the three-factor model and to estimate test-retest reliability. A multidimensional approach seems crucial for future development of the DPAQ, as well as for effective interventions. Future development should focus on increasing internal reliability by adjusting the number of items, rewording questions, and reformatting the instrument. This will allow the DPAQ to be administered to more diverse populations, which will enable researchers to identify misconceptions towards protein, which may ultimately improve the health of adults.

Consent for Publication
Not applicable.

Availability of Data and Materials
The datasets generated and/or analyzed during the current study are not publicly available due to participant con dentiality, but are available from the corresponding author on reasonable request. Any questions regarding the data should be directed to Parker Ackerman at packerman@twu.edu.