Mind the gap between English textbooks and test papers: a corpus-based discourse from Vietnam

doi:10.21203/rs.3.rs-2624635/v1

Download PDF

Research Article

Mind the gap between English textbooks and test papers: a corpus-based discourse from Vietnam

https://doi.org/10.21203/rs.3.rs-2624635/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Alignment in between curriculum and assessment remains an important, yet underexplored, topic. This study critically investigates the content (mis)match between English textbooks and English exam papers in Vietnam. This study explores an open question, often claimed by the ministers as the goal of the education reforms: Does the English Examination closely follow what was set out in the current English textbooks, in terms of the lexical content? Through corpus-based discourse analysis, this study evaluates 6 English textbooks and 150 English past papers for upper secondary students in Vietnam for 3 years. Results show a downward trend in content alignment between English exam past papers and the English curriculum. A subsequent discussion of underlying causes is included, and recommendations are offered for future research.

critical discourse analysis

corpus-assisted discourse research

Latent Dirichlet Allocation (LDA)

language testing

alignment studies

Alignment in education refers to a content match between what is laid out as learning outcomes in the curriculum and what is to be tested in the exam (Kurz, 2011; La Marca, 2000; Tindal, 2006; Webb, 2007). For subject experts and policy-makers, content alignment is key to determine the accountability and validity of a test in that test scores should reflect learners’ mastery of knowledge content and cognitive performance required in the syllabus (Ananda, 2003; Koretz, 2008; Messick, 1995; Newton & Shaw, 2014). There are different levels of cognitive skills ranging from remembering and understanding information to more advanced levels of synthesising, evaluating and creating information (Bloom, 1956). For teachers and learners at schools, a strong degree of alignment can ensure transparency and accountability between educational practice and outcomes (Baker, 2005; Hayes, 2003). Within the centralised system in Vietnam, pupils are expected to follow the National Curriculum programme in all subjects, and to take a high-stake assessment so-called the National High School Graduation Examination at the end of Grade 12 (Minh, 2007; H. T. M. Nguyen, 2017; T. M. H. Nguyen, 2019). Whilst the traditional approach towards alignment discourse includes expert review, teachers’ surveys, and document analysis (Case et al., 2004; Martone & Sireci, 2009; Webb, 2007), this article sets out to employ a new study design that combines an in-depth discourse analysis of six English textbooks, and a robust topic modelling analysis so-called latent Dirichlet allocation (LDA) (Blei et al., 2003) to capture thematic content from 150 English past papers from 2018 to 2020. This mixed-method approach towards data collection and data analysis aims to add analytical rigour, breadth, and depth to the current scholarship. This methodological choice also allows for reproducibility and extension when further data become available. The following section offers a critical review of alignment discourse in English language testing and learning in terms of concept, development. A brief description of the study context is also provided. I then present the processing of data and the methodological procedures of this article. The findings from both textbooks and exam papers analyses are then synthesised and critically evaluated for discussions and recommendations.

Alignment Discourse: Concept, Development, And Approach

Originated in the US, alignment studies were first developed against the backdrop of 2011 No Child Left Behind (NCLB), a movement that emphasised assessment accountability and transparency in the federal school systems (Näsström & Henriksson, 2008). Major frameworks that were designed for alignment evaluation include the Webb Model, the Achieve Model, and the Surveys of Enacted Curriculum (SEC) (Bhola et al., 2003; Case et al., 2004; Martone & Sireci, 2009; Porter et al., 2007; Webb, 2007). The Webb Model presents a holistic approach that looks at content focus, articulation, across grades, equity and fairness, pedagogical implications, and system applicability (Case et al., 2004; Kurz et al., 2010; Tindal, 2006). At a lower complexity level, the Achieve Model and the SEC offer a more focused scope on knowledge and skills compatibility between assessment and curriculum content (Roach et al., 2008). The choice as to which evaluation framework to be implemented largely depends on individual research focus, contextualised needs, and methodological preferences (Näsström & Henriksson, 2008). Research shows the Webb Model is deemed appropriate for mathematics and science assessment on large-scale accountability projects in US’s federal schools (Hajo, 2010; Roach et al., 2008; Webb et al., 2007). In comparison, the Achieve Model and SEC offer more flexibility and efficiency for non-science disciplines in wider contexts beyond the US (Kurz et al., 2010; Seitz, 2017). Methodologically, the evaluation process of these models includes an expert review panel on test-item ratings, teachers’ surveys on curriculum ratings, and document analysis for cross-comparison (Porter et al., 2007; Webb, 2007). Beyond the US context, alignment research is evolving in the field of English Language Teaching (ELT). Kabouha and Elyas (2015) showed a theoretical departure from the three aforementioned models in which the Biggs’ constructive theory (Biggs, 2003) was incorporated to build alignment between learning and testing practice for English foundational programmes in Saudi Arabia. Similarly, Kozikoğlu (2018) adapted the revised Bloom’s taxonomy (Bloom, 1956) to explore the cognitive demands gap between curriculum standards and assessment items in the English National Examination in Turkey. Nevertheless, both studies remained methodologically similar to the traditional paradigm in which document analysis and expert review were employed for the appraisal process. Das et al. (2014) investigated the link between the English syllabus and official exams with a triangulation design of classroom observations, semi-structured interviews, and focus group discussion on Bangladeshi schools. A gap of varying degrees between curriculum guidelines and assessment practice is found in all these three studies, calling for a change agenda in ELT policy to address misalignment issues. All existing studies (Das et al., 2014; Hajo, 2010; Kabouha & Elyas, n.d.; Kozikoğlu, 2018) seem to highlight the urgency to inquire into alignment assessment for educational effectiveness and accountability. A key concern with alignment discourse is to ensure fairness, rigour, and validity of the findings (Roach et al., 2008; Webb, 2007). Whilst the bias and low inter-rater reliability issues among reviewers remain unaddressed in traditional alignment research, a new methodology that allows for reliable and reproducible analysis is in need (Bhola et al., 2003; D’Agostino et al., 2008; Martone & Sireci, 2009). Whilst the corpus linguistics approach has been widely used in other areas of the language education field (Correspondence & Moser, 2020; Gilquin, 2020; Zaki, 2020), its application remains underexplored by alignment researchers. To fill this methodological gap, this study represents the first corpus-based of its kind to investigate the alignment issues in the context of English language testing in Vietnam.

A Close-up Account Of Alignment Discourse: Vietnamese Education Context

Vietnam is a multilingual country in Southeast Asia with a population of over 90 million people from over 50 ethnic groups (Bui & Nguyen, 2016). Whilst Vietnamese remains the official national language, English is increasingly recognised as the most important foreign language for the country’s modernisation and globalisation cause (Le Ha et al., 2014; M. Pham & Ba Ngoc, 2020; T. H. T. Pham, 2011). A VND 4.2 trillion (US$ 183 million) project so-called the National Foreign Language Project (NLFP henceforth) was launched in 2008 with an aim to advance English proficiency for Vietnamese young people by 2020 (Bui & Nguyen, 2016; Ngo, 2017). Subsequent reforms in the English national textbooks and the English National Examination for Vietnamese pupils were implemented in the national schooling system in a gradualist approach (Socialist Republic of Vietnam, Decision No.711/QD-TTg., 2012; Decision No.404/QD-Ttg, 2015; Communist Party of Vietnam, Resolution No.88/2014/QH13, 2014; MOET, Official dispatch No.4462/BGDDT-QLCL, 2017; No.32/2018/TT-BGDDT, 2018). A new series of English textbooks from Grade 1 to Grade 12 were first introduced from 2012 to 2018 (MOET, Official dispatch No.4462/BGDDT-QLCL, 2017; No.32/2018/TT-BGDDT, 2018). This curricular change marked a shift from the previous grammar-translation focus towards a new communicative competency approach in teaching and learning the English language for Vietnamese learners (Chi Nguyen et al., 2015; V. H. Nguyen & Hamid, 2020). In terms of knowledge content, the new series of English textbooks were claimed to provide ’a systematic, cyclical, and theme-based guidance’ that can equip learners to ’use English as a communicative tool’, ’to develop deeper language knowledge and understanding of international cultures’ and ’to use English beyond classroom contexts’ (MOET, 2018:5–7, author’s translation).

Until 2014, the national examination for Grade 12 students in Vietnam consisted of two elongated rounds that caused strains on students’ welfare and public resources (Hong, 2019). From 2015 onwards, the national assessment process has been synthesised into one single examination so-called the National High School Examination. The assessment takes form of a 60-minute test with 50 multiple choice questions (MOET, Circular No.02/2015/TT-BGDDT, 2015). There are 16 questions on grammar, and vocabulary; 2 questions on communication functions, 20 questions on reading comprehension, 5 questions on writing, 4 question on pronunciation, and 3 questions on sentence error correction (Duong, 2019). Table 1 demonstrates a component breakdown of 50 multiple-choice questions (MCQ) in an English National High School Graduation Examination paper.

Table 1

Component breakdown of exam questions of an English exam paperadapted from Duong, 2019
Grammar and vocabulary	Communication functions	Reading	Writing	Pronunciation	Sentence error correction
16 MCQ (32%)	2 MCQ (4%)	20 MCQ (40%)	5 MCQ (10&)	4 MCQ (8%)	3 MCQ (6%)

In recent years, Vietnamese authorities have widely emphasised achieving alignment as one of the goals of the recent reforms (Duong, 2019; Nguyen, 2019). However, to establish whether assessment and curriculum are aligned, it is important to investigate the test content for its fitness (N. L. Webb, 2007). Despite the concerns that the multiple-choice assessment might not fit with the curricular goals (Hamid et al., 2013), there has been no established evidence to-date to confirm this is the case. To fill this knowledge gap, and to verify the alignment claims from Vietnamese policymakers, a new empirical study that is focused on examining the alignment link between the English textbooks and the English test is required. Specifically, this paper aims to answer the following question: ‘To what extent are the content of the English National High School exam papers aligned with the English textbooks for Vietnamese upper-secondary students from 2018 to 2020?’

Data

This study analysed the content of six English textbooks and generated a corpus from 150 exam papers for the National High School Graduation Examination. For qualitative discourse analysis, I selected the e-copy version of 6 English textbooks from Grade 10 to Grade 12, published by the Vietnam Education Publishing House (2018 edition) for the books are nationally used in the public school system in Vietnam, covering over 15 million Vietnamese upper secondary students (London, 2011). For a topic modelling analysis, I selected and downloaded a total of 150 past and practice exam papers for the English National High School Examination from 2018 to 2020. Table 2 describes a yearly breakdown of selected English exam papers and their download sources.

Table 2: Distribution of English exam papers in 2018, 2019, 2020

Year	Official exam papers	Mock exam papers
2018	5	45
2019	5	45
2020	5	45
Total of official and mock exam papers		150

Source: www.moet.gov.vn; www.dethi.violet.vn; www.deluyenthi.vn; www.downloaddethi.vn

Methodological Procedures: Thematic Discourse And Topic Modelling Analyses

This study employed a corpus-based approach to investigate the alignment link between textbooks and test papers of English as a Foreign Language subject in Vietnam. First, I conducted a thematic discourse analysis of six selected English textbooks. The word ‘discourse’ in this study is understood as the sequence of sentences that compose meaning of the whole text, rather than as the pragmatics of language in context (Gee, 2014; Taylor, 2014). Hence, a thematic discourse analysis was used to capture key themes of knowledge and lexical items presented to learners. Then I performed a topic modelling analysis on a corpus of 150 English exam papers to discover the thematic content of all the exam items, including the exam question, the multiple-choice options and the reading texts, through a statistical model so-called Latent Dirichlet Allocation (LDA). LDA is an automated text-analysing technique that extracts and categorises large collections of texts into a pre-defined number of topics or themes (Brookes & McEnery, 2018; Jacobi et al., 2016; Rohrer et al., 2017).

There are three key assumptions for using LDA as the second step in this study. First, it is necessary to presume that each document in itself is a mixture of different smaller topics, and it is possible to attribute the probability of the word’s occurrence to one of the topics. Second, it is possible to automate a process to analyse the content of a large collection of texts so that different words in each document can be grouped into different topics based on their probability of occurrence. This assumption is important because it allows the use of computational algorithms to automatically assign each word into a topical theme based on its probability of occurrence. This automation offers high efficiency and, more importantly, ensures objectivity for the research process (Brookes & McEnery, 2018; King et al., 2017; Rohrer et al., 2017). Finally and relatedly, the high frequencies of words and lexical items in the corpus might indicate the content emphasis of the examined documents (Dang et al., 2020; Hsu, 2014; Nation, 2016). This assumption ensures that one could use the topics automatically identified by the LDA process to compare with the qualitatively identified topics from the national textbooks.

The chosen mixed-method design is inevitable for this study for two reasons. First, the qualitative analysis establishes the researcher’s knowledge of the pedagogical content and determine the number of topics to be aligned within the analysis of test papers. Indeed, the LDA process is not possible without the researcher’s qualitative interpretations of the results because LDA does not explicitly name the topics it automatically identifies, nor it can set the quantity of topics to be generated. Second, the quantitative process offers an economical means to analysing textual data at a large scale and presents a useful means of presenting the findings in forms of visual diagrams. Amid the reproducibility benefits of the corpus-based approach, an interpretive step remains vital to facilitate the workings of LDA initially and to give meaning to the results afterwards.

Methodological Considerations With Lda

There remain concerns that LDA may consider a large number of words that frequently occur in the English exam questions and yet convey little meaning and relevance to the thematic content of the English exams (Brookes & McEnery, 2018). In this study, to ensure consistency and comparability for analysis, a pre-processing of textual data was conducted before an actual LDA analysis of the test papers was performed (for a discussion, see Feldman & Sanger, 2007). Once all exam papers were imported into RStudio (the working platform of LDA), all non-English characters in the exam paper documents were removed using the ASCII code. All characters in the compiled documents were then converted to lower case for analytical convenience. I also created a list of stop words, which are words that frequently come up in texts but convey little substantive meaning to the analysis such as command words, articles, and possessive nouns (Rajaraman & Ullman, 2011). A full list of stop words used in this process can be found in Appendix A. The final process results in a corpus of words for the LDA analysis. In performing an LDA analysis, every document is modelled as a combination of topics, and topics are models as distributions of words. This automated process still requires a human factor in that the researcher first had to determine how many ‘themes’ were to be represented in the model. A prior discourse analysis of the English textbooks will identify the themes and then inform the LDA analysis of the n constant number of themes to be generated. After that, the LDA model will run the package ‘topic_modelling’ to generate an automated categorisation of n themes for every cleaned, textual word in which the frequency of the word represents the probabilities of a word to the distributed theme. Finally, gglot2, a visualisation package afforded by the LDA model was employed to generate visual presentations of data findings.

Findings

English textbooks: learning through themes

In examining six chosen English textbooks, this study found the knowledge content was structured into four themes namely ‘our lives’, ‘our society’, ‘our environment’, and ‘our future’. Within each theme, lexical items were further divided into 33 related topics. MOET (2012) officially justified two reasons for choosing the four listed themes. First, as the target learner groups are aged from 16 to 18, the thematic system aims to reflect a transition stage from teenagers to young adults by highlighting the importance of learners’ individual lives and their relationships with society, environment and future. Second, the thematic system takes into account a multicultural context in which Vietnamese can learn to use English as a global language. Amid a potential semantic overlap across the themes, the researcher decided to maintain the original structure of themes as indexed throughout the textbooks so that no textual data is lost during the alignment check process. Table 3 provides details of themes and topics covered in the examined textbooks.

Table 3

A summary of themes and topics covered in English textbooks
Theme	Our lives	Grade	Theme	Our Society	Grade
Topic	Family life	10	Topic	Serving community	10
	Your body and you			Inventions
	Entertainment			Gender Equality
	Generation gap	11		Caring for those in need	11
	Relationships
	Becoming independent			Vietnam and ASEAN
	Leaving school and career	12		Urbanisation	12
	Life stories			The mass media
	Cultural identity			Vietnam and IOs
Theme	Our environment	Grade	Theme	Our Future	Grade
Topic	Preserving the natural environment Cultural diversity	10	Topic	New ways to learn	10
	Eco-tourism
	People and the environment in conflict	11		Further education	11
	Global warming			Health care and longevity
	Preserving our heritage			The future of cities
	Endangered species	12		Lifelong learning	12
	Ecosystems			Artificial Intelligence
	The Green movement			The world of work

The ‘our lives’ theme includes topics that reflect the emotional, physical, and social aspects of learners’ lives. From grade 10 to grade 12, topics are arranged in an ascending order of emotional maturity and critical reflections that corresponds to the maturing nature and learning needs of learners. An indicative list of lexical items based on semantic relevance to each topic under the ‘our lives’ theme is summarised in table 4 below.

Table 4

Indicative lexical items and topics under ‘our lives’ theme
Grade	Topics	Lexical items	Example of suggested vocabulary
10	Family life	Related vocabulary: - Household chores, healthy lifestyles, and entertainment	Benefit, breadwinner, chore, contribute, enormous, equally shared, household finances, ailment, air, acupuncture, cancer, consume, audience, composer, dangdut, critical …
	Your body and you
	Music
11	The generation gap	Related vocabulary: - Family rules - Relationships between boys and girls - Personal views and opinions	Afford, attitude, bless, brand name, browse, burden, cognitive, constitution, counsellor, extended family, nuclear family, work out, argument, lend an ear, be reconciled with, sympathetic, self-esteem, self-reliant, self-discipline, studious.
	Relationships
	Becoming independent
12	Choosing a career	Related vocabulary: - Hopes, dreams and ambitions after school graduation - Life stories, cultural identity - Features of Vietnamese culture	Achievement, anonymous, dedication, diagnose, distinguished, humble beginnings, perseverance, respectable, assimilate, cultural practices, diversity, martial spirit, multicultural, national pride, worship, tuition, survival, institution, apprentice…
	Life stories
	Cultural identity

The ‘our society’ theme decentres individual learners into a larger community through topics about local surroundings, social issues related to the modern world, and the globalisation process. An indicative list of lexical items based on semantic relevance to each topic under the ‘our lives’ theme is summarised in Table 5 below.

Table 5

Indicative lexical items and topics under ‘our society’ theme
Grade	Topics	Language items	Example of suggested vocabulary
10	For a better community	Related vocabulary: - Volunteers and volunteer work, and priorities for community development - Inventions at present and in the future - The role of women and men in modern Vietnamese society	Announcement, apply, balance, by chance, community, concerned, creative, disadvantaged, employment, fortunate, handicapped, earbud, economical, headphones, imitate, inspiration, portable, submarine, Velcro, address, challenge, discrimination, effective, equality, favourable, alert, article, bridegroom, engagement, fertiliser, …
	Inventions
	Gender and equality
11	Caring for those in need	Related vocabulary: - Disadvantaged people - Recommendations to improve accessibility for disabled people ASEAN countries	Accessible, barrier, blind, barrier, care, charm, comfortable, community, control, disability, disapproval, discrimination, healthcare, mobility, support, assistance, association, bloc, economy, in accordance with, interference, legal, motto, principle, progress, state-owned…
11	Vietnam and ASEAN
12	Urbanisation	Related vocabulary: - Urbanisation - Different types of mass media and their functions - International organisations in Vietnam and describe their functions	Agricultural, centralise, cost-effective, counter-urbanisation, discrimination, downmarket, energy-saving, industrialisation, kind-hearted, sanitation, addicted, advent, cyberbullying, documentary drama, microblogging, tie in, website.
	The mass media
	Vietnam and International Organisations

The ‘our environment’ theme covers knowledge on preservation of the natural habitats and the cultural heritage, both in the Vietnamese and global contexts. The topic of ‘cultural diversity’ (for Grade 10) is strongly connected with the topic of ‘cultural identity’ in the previous ‘our lives’ theme (for Grade 12), indicating there is an overlap of topic-based knowledge in the textbooks across grades. An indicative list of lexical items based on semantic relevance to each topic under ‘our environment’ theme is summarised in Table 6 below.

Table 6

Indicative lexical items and topics under ‘our environment’ theme
Grade	Topics	Language items	Example of suggested vocabulary
10	Preserving the environment	Related vocabulary: - Different cultural groups in Vietnam - Knowledge of diverse cultural groups - Pollution and how modern life affect the natural environment - Eco-tourism	Affect, alert, altar, application, best man, bride, engagement, contrast, diversity, global warming, greenhouse effect, honeymoon, horoscope, influence, life partner, mass-media, chemical, confusion, consumption, degraded, depletion, eco-system, preservation, vegetation, assignment, eco-friendly, adapt, software, proposal…
	Cultural diversity
	Ecotourism
11	Global warming	Related vocabulary: - Issues about environmental destruction - Global warming - Vietnamese world heritage sites	Absorb, atmosphere, carbon footprint, clean-up, doctorate, drought, dynasty, eligible, landscape, archaeological, authentic, boost, catastrophic, college, complex, critical, cruise, diversity, emerge…
11	Our world heritage sites
12	The Green movement	Related vocabulary: - The endangered species - Living and non-living things interacting in ecosystem - A green lifestyle	Biomass, bronchitis, conservation, deplete, dispose of, long-lasting, mildew, preservation, purification, replenish, soot, sustainability, resurrect, overwhelming, opportunity, malfunction....
12	Endangered species

The ‘our future’ theme contains topics related to the personal, professional development of learners, and the general development of society in future time. Future-oriented topics are introduced mostly for Grade 12 students as they approach closer to high-school post-graduation prospect, indicating an age-appropriateness of the learning content. An indicative list of lexical items based on semantic relevance to each topic under the ‘our future’ theme is summarised in Table 7 below.

Table 7

Indicative lexical items and topics under ‘future’ theme
Grade	Topics	Language items	Example of suggested vocabulary
10	New ways to learn	Related vocabulary: - How electronic/ electrical devices can help in learning	Access, ancestor, concentrate, device, digital, disadvantage, educate, educational, identity, handkerchief, inequality, voice recognition, …
11	Further education	Related vocabulary: - Higher education - Health lifestyles and traditional treatments for common illnesses - Life in the future	abroad, academic, analytical, admission, achieve, baccalaureate, collaboration, CV, ecological, institution, liveable, magnificent, profession, subsequence, transcript, overcrowded, …
	Cities of the futures
	Healthy lifestyle and longevity
12	Artificial Intelligence	Related vocabulary: - Lifelong learning - The benefits and drawbacks of AI - The world of work	Well-spoken, unbelievable, tedious, shortlist, habitat, get to grips with, endangered species, algorithm, workforce, qualification, applicant, ambition, flexibility, career, approachable, administrator…
	The world of work
	Lifelong learning

Qualitative results identified 4 indicative lists of lexical items from 4 learning themes in 6 English textbooks for Grade 10 – Grade 12 students. With 4 as the pre-defined number of themes, a subsequent LDA analysis on English exam papers was conducted.

English Exam Papers: Testing Through A Mixed Picture

Figure 1 shows a topic modelling analysis on 50 selected English exam papers in 2018 in which the processed corpus of assessment items was categorised into 4 themes, automatically named as themes 1, 2, 3, and 4. With different words related to humanity, human activities (talk, look, progress, friends), or occupation (teacher, apply, learn, …), it is unclear to identify theme 1 as aligned with any of the 4 topics previously detected in the English textbooks. In contrast, there is a clear indication of alignment in thematic vocabulary between themes 2, 3, and 4 with the English curriculum. Specifically, under theme 2, 17 out of the top 20 relevant items such as ‘emission’, ‘destroy’, ‘absorb’, ‘habitat’, ‘forest’, ‘ecosystem’, ‘renewable’, ‘sustainable’ are words related to, and specified in, topics like ‘preserving the environment’, ‘ecotourism’ (Grade 10) and ‘our world heritage sites’ (Grade 11) under the ‘our environment’ theme in the current English curriculum. Similarly, a collection of family-related, and self-related words have been grouped under theme 3 including ‘responsible’, ‘breadwinner’, ‘parent’, ‘burden’, ‘self’, ‘respect’, ‘custom’, which are the lexical items listed for topics ‘family life’ (Grade 10), ‘the generation gap’ (Grade 11), and ‘cultural identity’ (Grade 12) under the ‘our lives’ theme in the English curriculum. As for theme 4, there is also a high occurrence of future-related words in the top 20 relevant items such as ‘technology’, ‘university’, ‘abroad’, ‘lifelong (learning), ‘reward’, and ‘career’, which indicates theme 4 identified in the English exam papers is aligned with the ‘our future’ theme in the textbooks. On the contrary, as for theme 1, the researcher found only 2 relevant items out of 20 (‘progress’, and ‘equality’) as aligned with the ‘our society’ theme. Therefore, the evidence suggests that the 2018 English exam papers correspond to three out of four themes covered by the English textbooks, namely the ‘our environment’, ‘our lives’ and ‘our future’ themes. This finding can be summarised in Table 8 below.

Table 8

4 themes derived from LDA topic modelling, and the most relevant items for the English exam papers in 2018
Automated theme	Aligned theme	Most relevant word items
1	Unidentified	Help, mean, learn, people, may part, new, good, many, teacher, look, start, talk, friend, progress, differ, access, apply, equality, world
2	Our environment	Emission, life, destroy, preserve, chemical, protect, pollute, adapt, absorb, environment, sustain, ecosystem, gas, earth, habitat, renew, forest, climate
3	Our lives	Self, responsibility, casual, breadwinner, burden, judge, donate, norm, body, children, parent, obey, identity, custom, articulate, initiative, career, pride, respect, stuff
4	Our future	Digital, technology, university, abroad, lifelong, world, time, reward, know, survive, career, intervene, entrepreneur, support, work, long-term, improvement, differ, job

Similarly, Fig. 2 shows a theme-based categorisation on 50 selected English exam papers in 2019. Items under theme 2 and theme 4 were identified as broadly consonant with the ‘our environment’ and ‘our future’ themes in the textbooks whilst results from theme 1 and theme 3 remained inconclusive and indicated an unclear pattern of alignment. Most of lexical items under theme 2 demonstrated a close alignment to environment-related topics under the ‘our environment’ theme, such as ‘environment’, ‘green’ (‘preserving the environment’, Grade 10; ‘the Green movement’, Grade 12), ‘reuse’, ‘ecotourism’ (‘ecotourism’, Grade 10), ‘famine’, ‘catastrophe’, ‘habitat’, ‘organic’, ‘renewable’ (‘global warming’, Grade 11). Similarly, under theme 4, high frequency of words such as ‘equality’, ‘modernity’, ‘technology’ demonstrated a content compatibility with such topics as ‘new ways to learn’ (Grade 10), whilst other items like ‘degree’, ‘stress’, ‘health’, ‘tuition’, ‘recruit’, ‘offer’, ‘school’, ‘shortlist’, and ‘attend’ showed its meaning consistency with topics like ‘further education’ (Grade 11), ‘healthy lifestyle and longevity’ (Grade 11), and ‘the world of work’ (Grade 12), all are under ‘our future’ theme. By contrast, theme 1 and theme 2 conveyed mixed results, which did not necessarily fall under any pre-identified theme in the textbooks. Findings suggest that the 2019 English exam papers unaddressed two out of four themes covered by the English textbooks, namely the ‘our environment’ and the ‘our future’ themes, leaving out the ‘our lives’ and the ‘out society’ as the two unaligned themes. This finding can be summarised in Table 9 below.

Table 9

4 themes derived from LDA topic modelling, and the most relevant items for the English exam papers in 2019
Automated theme	Aligned theme	Most relevant word items
1	Unidentified	Tabloid, support, character, practice, campaign, school, provide, clear, view, head, remote, mean, win, hours, advent, assist, population, profit, interest
2	Our environment	Environment, green, offer, charcoal, reuse, fresh, award, ban, ecotourism, acknowledge, catastrophe, famine, closest, drop, city, habitat, organic, learning
3	Unidentified	Strong, drop, accord, holiday, hard, method, newspaper, regret, press, confer, book, meeting, arab, centre, british, new, certain, many, book, mean
4	Our future	Equality, modern, technology, degree, stress, health, pursue, tuition, quality, workout, recruit, school, shortlist, offer, international, qualify, popular, attend, support

Similar to the previous analyses, figure 3 presents a topic modelling analysis on 50 selected English exam papers in 2020, including the most recent official papers held in August 2020. Across theme 1, 3 and 4, findings did not share any distinct similarity with the pre-identified themes in textbooks. However, results from theme 2 demonstrated half of the top 20 relevant items to fall under the ‘our environment’ theme including ‘environment’, ‘reuse’, ‘protect’, ‘renew’, ‘support’, ‘preserve’, ‘green’, ‘greenhouse’, ‘emission’ and ‘development’. This is a remarkable finding as it shows for the third year in a row, the ‘our environment’ theme in the textbooks appeared as the testing focus in the English exam papers. Conclusively, evidence shows that in 2020, the English exam papers seemed to only focus on testing the ‘our environment’ theme whilst there was less emphasis on the remaining three themes. This finding can be summarised in Table 10 below.

Table 10

4 themes derived from LDA topic modelling, and the most relevant items for the English exam papers in 2020
Automated theme	Aligned theme	Most relevant word items
1	Unidentified	Take, first, start, nature, world, meet, parent, problem, computer, think, moral, social, life, suit, straw, human, envy, make, say
2	Our environment	Environment, reuse, people, differ, build, protect, problem, renew, animal, support, good, help, preserve, need, green, (green)house, emission, development, character
3	Unidentified	Work, see, part, day, develop, play, hours, material, good, human, think, build, money, cause, success, make, text, left, character, use
4	Unidentified	People, use, differ, mean, even, look, text, spider, new, normal, requirement, concern, writer, well, positive, suggestion, quit, become, prize, committee

Findings from six textbooks and 150 exam papers analyses indicate that the testing content was increasingly out of alignment with the learning content of English language from 2018 to 2020.The number of matching themes identified between exam and textbooks fell from 3 in 2018, to 2 in 2018, and 1 in 2020 (out of 4). It is also interesting to notice that all three English exams from 2018 to 2020 consistently included the ‘our environment’ theme in the testing content. Furthermore, the LDA analyses could not detect any remarkable occurrence of the ‘our society’ theme in the English exam papers throughout the three years. Given there exists evidence of content alignment between English assessment and textbooks, this link is shown to be weakening over the recent three years.

The findings can be used to determine the construct validity of the National High School English Examination in Vietnam. Within a test, construct representation concerns the content of the assessment itself, which provides a crucial indicator of construct validity (Embretson & Yang, 2007; Messick, 1995). A test is considered valid when the interpretations and uses of test scores match the proposed purposes of the test (Kane, 2016; Messick, 1995; Stobart, 2008). As the English test represents a high-stake assessment on the national curriculum for Vietnamese high-school exitters, the test scores should reliably reflect students’ sufficiency on the written content of the National English Curriculum. However, this study identified a mislinkage between the intended curriculum and the tested content of English subject, pointing to a lack of construct validity within the current English testing regime.

The relationship between learning materials and test content is also of high relevance to the quality of knowledge gains. Relatedly, the ‘washback’ effect denotes that assessment exerts a powerful influence on what is taught and how it is taught in language education (Bailey, 1996; Buck, 1988). This is particularly the case with high-stake assessment in which improving results may be more important than leaning gains. To promote beneficial washback effect, an alignment between educational goals and the test requirements should be achieved (Buck, 1988). In this study, whilst the lexical items in the curriculum are found to reflect communicative, multicultural and pragmatic competence for learners, the test content did not capture the same goals. As a result, Vietnamese learners may have viewed the learning steps towards these two ends as competing rather than complimentary. Hence, the misaligned English test is likely to have had a negative washback effect on learners.

Amidst the recent rise of high-stake testing and standardisation towards school curriculum (Au, 2011), there emerged a heightened need for research on quality assurance in education. To this end, alignment research provides a useful framework towards ‘quality’ in terms of the three inter-related message systems of schooling: namely, curriculum, pedagogy and assessment (Bernstein, 2003). Whilst curriculum sets out the criteria for valid knowledge and content terms to be considered ‘standards’, pedagogy implies a transmission process of that knowledge and assessment ultimately represents the valid realisation of knowledge (Bernstein, 1975). As such, the inter-relationships between three dimensions are crucial to ensure coherence between knowledge production and evaluation. From the current findings, we were not able to reaffirm the educational quality in this context due to a spotted misalignment in the selection of organisation of knowledge content between the textbooks and the tests.

Three possible causes were identified to account for the existing misalignment issues. First, there exists a time-lag issue between reforms in the textbook and the testing system. Whilst the reform process of English textbooks was completed and formalised in 2018, Nguyen (2019) suggested changes to the English National High School Examination remain work-in-progress. Second, the national textbooks and the national examination for the English language were developed through separate mechanisms within the Vietnamese Ministry of Education and Training (MOET). Whilst the subject specialists, teachers, and interested stakeholders coordinated to decide on textbooks matters, the National Examination was mostly decided behind closed doors among test developers within MOET (Nguyen, 2019; Tran and Phan, 2014). Finally, the logistical costs of holding one single National Examination for all subjects, including the English language may have caused constraints on the research and development of the testing content itself, from which the issues of misalignment may arise. Subsequent policy recommendations are made to address the highlighted problem. At the implementational level, the task of synchronising changes in both textbooks and the assessment system of the English language should be prioritised to avoid further misalignment issues. At the institutional level, more co-ordination between decision-makers of both textbooks and exam development bodies needs to be facilitated. At the financial level, additional resources can be allocated to the work of researching and innovation to improve the current English testing system in Vietnamese schools.

In extending the literature on alignment issues in education, more validation studies on the existing gap between the English curriculum and English assessment (both formative and summative tests) are required. To answer the alignment-related question, a wider range of research methodology including expert reviews and surveys with key stakeholders should also be adopted. Besides the test content, other aspects including test structure and test difficulty also require more research attention. Further research on the impacts of the perceived misalignment on teachers’ and students’ English learning and testing practice will also fill the existing research gap in this field.

Declaration of conflicting interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/ or publication of this article.

Submission Declaration:

By submitting this assignment, I affirm that the work is my own and that any material derived or quoted from the published or unpublished work of other persons has been duly acknowledged. I confirm that I have read the journal’s guidance on plagiarism.

Ananda, S. (2003).Achieving Alignment. Leadership, 33(1),18–22.
Au, W. (2011). Teaching under the new Taylorism: High-stakes testing and the standardization of the 21st century curriculum. Journal of Curriculum Studies, 43(1), 25–45. https://doi.org/10.1080/00220272.2010.521261.
Bailey, K. M. (1996). Working for washback: a review of the washback concept in language testing. Language Testing, 13(3), 257–279. https://doi.org/10.1177/026553229601300303.
Baker, E. L. (2005). Aligning curriculum, standards, and assessments: Fulfilling the promise of school reform. Measurement and Research in the Accountability Era, 315. https://doi.org/10.4324/9780203823408
Bernstein, B. (1975). Class and pedagogies: Visible and invisible. Educational Studies, 1(1), 23–41. https://doi.org/10.1080/0305569750010105.
Bernstein, B. (2003). Class, codes and control: Applied studies towards a sociology of language (2 vol.). Psychology Press.
Bhola, D. S., Impara, J. C., & Buckendahl, C. W. (2003). Aligning tests with states’ content standards: Methods and issues. Educational Measurement: Issues and Practice, 22(3), 21–29. https://doi.org/10.1111/j.1745-3992.2003.tb00134.x.
Biggs, J. (2003). Aligning teaching and assessing to course objectives. Teaching and Learning in Higher Education: New Trends and Innovations, 2(4), 13–17. https://participativelearning.org/pluginfile.php/568/mod_resource/content/1/Aligning_Teaching_and_Assessing_to_Course_Objectives_John_Biggs%20%281%29.pdf
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. The Journal of Machine Learning Research, 3, 993–1022. https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf?ref=https://githubhelp.com.
Bloom, B. S. (1956). Taxonomy of educational objectives. Vol. 1: Cognitive domain. New York: McKay, 20(24), 1. https://dx.doi.org/10.4135/9781412958806
Brookes, G., & McEnery, T. (2018). The utility of topic modelling for discourse studies: A critical evaluation. Discourse Studies, 21(1), 3–21. https://doi.org/10.1177/1461445618814032.
Buck, G. (1988). Testing listening comprehension in Japanese university entrance examinations. JALT Journal, 10(1), 15–42. https://jalt-publications.org/files/pdf/jalt_journal/jj-10.1-2.pdf#page=15.
Bui, T. T. N., & Nguyen, H. T. M. (2016). Standardizing English for Educational and Socio-economic Betterment- A Critical Analysis of English Language Policy Reforms in Vietnam BT - English Language Education Policy in Asia (R. Kirkpatrick (ed.); pp. 363–388). Springer International Publishing. https://doi.org/10.1007/978-3-319-22464-0_17
Case, B. J., Jorgensen, M. A., & Zucker, S. (2004). Alignment in educational assessment: Assessment report. Lindon: Pearson Education Inc.
Chi Nguyen, L., Obaidul Hamid, M., & Renshaw, P. (2015). Current Issues in Language Planning English in the primary classroom in Vietnam: students’ lived experiences and their social and policy implications. Current Issues in Language Planning, 17(2), 191–214. https://doi.org/10.1080/14664208.2016.1089629.
Correspondence, J. M., & Moser, J. (2020). Evaluating Arabic textbooks: A corpus-based lexical frequency study. https://doi.org/10.1111/ijal.12321
D’Agostino, J. V., Welsh, M. E., Cimetta, A. D., Falco, L. D., Smith, S., VanWinkle, W. H., & Powers, S. J. (2008). The rating and matching item-objective alignment methods. Applied Measurement in Education, 21(1), 1–21. https://doi.org/10.1080/08957340701580728.
Dang, T. N. Y., Webb, S., & Coxhead, A. (2020). Evaluating lists of high-frequency words: Teachers’ and learners’ perspectives.Language Teaching Research,1362168820911189. https://journals.sagepub.com/doi/pdf/10.1177/1362168820911189
Das, S., Shaheen, R., Shrestha, P., Rahman, A., & Khan, R. (2014). Policy versus ground reality: Secondary English language assessment system in Bangladesh. Curriculum Journal, 25(3), 326–343. https://doi.org/10.1080/09585176.2014.909323.
Duong, T. M. (2019, November). The high school exit examination in foreign languages in Vietnam: current practices. Vietnam Language Assessment Symposium, Hanoi, Vietnam. https://www.britishcouncil.vn/sites/default/files/vlas_presentation_duong_thu_mai.pdf
Embretson, S. E., & Yang, X. (2007). Construct validity and cognitive diagnostic assessment. Georgia Institute of Technology. http://hdl.handle.net/1853/34250.
Feldman, R., & Sanger, J. (2007). The text mining handbook: advanced approaches in analyzing unstructured data. Cambridge university press.
Gee, J. P. (2014). An introduction to discourse analysis: theory and method. Routledge. https://go.exlibris.link/tylsm4y5.
Gilquin, G. (2020). Using corpora to foster L2 construction learning: A data-driven learning experiment. https://doi.org/10.1111/ijal.12317
Hajo, Z. (2010). Content validity and comparability of the Lebanese national examinations in chemistry:
An attempt to use the Webb alignment model in Lebanon. [Doctoral dissertation, University of Leicester]. https://figshare.com/articles/thesis/Content_Validity_and_Comparability_of_the_Lebanese_National_Examinations_in_Chemistry_
An_Attempt_to_Use_the_Webb_Alignment_Model_in_Lebanon/10097453
Hamid, M. O., Nguyen, H. T. M., & Baldauf, R. B. (2013). Medium of instruction in Asia: context, processes and outcomes. Current Issues in Language Planning, 14(1), 1–15. https://doi.org/10.1080/14664208.2013.792130.
Hayes, D. (2003). Making learning an effect of schooling: Aligning curriculum, assessment and pedagogy. Discourse: Studies in the Cultural Politics of Education, 24(2), 225–245. https://doi.org/10.1080/01596300303039.
Hsu, W. (2014). Measuring the vocabulary load of engineering textbooks for EFL undergraduates. English for Specific Purposes, 33, 54–65. https://doi.org/10.1016/j.esp.2013.07.001.
Jacobi, C., Van Atteveldt, W., & Welbers, K. (2016). Quantitative analysis of large amounts of journalistic texts using topic modelling. Digital Journalism, 4(1), 89–106. https://doi.org/10.4324/9781315115047.
Kabouha, R., & Elyas, T. (n.d.). Aligning Teaching and Assessment to Course Objectives: The Case of Preparatory Year English Program at King Abdulaziz University [Article].International Journal of Applied Linguistics & English Literature, 4(5),82–91. https://doi.org/10.7575/aiac.ijalel.v.4n.5p.82
Kane, M. T. (2016). Explicating validity. Assessment in Education: Principles, Policy & Practice, 23(2), 198–211. https://doi.org/10.1080/0969594X.2015.1060192
King, G., Lam, P., & Roberts, M. E. (2017). Computer-Assisted Keyword and Document Set Discovery from Unstructured Text. American Journal of Political Science, 61(4), 971–988. https://doi.org/10.1111/ajps.12291.
Koretz, D. M. (2008). Measuring up. Harvard University Press.
Kozikoğlu, İ. (2018). The examination of alignment between national assessment and English curriculum objectives using revised Bloom’s Taxonomy. Educational Research Quarterly, 41(4), 50–77.
Kurz, A. (2011). Access to what should be taught and will be tested: Students’ opportunity to learn the intended curriculum. Handbook of accessible achievement tests for all students (pp. 99–129). Springer.
Kurz, A., Elliott, S. N., Wehby, J. H., & Smithson, J. L. (2010). Alignment of the intended, planned, and enacted curriculum in general and special education and its relation to student achievement. The Journal of Special Education, 44(3), 131–145. https://doi.org/10.1177/0022466909341196.
La Marca, P. M. (2000). Alignment of standards and assessments as an accountability criterion. Practical Assessment Research and Evaluation, 7(1), 21. https://doi.org/10.7275/ahcr-wg84.
Le Ha, P., Ha, V. H., & Dat, B. (2014). Language policies in modern-day Vietnam: Changes, challenges and complexities. Language, education and nation-building (pp. 232–244). Springer.
London, J. D. (2011). Education in Vietnam. Institute of Southeast Asian Studies. Singapore: Pasir Panjang.
Martone, A., & Sireci, S. G. (2009). Evaluating alignment between curriculum, assessment, and instruction. Review of Educational Research, 79(4), 1332–1361. https://doi.org/10.3102/0034654309341375.
Messick, S. (1995). Standards of validity and the validity of standards in performance asessment. Educational Measurement: Issues and Practice, 14(4), 5–8. https://doi.org/10.1111/j.1745-3992.1995.tb00881.x
Minh, N. T. T. (2007). Textbook evaluation: The case of English textbooks currently in use in Vietnam’s upper-secondary schools. College of Foreign Languages Vietnam National University Hanoi.
Näsström, G., & Henriksson, W. (2008). Alignment of standards and assessment: A theoretical and empirical study of methods for alignment. Electronic Journal of Research in Education Psychology, 6(16), https://doi.org/10.25115/ejrep.v6i16.1296.
Nation, I. S. P. (2016). Making and using word lists for language learning and testing. John Benjamins Publishing Company.
Newton, P., & Shaw, S. (2014). Validity in educational and psychological assessment. Sage.
Ngo, X. M. (2017). Diffusion of the CEFR among Vietnamese teachers: A mixed methods investigation. Asian EFL Journal, 19(1), 7–32.
Nguyen, H. T. M. (2017). Models of mentoring in language teacher education. English Language Education. https://doi.org/10.1007/978-3-319-44151-1.
Nguyen, T. M. H. (2019). The current high school exit examination in foreign languages: future directions. Vietnam Language Assessment Symposium 2019.
Nguyen, V. H., & Hamid, M. O. (2020). The CEFR as a national language policy in Vietnam: insights from a sociogenetic analysis. Journal of Multilingual and Multicultural Development, 42(7), 650–662. https://doi.org/10.1080/01434632.2020.1715416.
Pham, M., & Ba Ngoc, D. (2020). English as a Medium of Instruction (EMI) in Vietnamese Universities: Policies of Encouragement and Pedagogies of Assumption BT - Higher Education in Market-Oriented Socialist Vietnam: New Players, Discourses, and Practices (P. Le Ha & D. Ba Ngoc (eds.); pp. 261–282). Springer International Publishing. https://doi.org/10.1007/978-3-030-46912-2_14
Pham, T. H. T. (2011). “ Doi Moi”(Renovation) and higher education reform in Vietnam. International Journal of Educational Reform, 20(3), 210–225. https://doi.org/10.1177/105678791102000302.
Porter, A. C., Smithson, J., Blank, R., & Zeidner, T. (2007). Alignment as a teacher variable. Applied Measurement in Education, 20(1), 27–51. https://doi.org/10.1080/08957340709336729.
Rajaraman, A., & Ullman, J. D. (2011). Mining of massive datasets. Cambridge University Press.
Roach, A. T., Niebling, B. C., & Kurz, A. (2008). Evaluating the alignment among curriculum, instruction, and assessments: Implications and applications for research and practice. Psychology in the Schools, 45(2), 158–176. https://doi.org/10.1002/pits.20282.
Rohrer, J. M., Brümmer, M., Schmukle, S. C., Goebel, J., & Wagner, G. G. (2017). “ What else are you worried about?”–Integrating textual responses into quantitative social science research.PloS One, 12(7), e0182156.
Seitz, P. (2017). Curriculum Alignment Among the Intended, Enacted, and Assessed Curricula for Grade 9 Mathematics. Journal of the Canadian Association for Curriculum Studies, 15(1), 72–94. Retrieved from https://jcacs3.journals.yorku.ca/index.php/jcacs/article/view/40286
Stobart, G. (2008). Testing times: The uses and abuses of assessment. Routledge.
Taylor, S. (2014). Discourse analysis (T. Teo (ed.)). Springer. https://doi.org/10.1007/978-1-4614-5583-7_80
Tindal, G. (2006). Alignment of alternate assessments using the Webb system.The Council Chief State School Officers,31.
Webb, N. L. (2007). Issues related to judging the alignment of curriculum standards and assessments. Applied Measurement in Education, 20(1), 7–25. https://doi.org/10.1080/08957340709336728.
Webb, N. M., Herman, J. L., & Webb, N. L. (2007). Alignment of mathematics state-level standards and assessments: The role of reviewer agreement. Educational Measurement: Issues and Practice, 26(2), 17–29. https://doi.org/10.1111/j.1745-3992.2007.00091.x.
Zaki, M. (2020). Self-correction through corpus-based tasks: Improving writing skills of Arabic learners. https://doi.org/10.1111/ijal.12312

Appendix A is not available with this version

Download PDF

Version 1

posted

You are reading this latest preprint version

Mind the gap between English textbooks and test papers: a corpus-based discourse from Vietnam

Status:

Version 1

Abstract

Figures

Introduction

Alignment Discourse: Concept, Development, And Approach

A Close-up Account Of Alignment Discourse: Vietnamese Education Context

Methodology

Data

Methodological Procedures: Thematic Discourse And Topic Modelling Analyses

Methodological Considerations With Lda

Findings

English textbooks: learning through themes

English Exam Papers: Testing Through A Mixed Picture

Discussions And Conclusion

Declarations

References

Appendix

Status:

Version 1