A Stakeholder-based Evaluation of Engineering ESP Courses’ Effectiveness

Investigating stakeholders’ perception about course success has been signied as a useful method for program evaluation. This study focused on evaluating English for Specic Purposes (ESP) programs in engineering departments at Iranian state universities through examining the view of stakeholders. Engineering students (N=100) and ESP practitioners (N=8) participated in interviews and non-participant observations were carried out to achieve a better insight into the actual learning and teaching contexts. Responses showed that some obstacles such as 1) low GE prociency of students and heterogeneous classes, 2) limited time, 3) ministry decision for making the course elective, 4) students’ low motivation, 5) teachers’ low motivation, 6) inappropriate materials and teaching methods hindered course usefulness and effectiveness from the perspective of the participants. The ndings urged the importance of comprehensive needs analysis to resolve the controversies among stakeholders about course objective and contents and to eliminate the mismatches between course objectives and students’ expectations.


Introduction
Language program evaluation is a research into the components of a learning context to satisfy different goals (Norris, 2016), such as ensuring the program's quality or modifying its elements to improve it. It can also show how effective is the instruction (Kiely, 2009). Master (2005) suggested looking into factors that lead to course success or failure and exploring required measures to improve course effectiveness as an important area of research in ESP. Students' judgement about the quality of their learning experience and their attitude toward this experience are important factors to be considered in program evaluation (Kiely, 2009).
Fenton-Smith, Humphreys and Walkinshaw (2018) introduced students' satisfaction about their language learning experience as an important criterion to measure program effectiveness. Kiely (2009) maintained that "the learning experience that is satisfying in a holistic way has the potential to engage, motivate, generate effort, and lead to desired outcome" (p. 107-108). Moreover, in the ESP approach to learning, language is learnt to achieve a speci c real world objective and if the course doesn't provide the learners with the competencies required to achieve the real world goals, the course effectiveness would be jeopardized (Basturkmen, 2010). Given that, course usefulness and relevance are important factors which make an ESP course effective. Students' interest and their positive attitude toward the course are paramount determinants of a course success. Teaching and learning practices, teachers' activities and knowledge and teaching methodology were found to be other factors that affect students' learning achievement in English courses in tertiary context (Binalet & Guerra, 2014). Moreover, students' attitude toward course materials could be a predictor of their engagement and also success (Rahimi & Hassani, 2011). Furthermore, Martinovic and Poljakovic (2010) reported self-e cacy and pro ciency level as two responsible factors in forming students' views toward the course.
This study was designed to examine ESP course effectiveness in engineering elds in Iranian state universities through investigating stakeholders' views and comments on key elements including satisfaction, course usefulness and relevance, interest and attitude toward the course, teaching and learning practices and factors that affect students' learning in ESP course based on the following research question: How effective are ESP courses in the view of Iranian engineering students and ESP practitioners?
Stakeholders in this study consist of engineering students, ESP instructors, ESP material authors, and program supervisors and decision making authorities. The nding consequently could help improve ESP course effectiveness. Kiely (2009) Kiely (2009) proposed gathering data through interviews as a method for obtaining information on students' perceptions about the quality of their learning experience. As he explained, these information provides precious insight into students' "concerns, wants, expectations and investment" in their learning activity (p.108). Kiely also considered this method of program evaluation as bene cial because it engages students in re ecting about the quality of instruction and the components of the learning context which is suggested as a useful activity for English language learners. Two groups of participants took part in this study: Iranian engineering students (N=100) and ESP practitioners (N=8). Students from 9 different engineering majors in three Iranian state universities, namely Sharif University of Technology, University of Tehran and Amirkabir University of Technology were randomly chosen to participate. The scope of this study was limited to engineering elds and the sampling was done in the engineering departments of three Iranian top state universities. Tomlinson (2003) suggested evaluating course material in three stages: pre-use, whilst-use and post-use. Based on this model of evaluation, course evaluation in this study was conducted in three phases: prestudy, whilst-study and post-study. Pre-study evaluation helped estimation of possible success of the course from the perspective of students before their enrollment. Whilst-study evaluation measured course effectiveness based on students' feedback whilst studying the course. Post-study evaluation aimed at assessing the actual impact of the course when students have already passed it. Accordingly, participants were chosen in three groups:

Materials And Methods
1. Group 1: Students at lower educational level-mostly second year-who did not pass the ESP course (N= 39).
2. Group 2: Students who were taking the ESP course at the same time they took part in the study.
These students were mostly from the fourth year and very few students who took the ESP course in the third or second year (N=38).
3. Group 3: Students in this group were in the last semester of their undergraduate study or early graduate study and had already passed the ESP course in their previous semesters (N=23).
Since students in the target engineering departments did not take the course in a speci c semester, it was not possible to control students' level (especially in the third group of the participants. Purposive sampling was used to choose eight ESP practitioners to take part in the study. The central criterion for choosing the participants was being active in ESP eld as an instructor, materials developer, supervisor, course or syllabus designer or program decision maker. The rationale behind the number of the participants in interviews was the saturation approach, proposed by Charmaz (as cited in Creswell, 2013) which suggested to stop collecting qualitative data when no fresh data including new ideas and themes are emerged. In addition, non-participant observations were conducted in 7 ESP classes to grasp a more comprehensive understanding of the classroom procedures and to acquire rst-hand experience of the elements of the learning context.
Two sets of open ended questions were developed to elicit students and ESP practitioners (Appendix A) views. Students' interview questions were developed to investigate views about course usefulness in developing ESP skills, its relevance to students' needs and students' satisfaction and attitude towards the elements of the learning context and the factors that affect their learning. ESP practitioners' interview questions seeks answers about the challenges that affect course usefulness and elements which affect students leaning in ESP course. In order to collect data through non-participant observation, an observation checklist which has previously been developed and used by Jafari Pazoki and Alemi (2019) was used to evaluate the factors involved in learning and teaching context that could affect students' perceptions about ESP course (Appendix B).
Interviews with students were conducted in pairs to make the interview atmosphere less threatening and the data gathering more e cient. The interviews were conducted in a exible way to elicit lengthy answers and to encourage new themes to emerge. Each interview took 15 minutes on average. In order to interview ESP practitioners, appointment requests were sent to the potential interviewees and meetings were scheduled. Each interview was conducted in one-on-one manner, and took 40 minutes on average. In order to provide rich details about their perspective on the topics being asked, the interviews were carried out in a exible and interactive fashion.
All the interviews were in Persian, the native language of the interviewees, so that the interviewees can respond conveniently. Interviews were audio recorded and transcribed and after scrutinizing, high frequency viewpoints were extracted as text segments and labeled with appropriate codes. Codes were examined and carefully grouped into main themes which revealed the factors that affected participants view about the course. The frequency and the percentage of the codi ed themes were calculated for reporting. Some signi cant sentences and phrases were selected and grouped into the thematic categories and the best quotes were chosen to be presented as excerpts.
In addition, seven ESP classes in three Iranian state universities were randomly selected for observation. Classroom were observed with the instructor's permission who were briefed about the purpose of the research. The observation data were used to interpret students' views about their learning experience.

Results And Discussion
Student participants were asked to comment on course usefulness. From total 100 students, 54% considered the course as not useful while 37% believed it was useful and the others were neutral about this issue. In order to put a valid interpretation on students' views, they were asked to elaborate on their reasons for considering the course as not useful. About 30% of the respondents complained that the course had not offered them any new knowledge. They believed that they could learn technical English automatically by reading their sources in English and many of the technical terms had already been acquired because professors explained them in their lectures or they learned them while reading their sources in English. Therefore, when ESP course was offered as an elective course many students refused to take it. One student explained: "I am not going to take the course. I asked students at higher levels who had passed the course. I've learned that it is not useful. I think it is important to learn technical English but I don't believe that I will learn technical English in one semester and in a 2-credit course. My friends advised me to study my source books in English. So I will learn the reading skills and the technical vocabularies. I think I can manage learning technical English without taking the ESP course." Students were asked to elaborate on how effective they consider ESP course in acquiring different ESP skills namely, writing and speaking scienti cally and understanding oral and written technical text in their eld of study. Almost 80% of the students believed that they could learn ESP reading skill without taking the course and that taking the course was not necessary for learning technical terms. Students criticized the course for too much focus on reading skill and they did not consider the course effective for learning speaking or listening. One student explained: "Learning just a bunch of new technical vocabularies is of no use. If I encounter a new technical term, I can simply refer to a dictionary to check its meaning. We need to develop our English academic communicative skills especially writing and speaking which are the most di cult skills for us" On the other hand, students did not consider developing writing skill as easy as reading skill so that they can acquire it without taking the course. One student stated: "I wish we had worked on writing articles. For example, how to write an abstract or the introduction part for a research paper. Writing and speaking are the main problems which were ignored in the syllabus. Moreover, we need to learn to speak academically to present in conferences and take part in scienti c debates." These comments unveiled discrepancies between students' wants and actual course contents and objectives. Non-participant observation helped to achieve a deeper insight into this issue. Based on the observations, many aspects of the teaching method in ESP courses was close to Grammar Translation Method which advocates using rst language as the medium of instruction, reading the passages line by line and translating the sentences into student's native language and focusing on key vocabularies and concepts mainly by giving the L1 equivalent (Richard & Rodgers, 2014). Alimorad (2019) endorsed that the method for teaching ESP reading in Iranian universities is not based on current theories and attributed the application of such an approach to the limitations imposed by the learning context such as students' low motivation. Furthermore, reading was used as a mean for teaching technical vocabulary. In few cases summarizing was used as a post reading activity and as a speaking or writing task rather than an exercise for developing reading comprehension strategies.
Moreover, ESP classes in engineering elds were characterized with high amount of teacher's talk and one-sided explanations. On the other hand, students' participations were limited to answering questions. In most observed classes no cooperative practices happened and classroom activities were not engaging.
Being taught with such an old-fashioned method, students believed that they can develop reading skills on their own because this approach to teaching reading did not facilitate students' academic reading skill. This fact revealed the importance of equipping students with reading strategies which has also been supported by Jafari Pazoki and Alemi (2019). They attributed employing inappropriate and outdated teaching method to the fact that most ESP instructors in engineering elds were subject specialists who were not trained about language teaching methods. Accordingly, they urged more cooperation between applied linguists and content specialists to eliminate the problem.
In contrast to students' views about course objectives, ESP practitioners emphasized that reading is the most needed skill for students at undergraduate level to read their sources in English while speaking and writing are mostly needed at graduate level or future occupational setting. One practitioner noted: "The main objective in the course is to improve students' reading comprehension skill. Developing writing skill is the main concern of graduate students. It is impossible to teach all four skills in a two-unit course. Therefore, we must focus on a single skill. Nevertheless, we 'd better work on other skills sometimes to add variety and avoid boredom." These ndings supported Hirvela's (2013) report that some needs analysis studies indicated that students regarded reading skill as the least challenging among other skills. However, he con rmed the signi cance of this skill as the most used skill in academic and occupational settings and placed emphasis on using an appropriate approach to teach reading skill which focus on developing reading strategies and draws on purposeful reading.
ESP practitioners complained that lack of needs analysis in designing courses has caused many challenges such as unintegrated teaching approaches and using inappropriate methodology and contents by ESP teachers. One practitioner explained: "In the absence of a thorough needs analysis, ESP teachers choose to focus on different skills based on their own intuition. … Needs analysis should be done to identify the language functions that students need and use in real word situation and these language functions must be taught through interactive and authentic classroom activities." A practitioner pointed out the changes in students' ESP needs in recent years and how it is increased due to the considerable rise in communication among the members of global scienti c communities. Moreover, she pointed to the availability of wealth of information and scienti c sources on the World Wide Web which an engineer needs to have access to and emphasized the need for modi cation in the course syllabi. These remarks con rmed Long (2005) statements about the dynamic nature of needs and that needs analysis must be renewed continuously and that the ideas of all insiders about needs should be taken into account.
In addition, most students (60%) complained that the skills and contents taught in the course are not in compliance with their needs. One participant stated: "I wish the course contents were more novel and challenging but the topics were very boring and banal." Moreover, 60% of the students were not satis ed with the contents of the course materials. Here are a few comments made by students: "The course book contains texts which are boring and dated but we need updated and profound topics and more challenging and interesting contents" "The texts in the book were replete with di cult academic words and structures which are not very common in the authentic texts and sources that we have studied in our subject classes. Authentic texts are uent and they don't use complicated structures." "The course was not useful. This course would be useful for someone who is not familiar with engineering concepts and terminology. The contents were rudimentary rubrics in our major." The comments revealed that students criticized the contents of the course materials because they were not authentic. On the other hand, Tsou and Chen (2014) emphasized that material and task authenticity was an important element to evaluate a course as effective.
It is assumed that ESP courses are offered when students do not have enough background knowledge in their disciplines, Therefore, introductory texts of different disciplines can make a good reading material for ESP courses. Such texts help students build up required schema and they also introduce the discourse pattern, lexico-grammatical features and text structure in that speci c discipline (Love, as cited in Hirvela, 2013). Besides, it would be very problematic when students encounter a discipline speci c English text that they have no background knowledge about its content (Hirvela, 2013) because student's content knowledge has been proved to be a determining factor for successful performance in understanding ESP reading (Estaji & Meihami, 2016). As a result, ESP courses should not be offered early in undergraduate level. While using less specialized and introductory texts seems to be a solution to this problem, Hyland (as cited in Hirvela, 2013) argued that relying solely on introductory textbook would be problematic because they do not familiarize students with all types of text structure used in other genres within a speci c discipline such as journal articles. Moreover, taking the course in the last year and encountering such rudimentary topics could be very demotivating and tiring for higher level students which have a good and updated subject matter knowledge.
More close analysis on students' responses revealed that educational level affected students' ideas about course effectiveness. From those who commented on course effectiveness, 39 students were in their rst and second year. In this group, 64% said that the course is useful and effective and 36% believed that the course is ineffective. On the other hand, 42 respondents were in their fourth and fth year. From this group, 16% believed that the course was useful and 83% believed that the course was not effective. The dissatisfaction among students at higher levels was mostly due to the discrepancies between students' needs at this level and the course contents. Students' ideas were different about what they wanted to learn in ESP courses based on their educational level. Students at higher educational level ordered the skill they expected to be covered in ESP courses as: writing (41.5%), reading (20.7%), speaking (20.7%), and listening (5.6%) and 11% wanted technical vocabulary to be taught in the course. On the other hand, students at lower educational level mentioned the skills in the following order: reading (30%), speaking (26%), writing (18%) and listening (4%) and 22% agreed that technical vocabulary is required to be taught.
One practitioner believed that usually the course is taken in the last semester because both departments and students do not take it seriously. Observations in this study revealed that the majority of engineering departments had no regulation to determine the time for taking the course and students had a tendency to delay taking the course up to the last year of their undergraduate study. The ndings in this study supported the importance of the time students take the course. Former studies also a rmed that while students showed high level of ideal self and instrumentality promotion motivations in all educational level, they have stronger motivation for improving their ESP knowledge to meet their obligation and degree requirements in lower academic levels (Jafari Pazoki & Alemi, 2019).
Responses revealed that 84.5% of the participants con rm the effect of GE pro ciency on their learning. Students with lower GE pro ciency considered the course as less useful. One students explained how low pro ciency had affected her ESP learning: "I know a lot of technical terms, yet I don't understand technical texts. It shows that the problem is my GE pro ciency. If I had good general reading skill, I could read all type of texts. Sometimes my problem is that I don't understand the grammatical structure or I don't know a vocabulary. I wish we worked on GE skills and essential vocabularies or some key grammatical lessons which improve our comprehension of the texts" In some responses students questioned course usefulness for those who have low GE pro ciency. One student stated: "I think if I had a good pro ciency level in English, I would learn technical English very easily because technical English is only learning technical terms and I can learn it by myself." ESP professors also supported good GE knowledge for effective learning in the course. One professor stated that: "…in the engineering departments that the course is offered as optional, students with low English level never risk taking the course because they don't want to get a low mark in an elective course. On the other hand, when the course is required, students with low GE nd the course di cult and not useful." Nevertheless, students with good GE believed that the course did not offer them any new knowledge. Students with low English pro ciency refuse to take the course when it is elective and those with good GE knowledge take the course in their last semester because they considered it an easy and least demanding course which can earn them a good mark. Heterogeneous classes and students' different general English level, were the other frequent complains which made the course ineffective from students' perspective.
The problems that ESP practitioners face can be a key to understand some reasons behind the course ineffectiveness. Consequently, ESP practitioners were asked to comment on the challenges in ESP programs. Students' low language pro ciency, heterogeneous classes and limited time in a two unit course as well as low motivation of students were the most frequent responses to this question. One interviewee attributed this lack of motivation to the fact that students don't take non-specialized courses very seriously. Lack of motivation in ESP courses in engineering eld has been attributed to students' negative view toward the elements of their learning experience such as course contents, materials, teaching methods and also ignoring students' needs based on their long term goals (Jafari Pazoki & Alemi, 2019) ESP teachers also had low motivation. One participant explained the reason behind teachers' low motivation: "Teaching this course requires expertise in two elds: language knowledge and language teaching expertise and content knowledge, yet ESP teachers are paid less. Teaching ESP course is not valued and appreciated and teachers consider it as having a lower status. Therefore, experienced and highly skilled subject teachers or language specialists are reluctant to teach the course." Two practitioners stated that despite the importance of ESP knowledge for engineering students, the Ministry of Science and Technology announced the course as elective and this measure added to the problem. They believed that if the course were compulsory, both students and departments would take the course more seriously.

Conclusion
The results of a program evaluation should provide a clear idea about the current function of the program and it should also indicate how shortcomings can be modi ed (Kiely, 2009). In this study, the elements of satisfaction, course usefulness and relevance, interest and attitude toward teaching and learning practices and other factors that affect students' learning constituted a framework to evaluate ESP courses in engineering from the perspectives of the stakeholders.
In terms of satisfaction and usefulness, the majority of the participants expressed negative views because they considered ESP course ineffective for developing reading comprehension skill and enhancing technical vocabulary knowledge. Students' dissatisfaction mainly stemmed from inappropriate and old-fashioned teaching methods and the fact that students' wants to develop writing and speaking skills had been disregarded in the course objectives. Regarding the element of interest, inauthentic and boring course contents and materials and improper teaching approaches impeded students' interest in the course. Moreover, discrepancies between students wants and needs and actual course objectives and course contents jeopardize course relevance. In addition, students' dissatisfaction with the course outcome and their achievements in the four language skills revealed that their expectations were not ful lled.
The participants acknowledged that some problems including heterogeneous classes, students' low GE competency, inadequate instruction time, inappropriate teaching methods, improper course contents and materials, lack of well-designed and need-based syllabus, ministry decision for making the course elective and the students' and teacher's low motivation challenged course success. Based on the ndings, lack of needs analysis compromised course relevance. Therefore, conducting comprehensive needs analysis would help to eliminate mismatches between students' wants and needs and the present course objectives and contents and to resolve controversies among stakeholders about students' needs. Furthermore, the current syllabi for ESP courses which focuses on reading skill does not match students' expectations and their goals for taking the course especially for higher level students. Consequently, appropriate methods should be used for teaching reading comprehension and other skills should be included in the course based on students' needs to improve students' satisfaction and sense of achievement.
Teacher training courses and cooperation between applied linguists and subject specialists improve the quality of instruction and more units and time should be allocated to ESP instruction. It is advisable to offer another course with focus on productive skills late in undergraduate study because these skills are mostly demanded at graduate level. Furthermore, interesting, authentic and updated materials, interesting classroom procedures and authentic learning and teaching tasks can improve students' interest.
Inappropriate time for attending the course brought about stronger dissatisfaction. The best time to offer the course is the second year when students establish a required background knowledge to comprehend discipline-speci c texts in English. Since the focus of ESP course is on reading skill which is mostly required early at undergraduate level for reading English sources, postponing the course to last semester have negative effect on students' motivation and consequently course effectiveness. In order to alleviate the problems related to heterogeneous classes, general English courses should be offered to help students with low pro ciency improve their GE knowledge to an appropriate level as a prerequisite to enroll in ESP courses. Moreover, making the course mandatory will place higher value on the course from the students and departments' perspectives.