Number and Usage of Vector-Related WeBWorK OPL Items
When considering online platforms, it is often possible, and important, to consider outcomes assessment reports (VanDieren et al., 2020) and usage to explore more about how often tasks are assigned and student performance on the tasks. To consider the first research question, we now report on the usage statistics for the WeBWorK OPL vector-related items, which are displayed in Fig. 1. The mean global usage for the 72 items was 1061. However, there were 7 outlier items, which are described in Table 5. The global usage ranged from 1 to 9767 with a median of 231. Note that these represent lower bounds for usage in one year since only a subset of WeBWorK users report their statistics. The mean number of attempts for the 72 items was 3.13 ranging from 1.35 to 9.77 with a median of 2.65. The mean status for the 72 items was 94 ranging from 69 to 100 with a median of 97.
We now consider the outliers for the usage statistics as displayed in Table 5. Considering the global usage statistics, there seems to be high demand for certain problem types. When considering the seven outliers for global usage, four are of the graphical/geometrical interpretation item type. Of the global usage outliers, five involve dot product, four involve angle, and three involve direction. All four tasks that were outliers for attempts require multiple responses. It could be the case that students were checking on the correctness of their responses after different parts; however, recall that there were 32 multipart items.
Of the four attempts outliers, all were either graphical/geometrical interpretation or contextual situation items. Of these outliers, three involve vector angle and three involve direction. When considering the three outliers for status, all three were contextual situation items whose prompts involved verbal descriptions of vectors but whose responses required non-vector answers.
Three of the outliers in Table 5 were outliers for more than one statistic. Item 34 was a multipart item involving rowing from north to south across a river, and one of the parts asks about which bank is reached first (but the required response is the north bank, which seems incongruous with the given information). Item 47 was an outlier for all three statistics; it was one of the few items that involved projection; so, students may not have had the opportunity to practice routine problems involving projection before attempting this item. Also, the physical work in the response to Item 47 may be negative (since the coefficients are randomized), which might have confused students. Note that item 50 had four parts, one of which required seven responses. There could be syntax issues in the required responses for the students typing the pairs of vectors that were required for the responses to this item.
Table 5
Outliers for OPL Item Statistics (including Global Usage, Attempts, and Status)
Item
|
Outlier
|
Dim
|
Item Type
|
Cog Comp
|
Key Vector Ideas
|
Prompt
|
Response
|
8
|
Global usage (1623)
|
3D
|
Computation/ Calculation
|
Low
|
VM-D, VA-D
|
Basis vectors
|
Angle
|
10
|
Global usage (9377)
|
3D
|
Computation/ Calculation
|
Low
|
VD, VM
|
N-tuple vectors
|
N-tuple vectors
|
14
|
Global usage (8126)
|
3D
|
Graphical/ Geometrical interpretation
|
High
|
VO-D
|
N-tuple vectors
|
Scalars
|
19
|
Global usage (9767)
|
2D
|
Graphical/ Geometrical interpretation
|
High
|
VA-D
|
Graphed vectors
|
Words
|
30
|
Attempts (5.81)
|
2D
|
Graphical/ Geometrical interpretation
|
High
|
VD-S
|
Basis vectors
|
Scalars
|
34*
|
Attempts (6.61),
Status (75)
|
2D
|
Contextual
Situation (distance, velocity)
|
Very High
|
VM-A, VA
|
Verbal description of vectors
|
Angles and verbal descript. of vectors
|
35
|
Status (72)
|
3D
|
Contextual
Situation
(velocity)
|
Very High
|
VM-A
|
Verbal description of vectors
|
Scalars
|
47
|
Global usage (8039), Attempts (9.44),
Status (69)
|
2D
|
Contextual
Situation
(e.g., force, work)
|
High
|
VD-P, VA-P,S-P, D-P, D
|
Verbal description of vectors
|
Scalars and basis vectors
|
50
|
Global usage (7652), Attempts (9.77)
|
3D
|
Graphical/ Geometrical interpretation
|
Low
|
VA-D, VD-S
|
Basis vectors
|
Basis vectors
|
95
|
Global usage (9683)
|
2D
|
Graphical/ Geometrical interpretation
|
Low
|
VM, A, S-A, S-B,
|
Graphed vectors
|
Scalars and n-tuple vectors
|
Note
The * indicates a multipart item with one part that does not make sense in the given context.
To consider if items with multiple parts had different usage statistics than items with a single part, we display the box and whisker plot comparisons in Fig. 2. There was a difference in medians across all three usage statistics, and the interquartile range from Q1 to Q3 was disjoint for multipart and single-part items in attempts. The attempts may be higher for a multipart item since students may submit the item to check the first part before completing the remaining parts of the item. Therefore, a high value for attempts may not correspond to a difficult item if the item had several parts.
Nature of Vector-Related WeBWorK OPL Items
To address the multiple parts of the second research question, we now consider specific characteristics of the vector items in the OPL. For this, we report on five different item characteristics: dimensions of vectors in the item, type of item, cognitive capacity of item, key vector ideas addressed in the item, and representations of vectors used in the item.
We first consider the dimensions used in the vector items. Of the 112 parts of items, 44 were coded as 2D, 62 as 3D, and six as 4D or more. For the 76 items, we used the highest dimension in any part of the item for coding. There were 30 items coded as 2D, 44 as 3D, and two as 4D or more.
For the item types, we found that more computation/calculation items (43 of 76) were in the WeBWorK OPL than any other item type. Next most common (23 of 76) were the graphical/geometric interpretation items. Multivariable calculus often emphasizes analytic geometry; so, it might be expected that this would be a larger percentage of the OPL vector items. Most (7 of 10) of the contextual situation items were based in a physics context. The contextual situation items (10 of the 76) in the OPL had a mean global usage of 1174, while the mean global usage for all OPL tasks was 1061. This might suggest that the OPL would benefit from the addition of more as well as more diversified contextual situation vector items. No items were found to emphasize justification of reasoning using proofs or logical deduction over the other possible item type categories.
We compared the distribution of item types in the WeBWorK OPL to previous findings of student reports on the types of items instructors assign in single-variable calculus. Specifically, Ellis et al. (2015) reported on the means for two groups of students, the selected group involving almost 600 students at universities where they were more likely to retain “confidence, enjoyment, and interest in mathematics (p. 269)” compared to the non-selected group involving about 1410 students at universities that were less likely to retain those three. Another study by Burn and Mesa (2015) reported the median responses from a survey of calculus instructors about the item types that they include on assignments. These findings are included in Table 6.
The first noted similarity between these studies and ours is that all reported at least 40% of the tasks assigned were computation/calculation tasks. The second is that graphical/geometrical interpretation tasks ranged from 20–33% in all the studies. Differences between these studies and ours could be attributed to many reasons. First, our study looks at actual problems, the Ellis et al. (2015) study uses student reports, and the Burn and Mesa (2015) study uses instructor reports. Second, our study looks at multivariable calculus, while the Ellis et al. and the Burn and Mesa studies were for single-variable calculus. Three-dimensional graphical analysis should play an explicit role in multivariable calculus (Martínez-Planell et al., 2017; McGee & Moore-Russo, 2015; McGee et al., 2015); this might be why graphical/geometrical interpretation tasks seem to be assigned more in multivariable calculus compared to single-variable calculus, apart from Ellis and colleagues’ selected group. Another factor to consider is the limitation of the types of problems that can be programmed into WeBWorK. The essay prompt and draggable proofs are newer features in WeBWorK. Therefore, there may be fewer items in the OPL that require a proof or justification. However, instructors may augment existing OPL items by adding an essay prompt asking students to provide reasoning, proof, or justification (VanDieren, 2021a). These adaptations may be assigned by instructors but may not appear in the OPL. Additionally, it is quite possible that instructors assign tasks involving reasoning, proof, or justification outside of the WeBWorK platform.
We now look at item cognitive complexity. Of the 112 parts of items, about two-thirds (73) were coded as low cognitive complexity, and about one-fourth were coded as high complexity. These were followed by very high (7) then very low (1) cognitive complexity. When the 76 items were coded, we coded by the highest cognitive complexity noted in any part of the item. Still, the results were similar with about two-thirds (51) coded as low cognitive complexity, and one-fourth (19) coded as high complexity, followed by very high (5) then very low (1) cognitive complexity.
Table 6
Item Type
|
This
Study
(n = 76 items)
|
Ellis et al.’s
Selected Students
(n = ~ 590)
|
Ellis et al.’s
Non-Selected Students
(n = ~ 1410)
|
Burn & Mesa’s Instructors
(n ranged from 345 to 355)
|
Computation/Calculation
|
57%
|
40%
|
51%
|
50%
|
Graphical/Geometrical Interpretation
|
30%*
|
33%
|
21%
|
20%
|
Contextual Situation (collapsed the two categories)
|
13%
|
55%
|
39%
|
30%
|
Reasoning, Proof, Justification
|
0%
|
14%
|
9%
|
10%
|
Note
The * symbol represents that students were asked to provide explanations justifying their answers to earlier parts of an item for two of the 23 items deemed to be graphical/geometrical interpretation items. However, the emphasis was not on the justification but rather on the graphical/geometrical interpretation of the item.
In Table 7 below we compare our results with those of White and Mesa’s (2014) study of the tasks assigned by five, two-year college, Calculus I instructors. Multivariable students may not have enough experience working challenging vector tasks, as compared to Calculus I students. They may also not have much exposure to higher cognitive complexity tasks prior to exams. This suggests the need to develop more WeBWorK vector items that are of very high cognitive complexity.
Key Vector Ideas
The items addressed all key ideas for vectors that were identified in the coding, as displayed in Table 1. This seems to suggest that the key vector idea list is comprehensive. However, some key ideas were much more prominent in the items than others as shown by the differing bar heights in Fig. 3. For example, vector magnitude stood apart as the most prominent key idea. It was addressed in 30 of the items, while cross product, subtraction, projection, and orientation only appear in 12, 6, 3, and 2 items respectively.
Table 7
Cognitive Complexity Comparison to Previous Research
Cognitive Complexity
|
This Study
(n = 76 items)
|
White & Mesa’s
Assignment Tasks*
|
White & Mesa’s
Worksheet Tasks
|
White & Mesa’s
Exam Tasks
|
Simple Procedure
(Very Low and Low categories)
|
68%
|
54%
|
54%
|
40%
|
Complex Procedure
(High category)
|
25%
|
21%
|
10%
|
11%
|
Rich Task
(Very High category)
|
7%
|
25%
|
37%
|
49%
|
Note
The * symbol refers to tasks included WeBWorK as well as textbook tasks, which had similar percentage breakdowns when separated.
Computation/calculation was prevalent in most of the key vector ideas as displayed in Fig. 3. Only vector angle, projection and orientation had three or fewer computation/calculation items. Graphical/geometric interpretation was found in most of the key vector ideas. Only special vectors, subtraction, projection, and orientation had three or fewer graphical/geometric interpretation items. Addition, vector magnitude, and scalar multiplication had several items with contextual situations. However, the key vector ideas of cross product, subtraction, and orientation had zero items with contextual situations, and only one projection item had contextual situations. We now consider the vector key ideas (subtraction, projection, cross product, and orientation) with either no graphical geometric interpretation or no contextual situations items in light of previous research.
Shaffer and McDermott (2005) reported that postsecondary students in physics classes were better able to subtract vectors when there was no physical context than when one was present. Vector subtraction is related to vector addition, which occurred in 18 items. However, while 8 of the 18 items involving addition were contextual situations, there were no contextual situation items that involved subtraction. Additionally, 4 of the 18 addition items involved a graphical or geometric interpretation while only 2 of the subtraction items involved a graphical or geometric interpretation. Paired with research studies by Heckler and Scaife (2015) and Carleschi (2016) that recommend students practice graphical problems in a variety of vector orientations, this points to the need of additional WeBWorK vector tasks involving subtraction in both contextual and graphical/geometric situations.
There is sparse research on student understanding of vector projection; yet Zavala and Barniol (2013) found that students perform poorly in both contextual and non-contextual problems. The three projection items (i.e., items 15, 47, 96) in our sample are frequently assigned (global usage: 812, 8039, 675 compared to the median 231), have high numbers of attempts (3.49, 9.44, 3.39 compared to the median 2.65), and lower success rates (status: 95, 69, 88 compared to the median 97).
Research on student understanding of the cross product indicates that students have difficulty with orientation (i.e., applying the right-hand rule) and often fail to recognize that the cross product is a noncommutative operation (Deprez et al., 2019; Kustusch, 2016; Scaife & Heckler, 2010; VanDieren et al., 2017; Zavala & Barniol, 2010). Additionally, students not only struggle with cross product and orientation in graphical problems but also in physical contexts (Deprez et al., 2019). Furthermore, Deprez and colleagues report that the context can help students understand the geometric aspects of the cross product. With no items in our sample addressing cross product or orientation in a contextual situation, and with only one item addressing both cross product and orientation, there is a need to develop additional WeBWorK vector tasks to better address these student difficulties.
Of the 76 items, 25 involved only a single key vector idea; the remaining items involved two or more key ideas. Of the 25 items that involved only a single key idea, 15 of these were assigned a vector code, and the majority of these tasked a student to find the components for a vector connecting two given points. For items coded with more than one key idea, the key ideas may have been connected, or they may have been addressed by themselves in different parts of an item. Table 8 displays that total number of connections and the number of key ideas to which each idea was connected. Since the vector (V) code was used for simple identification of vectors in their component forms without emphasis on any of the critical features of vectors or vector operations, no item part was coded with both V and any other key vector idea code, and no items were coded connecting vector with any other key ideas. Therefore, the vector (V) code does not appear in Table 8.
Although a maximum of 10 connections to other key vector ideas were possible, the maximum number of connections for any key idea was 7. The top three key vector ideas in terms of connections included: scalar multiplication (34 total connections to 7 other key ideas), dot product (23 total connections to 7 other key ideas), and magnitude (24 total connections to 6 other key ideas). Note that scalar multiplication, special vectors, direction, projection, and orientation never appeared by themselves in an item. However, there were only two connections that involved orientation, one to dot product and one to cross product.
We now investigate the key ideas in terms of critical features of vectors and vector operations. There were only eight instances where critical features of vectors were connected to different critical features. This happened in the following four ways: magnitude-special vectors (3 instances), magnitude-direction (2 instances), special vectors-direction (2 instances), and angle-direction (1 instance). Variation theory points to the importance of not only presenting students opportunities to focus on critical features in isolation, but it also emphasizes that “understanding the object of learning implies understanding the object as a whole and thus involves a simultaneous discernment of the defining aspects and their relationship” (Kullberg et al., 2017, p. 560). The lack of connections between different critical features may not provide students with as many opportunities to distinguish and make comparisons that would allow them to develop a robust understanding of vectors. For example, one would expect some connections between vector orientation, angle, and direction.
Table 8
Connections between Key Vector Ideas
Key Vector Ideas
|
Total
Connections
|
Connected to ___ Other Key Ideas
|
Critical Features of Vectors
|
|
|
Magnitude (VM)
|
24
|
6
|
Angle (VA)
|
16
|
4
|
Special Vectors (VS)
|
14
|
5
|
Direction (VD)
|
13
|
6
|
Orientation (VO)
|
2
|
2
|
Vector Operations
|
|
|
Scalar Multiplication (S)
|
34
|
7
|
Dot Product (D)
|
23
|
7
|
Addition (A)
|
17
|
3
|
Cross Product (X)
|
11
|
5
|
Projection (P)
|
8
|
4
|
Subtraction (B)
|
5
|
1
|
There were 21 instances where vector operations were connected to different vector operations, as displayed in the chord diagram in Fig. 4. While there was some diversity of connections between vector operations, there are certain vector operations that are not well connected. The number and diversity of connections with scalar multiplication is not surprising since addition, subtraction, dot product, projection, and cross product can each be defined in terms of scalar multiplication. In fact, scalar multiplication was connected to all other vector operations except cross product. On the other hand, apart from scalar multiplication and dot product, none of the vector operations are well connected to other operations even though there are many possible connections. For example, there are several properties related to both dot product and cross product that involve scalar multiplication and vector addition (e.g., linearity). Vector subtraction can be interpreted as a combination of vector addition and scalar multiplication. Additionally, a vector can be presented as the sum of its orthogonal component and its projection onto another vector. Therefore, there are many possibilities for additional items to be added to the WeBWorK OPL that address such connections.
There were 52 instances where critical features of vectors were connected to vector operations, as displayed in the Sankey diagram in Fig. 5. Of note, there are only five (of six possible) vector operations displayed on the right side of Fig. 5 since there were no connections between any of the critical features of vectors and vector subtraction. As might be expected, the critical feature special vectors, which can be defined in terms of scalar multiplication and vector addition, was connected to both. Also, it is no surprise that vector orientation had few connections since it was in few items. The remaining three critical features of vectors were each connected to at least three vector operations; however, there were still obvious omissions (e.g., connecting vector magnitude and projection). Despite that omission, vector magnitude was much more emphasized in the WeBWorK OPL than vector direction. Considering student confusion of a vector with its scalar (Harel, 2000), it is important for instructors to not over emphasize the length of a vector while ignoring its other component, vector direction, when assigning OPL items to students.
Vector Representations
We now consider which representations of vectors are used in the items. We first consider the representations used in the item prompts and then what was expected in the items’ responses. Note that due to the search criteria for the data set, it was not surprising that all the items’ prompts involved vectors, and 50 of the 76 items involved vectors in their responses.
Of the 76 item prompts, 58 used a single representation for vectors; 18 used 2 representations; and none had more than 2 vector representations. “Although more representations do not necessarily lead to greater understanding, numerous cognitive advantages associated with the establishment of links among various ways of representing a problem have been proposed” (Stein et al., 1996, p. 472). The most common vector representations in item prompts with a single vector representation were basis vectors and endpoints, as displayed in Table 9. However, for items with two vector representations in their prompts, the majority involved a verbal description in the prompt. When looking across all item prompts, vectors were represented using endpoints in 23 items, as basis vectors in 21 items, using verbal descriptions in 19 items, and with n-tuples in 18 items. Table 9 shows the distribution of the item prompts.
Of the verbal descriptions, 9 were prompts describing a geometric situation; 7 described a physics concept (e.g., force, velocity, work); and 3 were contextual word problems (e.g., homework grades, production output of a factory). The vector representations that occurred least frequently in the prompts were column and graphed vectors. The lack of graphed representations is likely due to the increased difficulty of programming graphs in WeBWorK items compared to using the built-in Vector Class of Math Objects in WeBWorK which easily handles basis, n-tuple, and column vector representations (see https://webwork.maa.org/wiki/Vector_(MathObject_Class)). The lack of column vectors may be due to the fact that we considered only multivariable calculus items for this study, the column vector representation is used more widely in linear algebra and linear programming.
Table 9
Mathematical Object in Item Responses by Vector Representation Used for Item Prompt (n = 76)
Vector Representation(s) in Item Prompt
|
Mathematical Object(s) in Item Response
|
Vectors
only
|
Vectors and Scalars
|
Vectors and Angles
|
Scalars
only
|
Angles
only
|
Other
|
Single Vector Representation Used
|
|
|
|
|
|
|
Graphed (n = 5)
|
2
|
1
|
0
|
0
|
0
|
2
|
Basis (n = 15)
|
9
|
2
|
0
|
2
|
2
|
0
|
Column (n = 5)
|
2
|
0
|
0
|
3
|
0
|
0
|
N-tuple (n = 11)
|
4
|
2
|
0
|
3
|
1
|
1
|
Verbal description (n = 7)
|
2
|
0
|
1
|
3
|
1
|
0
|
Endpoints (n = 15)
|
9
|
2
|
0
|
3
|
1
|
0
|
Two Vector Representations Used
|
|
|
|
|
|
|
Basis & verbal description (n = 6)
|
3
|
1
|
0
|
1
|
1
|
0
|
N-tuple & endpoints (n = 5)
|
0
|
0
|
0
|
0
|
0
|
5
|
Verbal description & endpoints (n = 3)
|
1
|
0
|
0
|
1
|
0
|
1
|
Column & verbal description (n = 2)
|
0
|
2
|
0
|
0
|
0
|
0
|
Graphed & n-tuple (n = 1)
|
1
|
0
|
0
|
0
|
0
|
0
|
N-tuple & verbal description (n = 1)
|
1
|
0
|
0
|
0
|
0
|
0
|
Totals for all items
|
34
|
10
|
1
|
16
|
6
|
9
|
Note: The Other column included words (both explanations and short answers), equations, and single points (all cases of points involved endpoints of a vector) |
Table 10 displays the distribution of the 50 items for which students were expected to respond with a vector in a particular representation. Of the 50 items, 28 used the same vector representation in the item response as was used in the item prompts. The most common “switch” in vector representations was from endpoints in the prompt to an n-tuple in the response of an item, which occurred in 10 items. Of note, students were never prompted to submit a vector in graphical representation. This may be because graphical inputs are a relatively new feature of WeBWorK, which requires more complex coding than
Table 10
Vector Representation Responses by Vector Representation Used for Item Prompt (n = 50)
Vector Representation(s)
in Item Prompt
|
Vector Representation(s) in Item Response
|
Basis
(n = 14)
|
Column
(n = 4)
|
N-tuple
(n = 26)
|
Verbal description
(n = 1)
|
Endpts
(n = 5)
|
Single Vector Representation Used
|
|
|
|
|
|
Basis (n = 11)
|
6
|
0
|
5
|
0
|
0
|
Endpoints (n = 11)
|
1
|
0
|
10
|
0
|
0
|
N-tuple (n = 6)
|
0
|
0
|
6
|
0
|
0
|
Graphed (n = 3)
|
0
|
0
|
3
|
0
|
0
|
Verbal description (n = 3)
|
2
|
0
|
0
|
1
|
0
|
Column (n = 2)
|
0
|
2
|
0
|
0
|
0
|
Two Vector Representations Used
|
|
|
|
|
|
N-tuple & endpoints (n = 5)
|
0
|
0
|
0
|
0
|
5
|
Basis & verbal description (n = 4)
|
4
|
0
|
0
|
0
|
0
|
Column & verbal description (n = 2)
|
0
|
2
|
0
|
0
|
0
|
Verbal description & endpts (n = 1)
|
1
|
0
|
0
|
0
|
0
|
Graphed & n-tuple (n = 1)
|
0
|
0
|
1
|
0
|
0
|
N-tuple & verbal description (n = 1)
|
0
|
0
|
1
|
0
|
0
|
Note
Shading indicates the same vector representation for the item prompt and response.
inputting endpoints, a column vector, an n-tuple vector, or a basis vector as a response. In light of their findings, Heckler and Scaife (2015) recommend the need to have students practice with a variety of vector orientations. They emphasize the importance of graphical representation for sense making as well as the need for using both the basis and graphed formats of vectors.
Understanding any mathematical topic, including vectors, requires an awareness of the ways the topic is represented (Duval 1999) and some flexibility in how individuals are able to move between those representations (Ainsworth, 2006; McGee & Moore-Russo, 2015). “Although more representations do not necessarily lead to greater understanding, numerous cognitive advantages associated with the establishment of links among various ways of representing a problem have been proposed” (Stein et al., 1996, p. 472), it seems important that there are sufficient WeBWorK items that require students to work with multiple representations concurrently and to switch between vector representations.
Usage Statistics by Item Characteristic
We created box and whisker plots for global usage, attempts, and status of items by the following item classifications: item types, cognitive complexity, number of key vector ideas, and vector representation. Since no patterns emerged for global usage, we only report on the findings of attempts and status by characteristic displayed in Fig. 6.
In terms of item types, contextual situation items tended to require a greater number of attempts than other items. This was followed by graphical/geometrical interpretation items then computation/calculation items. The status, which is an indicator of success on the item, was lowest for contextual situation items followed by graphical/geometrical interpretation items then computation/calculation items. This suggests that contextual situation items were the most difficult items, and computation/calculation items were the easiest. The difficulty of the contextual situation items might have been due to 7 of those 10 items requiring physics knowledge, as well as vector knowledge.
Since there was only one item that was coded as having very low cognitive complexity, a memorization item related to the general vector coding category, Fig. 6 displays only the low, high, and very high coding categories, with that one very low item included with the low category. As expected, the higher cognitive complexity items had more attempts and lower status.
As one might expect, the items with more key vector ideas (including both critical features of vectors and vector operations) had more attempts and lower status indicating these were more difficult for students. However, this was not as pronounced as was the case for item cognitive complexity.
The items with only verbal descriptions required more attempts and had lower status than items with other vector representations, including those having multiple representations. This suggests verbal descriptions are more difficult for students.
Item Cognitive Complexity by Key Vector Ideas
To address the third research question, we now consider if certain key vector ideas appeared more frequently in items with higher cognitive complexity. Since there was only one item that was coded as having very low cognitive complexity, a memorization item related to the general vector coding category, Table 11 displays only the counts for the low, high, and very high coding categories, with that one very low item included with the low category. For the general vector category, due to the coding definition, it was not surprising that most of the items were deemed as being of low cognitive demand (1 very low, 17 low). Recall that of the 76 items, 25 involved only a single key vector idea, and the remaining items involved two or more key ideas. Therefore, there are more than 76 entries in Table 11.
Table 11
Cognitive Complexity of Items by Key Vector Ideas
Key Ideas
|
Number of Items in Which Key Idea is Found
|
Cognitive Complexity Level of Items
|
Low
|
High
|
Very High
|
Vector (V)
|
18
|
18*
|
0
|
0
|
|
Critical Features of Vectors
|
|
|
|
|
|
Magnitude (VM)
|
30
|
12
|
14
|
4
|
|
Angle (VA)
|
15
|
6
|
6
|
3
|
|
Direction (VD)
|
14
|
5
|
7
|
2
|
|
Special Vectors (VS)
|
11
|
5
|
5
|
1
|
|
Orientation (VO)
|
2
|
0
|
2
|
0
|
|
Vector Operations
|
|
|
|
|
|
Scalar Multiplication (S)
|
26
|
20
|
5
|
1
|
|
Dot Product (D)
|
22
|
12
|
7
|
3
|
|
Addition (A)
|
18
|
10
|
5
|
3
|
|
Cross Product (X)
|
12
|
6
|
5
|
1
|
|
Subtraction (B)
|
6
|
6
|
0
|
0
|
|
Projection (P)
|
3
|
2
|
1
|
0
|
|
Note: The * signifies that the vector key idea was coded as low cognitive complexity in 17 items and very low in 1 item.
As is noted in the ternary plot in Fig. 7, one thing stands out. None of the key vector ideas were found in more than 20% of the items, and only one key idea was in more than 15% of the items categorized as very high. White and Mesa’s (2014, p. 686) analysis of the distribution of levels of cognitive complexity tasks assigned by instructors indicates that the very high cognitive complexity was noted in much greater percentages on the tasks assigned by instructors, ranging from 15–31% for bookwork, from 23–42% for worksheets, and from 26–68% for exams. This indicates a potential gap between the supply of very high items in the WeBWorK sample problem bank and the demand from instructors for all key vector ideas.
We first consider items that address critical features of vectors (i.e., all those in black except the asterisk) as graphed in Fig. 7. Except for vector orientation, there were items in our sample across all three cognitive complexity levels for the other four critical features (i.e., magnitude, angle, direction, and special vectors). However, even for these four, there were consistently more items that were of either low or high cognitive demand than there were items of very high cognitive complexity. This trend was even more pronounced when looking at the items that involved vector operations (i.e., scalar multiplication, dot product, addition, cross product, subtraction, and projection), which are denoted in Fig. 7 with grey symbols. The vector operations were noted predominantly in items that were of low cognitive complexity. There were fewer vector operations in items with high cognitive complexity, and, apart from the dot product, almost no vector operations in items of very high cognitive complexity. As might be expected, the key ideas addressed by fewer items (i.e., vector orientation, subtraction, and projection), tended to have the least spread across complexity levels. This points to a need for more items to be developed at all levels, especially at the very high cognitive complexity level.
Orientation is a critical feature of vectors that challenges students. With only two items addressing vector orientation, both of high cognitive complexity, there are no options in this sample of WeBWorK items for instructors to assign items with low or very high cognitive complexity. Despite research pointing to student difficulty in understanding vector orientation and the right-hand rule (Kustusch, 2016; Scaife & Heckler, 2010; VanDieren et al., 2017; Zavala & Barniol, 2010), there are limited opportunities within WeBWorK for students to become “acquainted” with the critical feature of orientation before going onto more complex tasks of “contrasting,” “generalizing,” and “fusing” this feature with others (Kullberg et al., 2017, p. 560). Therefore, not only is there a need for more WeBWorK items addressing vector orientation, but new items should include those of both lower and higher cognitive complexity so that students are better equipped to discern and understand this critical feature. One explanation for the lack of items concerning vector orientation is that this critical feature relies heavily on graphical representations, and there are barriers (although not insurmountable) within the WeBWorK coding environment to create exercises with graphical prompts and/or responses.
Vector Representations by Key Vector Ideas
To address the fourth research question, we now consider if key vector ideas were addressed in items with a variety of vector representations. Figure 8 below displays that there was some variety of vector representations used in item prompts for all key vector ideas. It appears in Fig. 8 that item prompts for vector items are represented as only graphed, endpoints, or multiple representations. However, for the 7 items whose prompts involved multiple representations, 5 were a combination of n-tuple and endpoints while 2 were column vector and verbal description combinations. Note that there are no instances of prompts for items coded with both the general vector coding category and basis vector representation, since this combination would be given a special vectors assignment as defined in the coding.
As displayed in Table 9 earlier, there are few item prompts (n = 5 as a single representation, n = 1 as part of a multiple representation) in the sample that used a graphed vector representation. Moreover, we note from Fig. 8 that some key ideas have no prompts with graphed representations. None of the prompts for items coded as addressing vector direction, orientation, projection, or cross product involved graphed vector representations (either as a single representation or paired with another vector representation). Each of these key vector ideas have graphical aspects that are not being addressed in the sample of WeBWorK items in this study.
The other vector representation that occurred with less frequency in item prompts is column vector. Column vectors were used as a single representation in prompts for items involving scalar multiplication, addition, and subtraction only. Column vectors were paired with other vector representations in item prompts coded with the general vector coding category but not in items involving the other eight key vector ideas.
Finally, we noted earlier that there are differences in the number of items that address vector addition and subtraction and in the cognitive complexity of items addressing these key ideas. Furthermore, addition also had a wider variety of vector representations and more occurrences of multiple representations used in their prompts compared to subtraction.