This post is a direct dump of the research that I undertook as part of my final semester in my teaching degree. The subject centred around mathematical and mathematical educational values, which are proposed by Alan Bishop as having a pivotal role in determining one’s intrinsic mathematical predispositions and motivations, and thus mathematical proficiency. My paper looked into the presence of these mathematical values in the NAPLAN high-stakes test. It is my view that our high-stakes tests should reflect the kind of mathematical proficiency that we expect people to practice and develop.
1. Introduction to the Study
Since 2008, students in Australian schools have been subjected to the nationally administered National Assessment Program – Literacy and Numeracy (NAPLAN) assessment. The assessment is regulated by the Australian Curriculum, Assessment and Reporting Authority (ACARA) and the numeracy component reflects the proficiency strands outlined in the mathematics curriculum (ACARA, 2016). The proficiency strands are considered a statement of mathematical values that are being emphasised in the curriculum (Seah, Andersson, Bishop, & Clarkson, 2016). Mathematical values are the deep affective qualities which education fosters through the school subject of mathematics (Bishop, 1996). Since mathematical values are implied within the proficiency strands, and the NAPLAN numeracy assessment is based on the proficiency strands, one would expect to see values promoted throughout the assessment itself. But which values?
Due to the researcher’s interest in authentic assessment and frustration with high-stakes tests, the NAPLAN assessment was selected to investigate just what mathematical values are promoted in this high-stakes test. It is this researcher’s persuasion that if our high-stakes tests do not reflect our society’s values, then they are not measuring what they should, in fact, be measuring and thus are somewhat defective. The researcher’s frustrations with high-stakes testing is not specifically directed towards NAPLAN and the general opprobrium it has received in recent years, but rather, as a way to make transparent the biases possessed while undertaking this research. To take the first step towards understanding, a content analysis will be performed on Year 5 Numeracy NAPLAN assessments from years 2008-2011. Each paper will yield a value score that reveals the mathematical values that they promote. The values of control and objectism were the most prominent throughout the assessment papers along with, to a lesser extent, rationalism. Each content strand is also evaluated for the values they most promote. It is hoped that this research can be a stepping stone, or perhaps pebble, towards future research that can influence reform in our methods of high-stakes testing.
2. Literature Review
2.1 Definition
The common definition of values throughout the literature refers to values in mathematics education as the deep affective qualities which education fosters through the school subject of mathematics (Bishop, 1996). Pertinent to this article are the values of mathematics and, to a lesser extent, the values of mathematics education (Bishop et al., 1999). Bishop (1988) identifies three sets of corresponding pairs of mathematical values, namely, rationalism and objectism; control and progress; openness and mystery. Any mention of values throughout this article relates directly to these corresponding pairs. Furthermore, the educational scope of this definition applies to curricula, specifically ACARA, and the mathematical values that ACARA promote.
2.2 Values in Curricula
The direction of this review now shifts its focus towards the importance of values in curricula. Tomlinson and Quinton (1986) contend that, when considering values, due attention should be paid to three elements: aims or intended outcomes; means or teaching, learning, processes; and effects or actual problems. This triad of elements maps neatly to the intended, implemented and attained curriculum. The complementary nature of institutional values shaping classroom mathematical values and vice-versa, support the call for focus on the triad of elements (Bishop, FitzSimons, Seah, & Clarkson, 1999). However, for the purposes of this article, we will narrow the focus to the intended and attained curriculum. In previous analyses of curriculum documents, it was found that values were only implied in curriculum material and were far from being a central guiding principle (Seah et al., 2016). Moreover, in Clarkson and Bishop’s (2000) examination of the 1990s curriculum, markers for both mathematics and mathematics educational values were found with some emphasis placed upon objectism, rationalism and openness. Seah et al. (2016) identify that the preference given to these values is still clearly evident in current curriculum documents. It is important to note that these values pairs are complementary with neither value in any pair being more significant than the other (Bishop et al., 1999). For a more empowering and relevant approach to curriculum reform to take place, Seah et al. (2016) propose the guiding of curriculum writers’ valuing from whatever sorts of beliefs, attitude, and interests they possess in order to better enable a more positive and productive view of mathematics learning.
2.3 NAPLAN Purpose
It is perhaps a sensible time to segue to the National Assessment Program (NAP) and its assessment: NAPLAN. The notoriety through which NAPLAN has received much publicity is not the focal point of this section. The aim is to position the purpose of the assessment and high-stakes tests in general. The NAP defines NAPLAN as a ‘measure through which governments, education authorities, schools and the community can determine whether or not young Australians are meeting important educational outcomes’ (ACARA, 2016). It claims to have two main benefits: driving improvements in student outcomes; and increased accountability. The numeracy element of the assessment is based on the proficiency strands of understanding, fluency, problem-solving and reasoning across the three content strands of mathematics (Kilpatrick, Swafford, & Findell, 2001). Due to the publishing of results on the MySchools website and the associated media coverage, NAPLAN can be regarded as a high-stakes test (Polesel, Dulfer, & Turnbull, 2012). Two criticisms of high-stakes tests include the encouragement of low-level thinking while promoting outcome measure rather than the intrinsic process of learning and acquiring knowledge; and focussing teachers’ efforts on areas in which the students will be tested of which the consequence is the isolation of largely unconnected facts and pieces of information (Polesel et al., 2012). Conversely, Perso (2011) acknowledges that it is impossible to authentically assess student numeracy on a national scale, and that the test itself does, to some extent, attempt to assess and promote numeracy.
2.4 Research Problem
At this juncture, this article aims to amalgamate the literature on mathematical values with the absence of research on illuminating mathematical values in high-stakes testing, chiefly, NAPLAN. The literature alludes to the presence of values in past curricula, and since NAPLAN is based on the Australian Curriculum, one would expect to see values present in NAPLAN. This amalgamation provides us with a premise for research, which leads to the following research question:
To what extent are mathematical values present in NAPLAN Year 5 Numeracy from 2008-2011?
3. Research Methodology
This study uses a descriptive and interpretive research methodology. Content analysis is the primary research method and deductive analysis is employed for the coding process (Bowen, 2009; O’Toole & Beckett, 2010). NAPLAN Numeracy papers are sourced for this study are from the Year 5 cohort in years 2008-2011. These papers were sampled for convenience since they were the only papers available to the researcher. The Year 5 cohort was selected to limit the scope of the study. For each paper, each assessment item will be assigned a content strand and associated with one or more of the mathematical values mentioned in section 2.1. Following the collection of data, a frequency count will be generated and converted into a scaled score for that test. The mathematical value scaled score will be compared from year-to-year and through each content strand, namely, Number & Algebra (NA), Measurement & Geometry (MG), and Statistics & Probability (SP).
| Table 3.1. Mapping of proficiency strands to mathematical values | |
| Proficiency Strand | Mathematical Value |
| Fluency | Control |
| Understanding | Objectism, Progress |
| Reasoning | Openness, Progress, Rationalism |
| Problem Solving | Mystery |
This researcher is aware of the subjectivity involved in associating values with assessment items. In order to increase consistency of judgment, a flowchart, which can be found in the Appendix, was designed and tested to standardise judgment prior to analysis. The bias and subjectivity is arrested following the creation of the flowchart thus allowing for consistent comparison between the years. In section 2.4, it was established that NAPLAN assesses the proficiency strands. The researcher saw this as an entry point into making values visible in each assessment as, evidently, there is some relationship and overlap between mathematical values and the proficiency strands. The researcher read through definitions of the proficiency strands and mathematical values and matched keywords or phrases that supported a connection between them (Bishop, 2001; Kilpatrick, Swafford, & Findell, 2001). Some relationships, such as fluency and control, are more transparent than others such as understanding and objectism.
When devising the flowchart, the researcher noticed that the mathematical value of mystery appears to have no clear entry point through the proficiency strands, and could itself exist in isolation. The researcher chose to use problem solving as an entry point to mystery by injecting attributes of the mystery value into the problem solving strand of proficiency. Problem solving also serves as a flexible entry point into other mathematical values.
It must be highlighted that the researcher is aware that this suggested relationship between the proficiency strands and mathematical values is not rigid in structure. The researcher also acknowledges that the use of the flowchart runs the risk of oversimplifying the relationship. Thus, to reiterate, the purpose of the flowchart is to make consistent the survey of mathematical values between each assessment paper.
The choice of a single secondary data source was due to the constraints placed upon the researcher. Namely, the researcher could neither gain ethical clearance in time to liaise with stakeholders nor could the researcher employ assistants to comb the data using the flowchart to reduce subjectivity. This study should be viewed as a preliminary survey into the presence of mathematical values in Year 5 NAPLAN Numeracy from which a more robust and triangulated approach could be employed.
4. Results
Figure 4.1. Percentages of values promoted across all questions within a single paper from years 2008-2011.
Each NAPLAN assessment contained a total of 40 questions that are sourced from each of the content strands NA, MG, and SP. A statistical composition of these strands is presented later in this section. Following initial nominal coding, each question of each Year 5 NAPLAN assessment was assigned one or more mathematical values that it appeared to promote. The values of control and objectism were the two most prominent values respectively. Figure 4.1 illustrates that the value of control was most promoted throughout the assessments. The value of control was detected throughout questions on all assessments as much as 80% to 90% of the time. The value of objectism was the only other value to appear more than 50% of the time throughout questions on all assessments.
The next two values of rationalism and progress appeared sporadically throughout questions on all assessments. On average, rationalism was promoted in just under one-third of questions on all assessments. The values of openness and mystery are the least represented in the results. In some years, no value of openness or mystery is promoted, and for all years, its promotion in the assessments falls under 10% of all questions. That translates to roughly one to two questions each per test where these values are promoted.
| Table 4.1. Percentage of mathematical values promoted in total number of questions (n) of each content strand in years 2008-2011. | ||||||
| Content Strand | Control | Objectism | Progress | Openness | Rationalism | Mystery |
| Number & Algebra (n = 68) | 97% | 44% | 19% | 3% | 26% | 3% |
| Statistics & Probability (n = 21) | 81% | 52% | 14% | 0% | 76% | 0% |
| Measurement & Geometry (n = 71) | 83% | 69% | 15% | 4% | 18% | 6% |
As mentioned previously, statistical composition of the content strands would be presented. Of all questions on all assessments, approximately 45% of questions came from the MG strand while just over 40% of questions originated from the NA strand with around just 13% of questions coming from SP. The value of control is the most promoted in each of the three content strands. It is in the NA strand, however, where control is promoted in almost every question at a staggering 97%. Perhaps the anomalous result is the 67% of SP questions that contain the value rationalism. SP is also the only content strand that does not at all appear to promote the values of openness and mystery through the assessment. Despite this peculiar percentage, data from Table 4.1 is mostly consistent with data from Figure 4.1.
| Table 4.2. Average number of values present per question in a subject strand in years 2008-2011. | |
| Content Strand | Average Number of Values per Question |
| Number & Algebra (n = 68) | 1.89 |
| Statistics & Probability (n = 21) | 2.25 |
| Measurement & Geometry (n = 71) | 1.98 |
Each question in every assessment was assigned a value score. The value score is the sum of all promoted values within that question. The minimum score could be zero with a possible maximum score of six. For any one question, a score of four was the largest detected. Out of 160 questions, this four-value-promoted question occurred just eight times. The average value score per question for each content strand is represented in Table 4.2. On average, questions belonging to the SP strand promote the greatest number of values in a single question. Finally, the sum of each question’s value score within an assessment year has been represented in Table 4.3. This table shows an increase of around 11% in value score from 2008 to 2011.
| Table 4.3. Value scores for each assessment year. | ||
| Year | Value score (Max 240) | Value score Percentage |
| 2008 | 67 | 27.92% |
| 2009 | 73 | 30.42% |
| 2010 | 83 | 34.58% |
| 2011 | 94 | 39.17% |
5. Discussion
The premise for this research asked to what extent are mathematical values present in NAPLAN, and more specifically, in the Year 5 numeracy assessments. It was established in the literature review that values of objectism, rationalism and openness were evident within the content descriptors of the three mathematical strands (Clarkson, 2000; Seah et al., 2016). We investigated the curriculum since NAPLAN bases its assessment on the four mathematical proficiency strands of fluency, understanding, reasoning and problem solving, which supposedly underpins the curriculum. Seah et al. (2016) write that these proficiency strands can be considered a statement on the mathematical values being emphasised. The proficiency strands thus became a lens through which to view values in the assessments. The findings of this research appear to corroborate the emphasis on objectism and to a lesser extent rationalism. Perhaps most surprising is the near absence of openness and almost complete emphasis on control.
5.1 Openness
Throughout the data collection process, the researcher experienced difficulty in satisfying the conditions for the value of openness. To establish a clear idea of the value let us revisit its definition:
Openness – I call another familiar value ‘openness’ because mathematicians believe in the public verification of their ideas by proofs and demonstrations. Asking students to explain their ideas to the whole class is good practice for developing the openness value. (Bishop, Clarkson, FitzSimons, & Seah, 2000, p. 150).
In order to assess openness in the NAPLAN assessment, the researcher appraised the assessment item against the question: Does the question ask the student to explain or justify their strategy or result?
So perhaps, the absence of openness in the assessments is more a reflection of the limitation of the platform through which the assessment is conducted. The students are not provided with an opportunity to publicly verify their results through a spoken or written means. Sure, the results are included as part of a school-wide result, but the interrogation of the assessment item is lost. Subsequently, the principal test format of multiple choice items lend themselves congenially towards the value of control whose presence is emphatically represented in Figure 4.1.
5.2 Content Strands
Following the collection of data, another question begged to be asked in relation to the original: is there a relationship between particular values and the content strands? The results of values promoted in the mathematical content strands of NA, MG, and SP mostly reflect the overall results. We can, however, inspect the results closer to find that, within NA, almost every single question (97%) promoted the value of control. While control is still the most prominent value in each of the other two content strands (MG & SP), its promotion was not as high as NA. Perhaps a reason for this is the calculative nature of NA questions. A study that compared mathematical textbooks for the presence of mathematical values from two Australiasian regions found that the value of control to be prominent throughout its content (Seah & Bishop, 2000). Could there be a relationship between the two formats that inherently promotes control?
Equally important was the prominence of rationalism promoted throughout SP questions. While the difference between the content strands for control was around 14%, the difference for the promotion of rationalism between SP and the two others of NA and MG was over 50%. Moreover, the content strand of SP appeared to, on average, contain more values per question than the other two content strands of NA and MG. Based on these results, it appears that questions from the SP content strand are the most rich in values. Yet, SP questions make up as little as 13% of total questions on all the assessments surveyed. So what do SP questions look like?
5.3 Most Valuable Questions
Figure 5.1. (Ministerial Council on Education Employment Training and Youth Affairs, 2009). A question from the Statistics & Probability content strand that promotes multiple values. Retrieved from http://www.nap.edu.au
Figure 5.2. (Ministerial Council on Education Employment Training and Youth Affairs, 2009). A question from the Statistics & Probability content strand that promotes multiple values. Retrieved from http://www.nap.edu.au
Let us now examine two questions from the SP content strand that are represented in Figure 5.1 and Figure 5.2. Initial appraisal of the questions assigned multiple values to each. Both questions ask the student to produce a number as the result, which appears calculative. However, both questions ask the student to make comparisons between different data values. In order to process these tasks, the student must perform several steps to reach the numerical answer. It is this attribute of comparison that breeds value rich questions. For both examples above, the student could generalise his or her solution to verify the answer, which would promote progress.
Figure 5.3. (Ministerial Council on Education Employment Training and Youth Affairs, 2009). A question from the Statistics & Probability content strand that promotes rationalism. Retrieved from http://www.nap.edu.au
Similarly, the question in Figure 5.3, which promotes rationalism among other values, asks students to make the same comparison between items, but in this case it is the solution that is interrogated. It is for this reason that this question promotes rationalism. The question is asking the students to prove which graph matches the data set. In doing so, they must cross-examine each graph. Let us reiterate that the most value rich questions demanded that students make comparisons between elements contained within the question. These observations lead to further questions for future consideration. Why are these value rich questions underrepresented throughout the assessments? Should questions, such as those shown above, have a greater presence in these assessments? What effect would that have on student performance, motivation and even fatigue?
5.4 Limitations
An apparent trend in the results suggests that values promotion is steadily increasing each year. It would be tempting to inspect this trend further, but the researcher is cautious to make any such claim as the result could come down simply to researcher bias. There is limited evidence to suggest such a relationship, but perhaps a survey from another cohort from the same years could assist in either supporting or refuting the trend. Moreover, the definitional understanding of each value influences value selection throughout the data collection process. In order to reduce this bias, the flowchart used to collect data should be interrogated by other researchers in order to standardise inferences. For instance, do the results reflect the ease of identifying control, objectism and rationalism through the use of the flowchart over the other three values? Such an interrogation is welcomed and a copy of the flowchart, as mentioned previously, can be found as an Appendix.
6. Conclusion
In summary, this research has taken a preliminary step into the investigation of the presence of mathematical values in the NAPLAN Year 5 assessments. It is not intended to add to the vitriol and opprobrium that has plagued NAPLAN over the years. It has merely tried to identify what it values in terms of numeracy. The values of control, objectism and, to a lesser extent, rationalism were three values supported the most in these assessments. The content strand of SP was identified as containing the most value rich questions. In other words, questions that contained the promotion of several values within a single assessment item.
This has been the first survey of its kind, and more surveys like it are welcomed and encouraged. Specifically, defining a standardised way to identify values within questions would considerably enhance the reliability of the survey. Secondly, it would be useful to survey the NAPLAN assessments from different cohorts within the same year. This would enable a comparison of value promotion across the different stages of schooling. Lastly, it would be of interest to define what assessment items should look like that promote each mathematical value given that each value has no higher emphasis placed upon it over another.
High-stakes tests, such as NAPLAN, should ultimately reflect what we value as a society. If mathematical researchers deem that six values be treated of equal importance, then this should be reflected in the intended, implemented and enacted curriculum. Teachers will teach to the test. Teachers will refer to the curriculum for guidance. If our government documents have some invisible values and our accountability instruments are not measuring what they are supposed to, then how can we expect our students to possess all that is required to become a numerate citizen who internally appreciates the discipline of mathematics.
7. References
Australian Curriculum Assessment and Reporting Authority (ACARA). (2016). National Assessment Program (NAP). Retrieved from http://www.nap.edu.au/
Bishop, A. (1996). How should mathematics teaching in modern societies relate to cultural values—some preliminary questions. Paper presented at the seventh Southeast Asian conference on mathematics education, Hanoi, Vietnam.
Bishop, A., Clarkson, P., FitzSimons, G., & Seah, W. (2000). Why study values in mathematics teaching: Contextualising the VAMP project. Retrieved January, 24, 2004.
Bishop, A., FitzSimons, G., Seah, W. T., & Clarkson, P. (1999). Values in Mathematics Education: Making Values Teaching Explicit in the Mathematics Classroom.
Clarkson, P., Bishop, A. (2000). Values and mathematics. In Ahmed, A., Kraemer, J. & Williams, K (Eds.) Cultural Diversity in Mathematics Education (pp. 239-244). Chichester, UK: Horward Publishing.
Kilpatrick, J., Swafford, J., & Findell, B. (2001). The strands of mathematical proficiency. Adding it up: Helping children learn mathematics, 115-118.
Ministerial Council on Education Employment Training and Youth Affairs (MCEETYA). (2009). National Assessment Program — Literacy and Numeracy: Year 5 Numeracy. Curriculum Corporation.
Perso, T. (2011). Assessing Numeracy and NAPLAN. Australian Mathematics Teacher, 67(4), 32-35.
Polesel, J., Dulfer, N., & Turnbull, M. (2012). The experience of education: The impacts of high stakes testing on school students and their families. Literature Review prepared for the Whitlam Institute, Melbourne Graduate School of Education, and the Foundation for Young Australians.
Seah, W. T., Andersson, A., Bishop, A., & Clarkson, P. (2016). What would the mathematics curriculum look like if values were the focus? For the Learning of Mathematics, 36, 1.
Seah, W. T., & Bishop, A. J. (2000). Values in mathematics textbooks: A view through two Australasian regions.
Tomlinson, P., & Quinton, M. (1986). Values across the curriculum: Taylor & Francis.
Appendix – Values Flowchart Assistant
