What will happen to teachers’ percentage estimations over the summer of 2020?
Some of you may have read what I have written about calculated grades in the past. If you would like to read more about the process, you might like to read other blog posts first. (links at the end)
Teacher estimations have been submitted by around 19th June 2020 (today) so over the next two months, the Calculated Grades Executive Office (CGEO) (with the help of many staff from the State Examinations Commission) within the Department of Education will be analysing the teacher’s estimations.
I imagine many parents and even some teachers, will be scratching their heads at this point. Surely the teacher’s percentage estimation is the grade the student is going to get? Surprise, surprise. This is not the case. Although there has been a lot of hype in the media about predicted grades and teacher’s estimations, most commentators have overlooked the fact that the teacher’s estimation is merely the starting point in a significant process of grade adjustment. If the teacher estimations were simply going to be awarded, they could probably award the grades next week.
Some grades will move up, some will move down, some will stay rather similar but my guess is that virtually none of the percentages that pupils will be awarded will match the teacher’s estimations.
In the Department of Education document on calculated grades (link below) the process of standardisation is half a page of a fifty-one page document (top half of p. 32). You would be forgiven for overlooking it because it feels almost like a footnote. It states that “teacher judgements…need to be examined and adjusted at a national level to ensure comparability across different schools and that a common national standard is applied.”
Therefore, the teacher judgement is going to be combined with some “historical data” to bring about a standard for 2020. There is no mention of exactly what this historical data is or how far back they are going to look to get it, but it is safe to say, that the CGEO is going to use everything they have to come up with an algorithm that produces the results they desire. Parents might not realise it but every piece of relevant data that is held is going to be put to use to ‘calculate’ the grade of the student in that subject. The teacher’s estimation is merely the starting point.
What the standardisation process 2020 might look like
I imagine that the CGEO is going to start looking at the percentage estimations that are coming in. This will be in a similar way to the way that the individual chief examiners do just after marking starts on ‘real’ exam papers. After a sample of papers are marked, the marking scheme in every exam paper is adjusted to get a distribution of marks that the SEC wants, namely the bell curve. The bell curve assumes that the bulk of students (and therefore) marks are around the middle. This process is called standardisation.
This year everything is being done backwards. Because there is no ‘paper’ or evidence for the assessment, the examiners in each subject area will most likely start looking at the percentage estimates coming in from teachers and will retrospectively change them to suit the bell curve they normally use in that subject. If the subject is normally taken by large or small numbers of students, they will have a different bell curve to apply. Normally, there is an exam paper upon which to base this but obviously this year is different because the starting point has no real basis.
These adjustments to the teacher estimations will continue within subject groups. I teach German at Leaving Certificate level so my top students will be competing in the same way against their peers to be towards of the top of the curve; the 6/7% of students each year who achieve a H1 grade. Everyone is competing to be within the group of students who achieves each individual grade. Parents may not know it, but the grades are predetermined; they simply need to be distributed as fairly and equitably as possible.
They GCEO will also be using other statistical information about students and school’s past performance to adjust (or as they say align) the grades. This is the “historical data” to which I referred earlier. This is the most opaque part of this process because no details about it have been released. Soon parents will become aware of a myriad of statistical information about schools and individual student’s attainment that the state keeps but about which they are in the dark. This data is going to be put to use this summer.
Individual student data
Remember that the SEC sets and marks every state exam in the country. It knows the exact results of every student in both Leaving and Junior Certificate exams going back decades. It knows the exact result of every student, in every school, in every subject every year. It knows and holds for example, the exact marks (more specific than percentages or bands of grades) individual students obtained at Junior Certificate level. I am fairly certain that the GCEO will be using this information this year to feed into the algorithm for grade adjustment.
The SEC can compare Junior Certificate and Leaving Certificate outcomes over decades to come up with a correlation or algorithm that takes account of the ‘usual’ or ‘normal’ progress of students from Junior to Leaving Certificate. This algorithm could therefore, make predictions about how individual students would have got on in each subject at Leaving Certificate 2020, had they had the chance to sit it. It will probably combine with the teacher’s estimated grade to form the calculated grade.
This is obviously questionable. Students when they were sitting their Junior Certificate two or three years ago, had no idea that how they got on might be used to adjust their score for a much higher stakes assessment. Although I imagine that the correlation to be made between Junior Certificate and Leaving Certificate performance in subjects might be contextualised in some way, my experience as a teacher makes me think of many teenage boys, who have little direction at the age of fifteen but much more nearer eighteen. Using this type of information if anything, hinders the outcomes of individual students who were about to defy the odds and we know that some students do that every single year.
Whole school data
The SEC also knows how students each year in each school fare in each subject. This means that they know how a school’s context and intake has a big impact on their outcomes in terms of grades. In other words, the SEC knows that students in DEIS schools are less likely to achieve higher grades, though obviously some always do. This is known as school profiling (link below) and as far as I understand, the CGEO be using a school’s normal expected achievement (based on the past three years) in Leaving Certificate examinations to make predictions about how students in that school usually fare in the exam. This information will also feed into the algorithm.
Although I object to school profiling in principle (see link below), in the context of calculated grades, I believe it is vitally important that this information is used. If the CGEO did not use this information to contextualise exam results, disadvantaged students would be even worse off than they are in the current calculated grades scenario.
Arrival at the ‘calculated grade’
The calculated grade will be ‘calculated’ by combining the teacher’s estimation together with the data I have mentioned above. There could also be other data of which I am unaware that may also be feeding into the algorithm as well. I am a classroom teacher of languages, so I fully acknowledge that my understanding of the mathematical process is limited.
However, in light of what I have detailed and imagined above, it is safe to say that the grade the pupil’s receive will most likely differ somewhat from what the teacher’s have submitted in their estimations. The extent to which they differ will depend entirely on how close the teacher’s predictions are to the normal bell curve distribution.
At the moment, nobody knows how close they are, except the CGEO. Fri. 19.06.20
Further reading on calculated grades
https://secretteacherirl.wordpress.com/calculated-grades/
https://secretteacherirl.wordpress.com/blog-3/
School Profiles
The Department of Education guidance on calculated grades is available here
https://www.gov.ie/pdf/?file=https://assets.gov.ie/74604/d9e27dc5986e49a5a1b24623e77308d3.pdf#page=5
Follow My Blog
Get new content delivered directly to your inbox.
Secret Teacher Ireland 