National Assembly for Wales

Good afternoon, everyone. Can I welcome Members to this virtual meeting of the Children, Young People and Education Committee? In accordance with Standing Order 17.46, after consultation between myself and the Presiding Officer, this meeting is being called in a week that is not a sitting week. I've also determined, in accordance with Standing Order 34.19, that the public are excluded from the committee's meeting in order to protect public health. In accordance with Standing Order 34.21, notice of this decision was included in the agenda for this meeting, which was published last Thursday. The meeting is, however, being broadcast live on Senedd.tv, with all participants joining via video-conference. As usual, a Record of Proceedings will be published. Aside from the procedural adaptation relating to conducting proceedings remotely, all other Standing Order requirements for committees remain in place. The meeting is bilingual and simultaneous translation from Welsh to English is available. If we become aware that there is a problem with the translation, I will ask you to pause for a moment while our meeting technicians reset the system.

Can I ask Members if there are any declarations of interest, please? No. Okay, thank you. Can I note, then, for the record, that if, for any reason, I drop out of the meeting, it's been agreed that Dawn Bowden MS will temporarily chair while I try to rejoin?

We move on, then, to item 2, which is an evidence session on the impact of COVID-19 on young people's examination results in 2020. I'm very pleased to welcome Ian Morgan, chief executive of WJEC, and Elaine Carlile, director of qualifications, assessment and responsible officer for the WJEC. The committee is really pleased that you've been able to join us today. We are very keen to provide constructive scrutiny of the processes to date and, above all, to ensure that young people are supported in the steps that they take over the days and weeks ahead. So, can I start, then, by asking you to make your opening statement?

Thank you. Good afternoon, everybody. Can we thank you for the opportunity to meet with you today in what are very difficult and turbulent circumstances? As you know, the WJEC is the main provider of general qualifications in Wales and is regulated by Qualifications Wales. Within that context and as an exam board, it's important that we understand what our roles and responsibilities are and how they fit in within the context of the political and regulatory environment in which we operate.

Clearly, as you're aware, education policy is set by Welsh Government; regulation and direction by Qualifications Wales, including the setting of rules for exam boards; and then the development and delivery of qualifications, assessment, training and support by the WJEC. As an examination board, we have specific responsibilities, as you will be aware. One of our responsibilities is the upholding and maintaining of standards. We build that into all the activities that the WJEC undertakes on an annual cycle and in every piece of work that we follow through with.

We have to develop qualifications and specifications, and within those contexts, we look at assessment objectives and we look at the holistic view of the specification to make sure that we are covering all the relevant areas of the curriculum and that we're able to assess in the most appropriate way. Within that context, we look at a range of assessment methodologies, be they non-examined assessment or coursework, or any written examinations that take place.

In order to fulfil our duties, we undertake an annual cycle of examination setting, marking and standardising using subject experts. I think the key word for me there is the 'standardising'. All the work that we do as an audit organisation is based on a series of standardisation activities and standardisation models. We undertake a year-to-year comparison to maintain standards and parity and that's based on a range of performance and statistical evidence. We moderate coursework as marked by teachers in centres to ensure that the standards remain clear across centres and within the centres so that we can maintain a level of standard. And, clearly, from an exams point of view, all scripts are marked in accordance with a mark scheme to ensure consistency so that we keep the standards and the parity, and all examiners, when marking, are sampled throughout that process. So, we maintain regular diligent checks of any activity that goes on in the marking of an examination script.

Clearly, in the specific context of what we're talking about today, the WJEC has had a number of activities and roles that it needs to fulfil: the specific issues that relate to the development for Qualifications Wales's approval; the standardisation approach to calculating learners' qualification grades and to apply it accurately and consistently; to provide guidance on those new processes; and establish and deliver an appeals process that is compliant. So, all those things came out of the consultation that was undertaken by Qualifications Wales and working with us, but there are some key activities that remain specifically for WJEC and I think it's important that we understand what they are.

I don't need to rehearse what's happened and how we've got to this point, but we know, in March, exams were cancelled and the principles for inclusion and centre-assessed grades were developed. We know we looked at the outline of the impact for year 10 and AS learners, which we built into the modelling work that we've undertaken, and we know there was a wide range of consultation by Qualifications Wales, and across the border by Ofqual, where there are a range of activities that were consulted on and feedback given from the education sector in which we currently operate. From a WJEC point of view, we've maintained regular dialogue and discussion with a range of key stakeholders, including headteachers.

When we look at the key aims that we were building in to underpin the standardisation model—these were set out following the consultation and were key aims in the outcomes: learners for whom a CAG and rank order were submitted get a grade; principle 2: national outcomes will be broadly similar to those in previous years to reduce the risk of unfairness for learners; final grades will be issued as a result of fair and robust processes; and as far as possible, the process for rewarding grades will not systematically advantage or disadvantage learners. 

In the context of centre-assessed grades, we as an organisation provided subject-specific guidance to teachers to aid their decision-making processes. We felt it was important that at a subject-by-subject level teachers were able to assess what was within the confines of the curriculum in the specification and what evidence they could use to determine the CAG. And in parallel with that we developed a system to collect and collate those CAGs and rank orders from centres and that opened and started at the beginning of June. There's a lot of activity within that period of time in terms of guidance and in terms of development of systems in order to collate and collect those CAGs.

If I move on quickly, then, to the approach to standardisation, clearly we've got a complex set of principles in which we're working here. There are high-level principles where when we drill down to the detail there are some complexities within it. There are two key approaches in Wales that we've taken in terms of identifying the basket of grades that could be used at individual centre level: the use of prior attainment and the use of previous centre performance from a range of sources. So, we know prior attainment—for us in Wales, we are in an advantageous position because we do have AS units that can be used to work towards A2, and in the context of GCSE, there will be some year 10 units that will already be cashed in and in the bank that could be used in order to determine the outcomes and the basket of grades that are available.

And within that over-simplified approach that I've taken there, there's clearly a lot of statistical evidence and data that is being used, which is looking at a broader range of information, e.g. what adjustments do we make for small centres, as an example. We need to look at the recognition of variation of centre performance all the time. And the allocation of those grades that have been determined from the process I've just identified is then based on the candidate rank order that is provided by the centre. So, we've determined the grades that are applicable to the centre, we've used the rank order that's been provided at the centre and then we allocate the grades on that basis.

In terms of the process, there's been a lot of rigour, challenge and sign-off at a qualification-by-qualification level both internally and with Qualifications Wales. This isn't something that's just happened in the background; there's been a huge amount of dialogue and debate around the iterations of the models that we've looked at to get to the point where we came up with the model that we used and was signed off by Qualifications Wales. And underpinning that activity clearly was the development of an appropriate appeals mechanism.

I think one of the things I'd just like to mention is—. I've already referenced standardisation, quality assurance, checks and balances—they're not new concepts for 2020; they're integral to all elements of our work in any annual cycle. And if I look at the standardised outcomes for A-level for this year, which are available in the public domain and you will have seen these, I'm sure—if you just look at the overall outcomes for A* to E, A* to A, and then A*, in the context of the standardised model that we utilised, A* to E up by 1 per cent from last year, A* to A up by 2.9 per cent from last year, and then A* up by 1.7 per cent. If I now look at that in the context of the decision that's been made in terms of awarding the CAG to those centres: A* to E goes up to 99.9 per cent of candidates getting an A* to E; A* start to A, the CAG goes to 40.4 per cent—an increase of 13.4 per cent on 2019 outcomes; and at A*, the CAG goes to 15.4 per cent—an increase of 6.3 per cent on 2019 outcomes.

So, whilst these outcomes, which will likely go up again once we've introduced the AS floor process that needs to be implemented—. Candidates who get their calculated grade, if higher than the CAG—clearly, this does not meet principle 2 of the aims, which is maintaining broadly similar outcomes. There is significant increase in the system there.

I'm also sure, in any normal year, if there was an increase in outcomes of this magnitude, I would be invited back to the committee to explain how that had happened, because you would want to know what happened in the examination system that would allow an increase of that kind to come to bear.

From a WJEC perspective, the issuing of CAGs does cause us concern, and the revised results need to be seen in the wider context of 2019, and also 2021 and 2022. There is a recovery activity that will need to take place moving forward, in terms of how we understand the impact of 2020 on outcomes and how that is reflected in 2021 in terms of those outcomes moving forward.

And whilst I understand that examinations would have been a preferred and more reliable form of assessment, there were a unique set of circumstances in which an alternative, albeit imperfect, approach needed to be taken. We should also remember that in any cycle of examination, there are learners whose outcomes reflect or exceed their personal and their centres' expectations, and there are those, for whatever reason, where they don't. I'm sure we can all recall the headlines on an annual cycle: 'Exams are too hard', 'Exams are too easy'. Every year, we have to deal with the fallout of an examination cycle.

We operate in a world where political and public opinion is played out through many social media channels, and, indeed, it is an avenue WJEC uses to convey its messages. That said, what also happens is comments made via these channels are not always reflective of the reality, or not always based on fact.

This isn't a case of the computer or algorithm saying 'no': the whole process is underpinned by a dedicated group of expert individuals who have worked tirelessly for the last three months using a broad range of knowledge, data, information sources and statistical evidence in order to do the best we can to provide as fair as possible outcomes whilst maintaining the validity and reliability of qualifications for this year, previous years and future years. Can I thank you for the opportunity to be with you today? I would welcome any questions.

Okay. Thank you very much for those opening remarks, and we'll go straight into questions now, and the first questions are from Siân Gwenllian. I'll remind Members again that we need to be brief, to cover as much ground. Siân.

So, I think to answer the question initially, the standardisation models were borne out of a serious amount of activity that was undertaken. So, we didn't just look at one or two models; there were probably 10 or 12 models in the round that were explored just to look at which of those best fitted what we were trying to work through in Wales. The models were tested using 2019 outcomes from centres. So, what we did was took 2019 data and information and we applied the algorithms to that to get to the point where we were in a better position to see whether those outcomes correlated, using proven data into the models that we decided to use. In terms of the process, it was an iterative process that we then worked with Qualifications Wales on, and went through, and the final sign off was via Qualifications Wales and the standards and advisory group. Elaine will probably give a little bit more detail.

Elaine.

I just want to say I think from our research what we've found is that the best indicator of how a student would achieve at A-level in general times, in a normal exam series—that it's very highly correlated that the AS grade is a very good indicator as to how they would achieve at A-level, and that correlation is really strong and that was borne out by the work that we did. So, the best method in order to get to the position to issue results on the days that we did in order to meet deadlines for universities was actually to look at that correlation of the information as to how students achieved at AS and how that then gets carried forward into A-level.

So, I think that that is the fundamental part of this particular standardisation process. It's looking at understanding that some students, as Ian said, might do slightly better or might do slightly worse, but that, then, was looked at in comparison to the rank order information that the centres gave us, which then realised that a student, perhaps, who had achieved a C at their AS may have been further down the rank than a student who achieved a little bit lower at AS. So, therefore, looking at those two pieces of information in the round we found to be the most sound way to issue results to time this year. 

I think the issue would be about what we're standardising, Siân. The information, if you recall, as part of the consultation process, was that we didn't have—. Not all students will have achieved the same types of work at that particular point in time. We provide exams and that is our benchmark—all students sit the exact same assessment on a particular day, and therefore we can be equal across looking at what students have achieved. If we look at the classwork that people had done, it's very difficult to understand what teachers had set at what time and under what parameters they had done that, so therefore, there will have been an inequality as to looking at what some teachers will have set their students and how they graded them and how we'd compare that to perhaps a more challenging exercise that somebody else had set for a student and graded that. And therefore, it would be very difficult to get to a situation where we'd have sufficient evidence to look at the work that teachers had given their students and the grades they had awarded them, and that wouldn't really be feasible or viable under the conditions. We hadn't set that everybody had to do a certain task at that time. 

I don't think I'm suggesting that at all. I think in the circumstances and the time we had in order to do the work, doing anything directly with the centres in relation to the CAGs wouldn't have been achievable. We were in extraordinary times and the capacity to check at an individual school level and an individual pupil level what made up the data sources and the information sources that generated the CAG would not have been possible in the current climate. So, the option that we've gone for was the best that was available within the time frame in which we were looking at it. If we had more time, we could've undertaken normal processes of looking at candidate work and doing moderation exercises. The reality is that that isn't there in the current climate; we had three months to turn the system around in order to meet the deadlines in terms of issuing results to candidates.

Thank you. 

No, you said—. I've got Dawn now.

Thank you, Chair. Welcome both of you, Ian and Elaine; it's nice to see you both. Can I just—? Following on from Siân, I want to get a better understanding of what happened both before and after the report was published in terms of processing the standardisation model to be used and so on. Because a lot of the information and the feedback that we've had is that there was considerable input from the education sector into how they felt this model should look, and a feeling that very little of that was taken on board. Heads in particular were saying that they didn't understand the mathematical model and they weren't really given the opportunity to input into that. So, I'd like a better understanding of that. But more importantly, I think, beyond that—I think you've answered that to a degree with Siân—but beyond that, once it was clear what you were doing and what the process was going to be and what the model was going to be, what discussions and representations were you getting at that point and did you respond to any of that and did you amend or change anything that you were seeking to do in the way that these results were delivered?

Okay. Can I start, Dawn, in response to that? And thank you for the questions. Clearly, from a dialogue and an interaction point of view, we've spoken with many stakeholders. I think it's fair to say—. We have a headteacher reference group that we meet with regularly, and there's probably about a dozen headteachers from across Wales represented there, and they are representatives of teaching associations across the broad range of spectrum, including Welsh medium and English medium. I've also been on a virtual tour of Wales in the last three months and I've met with every local authority and every secondary headteacher that would have been available on the day that I undertook those meetings, and I talked through in principle the work that we were doing in terms of the CAGs and I talked through in principle the high-level standardisation models that we were talking about. I've also attended three Welsh Local Government Association meetings with Cabinet members where, again, we've talked about the principles of the approach that we were taking—

Sorry, Ian, can I say—? Did they raise with you in those meetings their concerns about the process? 

I think inevitably in every meeting people raised concerns around certain aspects of it. I think one of the main concerns that you will be aware of is centres where they see themselves on a positive trajectory of performance increase and concerns around how the model could reflect that performance. Clearly, with a statistical model there are some challenges in that in terms of recognising how you apply any kind of changes in those circumstances. I have to say as well, in terms of all those meetings I've had, there was clearly optimism from headteachers—and there's no criticism in here—optimism in terms of this was going to be the year where things were going to increase. I didn't meet one headteacher where they were expecting performance to drop or performance to be equal to previous years; it was very much an optimistic, positive trajectory. And I understand that; I think everybody is working towards that kind of objective of raising educational standards, and examinations is the measure for that process. So, yes, we did get feedback, yes, there were some things that were raised, but we took those into account in terms of looking at how we developed the model. And to the best of our ability, within the circumstances and the time frames, we tried to apply those approaches where and when possible. 

So, were you surprised, then, at the strength of adverse reaction to the results outcomes when they were actually delivered? 

I think I probably was. The strength of view that has emanated from it, I think, is fairly significant. I think if I go back to one of my previous comments in my opening statement, we always have outcomes that don't match expectation in any year, even when exams are sitting. So, I think the expectation that this year there wouldn't be any kind of adverse effect on the system probably isn't something that we should have been thinking about collectively. It's important from our perspective that what we've tried to do, using the evidence and the information sources that we've got—we've tried to do the best that we can in the fairest way that we can, using the data and the information that we've got.

Very quickly, Dawn, because I want to bring Laura in.

Yes. I've got two quick questions. One is your assessment of the process used in Wales compared to England and Scotland—and I ask that for a specific reason because there was a different evidence base. So, are you still of the view, or are you of the view, that the process that we were going to use in Wales was still more robust than in the other two nations? That's my first question. And my final question, Chair, is around the guidance that the WJEC published on teacher assessments and where you had concerns about overgenerous assessments that you would intervene and look to moderate that, and we don't believe that that happened. If you could explain why.

Elaine, do you want to pick up on the comparison between the England and Wales models?

Yes. As you know, the WJEC does operate in both jurisdictions as well, so we will have run through A-levels there. It is a different model; as we've mentioned already, it is clearly based on evidence at AS. It's interesting to see, when you look at the AS grades coming in, and some of the centre assessment grades, looking at AS performance—in general, most of them are looking quite positive, and again, maybe aspirational, rather than perhaps the reality of what we would normally see in an exam series. So we considered that AS-level information that we had to be robust; it was sat under exam conditions, awarded to students last year, and therefore, we consider that was to be a robust and fair process for all. And it is different to what was used for A-levels in England.

And the issue of the guidance; who'd like to pick that up?

Yes, I'll pick that up. In discussion with stakeholders, we did look at options in terms of where there were outliers, as we've described them, at that point in time, and what approach we could take. There was some activity through the consultation with Qualifications Wales that said what sources of evidence could be used in order to support any kind of change of activity, and we saw from the outcomes of the consultation that there wasn't anything that wasn't linked specifically to a controlled sort of assessment that would be relevant or appropriate to use—back to Elaine's point in terms of what evidence would you use that isn't through a formal process—in order to kind of undertake something that would look at the outliers in those cases. Clearly, in terms of the scale of some of those outliers, there would be no capacity anyway. If we were talking three, five, 10 centres—. But the optimism within the system, as I've articulated, was significant, and we wouldn't possibly, within the time frame, have been able to get through all those centres. 

Siân. Really briefly now.

Elaine.

So, to say that again, we've tested this evidence through. There may be some students who might do better at A-level than AS, but in general terms, if you recall—I'm sure you know—the AS is worth 40 per cent, and the content is not as challenging as the content of the A2 units, and that of course is worth 60 per cent. So, in general terms, although a student may mature from their AS to A-level, the content is also challenging, and there is a greater weight put on the A2 units that they sit. So, with that 40 per cent, 60 per cent split, there will be some students who achieve better, and as I've said, the rank order that the teachers gave us then would have dealt with that so that we would look at those situations. And there were examples of students who would have achieved a B at their AS who were given an A-level grade of A this year, for example; equally, there were students—and we've seen students in the past—who would get a C grade at AS, but actually, if they're at the very bottom of that C grade at AS, they don't necessarily get a C grade when they go on to A-level. Sometimes, they would go to a D grade at A-level because of that rise in the challenge, and the 60 per cent weighting.

Okay. Thank you. Laura. 

Just unmuting. Sorry, Chair. Thank you, Chair. It seems clear from what you've both said that you're concerned with the significant rise in the higher grades from centre-assessed grades that have now come out, and that you support the algorithm that was used. You seem quite adamant that that was the best thing available, and, obviously, the Minister was extremely confident to come out and say how robust it was. Were you continually saying to her how robust it was and advocating it? I'd just like to know that.

My main question is what's been happening in recent events, really. When we saw what had happened in Scotland, did alarm bells not ring with you and your colleagues and did you not think this could possibly happen here in Wales as well? We had 10 days from our announcement—from 4 August, when they announced what had happened there. Did you not think that—? Were you advising the Minister? What were you advising the Minister at that point? Did you think that—? Were you still saying, 'It's robust. We've got every confidence in this algorithm, this system. Stick with it', or were you saying, 'There is potential that something could happen here'? Did you have any idea what was going to happen? Did the Minister approach you to seek assurances that it wouldn't happen here in Wales, or did you go to her with your concerns? I just want to know what happened and the process there, obviously, because we could have learnt significantly from what had happened in Scotland. That should have raised alarm bells—I mean, it did with a lot of us. Could all this have been avoided?

Okay. Ian, is that one for you?

Yes, I'll start. I think in terms of robustness and learning lessons from Scotland, clearly we tested the model that we've used significantly. We've tested it, as I said, against previous years' cohorts and previous data, and what we were finding was that the outcomes were comparable with outcomes from previous years in terms of the non-use of the model. Clearly, what we also had in the mix was that we knew what the CAGs were and we could see what the overarching outcomes and the increase of outcomes would be if we went directly to the CAG group. So, from an awarding body point of view, and our role and responsibility in terms of maintaining standards, we would stick to the approach that we've taken on the basis it does allow us to provide that continuity of standards across the piece from last year and into next year. The CAGs as issued now clearly pushed that out of the window in terms of that kind of comparability.

In terms of recent events, clearly we've had lots of conversations throughout this period, regular conversations, with the regulator. Our role, just to reiterate, was to develop the model and for the model to be signed off and agreed by the regulator. Any changes to that model would have to come from a flow of information from Government, from regulator, to us to do something different. And that's not me abdicating responsibility; I stand by what we've done and what our team have done and the way that they've done it. There are just roles and responsibilities in and around this here, and our role was to develop the model, run the model and output results on the basis of that model, which is what WJEC has done. 

So, did the—forgive me, Chair—did the Minister approach you to seek assurances that you still felt the same way after what had happened in Scotland? 

Our conversation would be more directly with Qualifications Wales through the standards and advisory group. 

Right, okay. The inflation of grades obviously is a massive, massive concern. There are things that are going to come of it, but surely the fact that, for our students, after having been through lockdown and everything that goes with it—the mental health impact and everything like that—that their grades would be a little bit higher rather than significantly lower out of this algorithm, surely that would have been a better way forward.

But I think we have to be clear that the overall outcome was an increase of 2 per cent at A-level across the board. So, the overall outcome for all A-levels was at least 2 per cent greater than in previous years, and there is an upward trajectory over the last five years that can be seen. Elaine, did you want to add something? 

Elaine. I can't bring you back in, sorry, Siân. Elaine.

I just want to say, as part of the decision making for each qualification, Ian mentioned that we ran some of our processes that we would normally. So, for every exam series, we look at each qualification and we sign off the outcomes. And being involved in that process, in those decision-making groups, in every single stage of the process we built in generosity where we could in order to avoid the students this year being under any disadvantage, because it was quite an unusual year. So, we looked at what the predicted outcomes would look like based on prior attainment. When we looked at last year's outcomes—2019—at every single point, where we could be generous, we were generous, whilst still trying to maintain comparable standards, which means that that related, then, to an overall outcome of almost 3 per cent at A* to A generosity. So, it is fair to say that we did take account of that throughout the process. 

Okay. Thank you. We're going to move on now to questions from Suzy Davies.

Thank you, Chair. Thank you, both, for coming today. Now, obviously, you're making the argument that your main purpose was to secure comparability across the whole of Wales and across years for the standards and robustness of the grades that we've had. Can you give us some indication about what the differences have been, just in recent years, between the top-grade predictions and the actual outcomes at A-level? Because you've both referred to the fact that these top grades, that 13 per cent to 14 per cent of students will be getting higher grades than you would have expected them to see, but what we don't have is any indication of what happens in previous years where teachers are also putting in their aspirational grades. What have those differences looked like? 

Elaine.

We don't actually collect teacher estimates anymore. I think there was a long time ago when teacher estimates were collected, but that doesn't happen. We are aware that teachers give some data perhaps to UCAS as part of that process, but it's not data that is given to us. However, it's clear to say, as well, as mentioned by Ian earlier, every year we know in general the ability of our cohort of learners. So, we know their mean GCSE score is usually a fairly good way of understanding whether this year's group of learners is so much better than last year et cetera. So, we look at that as part of a process of us understanding, in a general year, if their outcomes are coming down, whether it's because we created hard assessments, or, if outcomes are really going up, is it because we've created easy assessments, and looking at what the prior attainment is. So, there wasn't a huge variation this year in the prior attainment. There was nothing to show us that these were an exceptional group of students or vice versa. So, their prior attainment would match them to previous cohorts. There wasn't anything significant in that. So, therefore, it was part of our process to try and build in, as much as we could, benefit of the doubt to ensure the learners were benefiting from this unusual situation rather than penalising them in any way.  

Okay, thank you for that. In normal years, of course, the grade boundaries will move around a little bit, I accept that. In fact, I think it was two years ago in our committee that we had a short inquiry and asked questions about why 10 per cent of a difference between predicted grades and actual grades was a problem. At the beginning of August, we know that 35 per cent of students were getting lower than the grades that their teachers predicted they were going to get. What alarm bells did that ring at that time and what do you know about what Qualifications Wales told Welsh Government at that time, because it was pretty obvious at that stage that there would be quite a furore? As it's turned out, we were talking about 42 per cent of students who've had lower than anticipated grades. When we're talking about different iterations of this algorithm, I think I would like a little bit more detail on how many times that was changed in order to respond to the stats that were coming out to show that too many children were getting lower than their predicted grades. 

Elaine. 

We didn't adjust to too many children getting lower than their predicted grades. As I think I said, the model is looking at the evidence that we have of students and their attainment at AS and understanding their prior attainment, and looking at it from that perspective. So, we didn't feel, based on where we are with the standards, that it would be correct to adjust more to meet the targets of the centre assessment grades, because our modelling, our looking at that, showed us that the best approach we had was the banked evidence that we had at AS as an indicator of A-level outcomes at the end of the day. So, all we were looking at each time was ensuring that, as we ran the data through, it was checking against prior attainment, it was checking against the 2019 outcomes and giving the higher of as best we possibly could. We weren't using the centre assessment grades as the barometer to check against whether we felt these outcomes were accurate or not, but we did also always use the rank orders from the teachers. We never changed anything from the rank orders that the teachers provided to us.

Okay, that's really helpful, that certain things didn't change as a result. But, from my perspective as a layman, I think it looks kind of strange that, in order to try and use standardisation to overcome, shall we say, an in-built bias of 14 per cent, we've ended up with a bias the other way of now 42 per cent, and that worries me that that didn't get through to the Minister. I know it's not for you to make political decisions, but I would've thought that either WJEC or Qualifications Wales would've given the Minister that information because it was inevitably going to affect any possible political decision. And that goes back to Siân's question about why this couldn't have been stopped a bit earlier. 

Ian.

I think, Chair, from a WJEC perspective, we've undertaken the task that we were given and we've worked closely with Qualifications Wales in and around that in terms of the sign-off of all the outcomes. We've had no direct communication with the Minister during that process as that isn't the protocol in terms of how these things operate.

Okay. Thank you for that. I'll ask Qualifications Wales. My final one is a 'yes' or 'no' question, Chair, if that's okay. Was the algorithm applied subject by subject or across the whole school?

Subject by subject, yes.

Within each school as well.

Yes.

Okay, thank you.

No. I'd like to clarify that. The AS—

I was thinking of the A-levels specifically, but if it's different answers for both—

No. When we make decisions, we make the decisions globally when we look at it all. We look at the outcomes of the qualification level, but, of course, when we made those decisions, it was centre by centre, of course, that we had the information. So, all of the candidates' AS grades for each centre were looked at and they were awarded grades linked to their AS performance.

Within different subjects—?

Yes. 

I don't want to go over time, sorry, but we're going to have to ask you later about why physics has taken such a battering. Carry on—sorry. Thank you, Chair.

Okay. Hefin, you had a brief supplementary.

Very brief. I'd just like to ask the panel whether they have any reservations about the wider use of a statistical standardisation model that's used for moderation when, in higher education, you don't have statistical standardisation models; it's done by peer review at the individual centres and through links up with other moderators and other institutions. Isn't that a better way of doing it, a more human way of doing this anyway?

Elaine.

I think, looking at standardisation, we obviously have a system where we have comparable outcomes across the piece. It is again going back to the assessments we've set; maybe it's a bit different in higher education institutions, because everybody sits the same exam and we set the tasks that everybody has to sit, so we're not looking at the tasks that have been produced by the individual institutions and therefore have to do the standardisation for that process. It's back to whether it's a moderated task—we would set the task and set the assessment objectives and the mark schemes, so it's an application of that nationally across the piece. So, I think, in higher education, it works a bit differently in that respect. So, I think looking at it across the piece actually is the way to do it.

Okay. I'll bring Hefin in now to ask about unintended consequences, because of the time, otherwise we're not going to get through all we need to. Hefin.

Yes. A very simple question: what are going to be the unintended consequences of grade inflation?

I think to some extent, in terms of measuring performance over time and what next year looks like, there is a potential area of concern there, isn't there, that we as a collective will need to look at. I think in terms of going with the CAGs in their current form, it transfers pressure back to schools and back to colleges in terms of teachers and how they have to now interact with parents and with learners.

And I think, to some extent, there's probably a balanced view out there that feels that the standardisation process is the right thing to do, and, in the interests of learners, there'll be people who'll be saying, 'Well, the learners got the best outcomes they could get'. They aren't mutually exclusive, but I think there are real issues and concerns for us in terms of what next year looks like. We've got a year 10 cohort—I'm a parent of a year 10 child moving into year 11—and what does that look like in terms of outcomes for next year? We've made adjustments in terms of specifications and qualifications and looked at the content and how we can adjust assessment objectives, and how we can adjust what marking looks like for next year's examinations. But we've got to do that in the context of maintaining a qualifications standard that says if you've got a GCSE in English language, your GCSE in English language for this year is the same level as a GCSE in English language for next year. And I'm concerned about how we—. I get the exceptionality of what we're doing, I get that fact that we've now done the best for the individual learners, but how do we deal with that in an overarching process of saying there's equivalence between one year and the next year? I think we've got collective challenges in that, Hefin. 

Right. I've got Dawn on unintended consequences and then Siân. I'm going to have to keep this going at pace. I don't want any preamble, please, in questions. I want straight questions, please. Dawn. 

Thank you. Just on the unintended consequences of inflated grades, I'm just wondering what your thoughts and views are on the potential for students with inflated grades now opting, potentially, for courses that they are not best suited to and the longer term damage that that could cause. 

Elaine, do you want to pick that up, academically?

Yes. I think it can be a concern. I suppose the students will all be in the same situation. I do understand about the interjurisdiction thing. I think it is very important that in Wales we forge our own path as much as we can, and I actually am very, very pleased that the decision was taken in Wales to ensure that the AS was kept in place, and the A-level was there—so it's really good, and that's really helpful. So, one hopes that some students will have made the decisions about the subjects to pursue when they were in the lower sixth, based on their AS results, and took forward those that they were most able to achieve at A-level. I do accept that there will be students in Wales who will be competing for places in universities with those from other jurisdictions, so that can be challenging. I also believe that the universities will do their best to support those students as well, to ensure that they can achieve the best outcomes that they possibly can and offer them support and guidance throughout that process as well. 

Okay. Siân again briefly on the unintended consequences of yesterday's decision. 

Ian, is it, or—?

Elaine. 

I think I'll start. I think from—. So, the inequalities issue I think you went to, I think there weren't any inequalities issues that arose from this process this year. That's first thing I just want to say. I think one of the things that's important is how the students—. If we left their grades there and they feel that they weren't the most appropriate for them, there's the appeals process. So, if there were students—. Every year, we have reviews of marking, for example. So, if something goes wrong—you know, sometimes our examiners don't always mark to the standard we expect, so there is always a period after the issuing of results where we review scripts and correct any errors that might have been made in the marking, and university places are held open for those students until 31 August, usually, whilst that appeal, as it's called, a review of marking, is undertaken.

In this process, my understanding is that universities had extended their offers until 7 September so that we could get through any of the appeals that came to us and that we would look at those issues when they came up, to address any issues that would have arisen from the model, and therefore ensure that those individual students were given a result that reflected their ability had they sat the exams this summer. So, I believe those issues would have been resolved through the standard appeals process. 

No, no, no—Laura next, with a very brief, sharp supplementary, please. 

I will send the information that I have that contradicts what you're saying on equalities, if I may. Thank you. 

Okay, Siân, we've got to keep moving. Laura. 

Mine was on theme 2, so it's fine—on the algorithms. I'll ask it later.

Okay, right. I'm very keen that we move on to talk about the impact now for young people who are going to be going off to university. Can you tell us now, then, how it is going to work in terms of the change yesterday? Can you tell us a bit about how UCAS are going to be notified? What's the process going to be for those young people whose grades have improved yesterday?

So, we are in the process of running those grades through the system and we will be issuing new files to UCAS that will be completed overnight tonight and tomorrow, so UCAS will get the revised information based on the outcomes that were agreed by the change in policy yesterday.

Are you aware that last week, after the AS guarantee was given, that UCAS didn't update their site for quite some time and that there are some young people who have lost places because the UCAS site didn't reflect the higher grade that they'd got in their AS? 

I wasn't aware of that, personally. Elaine, were you aware of it?

All I can tell you is that when that came through, at any stage that anybody got in touch with us, we updated the files that we provided to copy across the AS grade to the A-level grade. We, again, have to go through that process. It's fair to say that nothing in life is really simple. There are students who have an AS grade because they claimed that AS when they were in 2019. There are some students and some schools who actually don't claim their AS grades even though the students may have had sufficient units that had been completed. So, it is quite difficult. It depends on whether—the term we use is 'cash in'. So, if the centre has cashed in or the students have cashed in their ASs, then there is clearly a grade in the system that can be quickly produced. If the centre hasn't actually cashed in the AS—and there's no obligation to do that because some just go on with AS and straight into A-level and those units carry forward—then we don't have a grade, so we have been working through ensuring that we check all historic data to ensure that every student who had sufficient units already completed and had never claimed their AS gets that and we provide that data. And, as Ian had said, that's being run and produced overnight.

Okay, thank you. Siân, you've got some questions on the new appeals process now. 

The appeals process is part of the regulatory framework that we were working towards—[Inaudible.] The current process for appeals was part of the consultation, first of all, that Qualifications Wales undertook and the guidance was published on the grounds for appeals. I have to say, I'm not sure at the moment. We will be having a meeting with Qualifications Wales later today. I think we were supposed to have one earlier today but we've moved it on because of this session, and I'm sure that at that point what the next stage of appeals will be will be discussed with us then. But that guidance is a regulatory guidance and we're not in a position at this stage to be able to give you any further information because we don't know. 

I think if I just respond picking up Elaine's answer to the last question, the appeals process was consulted on by Qualifications Wales and the outcomes of the consultation were what drove the appeals process that is currently put in place within the regulatory framework in which we operate. So, for us, we again, without abdicating any responsibility, will deliver any appeals system as defined. It comes out of guidance and instruction, to some extent, from the regulator. 

No.

Thank you. Hefin.

What's your plan to deal with next year?

So, as I've already said, we have already started—well, no, in fact, we have published two schools', two colleges' adaptations to specifications, which look at how we are able to kind of look at the content of the specification, but also at the assessment and the assessment methodologies for each qualification, to see where there are opportunities to perhaps build in optionality to questions, build in optionality in terms of non-examined assessment or coursework. But what we've tried to do, particularly for GCSE, is maintain the content in its current form.

Clearly, we're halfway through a cycle of teaching and learning, and what we wouldn't want to do is withdraw content from a specification that some centres might have already delivered and some centres might not have delivered. So, the current profile of activity is: we've released that to centres at the end of term, and that is with them to look at how they then implement that in their own settings from September.

I'm sorry, I only understood about one third of what you just said. I don't understand what you mean by 'optionality'. I didn't understand—. Can you put it into plain English? How are you going to deal with this problem next year?

Elaine, do you want to put it into plain English, then?

Okay. So, we haven't taken any content out of the specifications, but we've tried to streamline the assessments. So, therefore, for example, for English literature—let's take that as an example—there will be two papers: one is to do with modern plays and poetry, the other is to do with Shakespeare, or that sort of stuff. So, basically, for certain qualifications, the centres can choose to take one unit and do it as an examination assessment. So, that is one type of adaptation. In others, you have maybe questions, where you can answer on one area or the other, so that the students and the teachers can concentrate their time on certain areas. So, that is one thing we're doing.

And to just add to what Ian has said, we're also aware that the COVID pandemic—we don't know how it's going to play out, whether we'll get into a lockdown again. So, one of the things—once we've done any changes and advice to our schools and colleges about how to adapt for the summer, we have now started to think about, 'What if we're in the situation? What if this all happens again?' And we need to talk to teachers amongst our communities to say, 'Okay, how could we support you?' If we're in a situation, going back to where you mentioned at the very start of this session—'If we were in a situation where we had to ask you for assessment grades again, what would we have to do and what would we have to set you in order for us to be able to moderate actual work, and at what point would that actually have to happen?' And there'd be some sort of standard test we'd have to give everybody. How could we deliver it?

So, all those things are very challenging. So, we are this week starting to have those conversations, to get some viewpoints from teachers as to how they feel we could be supporting something if we are in this situation again next year. And it's going to be difficult, because people will have done different aspects of their qualifications already. So, that's something we're starting to look at.

I think it was that second point, that latter point, plan B, that I was driving at, Chair.

Yes. Thank you. Suzy, then—a very brief question. Can you unmute, please?

Yes, apologies. I think it's worth bearing in mind that next year's A2 set will be competing with this year's A2 set who didn't get their first choice of universities for places, and so getting comparability between those two years is going to be pretty important. I'm very worried to hear that centres have already been advised that practicals in science subjects will not be compulsory, so we're then looking at potentially a cohort of young people going to university without practical experience in the sciences that they're applying for. I think this is going to be an area of major work, and I'm keen to know how you're going to keep us updated on that.

Can I just say first of all, in response to that, Suzy, we're not saying that practical work shouldn't be done, we're saying it's not going to be assessed, because we're concerned about the burden for schools in making sure all the students have to be in a certain place at a certain time, because an assessment is something that happens based on a very timed sort of area? So, we're hoping that the schools and colleges will continue to deliver practical science to their students or will look at various ways of looking virtually at how science works practically, but it's just not subject to assessment. And we will be publishing the changes to the qualifications on our website. We again consulted with our headteachers on that. They didn't feel we should make those on our public website at this point in time because it could add to the anxiety of the students if they could read this information about next year without being able to speak to their teachers. So, once the term starts, we'll be publishing the information on our website, and happy—I guess Ian will say that—to provide you with any details.  

I think we'll have to ask about how they respond as well. 

Thank you very much. We would be grateful if you could keep the committee updated on the plans for next year. We have more than run out of time, really, so can I thank you both for your attendance today and for answering all our questions? We'll send you a transcript to check for accuracy following the meeting. Thank you again, both of you. 

Can I then propose, in accordance with Standing Order 17.42, that the committee resolves to meet in private for item 4 of today's meeting? Are Members content? Thank you. We'll now proceed, then, in private.  

We thank you for your time as well. It's really important for us as well that we're able to share and discuss with you. Thank you. Diolch. 

Okay. Can I welcome Members back, then? Our next item is a further evidence session on the impact of COVID-19 on young people's examination results in 2020, and this time with Qualifications Wales. I'm very pleased to welcome David Jones, who is chair of Qualifications Wales, Philip Blaker, chief executive of Qualifications Wales, and Jo Richards, regulation director at Qualifications Wales. Thank you all for attending. We recognise this is an especially busy time for you, and the committee is very keen to engage in constructive scrutiny that provides benefits for children and young people. So, if I can just start by asking you to make your opening statement, please. 

Okay. Good afternoon. Can you hear me? 

Yes. Thank you. 

Thank you for the opportunity for Jo, Philip and me to attend the committee today. This is a very difficult and upsetting time for everyone and we appreciate the opportunity to contribute and to listen to the committee and to answer your questions. We would like to recognise the hard work across the education sector in Wales: the professional, committed and hard-working staff at all of our schools and colleges and universities, the Welsh Government education department, WJEC, and not least our own colleagues at Qualifications Wales. This has been an extraordinarily challenging year for everyone and continues to be, and we all want to do the right thing for our young people. Nobody would have wished for this situation and we are sorry that the pandemic has had such a profound impact on the education system.

On 18 March, when the decision was made to close schools and cancel exams due to the COVID-19 pandemic, three options were available: have no qualifications in summer 2020 and therefore prevent learner progression, hold exams even though the nation was in the grips of a public health crisis, or award outcomes in some other way. 

On 6 April, we were directed to establish a fair and robust process for the awarding of grades this summer based on teacher estimates supplemented by a range of other evidence. In May, we consulted on aims to underpin the approach for awarding grades and an appeals system. We consulted in an open way and engaged widely with stakeholders, and we received more than 4,000 responses. We initially had five aims when we started the consultation and we reduced them to four, and in WJEC's evidence earlier on they covered the four aims that were eventually agreed. Most respondents agreed with our aims.

When considering responses, we recognised concerns about equalities from many individuals and representative organisations, not just about the standardisation model but also the potential for bias to be evident in teachers' judgments. We felt strongly that having broadly similar results to previous years was an important part of our consultation not in order to restrict results to any particular point, but so that the results could be considered credible and hold long-term value for learners throughout their lives. This would protect their interests in the longer term. It was not about protecting the system at the cost of learners; it was intended to be the very opposite.

We also recognised that significantly different results would present issues for progression, especially where selective admissions decisions would be made. Of course, this year's cohort has been badly affected, but fairness is not just about one cohort; it is about being fair to previous and to future learners too. We went on to develop what we believe is the fairest nationally regulated approach under these circumstances.

Qualifications Wales is a national regulator, so we have responsibilities to ensure that standards are maintained over time. National regulation involves setting standards, common assessment and national standardisation. The alternative—going with CAGs, centre-assessed grades—moves away from a model of national regulation to a model of self-regulation by schools and colleges. In normal times, a move from national regulation towards reliance on greater self-regulation would take considerable time to implement and would require full and effective engagement, consultation, training and support. Indeed, a move towards more self-regulation was proposed by the further education sector about 10 years ago, and perhaps our experiences this summer and the need to evolve our curriculum and assessment would support a national multistakeholder review of how self-regulation could play a stronger part in future qualifications.

Returning to the present, we know that any form of standardisation is never perfect, but we wanted this year's approach to be as fair as possible. We knew that there would be issues and that there would be some cases that seemed unfair but that these could have been addressed through the appeals process. We consulted on having an expedited appeals process so that issues could be corrected as quickly as possible. 

Moving on to last week's results, at the time of the results we received an addendum to the direction received from the Minister to adopt a new Welsh Government policy that no learner should receive an A-level grade lower than their corresponding AS-level result. This policy has subsequently been implemented and would have positively changed around 4,000 A-level results. On Saturday, we announced the broadening of the appeals process to allow schools and colleges to present their own internal assessment evidence to support the centre assessment grade. Since the weekend and following discussions with the Minister, our focus has shifted to implementing Welsh Government's policy that learners should now be awarded the better of the centre assessment grade or the grade derived through standardisation. That was their policy announcement that was made yesterday. This policy has been implemented for GCSE results day this Thursday and we believe that amended A-level and AS-level results will also be issued by the end of this week.

We have been clear that this move to a more self-regulated approach under these unprecedented circumstances brings with it different unfairness. So far, our work to analyse this year's results has focused on calculated results, but we are now urgently undertaking a granular analysis of centre assessment grades so that we have the full picture at a regional and local level. We published a national picture of CAGs two weeks ago and we will publish the more detailed analysis in the next few weeks.

Since last week's results were published, there has been sustained pressure across the UK to change the approach and accept CAGs as this year's results in these exceptional and very demanding circumstances. Recognising that there will be differences in the way that schools have approached centre assessment grades, there has however been a loss of confidence in the approach taken by national regulators. As Qualifications Wales, we now agree that the fairest approach for learners in Wales is to award centre assessment grades so that they're not disadvantaged relative to their peers in other UK nations. It is the right thing to do as we want learners in Wales to have the best chances in life. We also want to protect learners' well-being as well as confidence in qualifications. We can understand that this year's standardisation approach seems insensitive to individuals and clearly cannot replace actually taking exams, which give people a sense of fulfilment. The models adopted would never have been able to achieve everything and were always going to be limited in what they could do. Ultimately, any alternative to exams, be it through a standardisation model, using CAGs or anything else to replace actual exam performance will have flaws and unfairnesses of one form or another and will always be contentious. However, it is now clear that there is greater confidence in self-regulated teacher judgments this summer.

Receiving the results should feel like a reward for years of work. This year's cohort has gone through enough disruption. We are sorry that learners have had to go through this tumultuous time and know that what is normally an anxious time has been made even worse this year. We would also like to say sorry to all those learners whose grades will be changed. Everything that we've done has been to try and balance many factors so that learners get the fairest grade. Diolch yn fawr iawn. Thank you.

Thank you very much for that statement. We'll go now to questions from Members, and the first questions are from Siân Gwenllian.

Philip, do you want to come in to follow up on that?

Philip.

Just to answer the very specific question that Siân asked—good afternoon, everybody, by the way—the aims—. We did a consultation. We undertook a wide consultation; we had more than 4,000 responses to that, and half of those from young people because we did a young people's version of the consultation. So, very extensive responses. We did a full analysis of those, and the aims and decisions that we made in terms of controlling those aims were made by the Qualifications Wales board and then we published a full set of findings and decisions on our website. 

Can I answer?

Yes, go on.

Yes, you're right. That was a close-run question, but there was still a majority that agreed with our proposal. I think the issue there is around the role of the awarding body and how the appeals process can be managed by the awarding body. The particular issue was around could an individual make an appeal about the judgment that had been made by their school. So, this isn't a judgment that is made by WJEC in the normal way that an appeal would operate, because appeals would normally operate on the basis of decisions that WJEC are making on the basis of things like quality of marking of scripts. So, these are judgments, and particularly the thing that schools were asked for was to provide the grade the learner was most likely to get if they had sat the exam. Now, we deliberately didn't make that a reductive model where it was looking at a particular piece of evidence, like a mock result in isolation, but we asked schools to make a holistic judgment. Now, that holistic judgment would include a number of points of evidence and ephemeral information, so we didn't believe that WJEC would be in a position to be a party to all of the knowledge to be able to intervene in those decisions that schools had made. But we have made it clear that if learners wish to have a right to recourse about the centre assessment grades that have been submitted by the schools, then they would need to go to the schools and go through the schools complaints procedures.

We wanted to consult on options that we thought could be delivered, both in the timescale, and in the context that everybody was working in during the summer. So, the other options that might have been available might have been to do some work with schools in terms of looking at what the expectations might be for centre assessment grades, and frankly, there wasn't enough time in the process to be able to do that. So, centre assessment grades had to be provided to WJEC by the middle of June, so that they could be in a position to start processing and start developing and finalising the models that would ultimately be used. So, there was little time—in fact, insufficient time—to undertake any robust process with schools.

I think with more time—there's certainly no extra model of standardising schools so that they could come up with more consistent judgments. One of the things that we were most concerned about is the variability between schools. I think one of the important points that must be remembered is we are seeing some inflation in grades, as you'll be aware, as a consequence of moving to centre assessment grades. That is a concern, and that will be a concern in terms of maintenance of standards over the long term, but actually, probably our greatest concern is that that may interfere—that there may be a bigger problem underlying that, in relation to inconsistencies between schools. Some schools may have been very close—predictions may have been very close to results that learners may have achieved, and in other schools, there may have been a greater difference. Now, that isn't a criticism of teachers in any way; it's just evidence that we've seen that there is great variability between schools.

The other option that one could have looked at in terms of other processes and alternatives to statistical standardisation would be some approach of sampling materials—actual learner materials. But we didn't think that that was safe or appropriate given the fact that, certainly in April, when we were making those decisions, we were in the middle of a pandemic—a national health crisis—where asking teachers to go into schools to start preparing materials to be sent out to examiners, and then sending those materials on to examiners, would potentially have been hazardous and challenging. 

Okay, thank you. I'm going to go on now to Dawn. Dawn Bowden.

Thanks, Chair. A couple of things. First of all, in response to Siân, you were talking about the process of consultation leading up to the decisions that you made when you published your report in June. Once your report was published and it became apparent what you were going to do, what kind of response were you getting from stakeholders at that point? Did you revisit or look at any of the things that you put in that report to address some of those concerns, or, actually, did you not get representation from stakeholders? And I'm talking about the widest range of stakeholders, including people like us, as Members of the Senedd, councils, schools, education authorities, WLGA—you name it, all the relevant stakeholders. What were they saying once you published your report and realised this is what was going to happen and so, whatever had happened in the consultation process, you'd made your decision. What did they say and how did you respond to that?

Yes, certainly. After consultation, we received relatively few representations from organisations. We had some extended conversations with the children's commissioner and with the Equality and Human Rights Commission. We also had representations from a number of representative organisations. But, actually, the greatest concerns that were being raised after our consultation and, indeed, through our consultation—one of the major themes—were around concerns in equalities. Those were partly concerns associated with whether the model, whether standardisation would provide outcomes that addressed issues of equality. But, actually, the bigger concern was whether there would be potential for bias and discrimination in the centre assessment data—so, the grades and the rank order that were provided by schools. As a consequence, as a result of those representations, we updated our guidance to schools to remind them of their public sector equality duty, and also to make it very clear that when a headteacher signs off against those grades that are being submitted and that rank order that is being submitted, they are sure that those are a true and fair representation. So, those were the main areas.

In the consultation itself, we did have concerns raised with us, which we've spoken about in our findings and decisions report, around the nature of the data that would be included in the standardisation model, but we addressed those through our decisions report and the findings.

Yes. We heard from WJEC, actually, that, throughout the consultation period—I think the quote they used was this—there was an 'optimism in the system'. So, was it your assessment that the sector, heads, schools, local education authorities generally thought that the outcomes were going to be reasonable? Was that your sense of what they were saying?

I think it's difficult to say. We've seen in the analysis of centre assessment grades that there is some optimism or generosity in the results, which is why we will see outcomes increase considerably this year. What might lie behind some of that is difficult to interpret. I think what we've got to remember is that teachers are working—this is a novel procedure, it's something they haven't been asked to do before. We were all operating in extreme circumstances in terms of working environments and the like. I think teachers did a very good job in providing the data that they did. But we have seen that there is variability. We've seen that there is generosity and we've seen that there is variability between schools, and I think it's that variability between schools that is probably our biggest concern and remains a concern.

Yes. And of course the WJEC did have in their guidelines that they could intervene if that was the case, and they told us that they didn't have the time to do so, so that may be something that we need to flag up for the future.

My final question, Chair, is just around the discussions that you had perhaps with other parts of the UK, with Ofqual, in terms of the other nations trying to get some kind of standardisation across the four nations. Was that happening and did you feel that the system that we ended up with in Wales was more robust than we saw in other nations of the UK, or did it identify to you that we had similar flaws in our system? 

Jo, could I invite you to answer that question? 

We work closely with the other jurisdictions that also have the same brand as us, so both England and Northern Ireland where GCSEs and A-levels are taken. There are differences, as you're aware—the differences, particularly at A-level, where we have ASs here that we could use as evidence. So, we were aware of the models that were being used in other jurisdictions and we were aware of the differences due to divergence in education policy and therefore differences of evidence that we would all be able to use. So, as it stands, ourselves and Northern Ireland, both of whom have AS qualifications that contribute towards the A-levels, used that evidence, as we believed that that was the more robust evidence, in order to calculate A-level grades, whereas in England they no longer have AS-levels that contribute towards their A-levels and therefore could not use a similar model. 

Okay. Thank you, Dawn. The next questions are from Suzy. 

Thank you. Thank you, everyone, for coming as well. We heard from WJEC that teachers' estimates aren't collected any more and that obviously the decisions you've been making recently have been based on actual grades that have been acquired over the years, and I know that your primary purpose is to retain robustness and credibility across the years. Having got to the stage where you'd signed off the algorithm and run it through the system, tested against the actual CAGs, and you ended up at the beginning of August with 35 per cent of students getting reductions in their grades, which actually became 42 per cent later on—even though you're not a person or a body to make political decisions, did you have conversations with the Minister at that point saying this is going to be problematic in terms of the number of students who are not just affected, but in some cases affected very badly from what they've been led to expect by their teachers? 

We didn't have discussions directly with the Minister. As a sponsored body we have a sponsorship team and we liaise with officials and the sponsorship team through a mechanism that we call an information-sharing group. Jo's the principal officer from Qualifications Wales who goes to those information-sharing groups, so Jo, can I ask you to just sort of recount how those operate and what we shared?

Jo.

So, every summer, even in normal times, we have information-sharing groups where we would provide some high-level information to officials about how the exam series is progressing and we would give a flavour of what we are seeing coming out of the data as it comes to us. So, there were regular catch-ups during this series. We also spent some time briefing officials on both outcomes of our consultation and the standardisation models in the lead-up to the release of results. 

Okay. Thank you. When you talk about regular catch-ups, can you give us some indication of how regularly and when, and at what points this group will have been the conduit of information about the 35 per cent figure that I mentioned earlier and also the position that we also heard from WJEC that as a result of relying on AS grades have been told by WJEC as well that AS grades and A2 grades, actually, are pretty close? As a result of relying on AS grades, 4,000 people got better grades. How can we explain that difference, if ASs and A2s are usually fairly similar, that 4,000 people suddenly got better grades, when Qualifications Wales started looking at AS-levels? I just want to know how much the Minister knew and when. That's what it boils down to.

So, I can't off the top of my head give you the dates of the meetings, but we meet—. We have met in the lead up to this weekly, and as we get closer to results I would have regular contact, be it weekly or biweekly with our sponsor team, which, as Philip mentioned, is the conduit. I think we've probably, just to be clear—our aim in this process was not to hit the centre assessment grades. They were to be used as part of the evidence through this. And also, as we move through it, what we monitor and look at with WJEC in meetings, which were happening very regularly—we had reports at a subject level of what outcomes we're looking at. We would then put that information together, which would give us the global picture, but clearly we would needed to have monitored subject by subject to get to that position. 

Just to pick up on the point about the 4,000 grades, that number of 4,000 were—it's the figure that is the change due to the putting in of the AS floor that happened after the results had been released.

Thank you. And I'm not suggesting you set out to be unfair at the beginning of this; please don't take that the wrong way. So, I still don't quite understand this last point, though, that if AS-levels were part of the algorithm—let's call it that—that 4,000 figure wasn't already being reflected in the results at the end, once everything had been through the mathematical process. Because I appreciate it's just one factor, but it's quite an important factor if it is, as it subsequently became, a floor. You wouldn't expect anyone to be getting less in their AS-levels anyway I think was my point. 

Yes, I think it's interesting, isn't it, because I think there are cases where people get less than their AS-levels within there? But accepting that as a student you would probably expect to get higher than that, I think what we find is what we weren't particularly looking at was the proportions that were getting less than their AS grade and what we also weren't looking at globally was this 42 per cent figure. What we were looking at, it was on a qualification-by-qualification basis. 

That's very helpful, thank you.

Philip, are you okay with that?

Yes. I was just going to say it's always worth nothing that in any system where grades are allocated in these sorts of ways, there will be people that are close to the threshold between one grade and another and I think the AS floor probably affected some of that. So, some of those learners that were on the threshold between grades would have been affected and would have been uplifted. 

Thank you.

Okay, thank you. We'd like to ask some questions now about the unintended consequences of the decision yesterday, and Hefin David is going to start. 

Can I ask you how you're planning to deal with the issue now of grade inflation and how will it impact—? What will be the unintended consequence of grade inflation over the next academic year?

Okay. Well, I think there are many things that we need to think through and there will be an amount of work that we need to do. There are immediate issues that arise out of going to centre assessment grades, principally on admissions decisions that HE and FE will have to make at the moment. So, we've been liaising with UCAS to make sure that they're aware of decisions that have been made and they will be getting updated results from WJEC as soon as possible, so that those can be reflected through in their systems.

In terms of policy decisions relating to HE, those are with Welsh Government, but people will be aware of—certainly things have been reported around additional help and also around the flexibility universities are trying to show in those admissions decisions. But we know that there will be caps on the numbers of people that can get into places, albeit that those might be physical constraints or teaching constraints, rather than capped admission numbers, and that there may be a situation where learners have to be offered deferred places that would be available next year.

I think the other consideration that we've got is thinking about what the implications are for us next year and in terms of maintenance of standards, and we need to think about that not only in the context of one of our core roles in terms of maintaining standards of qualifications over time, but we've also got to think about that in the context of a cohort that in many ways will be even more challenged next year because of the impact of lockdown on their teaching and learning.

And another possible unintended consequence is the undermining of faith in the standardisation model and the system standardisation used in the future. One of the things you said was that you couldn't do sampling this year—human sampling of scripts and assignments—because of lockdown and the restrictions and the dangers associated with it. But do you think longer term perhaps a more human approach to moderation might be a better way of moderating than using a statistical model?

Certainly, in normal times that is the case. In the way that standards are maintained in a normal year, there are statistics that come into play, but there's also a large amount of human judgment. WJEC's awarding panels will look at material, they will look at scripts, moderation of non-exam assessments, so coursework and controlled tasks happen in a normal year, to make sure that there's standardisation across schools and that there is a single and consistent standard that's being applied. I think the issue is that for this year statistics have been relied on as the only source of moderation and there hasn't been an opportunity to look at judgment on learners' individual pieces of work as there would be in normal years, and, of course, that's one of the constraints that this process has and why it's not a perfect process.

Okay, Hefin. Dawn.

Thank you. So, to be clear, then, there's a standardisation process that we use every year and we've got algorithms that we use every year. So, the only difference this year is we don't have actual exams by which to use that standardisation process. We've had to use a different methodology. That's basically the key difference, yes?

I think the word 'standardisation' might be confusing here. There are mechanisms for maintaining standards, and those standards are made by judgments from examiners, but they are guided by statistics, and they are guided by actually looking at material, actually looking at scripts and work. So, it isn't inhumane—and I mean that in the nicest sense—it isn't an inhuman system in normal years; there is judgment by individuals. What we do as a regulator is we would set some expectations for how much we would expect grades to change from one year to the next. Now, those expectations don't mean that WJEC can't come to us and say, 'We have seen evidence of better performance or worse performance, which means that we have to go outside of those tolerances', and then we would consider that. But it is based on human judgment and it is based on the evidence of learners' performance either in exams or in non-exam assessments.

Okay, yes, I understand that. Just one other question, Chair, which is really around the impact on the attainment gaps, based on deprivation, or students from areas of deprivation, and whether you have concerns that they have been impacted upon unfairly, and if not, perhaps you could point to why not, in your view. And if you could perhaps explain why it is that schools seem to have not done as well in some of those areas as maybe further education colleges or sixth-form colleges. I'll use the example of my own constituency of Merthyr college, which did—you know, they've got a cohort of students, 65 per cent of which come from backgrounds with deprivation and yet have produced some of the best results for the last three years running and have done again under this system. So, I just wanted to get a sense of really whether you feel that there is an issue there. 

Philip.

So, we've looked at that in terms of the calculated results from last week and we published some information alongside the results that shows that there are some variations against the parameters that we looked at, which were age, gender and eligibility for free school meals. But those variabilities were in the normal range of flux that one sees from year to year anyway. We're in the process of doing that same work with the centre assessment grades now that centre assessment grades will be published. From an initial look at those, we're seeing similar fluctuations on the basis of those parameters when we look at centre assessment grades, with perhaps a slight widening of the attainment gap at grade A, I believe it is, at A-level for girls versus boys. So, apart from that, we're seeing sort of normal levels of variation and we don't believe that the model has driven in any particular issues in terms of equalities.

Now, I have to say that that's a limited initial analysis given the time that we've had results available to us to do that analysis. There is a plan to do more comprehensive analysis of results and to publish that within the coming weeks, and that would go on to look at ethnicity and other characteristics that we can derive from the data that has been made available to us. So, we are planning on a comprehensive equalities impact assessment and analysis in the coming weeks. 

Okay, thank you. If we can move on now to talk about the transition for learners. We heard from the WJEC that the UCAS information is going to be updated quickly based on the change yesterday. What is your understanding, then, of the likely difficulties that students are going to face? There were some young people, for example, who lost their places last week and they were told that the courses are full. Can you just tell us a little bit more about how you think this change yesterday is going to impact on people hoping to go to university in the autumn? 

So, I think it's difficult for us to answer some of that because it lies outside of our remit, so we would be answering it as individuals and, clearly, recognising the human element of this and the distress that it's causing to people and the worry that it's causing to people. I think our part in the system, and WJEC's part in the system, is to effect the changes that this change in policy has brought about as quickly as possible so that that information can get into the system in a way that the system can use it as quickly as possible. So, we're very grateful for the huge amount of work the WJEC have done to reprocess results. We understand that GCSE results will be published on Thursday with the centre assessment grade, or the better of the centre assessment grade and calculated grade being issued. I also understand from them that they're on track to be able to deliver updated AS and A-level files by the end of the week. For our part in the system, that's what we're trying to do and that's what we're focusing on to try and get this information into the system in a way that the system can deal with it most appropriately as quickly as possible.  

And you're confident that that is going to happen according to the timescales described. 

Yes. We're meeting with WJEC after this meeting to see where they are with processing. I think one area where we think there may be a little delay, but we'll need to confirm this, is that normally schools are provided with results at midnight on the day before results are published—so, midnight tonight—so that they can prepare results for students when they come in on results publication day. WJEC are working hard to get those results out to schools as quickly as possible. When we last spoke to them, we didn't believe that they would be able to make it by midnight, but they would be able to make it tomorrow morning and as early in the morning as possible. I think that delay is mitigated somewhat by virtue of the fact that because results will now be based on centre assessment grades, schools will know what grades they've got because they submitted those grades, or for those few learners that would have got higher grades out of a calculated model, they will get those calculated grades, so that they know what their position is and they know that there may be a few improvements on that. 

Okay. Can I just ask, then, about vocational qualifications? There are lots of young people doing things like BTECs who haven't had their results yet. When can we expect those to be issued, as far as you're aware?

So, we understand that there has been a delay to the BTEC results and we've had notification of that from Pearson, and we're working with them and are asking for regular updates. They're working through that data and we understand that students should expect that by the end of today.

Okay, thank you. Suzy, did you want to come in on this?

Yes, just one question from me, really. We've touched on it slightly already, but, bearing in mind that this year's experience will have produced some people who have decided just to defer a year and perhaps apply for university places next time, they'll be competing with next year's A2 cohort, and whether you're planning to have any conversations with the universities about how they will be viewing both cohorts of applicants for places and what they'll be taking into account other than the assessed grades in deciding whether they're going to make offers.

Who'd like to pick that up? Philip.

I think those are discussions that we'll need to move on to as we move out of this period now of just trying to get results out. Essentially, our focus and our capacity is focused entirely on trying to get through this results period as quickly as possible. I think, following this, we'll be making some decisions about how we maintain standards next year. WJEC have already looked at what they can do in terms of changing the assessment arrangements for qualifications next year so that they are as appropriate as they can be for the circumstances that learners have had to go through, and frankly, with the potential for future lockdowns because the pandemic hasn't gone away and can resurface at any point, then thinking about those sorts of things as we move forward. But we will be liaising closely with UCAS. UCAS is a good conduit for us to talk to higher education. We also have a member of staff within the organisation who's got very good and very strong links with universities. You'll remember that, as a result of our work on the Welsh baccalaureate, we have an engagement officer who goes around universities and talks to them not only about the Welsh baccalaureate but about our work more generally. We'll be focusing some of that effort into thinking about how next year's results will be considered. 

Okay, thank you.

Thank you. Siân, on appeals.

Philip.

I can answer that. So, on the grounds for appeal, we finalised guidance on that fairly recently. Clearly, we were asked to broaden the appeals and that was done over the weekend. But, before results came out, we issued guidance to WJEC, which was to look at what we have termed as 'wrong data'. So, what we've got to remember is that in a statistical process, what is getting through that outside of the judgments of schools are elements of statistics. So, those grounds were looked at before results went out and were adjusted, and we did this in parallel with changes that Ofqual were introducing in England, to look at a number of things that would, I guess, try and pick up the issues that we might have identified through results. So, an example there, for GCSEs and AS-levels, historic performance of the school or college was part of the equation that was used in standardisation, and part of the grounds for appeal were widened to include a circumstance where a school could say that a year was atypical for one reason or another and therefore should be excluded from that historic average. That could be for a number of reasons, such as a significant change in leadership or the like, but, in essence, it was looking at those grounds for appeal to make sure that they were as broad as they possibly could be, recognising that the role of the awarding body was in applying the statistical standardisation model. Now, we did want to keep a close eye on any issues that were coming out of the results base to make sure that those grounds for appeal were sufficiently comprehensive to address them, and those were considerations that went into our updated guidance, which was issued on Saturday, to broaden the appeals process.

Could I ask Jo to come in on that, because we were talking about this earlier?

We are currently reviewing, in light of yesterday's announcement, the grounds for appeal, and given that we are now in a position where it is centre assessment grades that are being awarded to students, then that is going to change some of the grounds for appeal, clearly. So, we need to continue to work through this and we're looking to update tomorrow our guidance around this. But, in essence, there will be grounds—again, it will be under the term 'wrong data'. So, there are two aspects to that: there could be a centre error in the data that they have supplied to WJEC—for example, the centre assessment grade—and what we would be clear on is that that cannot be a change in judgment; there needs to be evidence that there was an error. And the other aspect of the appeals process will be around WJEC procedural errors—so, for example, that there has been some issue in the way WJEC has transposed the data, that they haven't followed some conditions that we've put in place in order to get to this position. So, those will be the grounds for appeal.

Thank you, and—. I can't, Siân.

I just want to know if the Minister will have a look at those for final sign-off.

We can ask the Minister that in the next session. Hefin.

Just to recap anything you've said already, how are you going to plan for next year? What are plans A, B, and C for next year?

We certainly have plan A and plan B. Plan A is that exams continue as they would normally do, albeit that we've looked at, with WJEC, the assessment arrangements to make sure that they're appropriate for the circumstances. And I think one thing that has come out of this summer is the fact that exams are probably the best way forward. They're a system that people rely on and have confidence in, and all of our work looking at public confidence in exams supports that. But we know that there is the potential that we may be in the same situation again. Now, this time last year, it wasn't possible to foresee that we would find ourselves in these circumstances. So, everyone has been working very hard to do the best that they can, to come up with the best and the fairest solutions possible in the circumstances. We know that, next year, we could find ourselves in the same situation. Therefore, we have to look at how we can make the system more resilient and more robust. We will be working with WJEC, once the results are out, to look at what could be done to provide a framework and, potentially, materials for schools that can help to standardise at a school level and across schools, and to provide greater consistency between schools in their judgments, so that if these judgments need to come into play next year, we can place greater reliance upon them.

Okay. Hefin.

Could I just come back on that last point? So, what does that mean? You're not going to be using the standardised model; you'll be sampling. Is that what you just said?

I think we would be thinking about all of the options, but we would be thinking about all of the options now rather than doing so in the middle of a pandemic and trying to find the best way forward in the limited time that's available. So, what we will be doing is going back to the basics and thinking about all of the options that are available and having a plan B that is resilient, robust and as reliable as it can be if exams can't continue. 

How will that be arrived at? Will that be through engagement with unions, with stakeholders, with the Senedd? How's all that going to happen?

And will it involve engagement with young people?

Yes, absolutely. We'll be making sure that anything goes through consultation processes, certainly wider engagement and certainly with the involvement of teachers and headteachers. I think what we also have to remember, though, is we have to balance that, because an extensive engagement and consultation process might give us something that is too late to be implemented. So, we would need to move at speed, and there may be some compromises over some of the elements of consultation in order to get a solution that could be implemented, because, essentially, it would need to run through most of the next academic year; it's something that we couldn't leave until December to come up with.

Okay, thank you. Well, we have come to the end of our time. The committee is keen that we be kept updated on the planning for next year.

Can I just thank you all for your attendance today and for answering our questions? As usual, you'll be sent a transcript to check for accuracy following the meeting. I thank you all very much for attending today. Diolch yn fawr.

Can I propose, then, in accordance with Standing Order 17.42, that the committee resolves to meet in private for item 7 of today's meeting? Are Members content? Thank you. We will now proceed in private.

Can I welcome Members back to the Children, Young People and Education Committee for our final session of the day looking at the impact of COVID-19 on young people's examination results in 2020? I'm very pleased to welcome Kirsty Williams MS, who is Minister for Education, Georgina Haarhoff, deputy director of curriculum, and Sinead Gallagher, deputy director of higher education. Thank you very much for attending. We recognise this is an exceptionally busy time for all of you. Our intention as a committee is to provide constructive scrutiny and also to try and find out the support that's going to be available for children and young people in the days and weeks ahead. So, can I ask the Minister, then, to make an opening statement, please? 

Thank you very much, Chair. For our young people, just like everyone, the last few months have been, and continue to be, a stressful time. Many of us will know people who have been ill or have lost someone. I certainly do. It's been a time of anguish for people right across the country. And I am sorry that for some of our young people the results process has made that worse. That was not the intention of anyone; not me, not Qualifications Wales, not teachers, not the WJEC. But it is right that I apologise directly and unreservedly to our young people. I am truly sorry.

Recent months have thrown up unexpected, new and complex challenges. We've had to take decisions and design new ways of working at incredible pace: closing schools, cancelling exams and establishing a new results process. Qualifications Wales developed an alternative approach, and consulted publicly on it, to ensure that young people could receive their awards this year. Members of the committee will know that raising standards and providing targeted support and extending aspirations for students from disadvantaged backgrounds is what drives my reform programme for education in Wales. That is how we transform life chances, improve national prosperity and become a better country.

Working with Qualifications Wales and the WJEC, we sought an approach that provided fairness and balanced out differences in standards applied to judgments in schools. It brought together externally assessed prior attainment, teachers' and lecturers' estimates and a moderation of those across all centres to maintain standards. But as I announced yesterday, and given decisions elsewhere, the balance of fairness now lies with awarding centre assessment grades to students despite the strengths of the system in Wales.

I took the decision yesterday ahead of the results being published this week so that there is time for the necessary work to take place, and I'm pleased that the WJEC have confirmed that they are able to action this. For grades issued last week, all awards will also be made on the basis of teacher assessment. For those young people—around 17,000 grades—for whom the system produced the same or higher grades than those predicted by their teachers, the higher grades will stand.

Maintaining standards is not new for 2020, it's a feature of awarding qualifications every year in Wales and indeed across the United Kingdom. However, it is clear that maintaining confidence in our qualifications, whilst being fair to students, required me to make this decision. These have been, as we've heard many times, exceptional circumstances. Next week, I will be making a further statement on an independent review of events following the cancellation of this year's exams. I believe it's right that we wait until after this week's results before we review and learn lessons from any further disruption. This also gives the committee an opportunity, if it wishes, to offer advice on particular issues it would like considered following today's meeting and ahead of next week. 

To conclude, Chair, I want to congratulate and thank all of our children and young people. They have shown incredible strength and resilience during this pandemic, and I hope now that they can step forward, with confidence and ambition, into their next stages, whether that be education, employment or training. Diolch yn fawr.

Thank you, Minister, and thank you also for the offer that you made to the committee to feed into your review. We'll go now to questions from Members, and the first questions are on the decision-making and communication around the system, from Siân Gwenllian.

Thank you, Siân. At all stages of this process, we have tried to work in a timely fashion to protect the interests of Welsh learners, recognising that they are—especially at A-level—competing with students from across the United Kingdom. That's why we took the action that we did on the day before results, with regard to establishing the AS-level floor and to instructing a broadening of the appeal process.

Throughout this process, we have looked to ensure, wherever possible, that we could enhance the opportunities to address any flaws in the system, while trying to maintain, first, the currency of Welsh qualifications and, certainly, the ability of Welsh students to compete on a level playing field across the United Kingdom. When it became clear that confidence in the system, and the balance of fairness in the system, had shifted, we moved swiftly on Monday morning to make the adjustments to this week's qualifications and to signal the change to last week's qualifications.

No, not at all. We make our own decisions here in Wales. But, we have to be cognisant, do we not, of the fact that we share a qualifications system with other parts of the United Kingdom, and we do not want Welsh students to be disadvantaged. We, over the weekend, continued to pursue options, which included moving to centre-assessed grades if we felt that the balance of fairness had shifted. That was proactively worked on over the weekend, thus allowing me to make a decision on Monday morning, when it was clear that that was now the fairest way to proceed.

Yes, that's right, Siân. The timeline is an open and transparent one. This process began on 18 March, when I received advice from Qualifications Wales around the options for this summer's exams. Having taken the decision to cancel the exams because of the public health emergency, we issued a letter of direction outlining our approach and the principles by which Qualifications Wales were to deal with that situation. We continued to meet throughout this period with Qualifications Wales and take decisions to enhance their ability to do their job, so this isn't just about issuing directions to Qualifications Wales. So, for instance, agreeing that Welsh Government data would be made available to enhance their processes, whether that be key stage 3 teacher assessment or national test data. We also made amendments to the accountability regime, ensuring that teachers felt under no pressure from accountability to influence their centre-assessed grades. We then responded to the request by Qualifications Wales to approve their appeals and continued throughout this period to ensure that the principles outlined in our first direction were upheld. 

Well, it became clear to me—. I have checked the Record from the last time I was at the committee. The question was asked in relation to the speculation in newspapers of a significant amount of grade inflation and how grade inflation would be tackled, and I explained to the committee that the standardisation process was in place to deal with that situation. I became aware of the numbers involved on Monday of last week.

Thank you. Dawn Bowden.

Thank you, Chair. Can I just say to the Minister thank you for everything that you have done through this pandemic? It's been probably the most difficult thing that any Minister has had to do, and you were making decisions at a time when hundreds of people were dying, including people that were involved in the education system as well, and I think that needs to be recognised. Can I just ask you about when you were first alerted to the fact that England was about to change its system? Had you been in discussion with Gavin Williamson? I ask that because I'm interested to know if and how that impacted either directly or indirectly on your decision.

Thank you, Dawn, first of all, for the recognition that everybody involved in the education system, whether that be in our schools, Qualifications Wales or WJEC, or indeed in the education department here in Welsh Government, have been undertaking all of this work during that pandemic, and I'm grateful for your recognition of that.

Throughout this process, I have been in regular contact with all three other education Ministers in the United Kingdom through a regular series of quadrilateral phone calls. At the beginning of the pandemic, the Westminster Government also created a ministerial implementation group, which I joined from time to time if there were education issues on the agenda, and at least one of those included looking at the exam process. You'll be aware that I wrote on two occasions to Gavin Williamson around the date of the exam day because there was a push in England for the exams day to be brought forward and we wrote to urge the Westminster Minister to keep the exam dates the same, and we did that because we felt we needed to create that time for this process to be done properly, and any curtailing of the usual timing would be detrimental, and I'm glad that the traditional days became available.

I spoke to Gavin Williamson formally I believe on Tuesday of last week to discuss the impact of the decision that had been taken in Scotland on the Tuesday. So, as I said, the data became available to us on the Monday. I could hear Siân's gasp. I should say: in normal circumstances, as Welsh Ministers, we become aware of the data, of the results, at the same time the centres become aware of the data, of the results. That is not unusual; that's part of the safeguards in the system to stop political interference in the examination system. So, we saw the data, some of the data, on the Monday. I spoke to Gavin Williamson last on the Tuesday to discuss the impact of the Scottish decision, and I spoke to him again informally on Saturday. I had no—I certainly did not receive, and I'm not sure my officials received any formal notification of the decision to change in England. They were clearly grappling with the same issues we were. I gave a direction following discussions on the Saturday to prepare two scenarios: how we potentially could build safeguards into the GCSE system as a result of our learning when we saw the AS-level results, because many of the GCSEs were going to be standardised on the same model, and to prepare if necessary to go to centre-assessed grades, depending on, as I said, confidence in the system and the ability to protect the Welsh learners' interests.

Thank you for that and can I just ask—? In answer to Siân Gwenllian, you talked about when it became clear that confidence in the system had shifted and you said the balance of fairness had already shifted, so presumably those two things happened around the same time, when you started to see the data, when you saw the data. I'm not putting words into mouth, but I'm working on the assumption that that meant you saw the data and you thought, 'Oh, this wasn't what we wanted. This wasn't what we had expected', and so you moved to act in the light of that information. Can I ask how many people have been flagging with you as Minister their concerns that this could have been the outcome of the process that we were following in Wales, the assessment process? How many people: stakeholders, politicians, councillors, WLGA, schools, heads—you name it? Anybody who had an interest—how many of those have been flagging with you their concern that this could have been the outcome?