ANZCA MCQ Papers -Preparation & Marking
Primary Exam Changes
The description below outlines the standard ANZCA process for determining your mark on the MCQ paper. This may still be the way that the Final Exam MCQ paper is processed BUT the PRIMARY EXAMINERS have recently indicated the Primary MCQ Papers marking process will NOT be standardised. The effect of this is said to be "considerably higher MCQ scores", and of course they will know what will happen as they must have checked by comparing the two processes (i.e. with and without standardisation) on previous exams.
Because of this, they have changed the cut-off or minimum marks required in the MCQ and SAQ sections in order to be asked to attend the viva exams. The viva component of the exam is quite expensive to run (because of venue hire, examiner's expenses for airfare and accommodation), and of course expensive for the candidate as well (unless you happen to live in Melbourne). Consequently, there is thought to be little benefit in inviting candidates who cannot pass (based on poor scores on the MCQ and SAQ sedctions).
So from 2016 on-wards, you will need a score of 60% (or more) on the MCQ AND 40% or more on the SAQs.
What do they mean by standardisation? My initial assumption is it the process described in Step 4 below, as that is the context where the term "standardisation" has been used previously by ANZCA. This may be be wrong, though as I cannot see how removing just this step could result in such increased MCQ scores. The 'standardisation' of step 4 is (said to be) about adjusting for variation in difficulty of the exam as compared to previous ones. So if the current exam was more difficult, then your score would be adjusted upwards (and the opposite for an easier exam). So removing this upwards adjustment for difficult exams results in a lower score. There must be more to it than this. A fuller description of the change would be useful for transparency. Such poorly documented changes can result in adverse outcomes for candidates.
Determining the Score for each Candidate
Step 1: Mark the papers
The collected MCQ scoring sheets are marked electronically. This results in a "initial score" for each candidate.
Step 2: Start eliminating poorly performing MCQs
After the ANZCA MCQ papers are marked, a "point-biserial coefficient" (PBC) is calculated for each question.
- The PBC is a "discrimination index" because it providing a measure of how well each individual MCQ is at 'discriminating' the 'good' (knowledgeable) candidates from the 'poor' (less knowledgeable) candidates.
- This index can have a value from -1 to +1.
- Good questions have PBC >0.4 (>0.6 is really good).
All the MCQs with negative PBCs are eliminated from the marking. (A negative BSI means you are more likely to get the MCQ right if you don't know the answer - this is not considered good).
Step 3: Repeat this process until all poor MCQs are eliminated
- All the remaining MCQs are put through the program again and PBC values are calculated again.
- This process repeats until there are no MCQs with negative BSIs.
- All then have BSI >= 0 after recalculation.
- Since the part 2 Bank was improved I am told that typically only 1 to 3 questions are found with negative BSI after the first run.
- After removal of these and recalculation there are nearly always no MCQs with a negative BSI so the process is complete.
- The remaining score for each candidate is now referred to as the "raw score"
Step 4: Minimise variation due to variations in difficulty of different MCQ papers
The candidates' scores are "standardized", that is the 'raw scores' are converted to a distribution with a mean of 56% and a standard deviation of 12. The purpose of this process is to allow for variation in the difficulty of the Examination compared to previous Examinations.
Next a "KR20 reliability coefficient" is calculated. For ANZCA exams, this typically has a value of about 80.
Step 5: Correct for variations in quality of different exam cohorts
The process is known as the "The Marker Adjustment Process". It is done to allow for the fact that the 'ability' of each sitting cohort of candidates is different for different exams. Each exam paper has "marker questions" (the best MCQs from the previous exam) and the score on these questions is used to assess the candidate variability as a group and attempts are made to compensate for this.
After this, the scoring process is then complete and each candidate has an MCQ score that is a true reflection of their knowledge (ideally).
Some more details of this process as used for the Final ANZCA MCQ paper. This paper has 150 type A MCQs. This information was provided by ANZCA:
Fifty of the 150 questions in the current MCQ examination have been included as 'Marker Questions'. These questions were selected from the previous examination and repeated in the current examination based on their having the best point-biserials in that examination (i.e. they were the best discriminators). . For many reasons, the performance (mean difficulty) of the current cohort on the Marker Questions tends to be better than in the previous examination from which the questions were selected. This difference in Marker performance is called the "Current Marker Difference". . A moving average of the values of previous "marker differences" from the last 10 examinations is determined (called the "Expected Marker Difference"). The Expected Marker Difference varies from examination to examination because it is a moving average, but is usually 4-8 percent. The Expected Marker Difference is subtracted from the Current Marker Difference. If the current cohort of candidates are on average better performers than the cohort who sat the previous examination, then the Current Marker Difference would be greater than the Expected Marker Difference, and a positive figure (the Marker Adjustment) would result. Likewise, if the current cohort of candidates were on average poorer performers than the cohort who sat the previous examination, then the Current Marker Difference would be less than the Expected Marker Difference, and a negative figure (Marker Adjustment) would result. The Marker Adjustment value (usually within the range -2 to +2%) is added to every candidates' standardized score. . For example, if the Expected Marker Difference is 7 (over the past 10 examinations) and the Current Marker Difference for this examination is 8, then this cohort of candidates have performed 1 percent better on their marker questions. Thus the assumption is made that they are a more able group of candidates than the 'average' and the marks of all candidates in the current examination are raised by 1%. . Conversely if the Expected Marker Difference is 7 over the past 10 years and the Current Marker Difference for this examination is 6, then these candidates have performed 1 less on their marker questions. Thus the assumption is made that they are a less able group than the 'average' and the marks of all candidates in this group are reduced by 1%.
Step 6: Final score
The value for each candidate is the final percentage MCQ score. The Pass Mark is set at 50%.
The scores are then scaled to a number out of 25 (as this is the amount of marks available for the MCQ component of the exam.) For example if you obtained 50% then you receive a mark of 12.5.
Preparing the MCQ Paper
Before the next exam, a sub-group of the examiners prepares the next MCQ paper.
The Finals MCQ paper has 150 type A MCQs. These questions are selected as follows:
- Marker MCQs: The 50 MCQs with the highest PBCs from the exam just finished
- New MCQs: 50 MCQs are selected from among the new ones submitted by current Primary Examiners.
- Repeat MCQs: 50 MCQs are selected from the college bank.
An undocumented 'content balancing process' is undertaken when selecting the 100 non-marker MCQs from the 'College Bank' as the 50 markers may be unbalanced (i.e. more content in one area & less in another). This may just be based on personal judgement.
The content of the paper is then checked, the paper set and stored in electronic format on the College computers. I have heard that the Exam computer is separate from the college network but I am not entirely sure this is correct as I have heard that some MCQ Examiners have access from home.
Consider the situation of the MCQ coded as AB38:
- AB38 [2001-Apr ] MCQ-63 and [2001-Sep ] MCQ-55 and [2002-Mar ] MCQ-54
This MCQ has appeared on three consecutive papers. This means that when first asked in April 2001, it was one of the top 50 MCQs. Thus it was used again on the next paper and performed in the top 50 again, so was retained for yet another exam.
Conversely if an MCQ appears once then never again, it is because it was a poor performer, though possibly it may have been culled as being a stupid question. Such MCQs are usually eliminated from the ANZCA Bank.