Appendix A – Evaluator’s Companion Report
Report on the 2016 Decision Quality Evaluation: Refugee Appeal Division, Immigration and Refugee Board of Canada
Prepared by: Doug Ewart
September, 2016
Introduction
This Report provides an overview of the 2016 Decision Quality Evaluation of the Refugee Appeal Division of the Immigration and Refugee Board of Canada (the
RAD). Part I outlines the evaluation process and offers some context for and general commentary on the evaluations of individual decisions undertaken for this project. Part II sets out the interpretative guide that was developed for this evaluation. It then goes on to provide a relatively detailed assessment of the evaluation instrument that was used and to put forward a number of specific suggestions for the Board’s consideration for future evaluations.
Part I
The Project
The purpose of this project was to provide an independent and objective assessment of a sample of
RAD decisions using a set of evaluative criteria developed by the Board.
Although other Divisions of the Board have been assessed in this fashion for a number of years, this was the first assessment of the more-recently created
RAD. It was also the first to apply the Board’s evaluation criteria to an appellate process that is overwhelmingly based on a review of a file rather than on a hearing. As a result, the evaluator was asked to submit comments along the way about the applicability of the criteria, developed in oral hearing contexts, to this exercise.
It is to be stressed the evaluation was not designed to, and at no time did, consider whether a
RAD decision seemed correct or reasonable. The evaluation criteria focus only on how an appeal was conducted and how the decision was written. Ratings and comments were provided without regard to any views the evaluator may have had on the result in a given case.
Sixty-one
RAD decisions were evaluated over the course of the project. Four arose from oral hearings and the balance from exclusively paper proceedings. The geographic and linguistic distributions of the paper process decisions that were evaluated closely tracked the distribution of the
RAD’s caseload among the Vancouver, Toronto and Montreal offices, and between English and French proceedings. All four of the oral hearing matters arose from the Toronto office and were English language proceedings.
It is of note that towards to end of the time period from which the evaluated decisions arose the Federal Court of Appeal released two very significant decisions affecting the role of the
RAD. One provided important clarifications of the scope of an appeal at the
RAD, and the other clarified the approach to be taken to new evidence put forward in an appeal.
The evaluations took into account the fact that the legal expectations of Board Members accordingly changed during the review period.
The Process
This evaluation proceeded in three phases, on a part time basis, from early June through to September 2016: 1) attendance at the training program for new
RAD Members; 2) the evaluation of 61
RAD decisions, including the development, part way through the process, of an interpretative guide to the existing evaluation criteria and the development of additional criteria for the evaluations; and 3) the preparation of this final report.
The evaluations of the paper process decisions used a standard set of 13 criteria (20 after the additional criteria were added), each with either a three-level rating scale or a Yes-No assessment, depending on the nature of the criterion in question. As well, narrative comments could be, and frequently were, added to the numeric assessment of individual criteria. These comments were used to explain the rating or the relationship between or among the various ratings for different elements of a specific decision, or to note anomalies or issues of particular interest arising from the decision in question.
Some of the criteria provided for the evaluation proved challenging. Accordingly, after the first 9 evaluations had been completed an interpretative guide to the criteria was developed to address those challenges. It was used from then on, and the initial evaluations were adjusted as necessary in light of the new interpretations. As well, seven new evaluation criteria were added to the assessment process on a pilot project basis and were applied to all of the decisions evaluated.
Draft evaluations were submitted at intervals to provide the opportunity for feedback on the approach adopted, including the nature and level of detail in narrative comments.
Scope of Review
The files that were provided for the evaluation constituted the complete record of each proceeding before the
RPD as well as before the
RAD. The files ranged from about 2 inches in thickness to as many as 8 or 9. In the interest of economy, and based on the Board’s experience with other decision-evaluation exercises, it was determined that the evaluator would not review the entire file for a case.
Instead, for the first 25 paper process matters the evaluator reviewed in detail the Basis of Claim form, the
RPD decision, the Appellant’s Memorandum and the
RAD decision. In those rare cases where the Minister participated, the Minister’s Memorandum and the reply Memorandum, if submitted, were also reviewed. This meant that the hearing before the
RPD was not listened to, and that the administrative paperwork and the documentation on the
RPD and
RAD files was generally not considered in the evaluation.
In the interests of further economy, and because it was not providing important information for the evaluation beyond what was set out in the other materials, the Basis of Claim form was not reviewed after the first 25 evaluations had been completed.
The review process for the four appeal decisions where the
RAD had held a hearing differed in three ways. The
RAD hearing was listened to in its entirely; the
RAD’s administrative documentation leading up to the hearing was reviewed; and a more extensive set of evaluation criteria was applied.
Observations on the Process
1) Assistance of the Policy, Planning and Corporate Affairs Branch
Throughout the process the evaluator benefited significantly from the advice and support of the Policy, Planning and Corporate Affairs Branch of the Board. First, and most obviously, they developed the approach, and provided the criteria that were so important to the success of the exercise. As well, they arranged for participation in the
RAD training that turned out to be central to a full appreciation of the context behind many of the assessment criteria.
Officials in the Branch also provided a very helpful orientation to the checklist and the evaluation process overall, and were always available to assist with any questions as the evaluations proceeded. As noted above, they were also open to the creation and use of an interpretative guide to the assessment criteria to reflect the specialized features of the
RAD appellate process and supported the addition of seven additional criteria on a pilot project basis.
2) Value of the Criteria
While the initial criteria required some interpretation to take into account the unique aspects of the
RAD process, and to add some clarity where criteria appeared to overlap, they proved to be essential to the evaluation process. They focused and disciplined the assessment of each decision.
Indeed, the initial impression created by a review of a file and a
RAD decision not infrequently changed when specific criteria were applied to the decision. This very much parallels what adjudicators and judges have often stated: the process of writing a decision can change the result the decision maker had in mind after hearing the matter but before starting to write the decision. The use of specific criteria and a rating scale supported a reasoned assessment of each decision and accordingly played an important role in increasing the rigour of the assessment process.
As well, the ability to add written comments to the numeric assessments proved to be extremely valuable. The discipline of adding comments to explain or provide context for a numerical rating frequently had the effect of altering the rating. Thus, in addition to what value the comments may have when considered on their own, they clearly produced more accurate ratings on the numeric indicators.
3) Value of the Interpretative Guide
The guide proved to be very helpful, and sometimes essential, to providing consistent, non-overlapping applications of the various criteria. As well, once it was available as a reference the assessment of each file took appreciably less time.
4) Order of Review
For the initial assessments the evaluator reviewed the documents in chronological order, starting with the Basis of Claim form and ending with the
RAD decision. This followed the approach that a senior
RAD Member had indicated was applied to the appellate process itself. Part way through the evaluation process the order was reversed to determine if such an approach would contribute to a faster review process without compromising the quality of the evaluations. As it appeared to do so the approach was maintained throughout the balance of the project.
5) Numeric Ratings
While, as noted above, the use of a numeric rating system was important to the evaluation process, the ratings scales sometimes proved problematic. As is discussed in more detail in Part II, there were times when, because of the nature of a particular decision, almost any rating on the 3-point scale could be misleading. There were also criteria for which a Yes-No assessment seemed less helpful than the 3-point rating scale.
As well, the application of the 3-point scale to the criteria as currently worded can induce more subjectivity into the evaluation than is perhaps desirable. Basing a numeric assessment of criteria such as ‘plain’ or ‘easily understood’ on non-specific terms such as ‘acceptable achievement’ or ‘above-average’ assumes a shared understanding of these terms between the evaluator and the readers of the evaluation. It may also make it difficult to take into account changes in evaluators over the years.
As a result, the Board may wish to consider developing specific assessment standards for each criterion. A number of these are suggested in Part II of this Report.
As well, the evaluator was advised at the outset that the approach to numeric ratings was to be a relatively generous one, and in particular that there needed to be fairly serious flaws before a rating criterion could be said not to have been met. While this is a matter for the Board’s discretion, it could be that aggregated conclusions based on such an approach may fail to identify areas where, while decisions are not seriously flawed, improvements are very much to be desired.
6) Efficiency
In all, having clear interpretations of the criteria used in the evaluations, starting the review with the
RAD decision, and eliminating the review of the Basis of Claim form substantially expedited each assessment.
Some Very General Observations on Decisions
It is understood that the Board will use an aggregated assessment of the 61 evaluations as the basis to draw conclusions about the quality of
RAD decisions overall. It follows that un-aggregated, general observations have a limited role in assisting the Board move towards its objective of high quality decisions.
Nonetheless, in the event that they may be useful to the Board, a few general observations are set out here.
1) Quality in Relation to Available Time
The evaluations for this project were completed on the basis of the record and the decision alone. There was accordingly no basis to take into account contextual factors such as the time available to write a decision.
Since there can be a tendency in reviewing evaluations to focus on areas needing improvement, it is important that concerns be considered in relation to the workload expectations placed on Members, and accordingly the amount of time they have to write, and especially to re-write, their decisions. Given the expectation that, on average, Members will review a file and complete their decision in two working days certain identified shortcomings in decision writing could well be said to be an understandable consequence of that expectation.
2) Extensive Analyses of Legal Issues
Some Members showed a tendency to write at length about the legal standards that define the role of the
RAD or the legal standards for new evidence or oral hearings, or the law establishing legal standards for substantive matters such as credibility assessments or state protection. While the justification for this practice is outside the scope of this evaluation, it is of note that these analyses sometimes dealt with aspects of those standards that seemed to have no relevance to the decision in question. For example, a decision might discuss in some detail the appellate role of the
RAD on credibility issues even though no such issues arose in the appeal, or discuss the legal standards for an aspect of the new evidence test that similarly was not relevant to the specific case.
These practices raise two issues. First, where the analysis addresses matters not relevant to the appeal it can give the parties the impression that the decision is a generic one rather than one focused on their particular circumstances and concerns. And, second, even where all issues dealt with in a legal analysis can be said to pertain to the appeal, writing at length on these matters requires the reader to work through a quite-complicated and lengthy legal analysis before getting to the decision on the merits. In most cases this will not contribute to the Appellant’s ability to understand the decision.
3) Use of Preliminary Overviews
Moreso in the Montreal decisions than the Toronto decisions reviewed there was a tendency to start decisions with separate, and sometimes relatively lengthy, overviews of the
RPD decision, the Appellant’s arguments and/or the facts. In some instances little of this was used in discussing the merits of the appeal; in others, significant parts of it were repeated in that analysis. Rarely, if ever, did this practice make the decision easier to comprehend.
4) Demonstrating the Deliberative Process
In some instances it appeared that a Member had written a significant amount of text in coming to a decision that then turned out not to be necessary to the decision that was made. They then left the text in the decision rather than editing it out of the final draft.
Similarly, some decisions appeared to be written to show the deliberative process that was gone through to get to the decision rather than being written to explain and support the decision that had been made. This tendency to write towards a decision to be made rather than from decision that has been made often added extraneous material and made the path to the result more difficult to follow.
It should be stressed that these are not serious flaws in decisions. They do not reflect concerns about the completeness of the reasoning process, but instead reflect writing styles that can make it more difficult for a reader, especially a lay reader, to get through a decision and readily appreciate the bases on which it was made.
Other Matters
1) Training on the evaluation criteria
Based on attendance at the
RAD training program for new Members, and a review of the materials provided to Members in the sessions on decision-writing, it appears that the criteria used in Board evaluations of decisions are not provided to, or at least not taught to, the Members. If this is not addressed elsewhere, then given that the criteria have been approved by the Board as formal indicators of decision quality there would seem to be some logic in including the criteria in the training for Members.
This is not to say that similar aspects of decision-writing are neglected in the training, but only to suggest that making clear to the Members the Board’s formal indicia of decision quality could be very helpful to advancing that goal.
2) Substituting Decisions and Providing Directions
There appeared to be limited use of the
RAD’s power to substitute a decision or to provide directions for a rehearing. The former is of course determined to a considerable extent by the state of the record of the
RPD proceedings, but there could still be value in measuring whether the use of the power changes as the
RAD develops over time.
Similarly, there was very limited use of the power to provide directions for a rehearing at the
RPD. This would seem to be a missed opportunity to add value to the appellate process, and efficiency at the rehearing, by providing expert guidance based on a review of the entire record and of the specific errors found at first instance. It too might be something that the Board would like to measure in future evaluations.
3) New Members
It appears that the recently-appointed
RAD Members differ from the existing complement of Members in not having previously served as
RPD Members. The coming year accordingly provides an opportunity to assess whether there are any statistically significant differences in decisions based on that distinction.
Conclusion
To paraphrase a well-known author, all sound decisions are sound in the same way, but all flawed decisions are flawed in their own way.
This reality fully validates the detailed assessment criteria used by the Board in this important exercise. This approach moves beyond impressionistic assessments. It not only disciplines the evaluation process but as well provides the Board with quite specific information on areas where increased training and or new standards will produce significant quality enhancements.
The Board is to be congratulated for undertaking this kind of review, something that is regrettably rare in other parts of the administrative justice world. It was a pleasure to have the opportunity to take part in this exercise, and it is hoped that the results of the review, as well as the specific suggestions for future such exercises, are of value to the Board.
Part II
Part II of this Report begins by setting out the interpretative guide developed for this evaluation. It then goes on, in a separate section, to review and offer comments on the criteria used in the evaluation, both those initially provided (Question #1 to Question #34) and those added as the process unfurled (Question S1 to Question S7). [Note: The separate section of Part II was submitted to the
IRB and included a detailed analysis of the checklist. Because of the technical nature of that analysis it is not included here and instead will be brought forward when updates to the methodology are considered.]
Interpretative Guide to the Checklist for One Party (Appellant) Appeals - July 18, 2016
‘Returned with instructions’ means substantive directions beyond being heard by a different panel. May differ from disposition record on the file.
Assesses how the Member proceeds when raising a new issue from the record or where specialized knowledge is used (see
RAD Rule 24) or where a change in country conditions is relied upon. Per Legal Services, it is a breach of natural justice to deal with an issue that was not addressed in the
RPD decision or the appellant’s memorandum, even if fully canvassed at the
RPD hearing, without giving notice and an opportunity to make submissions. [Note that Question
S2 addresses
whether a Member raises a new issue; i.e. goes to the thoroughness of the review rather than to procedural fairness].
A bare statement of the issues is treated as falling short of the standard and rated as 1, with a basic summary as 2 and a particularly helpful summary as 3. An example of a bare statement of the issues would be: “The issues in this appeal are credibility and whether there is an IFA”.
Means that the Member addresses the parties’ arguments positions, with the scale assessing whether the thoroughness with which they are addressed matches their relevance to the outcome. The question does not assess whether the findings of the
RPD are addressed.
Note that credibility overall is treated as a fact for the purposes of this question and Question #6.
Note that the issue is findings of fact, not findings of error.
This will apply less frequently where the appeal is allowed.
Includes case law: answer in the negative when the only case law referred to is that concerning the
RAD’s jurisdiction.
On guidelines, the issue is whether the standard is referred to and its application explained, not whether it is formally cited.
Note that there are no Jurisprudential Guides or persuasive decisions for
RAD.
The core issue is whether potential biases concerning socio-cultural background were adverted to and filtered out from the assessment of how the claimant’s or another witness’s evidence presented at the
RPD.
Does not include taking into account the cultural and social conditions of a country.
This does not involve a formal plain language assessment. A decision will be rated 2 unless it used challenging, technical or obscure language or was written in a particularly complex structure.
To avoid an overlap with Question #10, ‘clear’ will address whether the reasoning, as opposed to the language or structure used, is clear: this question assesses whether, despite some difficult language or structure, the reasons explain the conclusion(s).
To avoid an overlap with Question #13, ‘conciseness’ is treated as referring to the length of the analysis of each issue as opposed to the length of the reasons overall.
Given that ‘easily understood’ and ‘logically-sequenced’ could be seen to overlap with ‘clear’ in Question #11, the approach will assess whether the reasons are
structured to be easily understood: i.e., easily understood because logically structured and ideally sign-posted throughout.
Given the potential overlap with Question #11 regarding conciseness, the assessment will look at the length of the reasons overall and whether matters not necessary to support or explain the decision were dealt with.
Will also consider the application of the appropriate tests for an oral hearing if new evidence is admitted.
The assessment will look for
whether the member finds a new issue in the record. [Compare to Question #1]
The assessment will look for a clear articulation of the distinct issues in the appeal and for reasons that address all relevant considerations and information under each distinct issue in question---evidence, law, submissions,
RPD findings and
RAD conclusions.
The assessment looks for focused reasons that demonstrate and ideally explain a clear line between each issue and how it was resolved. A decision could meet Question S3 in that it provides an issue-by-issue analysis but not meet this standard.
The assessment will consider whether the reasons do more than support the result and instead explain to the Appellant how and why they lost or won,
The standard is applied less strictly in cases where the Appellant won their appeal.
The assessment will consider whether the reasons provide information that is likely to assist the
RPD to determine the matter, if returned, and/or to assist others in resolving similar issues once the decision is posted on CanLII.
The assessment will consider whether this is demonstrated by independent engagement with the evidence; by an independent analysis of credibility; by references to the recording, submissions or additional documents from the record; and by the avoidance of terms like “the
RPD reasonably found’ or ‘it was open to the
RPD to find’. It may also be expressly stated as the approach used.
Note that the FCA decision conclusively deciding the
RAD’s role was released on March 29, 2016, in the midst of the time period from which decisions for this evaluation were drawn. As a result,
RAD decisions may reasonably vary on this issue depending on whether released before or after the FCA decision.