The Service-Learning Quality Assessment Tool (SLQAT) was designed to provide a mechanism to evaluate the quality of design and implementation for credit-bearing, academic service-learning courses. The tool takes into account 28 elements that the service-learning literature supports as essential for high quality service-learning promoting positive academic and other outcomes for students, and organizes these elements into five dimensions. Each element also has a numerical value or weight, representing the importance of its contribution to quality service-learning course design and implementation.
The SLQAT may be used for different purposes, such as instructor self-study, course design, faculty development, and as a research instrument providing dependent (outcome) or independent variables. Each of these purposes is valuable, but may imply different applications; for instance, use with faculty in creating a new service-learning course will likely focus on ensuring inclusion of all elements, rather than scoring per se.
For scoring uses, the SLQAT provides numerical values for each element, with a baseline value (ranging from 1.29 to 3.0) representing the hypothesized importance of that element’s contribution to service-learning course quality and implementation. (For instance, while both are important, Element #11, Academic Content Learning, carries a higher base value than Element #17, Appropriate Duration/Intensity of the Service.) If an element is absent, that component receives a score of zero. If present, depending on how well developed and implemented the element is, each element can be scored with different possible implementation “weights”:
A base (middle) score if there is evidence of adequate or baseline implementation;
Two possible greater values (adding 10% or 20% to the base score) for exemplary implementation;
Two possible lesser values (subtracting 10% or 20% from the base score) for partial or inadequate implementation.
Because every element is considered important for service-learning quality, a score of zero (absent) for any element will substantially reduce the overall final summed Quality Score.
Scoring is based on the overall evidence provided about the course (e.g., course syllabus, course assignments, descriptions of service-learning opportunities, interview or discussion with course instructor or campus service-learning staff, observations, evaluations, etc.). Instructions for how these scores are applied, and more information about data sources, are presented below.
While other stakeholder outcomes are also important for service-learning, this tool is focused on elements that influence student outcomes.
Some sort of service-learning activity is assumed to be a required component of the course being scored.
Each element is assumed to be essential to all types of course-based service-learning (regardless of scale and scope) in that it contributes to the overall quality of service-learning. However, not all elements are assumed to contribute equally to service-learning quality, represented in the base score values that indicate each element’s level of contribution.
Other factors likely influence the quality of service-learning courses and implementation (e.g., faculty teaching experience, size of the course, length of term, students’ prior experience with service-learning, access to transportation, community and institutional characteristics, etc.). These factors typically cannot be adjusted at the course level, or are out of the instructor’s control, and are not included in the SLQAT.
The information contained in the data sources analyzed (such as the syllabus) is assumed to represent actual practice in the delivery of the course, and they are assumed to be valid sources for determining the presence or absence of each element.
The course is assumed to have been taught prior to scoring. (For course development purposes, focusing on the elements, rather than attempting to ascertain a score, is appropriate.)
Higher scores on the SLQAT are assumed to represent a higher quality of service-learning course implementation, which in turn is assumed to produce more positive outcomes for students.
The SLQAT scoring is based on a review of both foundational data sources and of supplemental data sources.
The foundational sources for scoring the SLQAT are the course syllabus and all course-specific materials that are provided to students (e.g., assignment guidelines not incorporated into the syllabus; student contracts for service-learning; information about community partners, placements, or projects; pertinent service-learning handouts from the institution’s service-learning office, etc.).
Supplemental data sources for the SLQAT rating include at least one of the following: interviews with/statements from the instructor; information from the campus service-learning office, the community partner, and/or students who took the course; deliverables from the service-learning activity; student reflections; etc. If needed and available, more than one of these supplemental data sources should be secured and reviewed to help enhance the accuracy and confidence of ratings.
For “low-stakes” purposes (e.g., self-study, faculty development, etc.), the SLQAT may be used with only the foundational sources. However, these foundational materials alone will likely not provide sufficient evidence to determine the presence/inclusion of particular service-learning elements. (In this case, the ratings should be used primarily for discussion around areas of strength and of potential improvement, etc.; while the element weight scores could be summed for an approximate total score, this should not be considered reliable or valid.)
When using the instrument for research and evaluation purposes:
Foundational sources plus at least one supplemental data source (#2, above) must be included in the review and rating, and should be consulted to confirm the accuracy of the scoring of the course materials.
At least two raters should use this instrument to independently score a given course. This enhances objectivity within the evaluation as it provides a means to reduce potential rater bias or error while strengthening the reliability of the scoring process.
Depending on the intended use of the SLQAT, two rounds of scoring are recommended:
First, each rater carefully reviews the initial course syllabus and course-specific materials (#1 foundational sources above), at minimum. Each rater independently scores each element in the SLQAT, noting evidence supporting each rating. For elements where the data provided do not allow the rater to decide if the element is truly “absent”, a preliminary indication of “insufficient evidence to rate” may be noted, with no score assigned (i.e., leave blank).
Next, the raters’ individual assessments should be compared, and then through discussion between the raters and consultation of all course materials and supplemental data sources available, an agreed-upon final score for each element should be determined. If a consensus cannot be agreed upon then the average score of the individual raters will be the final score for that element. For this final scoring, no rating of “insufficient evidence to rate” should be included; instead, a score of zero (0) should be assigned for any element which is determined to be absent, or which is still not evident from the thorough review and discussion of the full set of available data sources.
To establish a total Service-Learning Quality Scorefor the course, the scores for each of the 28 individual elements are summed. Because these elements have different base values representing their contribution to quality, and these values are modified by level of implementation, the overall SLQAT score for a given course could range from zero (none of the elements present) to 70.15 (all elements present and exemplary). Element ratings of zero (“absent”) will also substantially impact the final score. While additional research will be needed to provide scoring norms and comparisons, we propose an initial interpretation as follows:
Exemplary implementation: an overall score of 60 and above.
Adequate implementation: an overall score between 50 and 60.
Emergent implementation: an overall score of below 50.
For more informal uses (e.g., self-study by an instructor, instructional review of syllabi, faculty development), the scoring process and procedures may be adjusted, keeping in mind that such adjustments may alter the reliability or validity of the scores obtained.
To use the SLQAT to rate a service-learning (SL) course, begin by considering the descriptor and question to decide if there is evidence of each element’s presence in the course.
Upon first review, if no evidence is available or provided, leave the rating blank.
If the evidence provided is sufficient to determine presence/absence, but the element is not present, assign a score of zero [“0”] for this element.
If evidence is provided that the element is present in the course, review each of the guiding statements to decide the quality of implementation or presence of the element. Select the best statement and assign the score associated with the statement that is best aligned with the quality/level of the element’s presence and implementation, given the information and data reviewed about the course. (These scores represent +20%, +10%, baseline, -10%, or -20% of baseline.)
Where possible, for each element, enter comments regarding the particular evidence that was used to justify the score assigned.
The raters’ individual assessments should be compared, and then through a conversation between the raters and review of all evidence, a final agreed upon consensus score for each element should be determined (if consensus is not possible, the recommended alternative is to calculate the average of the raters’ scores).
After a final score for each element is determined, these are summed to establish the overall Service-Learning Quality Score.