Combining course- and program-level outcomes assessments through embedded performance assessments at key courses: A proposal based on the experience from a Japanese dental education program
This paper addresses how to combine the course- and program-level assessments and presents a new method illustrated by a case of dental education program in Japan. Performance assessments are considered effective for evaluating knowledge integration and higher-order skills, while placing a burden on faculty, hence their feasibility as the program-level assessment is regarded lower than standardized tests or questionnaire surveys. We have developed several performance assessments at the course level, such as Modified Triple Jump for the PBL course. Based on this experience, we propose Pivotal Embedded Performance Assessment (PEPA) as a method for combining assessment at the course and program levels. The method limits the range of performance assessment to key courses directly linked to program goals and placed at the critical juncture points of curriculum, while entrusting the assessment of other courses to expert judgment of individual teachers. PEPA consists of the following procedures: systematization of curriculum and selection of key courses; design and implementation of performance assessments by a faculty team; setting passing criteria with incorporating the function of formative assessment; certifying the completion of the degree program. PEPA thus enables maintaining assessment feasibility and compatibility with a credit system, while ensuring assessment validity and reliability.
Received: 27 September 2018
Accepted: 13 November 2018
Published online: 29 November 2018
Akiba, Nami, Masako Nagasawa, Kazuhiro Ono, Takeyasu Maeda, and Katsumi Uoshima. “An Introduction to the Undergraduate Comprehensive Model Practice Course at the Faculty of Dentistry, Niigata University.” The Journal of Japanese Dental Education Association 33 (2017): 106-14. [In Japanese.]
Alverno College Faculty. Student Assessment-as-Learning at Alverno College. Milwaukee: Alverno College Institute, 1994.
Banta, Trudy W., and Catherine A. Palomba. Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education. 2nd ed. San Francisco: Jossey-Bass, 2015.
Barrows, Howard S. “The Essentials of Problem-Based Learning.” Journal of Dental Education 62 (1998): 630-33.
Biggs, John, and Catherine Tang. Teaching for Quality Learning at University. 4th ed. Berkshire: The Society for Research into Higher Education & Open University Press, 2011. Kindle.
Blake, Jennifer M., Geoffrey R. Norman, and E. Kinsey M. Smith. “Report Card from McMaster: Student Evaluation at a Problem-Based Medical School.” The Lancet 345 (1995): 899-902.
Cummings, Rhoda, Cleborne D. Maddux, and Aaron Richmond. “Curriculum- Embedded Performance Assessment in Higher Education: Maximum Efficiency and Minimum Disruption.” Assessment & Evaluation in Higher Education 33, no. 6 (2008): 599-605.
Earl, Lorna M. Assessment as Learning: Using Classroom Assessment to Maximize Student Learning. Thousand Oaks: Corwin Press, 2003.
Ewell, Peter, Karen Paulson, and Jillian Kinzie. Down and In: Assessment Practices at the Program Level. Champaign: National Institute for Learning Outcomes Assessment, 2014. http://www.learningoutcomeassessment.org/documents/NILOAsurveyreport2011%20-%20Down%20and%20In%2010-20.pdf.
Fujii, Noritaka, Syoji Takenaka, Koichi Tabeta, Naoko Sato, Nami Akiba, Yohei Oda, Yuji Katsumi, Kazuhiro Ono, and Takeyasu Maeda. “Competency Assessments for Undergraduate Students in Clinical Clerkships at the Faculty of Dentistry, Niigata University.” The Journal of Japanese Dental Education Association 33 (2017): 4-11. [In Japanese.]
Kuh, George D., Natasha Jankowski, Stanley O. Ikenberry, and Jillian Kinzie. Knowing What Students Know and Can Do: The Current State of Student Learning Outcomes Assessment in U.S. Colleges and Universities. Champaign: National Institute for Learning Outcomes Assessment, 2014. http://www.learningoutcomeassessment.org/documents/2013%20Abridged%20Survey%20Report%20Final.pdf.
Lane, Suzanne. “Performance Assessment: The State of the Art.” In Beyond the Bubble Test: How Performance Assessments Support 21st Century Learning, edited by Linda Darling-Hammond and Frank Adamson, chap. 5. San Francisco: Jossey-Bass, 2014. Kindle.
Matsushita, Kayo. “Making Learning Outcomes Visible.” Japanese Journal of Higher Education Research 20 (2017): 93-112. [In Japanese.]
Matsushita, Kayo, Kazuhiro Ono, and Yusuke Takahashi. “Development of a Rubric for Writing Assessment and Examination of Its Reliability.” Journal of the Liberal and General Education Society of Japan 35, no. 1 (2013): 107-15. [In Japanese.]
Middle States Commission on Higher Education. Student Learning Assessment: Options and Resources. 2nd ed. 2007. https://www.msche.org/publications/SLA_Book_0808080728085320.pdf.
Miller, George E. “The Assessment of Clinical Skills/Competence/Performance.” Academic Medicine 65, no. 9 (September Supplement 1990): S63-S67.
Ministry of Education, Culture, Sports, Science and Technology. “Japanese University Reforms Including Those in Educational Contents.” November, 2017. http://www.mext.go.jp/a_menu/koutou/daigaku/04052801/__icsFiles/afieldfile/2017/12/13/1398426_1.pdf. [In Japanese.]
Model Core Curriculum Revision Coordination Committee and Model Core Curriculum Revision Specialist Research Committee. Model Core Curriculum for Dental Education: AY 2016 Revision. 2017. [In Japanese.] http://www.mext.go.jp/component/b_menu/shingi/toushin/__icsFiles/afieldfile/2017/12/26/1383961_02_3.pdf.
Mtshali, Ntombifikile G., and Lyn Middleton. “The Triple Jump Assessment: Aligning Learning and Assessment.” In New Approaches to Problem-Based Learning: Revitalising Your Practice in Higher Education, edited by Terry Barrett and Sarah Moore, 187-200. New York: Routledge, 2011.
Newman, Mark J. “Problem Based Learning: An Introduction and Overview of the Key Features of the Approach.” Journal of Veterinary Medical Education 32 (2005): 12-20.
Oda, Yohei, Kazuhiro Ono, Noritaka Fujii, Tadaharu Kobayashi, and Takeyasu Maeda. “Development and Use of a Web-Based E-Portfolio for Dental Clinical Training.” The Journal of Japanese Dental Education Association 33 (2017): 65-73. [In Japanese.]
Ono, Kazuhiro, Kayo Matsushita, and Yugo Saito. “Prospects for Direct Assessment of Problem Solving Competence: Development of Modified Triple Jump in Problem-Based Learning.” Journal of the Liberal and General Education Society of Japan 36, no. 1 (2014): 123-32. [In Japanese.]
Ono, Kazuhiro, and Kayo Matsushita. “Assessment of Writing in First-Year Education.” In Assessment of Active Learning, edited by Kayo Matsushita and Terumasa Ishii, 26-43. Tokyo: Toshindo, 2016. [In Japanese.]
_____. “PBL Tutorial Linking Classroom to Practice: Focusing on Assessment as Learning.” In Deep Active Learning: Toward Greater Depth in University Education, edited by Kayo Matsushita, 183-206. Singapore: Springer, 2017.
Pike, Gary R. “Limitations of Using Students’ Self-Reports of Academic Development as Proxies for Traditional Achievement Measures.” Research in Higher Education 37, no. 1 (1996): 89-114.
Rhodes, Terrel. Assessing Outcomes and Improving Achievement: Tips and Tools for Using the Rubrics. Washington D.C.: Association of American Colleges and Universities, 2009.
Rhodes, Terrel, and Ashley Finley. Using the VALUE Rubrics for Improvement of Learning and Authentic Assessment. Washington D.C.: Association of American Colleges and Universities, 2013.
Richman, W. Allen, and Laura Ariovich. All-in-One: Combining Grading, Course, Program, and General Education Outcomes Assessment. Champaign: National Institute for Learning Outcomes Assessment, 2013. http://learningoutcomesassessment.org/documents/Occasional%20Paper%2019%20FINAL.pdf.
Rohlin, Madeleine, Kerstin Petersson, and Gunnel Svensäter. “The Malmö Model: A Problem-Based Learning Curriculum in Undergraduate Dental Education.” European Journal of Dental Education 2 (1998): 103-14.
Saito, Yugo, Kazuhiro Ono, and Kayo Matsushita. “Correlations of Direct Measures Based on Performance Assessment and Indirect Measures Based on Student Self-report.” Japan Journal of Educational Technology 40 (Suppl.) (2016): 157-60. [In Japanese.]
Suskie, Linda. Assessing Student Learning: A Common Sense Guide. 2nd ed. San Francisco: Jossey-Bass, 2009. Kindle.
Winning, Tracy, Elaine Lim, and Grant Townsend. “Student Experiences of Assessment in Two Problem-Based Dental Curricula: Adelaide and Dublin.” Assessment & Evaluation in Higher Education 30, no. 5 (2005): 489-505.
Authors are required to sign and submit a copyright transfer agreement after acceptance but before publication of their manuscript. To that effect, they receive, from the Managing Editor of Tuning Journal for Higher Education, a standard copyright assignment form designed along the following lines:
The author who signs the copyright transfer agreement must be the sole creator of the work or legally acting on behalf of and with the full agreement of all the contributing authors.
2. Copyright and Code of conduct:
a) Authors warrant that their work is original; has not been previously copyrighted or published in any form; is not under consideration for publication elsewhere; its submission and publication do not violate TJHE Ethical Guidelines for Publication and any codes (of conduct), privacy and confidentiality agreements, laws or any rights of any third party; and no publication payment by the Publisher (University of Deusto) is required.
b) Authors are solely liable for the consequences that may arise from third parties’ complaints about the submitted manuscript and its publication in Tuning Journal for Higher Education (TJHE).
c) Authors grant to the Publisher the worldwide, sub-licensable, and royalty-free right to exploit the work in all forms and media of expression, now known or developed in the future, for educational and scholarly purposes.
d) Authors retain the right to archive, present, display, distribute, develop, and republish their work (publisher's version) to progress their scientific career provided the original publication source (Tuning Journal) is acknowledged properly and in a way that does not suggest the Publisher endorses them or their use of the wortk.
e) Authors warrant that no permissions or licences of any kind will be granted that might infringe the rights granted to the Publisher.
Tuning Journal for Higher Education is an Open Access publication. Its content is free for full and immediate access, reading, search, download, distribution and reuse in any medium or format only for non-commercial purposes and in compliance with any applicable copyright legislation, without prior permission from the Publisher or the author(s). In any case, proper acknowledgement of the original publication source must be made and any changes to the original work must be indicated clearly and in a manner that does not suggest the author’s and or Publisher’s endorsement whatsoever. Any other use of its content in any medium or format, now known or developed in the future, requires prior written permission of the copyright holder.