Combining course- and program-level outcomes assessments through embedded performance assessments at key courses: A proposal based on the experience from a Japanese dental education program

DOI: https://doi.org/10.18543/tjhe-6(1)-2018pp111-142
Keywords: performance assessment, embedded assessment, program-level assessment, curriculum, PBL (problem-based learning), dental education

Abstract

This paper addresses how to combine the course- and program-level assessments and presents a new method illustrated by a case of dental education program in Japan. Performance assessments are considered effective for evaluating knowledge integration and higher-order skills, while placing a burden on faculty, hence their feasibility as the program-level assessment is regarded lower than standardized tests or questionnaire surveys. We have developed several performance assessments at the course level, such as Modified Triple Jump for the PBL course. Based on this experience, we propose Pivotal Embedded Performance Assessment (PEPA) as a method for combining assessment at the course and program levels. The method limits the range of performance assessment to key courses directly linked to program goals and placed at the critical juncture points of curriculum, while entrusting the assessment of other courses to expert judgment of individual teachers. PEPA consists of the following procedures: systematization of curriculum and selection of key courses; design and implementation of performance assessments by a faculty team; setting passing criteria with incorporating the function of formative assessment; certifying the completion of the degree program. PEPA thus enables maintaining assessment feasibility and compatibility with a credit system, while ensuring assessment validity and reliability.

Received: 27 September 2018
Accepted: 13 November 2018
Published online: 29 November 2018

Downloads

Download data is not yet available.

Author Biographies

Kayo Matsushita, Kyoto University, Japan

Corresponding author, has been a professor at the Center for the Promotion of Excellence in Higher Education and the Graduate School of Education, Kyoto University, Japan, since 2004. She received her B.A., M.A., and Ph.D. in education from Kyoto University, Japan. Her research themes include teaching & learning, curriculum, and assessment in higher education. She advocates “deep active learning,” which combines active learning and deep learning, in her edited book Deep Active Learning: Toward Depth in University Education (Springer, 2017). Classifying learning outcomes assessment into four types along two axes, direct vs. indirect and qualitative vs. quantitative, she mainly focuses on performance assessment. She has developed several performance assessments collaborating with faculty members in fields such as dentistry, physical therapy, and philosophy. She is currently leading a project “Building Disciplinary Reference Points for Curriculum Design and Quality Assurance of University Education” in the field of education studies, as a member of the Science Council of Japan. She is also a council member of the Tuning Japan National Center. Mail: matsushita.kayo.7r@kyoto-u.ac.jp

Kazuhiro Ono, Niigata University, Japan

Received D.D.S., Ph.D. from Niigata University, Japan in 1990. He is a professor at the Division of Oral Science for Health Promotion and the Division of Dental Educational Research Development (concurrent post), Graduate School of Medical and Dental Sciences and a chair of the Student Affairs of the Faculty of Dentistry, Niigata University. He specializes in oral surgery and dental education. He has been leading curriculum and assessment reforms at the Niigata University Faculty of Dentistry, and is one of the influential members in charge of educational programs of both the Department of Dentistry and the Department of Oral Health and Welfare which comprise the Faculty of Dentistry. His recent research topics include active learning, especially problem-based learning, performance assessment of higher-order integrated abilities, and program design. Having been a vice president of Niigata University since 2018, he is also in charge of establishing a system of quality assurance for higher education at Niigata University. Mail: k-ono@dent.niigata-u.ac.jp

Yugo Saito, Aino University, Japan

Received B.A., M.A., and Ph.D. in education from Kyoto University, Japan. He is an assistant professor at the Department of Physical Therapy, Faculty of Health Science, Aino University, Japan. He is a member of the Japanese follow-up program “The Assurance of Higher Education through the Development of a Tuning Test Item Bank Global Quality.” He works with a team of engineers seeking to develop a shared understanding of expected learning outcomes in the field of mechanical engineering. His research topics include assessment of higher education learning outcomes, performance assessment, institutional research, learning analytics, and deep active learning. The focus of his research is on how to bridge and combine direct and indirect as well as quantitative and qualitative assessments to support students’ learning and development. In addition, he teaches statistics and information science to paramedical students using “deep active learning” to increase students’ research ability and higher-order thinking skills. Mail: ugo.saito@gmail.com, y-saito@pt-u.aino.ac.jp

 

References

Akiba, Nami, Masako Nagasawa, Kazuhiro Ono, Takeyasu Maeda, and Katsumi Uoshima. “An Introduction to the Undergraduate Comprehensive Model Practice Course at the Faculty of Dentistry, Niigata University.” The Journal of Japanese Dental Education Association 33 (2017): 106-14. [In Japanese.]

Alverno College Faculty. Student Assessment-as-Learning at Alverno College. Milwaukee: Alverno College Institute, 1994.

Banta, Trudy W., and Catherine A. Palomba. Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education. 2nd ed. San Francisco: Jossey-Bass, 2015.

Barrows, Howard S. “The Essentials of Problem-Based Learning.” Journal of Dental Education 62 (1998): 630-33.

Biggs, John, and Catherine Tang. Teaching for Quality Learning at University. 4th ed. Berkshire: The Society for Research into Higher Education & Open University Press, 2011. Kindle.

Blake, Jennifer M., Geoffrey R. Norman, and E. Kinsey M. Smith. “Report Card from McMaster: Student Evaluation at a Problem-Based Medical School.” The Lancet 345 (1995): 899-902.

Cummings, Rhoda, Cleborne D. Maddux, and Aaron Richmond. “Curriculum- Embedded Performance Assessment in Higher Education: Maximum Efficiency and Minimum Disruption.” Assessment & Evaluation in Higher Education 33, no. 6 (2008): 599-605.

Earl, Lorna M. Assessment as Learning: Using Classroom Assessment to Maximize Student Learning. Thousand Oaks: Corwin Press, 2003.

Ewell, Peter, Karen Paulson, and Jillian Kinzie. Down and In: Assessment Practices at the Program Level. Champaign: National Institute for Learning Outcomes Assessment, 2014. http://www.learningoutcomeassessment.org/documents/NILOAsurveyreport2011%20-%20Down%20and%20In%2010-20.pdf.

Fujii, Noritaka, Syoji Takenaka, Koichi Tabeta, Naoko Sato, Nami Akiba, Yohei Oda, Yuji Katsumi, Kazuhiro Ono, and Takeyasu Maeda. “Competency Assessments for Undergraduate Students in Clinical Clerkships at the Faculty of Dentistry, Niigata University.” The Journal of Japanese Dental Education Association 33 (2017): 4-11. [In Japanese.]

Kuh, George D., Natasha Jankowski, Stanley O. Ikenberry, and Jillian Kinzie. Knowing What Students Know and Can Do: The Current State of Student Learning Outcomes Assessment in U.S. Colleges and Universities. Champaign: National Institute for Learning Outcomes Assessment, 2014. http://www.learningoutcomeassessment.org/documents/2013%20Abridged%20Survey%20Report%20Final.pdf.

Lane, Suzanne. “Performance Assessment: The State of the Art.” In Beyond the Bubble Test: How Performance Assessments Support 21st Century Learning, edited by Linda Darling-Hammond and Frank Adamson, chap. 5. San Francisco: Jossey-Bass, 2014. Kindle.

Matsushita, Kayo. “Making Learning Outcomes Visible.” Japanese Journal of Higher Education Research 20 (2017): 93-112. [In Japanese.]

Matsushita, Kayo, Kazuhiro Ono, and Yusuke Takahashi. “Development of a Rubric for Writing Assessment and Examination of Its Reliability.” Journal of the Liberal and General Education Society of Japan 35, no. 1 (2013): 107-15. [In Japanese.]

Middle States Commission on Higher Education. Student Learning Assessment: Options and Resources. 2nd ed. 2007. https://www.msche.org/publications/SLA_Book_0808080728085320.pdf.

Miller, George E. “The Assessment of Clinical Skills/Competence/Performance.” Academic Medicine 65, no. 9 (September Supplement 1990): S63-S67.

Ministry of Education, Culture, Sports, Science and Technology. “Japanese University Reforms Including Those in Educational Contents.” November, 2017. http://www.mext.go.jp/a_menu/koutou/daigaku/04052801/__icsFiles/afieldfile/2017/12/13/1398426_1.pdf. [In Japanese.]

Model Core Curriculum Revision Coordination Committee and Model Core Curriculum Revision Specialist Research Committee. Model Core Curriculum for Dental Education: AY 2016 Revision. 2017. [In Japanese.] http://www.mext.go.jp/component/b_menu/shingi/toushin/__icsFiles/afieldfile/2017/12/26/1383961_02_3.pdf.

Mtshali, Ntombifikile G., and Lyn Middleton. “The Triple Jump Assessment: Aligning Learning and Assessment.” In New Approaches to Problem-Based Learning: Revitalising Your Practice in Higher Education, edited by Terry Barrett and Sarah Moore, 187-200. New York: Routledge, 2011.

Newman, Mark J. “Problem Based Learning: An Introduction and Overview of the Key Features of the Approach.” Journal of Veterinary Medical Education 32 (2005): 12-20.

Oda, Yohei, Kazuhiro Ono, Noritaka Fujii, Tadaharu Kobayashi, and Takeyasu Maeda. “Development and Use of a Web-Based E-Portfolio for Dental Clinical Training.” The Journal of Japanese Dental Education Association 33 (2017): 65-73. [In Japanese.]

Ono, Kazuhiro, Kayo Matsushita, and Yugo Saito. “Prospects for Direct Assessment of Problem Solving Competence: Development of Modified Triple Jump in Problem-Based Learning.” Journal of the Liberal and General Education Society of Japan 36, no. 1 (2014): 123-32. [In Japanese.]

Ono, Kazuhiro, and Kayo Matsushita. “Assessment of Writing in First-Year Education.” In Assessment of Active Learning, edited by Kayo Matsushita and Terumasa Ishii, 26-43. Tokyo: Toshindo, 2016. [In Japanese.]

_____. “PBL Tutorial Linking Classroom to Practice: Focusing on Assessment as Learning.” In Deep Active Learning: Toward Greater Depth in University Education, edited by Kayo Matsushita, 183-206. Singapore: Springer, 2017.

Pike, Gary R. “Limitations of Using Students’ Self-Reports of Academic Development as Proxies for Traditional Achievement Measures.” Research in Higher Education 37, no. 1 (1996): 89-114.

Rhodes, Terrel. Assessing Outcomes and Improving Achievement: Tips and Tools for Using the Rubrics. Washington D.C.: Association of American Colleges and Universities, 2009.

Rhodes, Terrel, and Ashley Finley. Using the VALUE Rubrics for Improvement of Learning and Authentic Assessment. Washington D.C.: Association of American Colleges and Universities, 2013.

Richman, W. Allen, and Laura Ariovich. All-in-One: Combining Grading, Course, Program, and General Education Outcomes Assessment. Champaign: National Institute for Learning Outcomes Assessment, 2013. http://learningoutcomesassessment.org/documents/Occasional%20Paper%2019%20FINAL.pdf.

Rohlin, Madeleine, Kerstin Petersson, and Gunnel Svensäter. “The Malmö Model: A Problem-Based Learning Curriculum in Undergraduate Dental Education.” European Journal of Dental Education 2 (1998): 103-14.

Saito, Yugo, Kazuhiro Ono, and Kayo Matsushita. “Correlations of Direct Measures Based on Performance Assessment and Indirect Measures Based on Student Self-report.” Japan Journal of Educational Technology 40 (Suppl.) (2016): 157-60. [In Japanese.]

Suskie, Linda. Assessing Student Learning: A Common Sense Guide. 2nd ed. San Francisco: Jossey-Bass, 2009. Kindle.

Winning, Tracy, Elaine Lim, and Grant Townsend. “Student Experiences of Assessment in Two Problem-Based Dental Curricula: Adelaide and Dublin.” Assessment & Evaluation in Higher Education 30, no. 5 (2005): 489-505.

Published
2018-11-29
How to Cite
Matsushita, Kayo, Kazuhiro Ono, and Yugo Saito. 2018. “Combining Course- and Program-Level Outcomes Assessments through Embedded Performance Assessments at Key Courses: A Proposal Based on the Experience from a Japanese Dental Education Program”. Tuning Journal for Higher Education 6 (1), 111-42. https://doi.org/10.18543/tjhe-6(1)-2018pp111-142.
Issue
Vol. 6 No. 1 (2018): Academics, programmes, and methodologies for fostering students’ competences
Section
Articles

Authors are required to sign and submit a copyright transfer agreement after acceptance but before publication of their manuscript. To that effect, they receive, from the Managing Editor of Tuning Journal for Higher Education, a standard copyright assignment form designed along the following lines:

1. Authorship:

The author who signs the copyright transfer agreement must be the sole creator of the work or legally acting on behalf of and with the full agreement of all the contributing authors.

2. Copyright and Code of conduct:

a) Authors warrant that their work is original; has not been previously copyrighted or published in any form; is not under consideration for publication elsewhere; its submission and publication do not violate TJHE Ethical Guidelines for Publication and any codes (of conduct), privacy and confidentiality agreements, laws or any rights of any third party; and no publication payment by the Publisher (University of Deusto) is required.

b) Authors are solely liable for the consequences that may arise from third parties’ complaints about the submitted manuscript and its publication in Tuning Journal for Higher Education (TJHE).

c) Authors grant to the Publisher the worldwide, sub-licensable, and royalty-free right to exploit the work in all forms and media of expression, now known or developed in the future, for educational and scholarly purposes.

d) Authors retain the right to archive, present, display, distribute, develop, and republish their work (publisher's version) to progress their scientific career provided the original publication source (Tuning Journal) is acknowledged properly and in a way that does not suggest the Publisher endorses them or their use of the wortk.

e) Authors warrant that no permissions or licences of any kind will be granted that might infringe the rights granted to the Publisher.

3. Users:

Tuning Journal for Higher Education is an Open Access publication. Its content is free for full and immediate access, reading, search, download, distribution and reuse in any medium or format only for non-commercial purposes and in compliance with any applicable copyright legislation, without prior permission from the Publisher or the author(s). In any case, proper acknowledgement of the original publication source must be made and any changes to the original work must be indicated clearly and in a manner that does not suggest the author’s and or Publisher’s endorsement whatsoever. Any other use of its content in any medium or format, now known or developed in the future, requires prior written permission of the copyright holder.