Teaching and Learning: Evaluation and Improvement: Core Component 4.B

4.B - Core Component 4.B

The institution demonstrates a commitment to educational achievement and improvement through ongoing assessment of student learning.

The institution has clearly stated goals for student learning and effective processes for assessment of student learning and achievement of learning goals.
The institution assesses achievement of the learning outcomes that it claims for its curricular and co-curricular programs.
The institution uses the information gained from assessment to improve student learning.
The institution’s processes and methodologies to assess student learning reflect good practice, including the substantial participation of faculty and other instructional staff members.

Argument

4.B.1 The overarching goals for an ASU education are:

Graduates who have achieved the skills and abilities to think critically, communicate effectively, solve quantitative problems, demonstrate information and digital literacy, and understand how to make ethical decisions.
Graduates who possess the credentials, knowledge, and abilities necessary to advance in their chosen careers or fields of study.

At ASU, assessment is a structured, ongoing process designed to measure the extent to which graduates leave the institution with the knowledge and skills expected of majors and of the general education program. Further, through the assessment process, faculty and staff identify changes to existing instructional strategies, content, and learning experiences that will better facilitate the achievement of curricular and co-curricular learning outcomes.

ASU faculty and staff assess student learning outcomes, programs, and support services at multiple levels: the institution, academic programs, courses, and support units. ASU’s robust assessment process involves developing and/or refining assessment plans containing learning outcomes, measuring student performance on those outcomes, and using the information to refine curricula, instructional practices and support services to improve student learning and experiences. Each assessment cycle provides information about the degree of success from the previous cycle and informs decisions and activities in subsequent cycles. A variety of tools, mechanisms, and consultation from professionally trained staff from UOEEE are available to assist faculty and staff in their assessment practices.

Assessment is a core practice for ASU faculty at both the classroom and academic program level, including assessment planning, data collection and analysis, and use of assessment data for curricular and programmatic decision-making. ASU faculty conduct assessment in nearly every individual class using a variety of embedded, direct assessment measures to determine the level of achievement of predetermined student learning outcomes, including such measures as quizzes, tests, case studies, labs, performances and practical demonstrations, projects, and presentations. Individual faculty also use assessment data retrieved from student course evaluations to enhance their courses and to facilitate individual student learning. In some areas, faculty committees collectively examine aggregate assessment data across sections of the same course, as well as in program-level course sequences to gauge the effectiveness of the courses (and sequences) in facilitating students’ acquisition of institutional learning outcomes. Faculty also use the assessment data to modify individual courses and course sequences.

As previously discussed in 3.C.5, Adaptive Learning is being used to replace the common lecture in ten lower division general education classes with plans to expand beyond these ten. Instead of attending a lecture, students study content online demonstrating their mastery. Students then attend a flipped classroom in which they engage in group activity involving inquiry based learning using their mastery of the learning objectives gained in their adaptive learning assignments. By so doing in repeated classes, students are expected to improve their general education goals of critical thinking skills, communication skills, collaboration abilities and quantitative problem solving. An added benefit is that extensive data is available to instructors for assessment purposes. Most importantly, each student's proficiency on specific learning objectives can be measured and is readily available in the courseware beyond the normal graded assignments and exams. For example, beyond graded assignments, the data indicate students in college algebra who completed the course achieved 95.6% mastery on the learning objectives using a definition for mastery as 90% proficient. This technology also permits an intense examination of the quality of the content and the learning objectives themselves. For instance, in the college algebra course, the data indicated that on some lessons students experienced more difficulty in achieving the required proficiency level both in terms of time on task and the average proficiency score attained. In response, the faculty reviewed content, made revisions to the content, and inserted additional supplemental learning lessons such as short video explanations. This type of detailed data is usually unavailable in other courses. Also after reviewing data, the faculty determined some content would not be applied in any other math course in the sequence or any other course more generally. In response, they decided as a group to reduce the number of learning objectives.

Learning outcomes assessment at the program level is an integral component of the work done by curricular committees, and it is reflected in course sequences for majors and their resulting major maps. For example, English faculty actively share their successes and lessons learned as part of program assessment practices. As a result, faculty are learning from each other and are integrating new information, tools, and practices into their courses and programs.

A robust assessment plan is required for every academic program, and all new programs develop assessment plans as part of the program approval process. These plans focus on measurable and essential knowledge and skill outcomes with the aim of producing accurate information about student learning. Academic units update program assessment plans on an annual basis. An example of an assessment plan is provided for the BSE in Chemical Engineering degree program. These assessment plans include a minimum of two desired learning outcomes with corresponding measures and performance criteria for each of their degree and certificate programs. Programs may continue to assess the same learning outcomes from one assessment cycle to the next, or they may choose to add or rotate outcomes during subsequent cycles.

4.B.2 At the program level, multiple measures are expected for each learning outcome (see examples below), including at least one direct measure of student performance. Faculty use a variety of direct measurement tools and processes to assess student learning, including portfolios; rubric-driven reviews of performances, practical demonstrations, presentations, papers (e.g., assignments, theses, dissertations), projects, labs, and case studies; embedded exam items; and evaluations by internship supervisors. In addition, departments also may use one or more indirect measures to assess performance with respect to a learning outcome (e.g., surveys about job placement, enrollment in further education and degree completion, and certification and licensure). Course grades, GPAs, and course completion rates are important, but they do not constitute a program assessment. For each measure, departments are asked to identify a performance criterion or level of performance at which faculty can conclude that program graduates possess the knowledge or skill identified in the outcome. Assessment plans are submitted on an annual basis using ASU’s Assessment Portal. The following examples represent the diverse approaches ASU academic units use to assess student learning:

Elementary Education, STEM (B.A.E.) – Program faculty use the Professional Knowledge Arizona Educator Assessment, the Subject Knowledge Arizona Educator Assessment, and the TAP rubric to gauge graduates’ ability to design, implement, and evaluate instruction effectively for diverse learners. Faculty also have students design and implement a problem-based learning experience for culturally and linguistically diverse learners, and they provide feedback on students’ performance using the TAP rubric.

Nursing (B.S.N.) – Traditional and post-baccalaureate pre-licensure nursing students’ ability to provide safe and effective nursing care is assessed using their first time pass rate on NCLEX-RN exam and a clinical performance assessment. Faculty gauge students’ ability to apply research findings to deliver evidence-based client care using a quantitative critical appraisal assignment and a capstone evidence-based practice application project. Students complete a leadership paper and a case scenario assignment to demonstrate personal and leadership characteristics appropriate for innovative designer providers, managers and coordinators of care.

American Indian Studies (B.S.) – Faculty use a ‘contemporary issues of American Indian Nations’ research paper and a group project on ‘Preferred futuring: Visions, Goals, and Strategies for the future’ to assess students’ ability to critically analyze and interpret issues related to American Indian nations and populations. Faculty also have students complete an essay exam and a group project on ‘Determining community support and readiness’ to gauge students’ ability to ‘interpret and assess the sovereign powers exerted by American Indian nations prior to and after colonization.'

Counseling (M.C.) – Graduates’ knowledge of the core content areas in clinical mental health counseling is assessed using a national exam (National Counselor Preparation Comprehensive Examination - CPCE) or a master’s thesis, as well as an ethics project. Students’ counseling skills are also gauged using supervisor evaluations completed during a required counseling practicum and a 600-hour internship.

English (M.A.) – Faculty gauge students’ knowledge of past and present perspectives on specific problems of literary analysis through papers (i.e., literature review; defense of a thesis that identifies a critical or textual problem engaging the work of other scholars; and a summary of a journal article, including the argument of a specific point about a work of literature). Faculty also use presentations of an academic paper, lesson plan, anthology or textbook proposal, or other evidence of in-depth engagement with pedagogical theory to assess students’ proficiency with (1) current theories of pedagogy and (2) current critical theory, including the intellectual traditions informing contemporary theory.

Law (J.D.) – Program graduates’ ability to identify and analyze legal issues is assessed using their first time pass rate on the Arizona Bar exam as well as their performance on a real life or simulated legal analysis problem. Faculty also evaluate draft written motions to determine students’ ability to write clearly about legal problems and analysis, and to present effective legal arguments.

4.B.3 At the end of each assessment cycle, units submit annual assessment reports describing learning outcomes and related assessment practices; results for the academic year; factors to which faculty have attributed the results; and any planned curricular or instructional changes, support activities, or other items that specifically target the improvement of student learning on the stated outcomes. Academic units submit annual reports each fall using ASU’s Assessment Portal. An assessment report for the MS in Nutrition degree program is provided as an example. Feedback is summarized at the unit and college levels, and summary reports are shared with the Provost.

1. Assessment data are used by the academic colleges, schools, and units to inform decisions on curriculum, instruction, and academic support services. Several examples that highlight the effective use of assessment information by ASU academic units are summarized below.

Journalism and Mass Communication (B.A.) – For the Walter Cronkite School of Journalism and Mass Communication, assessment is a continuous process. Results of the 2015-16 assessment cycle, including data from pre-/post-assessment of students’ skill in writing news stories, informal student polls and feedback from internship supervisors, have been used to modify undergraduate coursework to meet the professional demands on journalists. For example, faculty added a new course in fall 2016 (JMC 102, Coding for Journalists) due to the increasing importance for journalism students to have basic html and coding skills. To better prepare students for careers in the media, faculty expanded a required course (JMC 305, Multimedia Journalism) to include more audience analytics, mobile reporting and visual journalism; added a module on data journalism to JMC 201 (News Reporting and Writing); and included social media/audience engagement analytics outcomes to several other courses. Faculty also provided more time for in-class writing assignments by expanding the ‘flipped classroom approach’ to more courses, and added a module to introduce students to the use of data in reporting and to develop further their skill in using Excel.

Asian Pacific American Studies (B.A.) – Faculty members in the School of Social Transformation (College of Liberal Arts and Sciences) reviewed assessment data from lower division coursework and have begun development of a required upper-division mixed-methods research course that students will take along with the program’s required capstone and/or internship experience. This new course will enable students to develop knowledge of ‘diversity in the United States’ as well as transferable workplace skills they can demonstrate to employers.

Engineering (Automotive Systems) (B.S.E.) – As part of comprehensive continuous improvement efforts, the Ira A. Fulton Schools of Engineering undergraduate faculty met to evaluate automotive systems engineering students’ attainment of course and overarching student outcomes. As a result, faculty made changes to course prerequisite requirements and modified the curriculum (e.g., adding C programming language content to EGR 219, improving the assessment of teaming in specific junior-level project courses, and revising strategies for assessing problem solving in EGR 338 and EGR 431). Faculty continue to improve rubrics for assessing students’ design and communication skills, and they have expanded their assessment strategy to include the collection of industry partners’ perceptions of students’ design outcomes and effectiveness of communication in capstone projects.

Real Estate Development (MRED) – The W. P. Carey School of Business assesses student learning on an individual and a program level and involves industry professionals in the process by soliciting feedback on project reports and presentations. Based on assessment data, faculty have made several changes to the program’s curriculum. First, faculty created a stronger emphasis on design and sustainability in RED 511 (Design of the Built Environment) to enhance real-world applications of course content. Second, faculty shifted the focus of one of the program’s three synthesis projects from teams to individual students to better assess knowledge of pertinent subject matter and development of strategic and critical thinking skills. The new approach allows faculty to assess each student’s strengths, deficiencies, and ability to apply concepts and skills at the midpoint, and to provide guidance to strengthen areas needing improvement. Faculty have created rubrics and are using TurningPoint software to provide more immediate feedback.

Psychology (M.S.) – New College of Interdisciplinary Arts and Sciences faculty reviewed assessment data from two quantitative research methods courses and a course on professional issues in psychology, and they decided to add two courses to the program of study to help prepare students planning to pursue doctoral degrees in psychology. Specifically, faculty added a third required quantitative course (Multivariate Statistics) for second-year students, and they are currently developing a required graduate-level research methods course. In an effort to strengthen graduates’ presentation skills, students are now required to deliver annual brown bag seminars. The psychology program provides a statistics ‘refresher’ and quiz for incoming master’s students to ensure preparedness for the program’s first quantitative course. Finally, program faculty developed and adopted a rubric for prospectus and thesis defenses that communicates expectations to both students and faculty.

Solar Energy Engineering and Commercialization (P.S.M.) – Ira A. Fulton Schools of Engineering graduate faculty and staff conducted student surveys, met with an Industry Advisory Board, evaluated students’ collective progress, and maintained connections with program alumni. Based on recommendations and student assessment data, program faculty continued to offer successful networking experiences for students and graduates, including a spring seminar series during which panels of industry experts discuss current issues, and a ‘Shark Tank’ event during which students pitch applied project concepts to advisory board members and receive feedback on their ideas. Faculty are also currently developing a new photovoltaic systems operation and maintenance course, delivering hands-on training for software platforms used in the solar industry, and providing more access to writing resources. Additional ideas under consideration include resume writing and interview skills workshops.

2. ASU is advancing the measurement of student learning outcomes to include the adoption and use of digital portfolios. Students construct these portfolios using materials from their coursework that demonstrate their growth and development in a variety of learning outcomes, and they provide a holistic view of student development. The university is pursuing an initiative that will complement the current assessment process with enhanced analyses of elements of general education and the value of an ASU education in a manner that is aligned with the ASU Charter and Design Aspirations. Through digital portfolios, all ASU graduates will be assessed in the core principles of critical thinking and ethics, plus quantitative reasoning, creative thinking, and/or information literacy, as determined by the program faculty. The project is based on the VALUE rubrics as part of the Liberal Education and America's Promise initiative by the Association of American Colleges & Universities. ASU faculty will develop analytic rubrics and employ cross-sectional and longitudinal sampling of portfolios to measure university-wide and program-specific outcomes, respectively.

3. As noted in Criterion 3, instruction in ASU courses is evaluated at the end of each term, and these student evaluations are reviewed at the academic unit and college levels not only to assess instructor performance but also to assess how successful the courses are at promoting learning. Course evaluations invite students to provide their subjective measurements of the learning environment as well as the quality of learning that takes place in the classroom. These results complement the program- and institution-level assessment activities.

4. Each college and school has an assessment delegate who serves as a liaison to UOEEE. Assessment delegates coordinate faculty participation in all university assessment activities; serve as a resource to faculty; review and provide feedback on assessment strategies and results; and review, approve, and submit all assessment plans and reports to UOEEE.

5. UOEEE supports ASU faculty and college delegates in every phase of the assessment cycle by offering workshops, small group or one-on-one consultation, personalized review and feedback of draft assessment plan and report submissions, online resources, and a variety of customized support options.

6. A suite of assessment resources is available on the UOEEE assessment website:

ASU’s comprehensive assessment management platform, the ASU Assessment Portal, supports every phase of the ASU assessment cycle including facilitating the exchange of assessment documents, the sharing of survey data, and providing a platform for storing historical assessment records.
The Assessment Handbook is a comprehensive, step-by-step workbook that guides readers through the development of an assessment plan, as well as through the data collection, analysis, reporting, and decision making processes of learning outcomes assessment.
Assessment FAQs: The UOEEE website contains a list of responses to frequently asked questions about assessment.
Institutional data: The UOEEE website provides access to a variety of institutional data sets, including enrollment data, survey data (via a password protected analytics site), and ASU’s Academic Program Profile.

7. Ongoing assessment is also conducted within ASU’s Educational Outreach and Student Service programs. Three operating principles inform EOSS decision making and guide daily interactions with students through programs, services, and experiences:

Achievement: Academic success and personal growth for each student.
Engagement: Intentional and meaningful opportunities for student engagement, involvement, and participation in the broader university community.
Responsibility: Structured experiences that create values driven decision making and experiences that reinforce self-discipline and a developed sense of civic responsibility.

Key co-curricular learning outcomes that support the core objectives are:

Students will excel and succeed academically.
Students will experience and understand the intersection of academic and co-curricular life through the residential college model.
Students will access support systems to enhance their individual success.
Students will be involved with faculty outside of the classroom environment (undergraduate research, service learning).
Students will apply their education and co-curricular experience in professional settings and experiences (internships, co-ops).
Students will be engaged, educated, and empowered to influence their lifelong career development and professional ambitions.
Students take responsibility for their own success and contribute to the success of others.
Students will be involved in civic engagement and community based experiences.
Students will participate in the intellectual life of the university (lectures, debates, competitions).

Co-curricular learning outcomes are assessed by Educational Outreach and Student Services (EOSS) using a variety of indirect methods, including surveys, institutional data, program participation data, and student forums. Assessment data (e.g., Connections Survey, Senior Survey, Services Survey, Career Outcomes) drives discussions among leadership and program personnel, prioritization of staff time and effort, and planning and improvement of programs and services. Each department submits objectives and quarterly and annual reports to the Senior Vice President for Educational Outreach and Student Services. Reports include reflection on the extent to which programs and services met goals and objectives, as well as a cost/benefit analysis of program intent and actual impact.

Teaching and Learning: Evaluation and Improvement: Core Component 4.B

2017 Assurance Argument

4.B - Core Component 4.B

Argument

Sources