This is the Arizona State University Academic Program Review (APR) online manual. This online manual serves to outline procedures for the self-study committee and the site visit team in the successful completion of a program review.
The sections that follow are divided into four phases of the APR. Phase one describes the process the UPRA Office follows in notifying and preparing the unit for the upcoming review. Phase two is instructions to the unit for the preparation of the self-study document. Phase three provides instructions to site visitors regarding the visit and the site visit report. Phase four addresses the activities that occur after receipt of the final site visit report.
Phase I: Preparatory phase
Notification to unit
The UPRA Officer will notify the head of the unit and the dean of the unit under review that a review has been scheduled. Letters of notifications are sent out a year in advance of the actual review cycle.
APR Preliminary Information
The head of the unit should complete the APR Preliminary Information form (Appendix 1) to submit the faculty self-study committee names, the site visitor honoraria, and other pertinent information, along with the dean’s approval, to the UPRA office by May 1st of the academic year before their review.
The self-study committee will be responsible for preparing the self-study report (Appendix 2) as well as organizing and conducting the review activities. The committee consists of a chair and a maximum of six faculty members who are representative of the unit.
On the APR Preliminary Information Form, the head of unit will also designate the amount of the honoraria for the local site visitor and the non-local site visitors. There is not a required amount for the honoraria, but the unit should keep in mind that the visitors are committing to a 2-day visit, travel to and from Arizona, and writing a thorough site visit report.
This form submission should be approved by the dean either through signature or by email with the completed form attached to firstname.lastname@example.org.
*Reimbursements for travel expenses and payment of honorariums are the responsibility of the unit under review once the final site visit report has been received.
Meeting with the Dean
The head of the unit and self-study committee are encouraged to meet with their dean to ascertain if there are particular issues that the dean would like included or addressed in the self-study report.
Orientation meeting with the UPRA Office
In both the spring and fall semesters the UPRA Office organizes an Orientation Luncheon Meeting for the units currently under review and the units that have upcoming reviews. The unit heads are encouraged to invite the self-study chair, administrative support person, and the business office manager to attend as well as any other key players.
The orientation luncheon meeting will provide an overview of the APR process including preparation of the self-study report and its required format, locating relevant university data, payment of reimbursements and honorariums, as well as providing general direction and answering questions regarding the process.
Questions regarding the program assessment plans for the curricular offerings and assessment reports on the student learning outcomes should be directed to the University Office of Evaluation and Educational Effectiveness (UOEEE). Questions regarding official university financial data for the unit under review should be directed to the Office of Institutional Analysis (IA). Official data for the self-study tables is available through the Academic Program Profile.
Any unit undergoing review may request an individual meeting with the UPRA Office at any time to discuss questions regarding the APR process.
Dates for the site visit
The self-study report is written during the fall semester and the site visits are scheduled during the spring semester. UPRA office will provide all units under review several date options to choose from for their two-day site visits. Units will receive the potential site visit dates via email, and the dates will be confirmed on a first-come, first-served basis. Each unit will be asked to select a first and second choice of dates.
Guidelines for nominating site visitors
Using the Site Visitor Nomination Form (Appendix 3), each unit under review nominates six non-local academic site visitors who are professionals in the discipline of the unit along with three local community members who are, preferably, recent graduates of the unit. Please attempt to diversify your nominations (gender and race/ethnicity). Please do not nominate multiple individuals from the same Institution. If the academic unit has specialized areas (e.g., professional/disciplinary, basic/clinical), please indicate which site visitors are appropriate for the various specialty areas, and indicate the order in which visitors should be invited.
- Faculty nominees should be active members of their profession and/or the academic community particularly in the areas of specialization that are important to the unit under review
- Recent graduate/community nominees should have strong familiarity with the program, should not have financial association with ASU, and should reside in the Phoenix metropolitan area, if possible.
- In order to avoid conflict of interest, do not forward names of individuals such as former/current faculty of the unit under review, mentors/friends, or previous ASU teaching faculty/staff. Please do not nominate former/current job applicants, donors, contractors, or administrative council members.
The UPRA staff will extend invitations to nominated site visitors. The unit should not make contact with potential site visitors before the site visit team has been identified by the UPRA office. A confirmation email identifying the site visit team will be sent from the UPRA office to the unit. Once the confirmation email has been received, the unit will make contact directly with the site visitors to make hotel/travel arrangements as well as forward the expense reimbursement and honorarium paperwork. (See Appendices 4A and 4B, non-local and local logistics letters.)
The site visit team plays an important evaluative role in the process by providing objectivity and helping the unit and university determine where the program fits within the discipline at regional, national, and international levels. The unit under review is best served by nominating qualified academics and professionals experienced in the discipline(s) under review.
Nominations should be submitted with complete current contact information (including physical address). The UPRA office will initiate all contact with potential site visitors.
Site visitor nominations should be submitted by the head of the unit under review, in consultation with the self-study committee faculty, along with the dean’s written approval.
- Please use the Nomination Form (Appendix 3) to submit nominees to the UPRA Office. One form for each nominee. Please complete all fields on the form and email to email@example.com.
- Please rank your list of nominees in order of invitation preference. Use the section at the top of the form to rank 1-6 for your preferred order of invitations. The UPRA Office will invite nominees in the order specified until two non-local and one local visitor has accepted. A unit may elect to have additional non-local visitors to cover disciplinary needs.
Site visitor confirmation
Once the site visitors have accepted, the unit under review will receive a confirmation email from the UPRA Office identifying the site visit team.
At that point the unit should directly contact the non-local members of the site visit team to provide them with the reimbursement claim forms, request a copy of their travel itineraries, and notify them of their hotel arrangements and ground transportation to and from the airport. Hotel arrangements for non-local site visitors should be provided via the Site Visitor Logistics Template, Appendix 4A.
Units contact the local site visitor using the local site visitor logistics template in Appendix 4B. You will provide the local visitor with directions, parking information and validation instructions, and attach the appropriate forms for reimbursement, if necessary.
*Reimbursements for travel expenses and payment of honorariums are the responsibility of the unit under review once the final site visit report has been received.
Phase II: Self-study report—development and preparation
Preparation of self-study
The self-study report is an interpretive document that uses data to assess progress since the last review, current program status, and future directions (template found in Appendix 2).
Datasets for the self-study template are found at the Academic Program Profile data site. The dataset is tightly integrated into the template. Data should be analyzed and discussed in relation to a unit’s mission and goals.
Although the report is compiled and written by the self-study committee, the head of the unit is responsible for the content, accuracy, and completeness of the work, and should actively oversee the report preparation.
In this section: Assessing Student Learning Outcomes
UOEEE works with programs in a two phase process to help programs prepare for self-study: evaluate past data and prepare for future assessment of student learning.
PART ONE: Analyzing Past Assessment Data
- To help programs gather past data, the UOEEE assessment portal (https://uoeee.asu.edu/aportal) has developed a new feature that allows programs to access aggregated data as well as download past assessment reports. To help guide data interpretations, programs are asked to respond to the five prompts provided below as part of their analysis. These prompts should help you establish how well students are achieving the established learning outcomes. This outcome is essential to the assessment process.
- Please discuss the results of your quantitative and qualitative assessment data for each outcome and measure.
- Are your students achieving at the levels of performance you had expected? How well did they meet your performance criteria?
- What plans do you have in place for students who are not achieving the desired level of performance?
- What actions have you taken or will you take based on your assessment data?
- How will the results be shared with program faculty and students?
- It is also recommended that programs utilize data from the various student report cards, first destination (Formerly Recent Graduate), and alumni surveys to address career placement outcomes, career preparation, and who is employing graduates over the period since the last review.
Please address each program learning outcomes (PLOs) using these reflectivequestions as a guide:
PART TWO: Preparing for the next cycle
Programs will begin to prepare for APR the year before their actual review. By the start of the academic year prior to the review, UOEEE will have reviewed each program’s assessment plan and provided feedback so the plan comes into compliance with the ABOR requirements. All programs are then asked to review the feedback and make the necessary adjustments to meet the current needs of their program as well as the current suggested criteria established by ABOR. Each assessment plan must be approved by UOEEE through the portal prior to submitting the final Academic Program Review.
Submission and approval of the self-study report
The dean should review the self-study report and executive summary, and provide written approval, before it is forwarded to the UPRA Office.
The completed self-study report should be submitted to the UPRA office and can be emailed to firstname.lastname@example.org along with the written approval of the dean (which may be made by an accompanying email). Should the report and appendices be too large for email, the UPRA office will accept the documents via a shared folder (Google docs or Dropbox).
The UPRA office will review the self-study report. When necessary, suggested changes or improvements will be returned to the self-study committee for revision.
The final, dean-approved, self-study and appendices should be submitted to the UPRA office a minimum of three weeks prior to the site visit date. In addition to the self-study and appendices, the unit should submit the final, dean-approved, site visit schedule. See Appendix 6 for an example of a site visit schedule.
Distribution of documents to site visitors
Once submitted, the UPRA office will convey all relevant documents to the visitors at least two weeks prior to their visit. The UPRA office will upload the documents to a shared folder and provide access for the site visit team.
Phase III: Site visit and site visit report
Site visit overview
Academic Program Reviews (APR) serve many purposes, the most important of which is to provide information to academic units and the university for continuous program improvement. The review should identify strengths and areas for improvement and also identify opportunities for future development. The site visit team plays an important evaluative role in the process by providing objectivity and helping the unit and university determine where the program fits within the discipline at regional, national, and international levels.
Site visit schedule arrangement and coordination
Entry and Exit Meetings
The site visit is scheduled for two full days. Please refer to Appendix 6 for a sample site visit schedule.
The site visit begins and ends with entry and exit meetings. The UPRA office sets the time of the meetings. Entry and exit meetings are the first and last meetings of the visit and are scheduled at 8:30-9:15 a.m. on day one and 2:30-3:30 p.m. on day two. The entry meeting attendees are the dean, executive director UPRA, associate director UPRA, unit head, and any additional people at the Dean’s discretion. The dean will determine if any additional people should be included in the exit meeting. The unit should provide the dean-approved site visit schedule to the UPRA Office at least three weeks prior to the scheduled visit.
During the construction of the site visit schedule please allow the site visitors at least one hour preceding the exit meeting to discuss major issues and preliminary findings in preparation for the site visit report and exit meeting discussion points. The unit should provide a quiet room with computer access for the site visitors.
- It is the responsibility of the unit to make the non-local site visitor hotel reservations and arrange for ground transportation to and from the airport. Parking reservations should be made for the local visitor and the unit should communicate parking instructions (see Appendices 4A and 4B).
- The unit is responsible for reimbursement of expenses and payment of the honorarium for the site visit team immediately upon receipt of the Site Visit Report. If the unit makes arrangements for entertainment, receptions, or other social functions, the costs of those activities will also be the responsibility of the unit. The unit is not obligated to provide entertainment.
- Since the community member of the site visit team should come from the metropolitan Phoenix area, it is not expected that they will incur expenses beyond parking and any meals scheduled during the site visit. Should the unit choose to nominate a recent graduate who does not reside locally, the unit will reimburse for travel and lodging.
- It is the responsibility of the unit to arrange a tour of its facilities; time for reviewing honors theses, master’s theses and doctoral dissertations; and to schedule the appropriate meetings with faculty, students, and staff. Junior faculty members and senior faculty members should have time with the visitors without the presence of the other group to facilitate candid conversation. The same is true for undergraduate students and graduate students who have different needs and perspectives. Staff should have time with visitors without supervisors present.
- The unit will ensure that all meetings are on the calendars of the appropriate participants.
- Lavish entertaining of the site visitors is not expected nor encouraged. They will have a busy day reviewing the academic unit and will need time to discuss findings. Please leave dinner and evening hours free whenever possible.
- If faculty members wish to go out to dinner or lunch with the visitors, they or their unit will be responsible for those expenses.
- The unit should consider arranging a representative to escort the non-local visitors to and from the hotels, scheduled meetings, and activities on campus.
Visits with students
- Careful planning should be employed to set up meetings with a representative and a sizable group of undergraduate and graduate students (separate meetings are best). A faculty member and student representatives should be given responsibility for organizing these meetings. These meetings, if left to last minute planning, may be poorly attended, and can leave a lasting (possibly negative) impression on the site visit team. The schedule should be finalized at least three weeks prior to the scheduled visit.
- Brown-bag lunches, coffee hours, class meetings have worked well. Because students often bring up questions for which the site visitors will want to seek answers, these meetings should be set up fairly early in the schedule.
- After the visitors are introduced and their purpose explained, the unit meeting coordinator and/or faculty member should leave so that students feel free to have a candid discussion with the site visitors.
- Some time should be set aside for the site visit team to review dissertations, theses, and honors projects.
Visits with faculty members
- Depending on the size of the faculty, two or three small group meetings might be desirable. Meetings with individual faculty members are discouraged. The head of the unit should not attend the meetings with faculty.
- Please make sure that tenured, tenure track, and non-tenure track faculty have the opportunity to meet with the site visit team separately.
- Where appropriate, there should be meetings with other relevant faculty groups (e.g., graduate advisory committees, undergraduate curriculum committees whose work relates to the program review).
- If possible, it is strongly recommended that site visitors meet with faculty from other related disciplinary areas with whom the unit under review interacts.
Visits with unit administrators
- Immediately following the entry meeting, visitors should have the opportunity to meet privately with the dean to ascertain any specific requests for the visiting team.
- At least one-half hour should be scheduled for the site visitors to meet with the head of the academic unit. Because the site visitors will have developed a better understanding of the unit under review, this meeting should occur in the afternoon of the second day of the site visit.
Visits with university administrators
- The UPRA Officer and Associate Director will attend the entry and exit meetings. (See more information on entry and exit meetings above.)
Instructions to site visitors
Prior to and during the site visit, the visiting team will analyze the self-study report, collect additional relevant information as well as meet with faculty, students, alumni, community representatives, and unit administrators (e.g., dean, provost’s office, when applicable). The team will prepare a report identifying program strengths, concerns, and recommendations. In addition, the team will also tour the university to examine facilities and other resources if relevant for their review.
One of the site visitors should take the lead in drafting the report, but all are full participants and contributors to the report. The following information describes specific site visitor functions, responsibilities, and suggestions for maximizing the effectiveness and outcome of the APR process.
Site visitors will be granted access to a shared folder containing the relevant documents as a shared space to collaborate on producing the final site visit report.
There is a checklist of responsibilities available for the site visitor's use and convenience (see Appendix 11).
Communication between the site visit team and unit under review
The appropriate business paperwork should be sent to the visitors by the unit under review along with the site visit logistics letter (See Appendix 4A and 4B) shortly after the site visit team is confirmed. Site visitors should send the unit their original travel receipts along with the completed business paperwork at the conclusion of their visit.
The unit under review will reimburse any appropriate site visit expenses and an honorarium for each visitor upon receipt of the final site visit report.
Two weeks prior to the site visit
Two to three weeks prior to the site visit, the UPRA office will upload the self-study report, along with any necessary appendices, and the final site visit schedule to the shared folder for the visiting team.
Three to four weeks after the site visit
The site visitors will submit a site visit report that summarizes program evaluations and recommendations to the UPRA office at email@example.com. The next section provides instructions pertaining to the site visit report. The UPRA office will distribute copies to the head of the unit, the dean, and others as appropriate.
Once the site visit report has been received (along with a signature page from each visitor, see Appendix 7), the UPRA office will notify the unit under review so they can process expense reimbursements and honorariums.
Site visit report
The site visit team is asked to provide an assessment of the future direction and strategic initiatives of the unit as they relate to the unit’s mission and vision for its programs. The site visit report should be submitted electronically to the UPRA office approximately three weeks after the site visit.
The UPRA office will distribute copies to the dean, the head of the unit, and any other appropriate parties. The site visit report should be accompanied by a signature page (Appendix 7) executed by each visitor indicating their concurrence or disagreement with the report. There is space provided for context for disagreement.
Guidelines for preparing the site visitors report
There is not a specific format requirement for the site visit report. The contents and length vary depending on the nature and size of the program under review. Although we ask that the report address each of the substantive areas described below, other program dimensions important to program quality and future development may also be addressed.
Mission and goals
- Are the mission and goals adequately addressed? Do they seem appropriate given the university mission? Are there on-going mechanisms in place within the unit to evaluate the currency and relevance of those goals?
- Please assess the contribution of the unit in advancing the state of the discipline/profession and the program’s position relative to peer institutions.
Strategic initiatives and future direction of the unit
- Please provide an assessment of the strategic initiatives identified by the unit and the future direction of the program as articulated in the self-study report — the state of the discipline, emerging areas in the national scene, peer institutions and aspirational peers as identified by the unit should be taken into consideration as this assessment is done.
Learning objectives and curricular effectiveness
- Does the unit have a clear understanding of curricular content and sequence? Are there appropriate learning objectives and outcome measures?
- Based on your knowledge and/or understanding of the discipline, is the undergraduate and graduate curriculum current and does it provide adequate training/education for graduates of the program?
- Are areas of program emphases (e.g., concentrations, certificates, tracks) appropriate in view of available resources (human and physical), and do they address community, regional, or national needs? If not, what would the site visit team recommend?
- Does the unit have adequate assessment plans in place to evaluate the effectiveness of their undergraduate and graduate programs (i.e., outcome measures)?
- How successful is the unit in using the assessment reports of student learning to improve its undergraduate and graduate programs?
Student recruitment, retention, and placement
- Are there unit processes in place to ensure quality recruitment and retention practices? In what ways does the unit address student retention? Is the diversity of the student population appropriate or should efforts be undertaken to further diversify the program?
- Assess the depth and quality of efforts by the unit to retain students and foster graduation rates, reduce the time to degree and increase faculty involvement with undergraduate and graduate students in both teaching and research.
- Is there adequate attention placed on career advisement, development, and placement for both undergraduates and graduates?
Faculty and staff quality
- How does the level of faculty research/scholarly activity compare with peer institutions? Are external funding levels appropriate? Please note any exceptional faculty contributions to the mission of the unit and university.
- Is the faculty research/scholarly agenda appropriately integrated into student learning? Are there missed opportunities that should be considered?
- Assess the staffing levels, diversity, and quality.
- Are there unit processes in place to ensure quality recruitment and retention of faculty? In what ways does the unit address faculty retention? Is the diversity of the faculty population appropriate or should efforts be undertaken to further diversify the program? How does this affect or not affect student diversity?
- Does the unit make appropriate use of existing resources including physical (e.g., lab, office space), educational (e.g., university instructional, advising), and internal fiscal resources?
- Do all students and faculty have equal and fair access to those resources?
- Are there specific resources set aside for underpresented groups
- Assess the level of engagement and impact with community professionals/constituents. If improvement is necessary, what steps might the unit take to foster stronger relationships?
- New curricular initiatives. If the unit has proposed, or plans to propose, a new program (e.g., degrees, certificates), please evaluate the potential for quality, success, faculty workload responsibility, resource adequacy, and quality of student recruitment and placement.
Conclusions and recommendations
- What do you consider to be the major strengths of the program?
- What areas need greatest improvement?
- What opportunities exist for future development? What do you consider to be the major limiting factors for future growth and development of the unit?
Submission of site visit report
Once all members of the site visit team have reviewed and concur or disagree with the contents of the report (signified by individually executed signature pages, Appendix 7), the report and signature pages should be emailed or mailed to the UPRA office:
U.S. Mail to:
University Program Review and Accreditation Office
Executive Director for Accreditation and Academic Program Review
Office of the University Provost
Arizona State University
PO Box 877805
Tempe, AZ 85287-7805
*Upon receipt of the site visit report and a signature page executed by each site visitor (Appendix 7), the UPRA Office will notify the unit to reimburse site visitor expenses and honorariums.
Phase IV: Unit response report and wrap-up phase
Unit response to the site visit report
Once the site visit report is received, it will be reviewed and shared with the head of the academic unit under review and the dean.
The head of the unit should review and discuss the site visit report with the faculty and prepare a unit response report for the Provost that addresses program strengths and deficiencies, faculty and student quality, resource needs, future plans, strategic initiatives, etc.
The unit response report should be submitted to the UPRA Office by the date specified in the memorandum accompanying transmission of the site visit report (usually three weeks after distribution of the site visit report).
Note: The unit response report will be shared with university officials as appropriate. The unit response and dean's response must be independent from each other, not a collaborative report.
Dean's response report to the site visit report
This should be a focused statement about the value derived from the review, any issues that arose, and the recommendations from the external reviewers. Please provide a bullet point list of each recommendation alongside the academic dean’s response to each. The dean’s report should be submitted to the UPRA Office by the date specified (usually three weeks after receipt of the site visit report).
Note: The unit response and dean’s summary should be independent reports.
Outline of the unit response report
Briefly review and respond to major strengths, weaknesses, and opportunities discussed in the site visit report.
II. Strategic plans to address concerns
A. Issue 1 (identify issue)
- Proposed action, expected outcome
- Cost and resource implications
- Source of funds and resources
- Benchmark and timeline for solution
B. Issue 2 (identify issue)
- Proposed action, expected outcome
- Cost and resource implications
- Source of funds and resources
- Benchmark and timeline for solution
III. Additional information
Discuss any other program changes and developments related to the APR generally and the site visit report specifically.
The wrap-up phase will include an assessment of the site visit team’s findings and the unit’s response by university officials. This phase may also include a meeting with the head of the unit, the UPRA Officer, and the dean if there are concerns or if further clarification is deemed necessary before the final wrap-up of the APR.
Permanent record of the program review
The self-study report, site visit report, dean’s report to the Provost, unit response report, and any wrap up reports will be considered the permanent record of the APR process and stored electronically with the UPRA office. A summary report prepared by the UPRA Office will be forwarded to the Arizona Board of Regents.