Academic Audit FAQs

What, exactly, is an "academic audit"?

An academic audit reviews the processes or procedures that faculty members use to provide a quality education in their department. Unlike some other approaches to evaluation or program review, academic audits do not try to second-guess decisions made by the faculty about their department curriculum nor attempt to "micromanage" the way they teach.

Academic audits concern themselves more with the processes that faculty use to think about their curriculum decisions and how they carry out these activities in the best interests of the discipline and student learning. Academic audits also focus on how faculty members organize these activities and how well they perform them.

Given that an "academic audit" just focuses on strategies that a department uses to ensure quality education, and not their actual teaching, how can it improve student learning? Wouldn't a more direct evaluation approach be better?

People who work to improve the quality of academic programs often report that, "good people working according to good processes accomplish more than good people working with poor ones." The processes used to accomplish tasks reflect the way people organize their work and the kinds of data they use to make decisions.

It's true, of course, that good processes are not sufficient. To be more complete one would also have to look at department resources, whether the department uses its available resources effectively, and what kinds of activities the department faculty members focus their efforts on.

A key principle of the academic audits relies on the following simple tenet of academic life that many faculty members believe: "Department faculty members want to provide quality education and they will do so when supported by good processes." The purpose of an academic audit is to encourage departments to strengthen the techniques and processes they have in place to improve the quality of their work.

This is all a bit confusing. Exactly what kinds of "processes" are we talking about?

There are five areas that are examined in an academic audit. Each contains a set of questions that should be asked and answered as part of any department's educational activities if it hopes to improve its educational efforts.

1. Determining desired learning outcomes. What should a student who successfully completes the course or departmental major know and be able to do? How will the course or major program build on the student's prior knowledge and abilities? How will it contribute to the student's future employment opportunities, capacity to make social contributions, and quality of life?
2. Designing course content and department curriculum. What will be taught, in what order, and from what perspective? How will the content contribute to the overall students' overall knowledge and learning? What course materials will be used? How will these materials relate to other parts of the students' program?
3. Designing teaching and learning. How will teaching and learning be organized for students? What methods will be used to exposure students to the material for the first time, for answering questions and providing interpretation, for stimulating involvement, and for providing feedback on student work? What roles and responsibilities will the faculty members assume? What other resources will be required and how will they be used?
4. Developing student assessment. What measures and indicators will be used to assess student learning? Will they compare performance at the beginning and end of the term, or simply look at the end result? How will the long-term outcomes of the students' experiences be determined? Will baseline and trend information be available? Who will be responsible for assessment? How will the assessment results be used?
5. Implementing quality education. How will faculty members assure themselves and others that content is delivered as intended, that teaching and learning processes be implemented consistently, and that assessments are performed as planned and their results used effectively?

These are not necessarily areas traditional professors focus their attention on in research universities. How do faculty members who have participated in an audit react to questions like the above?

The education of the students is one of the most important jobs university professors do. Even in research universities like the University of Missouri it accounts for an average of 40% of a professor's time. Addressing teaching and learning with less rigor than is used to judge their "scholarly research" through peer reviews indicates that a less rigorous standard is used for teaching than for scholarly work.

Professors usually think these are reasonable common-sense questions that cause them to think more about their department curriculum and how it affects their students. Some questions are familiar and some are not, but few say the answers are not important for educational quality. Even if faculty members have not dealt with these issues, they are more likely to focus on improvement than to dismiss the questions as irrelevant. By the way, these are also the same kinds of questions likely to be asked by academic auditors.

It appears that the academic audit only focuses on teaching and learning. What about the other significant work faculty members do in the areas of research and service?

Yes, the initial academic audits will focus more on the teaching and learning process. Conceptually an academic audit is designed to help faculty members look at the processes their department uses that will make them more effective. Although the academic audit initially focuses on teaching and learning, we envision that it can later be expanded to address the research and service functions as well. Once in place, the processes could easily be expanded into other areas like research and scholarship where the expected outcomes are more established (e.g., papers published, grants received, etc.).

The word "audit" suggests a judgmental, after-the-fact evaluation, yet the reasons and advantages for conducting audits seem largely developmental. Is this a contradiction?

No. Academic audits invite departments to describe the strengths and weaknesses of their efforts to improve the academic quality of their programs and identify plans for improvement. Such activities are clearly developmental. The audit visit consists mainly of conversations between team members and faculty. Experience shows that the conversations evolve in a developmental way, even when there is an element of judgment at the end. The audit report identifies areas where the quality efforts are exemplary, and also where they need improvement. While calls for improvement inevitably have a judgmental character, criticism of a department's quality efforts is less likely to undermine the evaluation's developmental goals than, say, citing a professor for poor teaching. As is typical with audit, the University of Missouri has declared that audit's developmental aspects should be paramount in program design.

Some kind of self-study or report must be required before the site visit. What is expected in the self-study?

The Self-Evaluation should describe the current state of the department's efforts to improve educational quality. It should cover the five domains listed above and supporting documentation should be cited and described briefly, but not included in the report.

The Improvement Plan should list priorities and focus on the areas identified in the self-evaluation as needing improvement. In particular, it should specify improvement actions the department will take during the fall semester of 2002 and later in the 2002-03 academic year, and their expected results.

Draft guidance notes for the Self-Study are available. They will be discussed at the February meetings with department faculty and modified as appropriate.

Who are the auditors and what will the audit visit be like?

Most, if not all auditors will be recruited from within the University of Missouri. As an academic audit focuses on common sense questions about how the department conducts its work, the process does not require expertise in the department's academic discipline, though in some cases it helps. In the interest of full participation and communication, the teams will include members from the departments being audited during the pilot. The length of the audit visits has not been determined, but will probably not exceed two days.

The auditors will meet with the departmental leadership group, with the cognate faculty committees and teams, with individual faculty members, and with students. There will be no classroom visits.

What kinds of questions will the auditors ask and what kinds of documentation will they seek?

Auditors usually begin by asking about the department's philosophies and approaches to the five areas noted above (i.e., learning outcomes, course and curriculum content, teaching and learning, assessment, and implementing quality). Exemplary departments normally will have developed descriptions of how the seven principles that enhance quality should be applied in the various areas. The auditors will be interested in the faculty members' descriptions because they demonstrate forethought and commitment to quality within their department.

The auditors will also ask for examples of how these principles have been applied. For example: "What have you learned from former students and employers?" "What changes have been made recently in your curriculum, what was the impetus for the changes, and what facts were used to make these decisions?" "What research went into designing the student assessment, and how do the assessments relate to what you are trying to accomplish?" "How do colleagues hold each other accountable for implementing these processes and improving the quality of the education for students?"

Experience shows that faculty members in departments that have good procedures to ensure the quality of their work will enthusiastically describe what they're doing-just as they would describe their research to an interested observer. However, those who have not engaged in any kind of process to improve their academic programs will find it hard to be specific. Departments with active discussions and procedures to improve their academic program will have produced a rich set of materials for internal purposes, whereas materials produced by departments that have not attended to this will be sketchy and generic. Another key point is that these materials will be department specific, generated by the faculty members in the academic discipline. No externally created generic guidelines will be imposed on the department."

How important is it to document one's work?

Documents can demonstrate commitment and progress, but this can be done in other ways as well. While an absence of writing will trigger questions, there is no prescribed form of documentation. Professors and departmental teams should write up their activities in ways that help focus issues, further data analysis, and facilitate collegial communication. Most important, they should be able to engage the auditors in well-informed discussions about a broad range of quality process activities.

How will the auditors describe a department's progress in developing its educational efforts?

There will be no scoring during the pilot. The audit reports will describe the department's quality processes, highlight areas of strength, and identify areas that need improvement. They will not compare one department with another, although a careful reader may be able to draw conclusions about relative progress. How progress will be reported in future audits has not been decided.

There is no one-way, but one common-sense method is to rate programs along the following scale.

• No effort means there are no organized processes used in the department to insure quality programs.
• Firefighting means that departments respond to problems, but mostly with ad hoc methods. The seven principles for enhancing quality (see below) receive little attention.
• With informal effort, one sees individual initiatives and experimentation with the seven principles in one or more of the five areas used.
• With organized effort, quality process initiatives begin to be planned and tracked, work methods are systematically rooted in the seven principles, and the department has begun to develop ways of gauging performance and departmental norms that seek to improve quality.
• With mature effort, departmental procedures have been embedded in departmental culture and become a way of life in the department, and continuously seeking to improve the department is fully established.

Suppose a department decides to really focus its attention on its curriculum. Are there some key principles to use when trying to improve the quality of a department's academic programs?

Yes. Departments can use seven common-sense principles to improve the quality of their academic programs. While these practices have some roots in business, government, and health care, these specific practices come from academia.

1. Define education quality in terms of outcomes. The quality of student learning, not teaching per se, is what ultimately matters. Learning should pertain to what is or will become important for the students enrolled in the program (not some an "ideal" student). Exemplary departments figure out what their students need and then work to meet these needs.
2. Focus on the actual process of teaching and learning. Departments can carefully analyze the methods that faculty members use to teach and how students learn. They can consult the literature in their academic disciplines and look at what works and what does not. They can collect data where possible to see how their students are doing. They can try enhancing active learning and making better use of information technology. Faculty members can experiment with new ways of looking at student learning on a regular basis; methods that have been proven effective in their academic disciplines. They can adopt successful innovations, which become part of the department's modus operandi and form the baseline for future experimentation and improvement.
3. Strive for coherence in their department curriculum and the educational activities they use. Departments should view learning through the lens of the student's entire educational experience. This clearly applies to the department's curriculum, where courses should build upon one another to provide the desired depth and breadth, and how this learning provides the basis for other courses. It also applies to the range of class sizes and learning approaches experienced by the typical student. For example, a mix of large lectures and small seminars may well produce better learning than a succession of medium sized classes that consume the same amount of faculty and student time. Additional enrollments make little difference once a lecture exceeds a certain size because the communication is one-way anyhow. However, larger lectures can make room for small seminars that produce more active learning than medium-sized classes with fewer interaction opportunities. Writing, oral presentations, group work, and learning activities that can be used in a range of classes to provide other opportunities for building coherence.
4. Work collaboratively to achieve mutual involvement and support. Faculty members can demonstrate collegiality in teaching, just as they do in their research. Departments can encourage teamwork in order to reinforce collegiality and bring a broader array of talent to bear on difficult problems. Team members can encourage strong efforts by their peers and take each other to task when shortfalls occur. In the end, collegiality and teamwork make the department a "learning organization" with respect to the education quality process.
5. Base decisions on facts wherever possible. Departments should base educational decisions in data instead of simply adopting traditional criteria. For example, a department might collect data on student preparation, learning styles, and probable requirements for employment. Faculty can also obtain feedback from past students and employers to evaluate the relevance of course and program content. In many cases, traditional disciplinary training may not answer these questions except as they pertain to aspiring PhD students. Faculty members can analyze the data carefully in light of disciplinary standards and one's own professional experience, and then incorporate the findings in the design of curricula, learning processes, and assessment methods.
6. Identify and learn from best practice. Departments should seek out examples of good practice from their disciplines and adapt the best of these practices to local circumstances. They can regularly compare good versus average or poor performing programs and students; assess the causes of the differences and then seek ways to mitigate them. Faculty members can attempt to move poor performance toward average performance and average performance toward exemplary performance.
7. Make regular improvement of the department's programs and offerings a high priority. Departments can strive to improve the quality of teaching and learning on a regular basis. While departments may place strong emphasis on research as well as on teaching, they recognize that continuous improvement requires that they also focus on the educational efforts of the department too. Promotion and tenure committees and departmental personnel committees should strike a new balance between teaching, broadly defined, and research. These simple efforts can most importantly affect the attitudes and day-to-day behavior of professors.

Sample Academic Audit Questions