Feedback Questions as Course Scaffold

The semester is winding down.  Soon we’ll face the sometimes anticipated but often dreaded task of reviewing students’ course feedback. While many schools refer to them as student evaluations of teaching (SET) or SRTEs (student ratings of teaching effectiveness), they aren’t really meant to be evaluations and shouldn’t be viewed as ratings. The instruments are only one source of feedback. The unfortunate truth is they’re often assigned undeserved weight and meaning.

I’ve spent a few months reading the course feedback literature.  Much of it is conflicting and I’m not confident enough in my understanding to write about best practices yet. So for now, let’s put aside the hot button issues related to ratings differences and biases across major, gender, student-level, discipline and other concerns.

Instead, I’d like to focus on the questions. Do you have the authority to set or change some or all of the questions on your feedback instrument? If so, when was the last time you did? Do you ask the same questions in different courses? Why or why not?  I recently examined my forms, reviewed our university’s question pool, and made significant changes. It was interesting and eye opening. I highly recommend going through this exercise during a break, not mid-semester (speaking from experience).

Like many schools, our form specifies some questions.  Beyond those mandated by the university, college, and program I have a lot of choices.  Here are a few that capture my instructional goals. Rate the instructor’s skill/effectiveness in

• Relating course material to real life situations
• Making class sessions intellectually stimulating
• Helping students answer their own questions
• Encouraging students to apply concepts to demonstrate understanding
• Emphasizing learning rather than tests and grades
• Guiding students to be more self-directed in their learning

These items reflect instruction involving more than content coverage. I want to develop students as learners while they master accounting or economics.

The questions asked in prior semesters primarily focused on the administrative aspects of instruction such as my availability during office hours, preparation for class, skill in stressing important material, and presentation clarity. These are important, but they present a very narrow interpretation of teaching.  So much more should be happening in vibrant, engaging learning spaces; course feedback should reflect that.

Unfortunately, I cannot predict how students will react. I may, as a dear friend suggested, need to sip a good glass of wine when reviewing them. Students, unfamiliar with these questions, may not know how to respond. They may have never thought about the educational process in these terms.  This made me realize how the questions can and should be used for more than obtaining feedback at the end of the semester.

Teachers can use them as a framework for scaffolding instruction. Here’s an example to illustrate. On the wall opposite my desk at home is a sheet of paper titled The Far Goal. It lists four questions that drove my scholarly activity over the past several years.  This was a period where I shifted away from economics scholarship toward faculty development, mentoring and consulting.

• What publishing, writing, and thinking positions me for the far goal?
• What publishing projects will best support writing a book?
• What publishing projects will open more consulting opportunities?
• What publishing projects will continue the quest to be the best teacher I can become?

Each time I sat at my desk, the Far Goals were there, providing a visible reminder of what I want to achieve. The Far Goals provide a framework from which consulting and research opportunities and projects were evaluated.

Similarly, course feedback questions can scaffold instructional decisions in many ways: choosing how and when to cede control; selecting  formative and summative assessments; balancing student- v. teacher-selected topics; and instructional practice decisions like developing a pool of response strategies aimed at helping students answer questions for themselves.

Teachers can also share their questions with students at the start of the course.  They can be part of the syllabus and class discussion.  Building on this process may lead to asking students to select items from a pool of choices, thereby sharing ownership, responsibility and control of learning even more.  If you haven’t thought about course evaluation questions or the syllabus in this way before, please considering reading Learner Centered Syllabi for details about my syllabi, and beliefs about sharing control with students.  Reservations & Setting the Table for Learning considers how tone and underlying messages can detract from learning even before the course starts.

I also plan to reframe the questions as Instructional Goals to be prominently posted in my school office. The Instructional Goals will orient my choices and remind me of my teaching/learning priorities before I head to the classroom each day.

Does your course evaluation incorporate particularly good questions? Are you using them in unexpected ways? Please share!

We’ll examine the results and their implications in January.   Until then, thank you for sharing your ideas and feedback in the comments and through email this year. I appreciate learning from and with you.  Happy Holidays!

Advertisement

Good Intentions & my “Epic Fail”

I recently wrote about a cheat sheet and quiz wrapper strategy. Students are allowed to prepare one side of a 3×5 index card to be used during a quiz that’s worth 10% of the course grade.

Good Intentions

I intentionally refer to the assessment as a quiz, not an exam. I stopped calling it a test when I sensed the names “exam” and “test” increase anxiety in unproductive ways. I also wanted to make a clear distinction on the syllabus. Learning in this course is assessed in a variety of ways: homework, classwork, essay exams, multiple-choice quiz, paper, and project.

The multiple-choice format is also an intentional decision. The quiz assesses students’ ability to perform computations, interpret them, interpret related graphs, and understand the implications of a specific set of characteristics. The quiz assesses lower-level Bloom’s taxonomy learning. Understanding the concepts and preparing computations is the foundation for more advanced thinking and analysis in the second part of the unit. Unlike the essay exams, where I want students drawing graphs and explaining in writing, the mechanical basics addressed in this part of the course can be assessed conveniently with bubble sheets.

I introduced the cheat-sheet index card policy a couple of years ago, as a way to reduce stress during the quiz and promote active study strategies. The goal was to get students thinking about the material earlier and differently, prepare more effectively, and perform better on the quiz.

Unfortunately the strategy is,

as my kids would say,

an Epic Fail.

Calling it a quiz has not reduced anxiety. Because this is the first and only multiple-choice assessment, students don’t know what to expect. Lack of familiarity increases apprehension, regardless of what I call the assessment.

To give students an idea of what to expect, I provide a “practice quiz.” Perhaps it should be called “sample questions.” The practice quiz has the unintended consequence of limiting the scope of material studied. At least one student noted they didn’t study anything that wasn’t in the practice quiz.

Calling this assessment a quiz may actually produce more harm than good. “Quiz” may be less stressful than “test” or “exam.” But some stress is good. An unintended consequence of the name change: students may study less when it’s “only” a quiz.

Because quizzes may be seen as less important than exams or tests, some students may conclude the “cheat sheet” notecard is unnecessary. Thus, some students were insufficiently motivated to prepare one. In one case, a student noted they forgot about the quiz. Another prepared a card but neglected to bring it. Overall, about 10% of the class didn’t have one.

To be effective in promoting learning and improving scores, the card needs to be prepared in advance. Unfortunately, I noticed some students writing on theirs in the few minutes before the quiz. Others turned in cards that were incomplete or disorganized. I haven’t been able to analyze the data yet, but early findings are clear: many students aren’t preparing the cards or studying for the quiz as I intended.

From the student perspective, the purpose of a cheat sheet card is to improve their quiz score. Unfortunately, anecdotal evidence (which is all I have at this point) doesn’t bear that out. Grades on the quizzes aren’t much, if at all higher than before I allowed the cards.

That’s at least partly due to the observations above. I tried to convince students that the act of preparing the card promotes learning. The value of the card during the quiz is directly related to the quality of time and effort that went into preparing it. The message didn’t get through. Worse, feedback suggests students may have shifted from thinking about concept interrelationships toward putting basic definitions on the card.

Readers of this blog may be surprised (disappointed?) by this post’s focus on an ineffective strategy. But there is much you and I can learn from epic fails. Here are two quick takeaways:

1. Wrappers or other mid-semester feedback is vital to understanding our students. I’m gaining valuable insight from the honest admissions about study time and strategies used. A lot of it depressed me (more about that soon). But I can’t improve instruction or enhance learning if I’m unaware of where my students are as learners.
2. We learn a lot from mistakes.  Be brave. Pick one instructional strategy and critically examine the intended and unintended consequences. What are your assumptions? What are your intentions? What evidence can you gather to test how well they are or aren’t being met?

I’d appreciate learning from your “epic fail.” Please share what you learned from a strategy or policy that didn’t work. Let’s learn from and with each other.

SWOT: A strategy for mid- & late-career teachers

One of the most common analytical approaches taught in business programs is SWOT Analysis. Students are asked to identify an organization’s Strengths, Weaknesses, Opportunities, and Threats.  The results identify strategic, proactive actions the firm can take to improve.  Andy Molinsky, author and speaker, recently blogged about the need for entrepreneurs to apply the technique to themselves, asking entrepreneurs to consider their blind spots, weaknesses and opportunities.  You can find the source article here. I think SWOT can be a useful exercise for mid-career teachers.

Strengths.  What are your teaching strengths? Are you organized? Do you plan well? Are you an engaging speaker? Do you have a way with words? Are you personable? Are you tops in your field? Are you reliable, dependable, trustworthy? Are you effective at motivating and inspiring students? Are you savvy with technology? Can you find that sweet spot of pushing students hard without discouraging them? In economic terms- what are your comparative advantages?

Weaknesses. Be honest. What aspects of teaching or work as a teacher is a struggle? Are you disorganized? Impatient? Do you have trouble learning students’ names? Have trouble managing teaching and learning at a reasonable pace? Does your teaching feel stale? Do you procrastinate? Return papers late? Miss opportunities for timely feedback?

Opportunities. Molinsky sees opportunities as “chances to build on your strengths and rectify your weaknesses.” Look at the patterns and relationships between and among your strengths and weaknesses. Are there ways to minimize a weakness by exploiting a strength? For example: Could you apply a technology skills advantage toward becoming more organized, managing paperwork, or grading more timely?

Threats. Threats can be internal, personal, departmental, or institutional. For contract faculty, job insecurity may be a significant threat. For tenure-line faculty slow publication acceptances and manuscript rejections can be significant professional hazards.  Decreases in funding, declining enrollments, changing demographics, new leadership, or a lack of institutional leadership can be threats to teaching vibrancy and effectiveness.

Look at the patterns and relationships between and among the SWOT categories.  How might you use this information to invigorate your teaching?

Additional resources:

Mid-Career Faculty: How to Stay Engaged Fulfilled and Productive. White paper from Faculty Focus, https://www.facultyfocus.com/free-reports/teaching-and-learning-free-reports/mid-career-faculty/

Phelps, Patty. Nov 12, 2014. Climbing the Stairs: Observations on a Teaching Career. Faculty Focus, https://www.facultyfocus.com/articles/teaching-careers/climbing-stairs-observations-teaching-career/

Weimer, Maryellen. (2010). Inspired College Teaching: A Career-Long Resource for Professional Growth. Jossey-Bass: San Francisco.