In spring 2019, UW-IT conducted a survey of instructors and students at the University of Washington (UW) to learn about satisfaction levels, impact on teaching and learning, and impediments to using Panopto. This report reviews the survey findings and offers recommendations to improve the Panopto experience at the UW. Among the positive findings from the 2019 surveys:

• Use of Panopto continues to improve teaching and learning
• Use of Panopto does not, for the most part, encourage students to skip class. Most students who reported missing class had legitimate reasons and appreciated the safety net that recorded lecture provides
• Instructors report that conversations in class and office hours go deeper into course material when students are able to review class lectures at their own pace
• Students report that having the ability to review lectures provides a firmer grounding in course material

Since its release to all three campuses in 2014, Panopto usage has grown steadily. In 2015, the first full year of Panopto at the UW, students watched nearly 138,000 hours of recordings; in 2019, the number was over 500,000 hours. Similarly, the recordings created have increased steadily from over 7,000 in 2014 to over 37,000 in 2019.

Participants: Instructors

In spring quarter 2019, 142 instructors completed the Panopto Instructor Experience Survey. The survey invitation was sent to 650 instructors who had recorded over one hour of content in the last two quarters. Twenty-two percent of recipients responded. Thirty-five percent of these were from the College of Arts & Sciences, 14% from Engineering, 10% Public Health, and the remainder were from 12 other schools/colleges. Eighty-two percent have used Panopto for two or more quarters.

Participants: Students

In spring quarter 2019, 6,000 students received invitations to participate in the Panopto Student Experience Survey. The initial response rate was low: only 461 students responded. We then posted the survey invitation in the Canvas Learning Management System. This method drew 1,271 responses, for a total of 1,732 students who completed the survey. All class levels and most schools were well represented. Students responding, by college/unit   Class standing of student respondents

Findings

Instructor uses

When asked about how they used Panopto for both a specific course and other courses, instructors reported that the most common uses for Panopto on average were:

• Recording classroom lectures manually (62%)
• Recording lectures outside of class (55% - up from 35% in 2016)
• Recording classroom lectures automatically (~20%)

Only four percent of instructors who responded required  students to create recordings for assignments. When asked about other uses of Panopto (beyond recording), instructors reported:

• Uploading recordings (66%)
• Editing recordings (43%)
• Sharing recordings with colleagues (35%)
• Reviewing analytical data (40%)

Student viewing

When responding to the first set of questions, students were asked to keep one particular course in mind. A majority of students viewed half or more of all the recordings from their course. Of the recordings viewed, over half watched the entire recording.

Student goals for viewing recordings

Objectives for viewing recordings vary, but all are aimed at understanding course material better. Many reported studying for an exam (64%), reviewing what they did not understand (62%), or reviewing a missed class (59%). Nearly half take notes within a recording (42%). The UW was closed for several days due to a snowstorm, and 33% reported viewing recordings for cancelled classes. Almost a third use recordings to prepare for a class (30%).

Support

Instructors are availing themselves of UW-IT support. When instructors need help with Panopto, many use UW-IT documentation (52%), use Panopto's documentation (49%), and send messages to the UW-IT help desk, help@uw.edu (40%). Instructors also report asking a colleague for help (30%). Most students (80%) indicated they did not need help with Panopto. Those who did need help frequently asked people they knew (an instructor, TA, classmate or friend) for assistance.

Satisfaction

Most instructors were satisfied with the reliability (85%), ease of use (86%), and function across platforms/devices (84%). One percent reported having problems significant enough that they would not use the service again.

	Very satisfied: No problems, great to use	Satisfied: Minor problems, none that got in way of use	Dissatisfied: Major problems, but still using	Very dissatisfied: Major problems, will not use again
Reliability	33%	58%	7%	1%
Ease of use	32%	57%	9%	1%
Function across platforms/devices	35%	56%	8%	1%

Instructor satisfaction levels Most instructors (91%, up from 79% in 2016) would recommend Panopto to a colleague who is interested in lecture capture. An additional 6% would recommend it under certain conditions.

Impact on Teaching and Learning

Survey responses indicated that, overall, Panopto has had a positive impact on teaching and learning.

New approaches to teaching

Instructors reported mostly positive changes in their teaching due to Panopto. Some of the most commonly reported changes included:

• Flipping their classroom/including more active learning (33 wrote about this).
• Recording guest speakers.
• Giving more quizzes in class to encourage attendance.
• Supporting absent students more effectively and efficiently, and providing lectures when the instructor had to miss class (24 mentioned this). This was especially important when the UW was closed for snow days.
• Covering more content and having deeper discussions. Several mentioned supplementing their lectures.

Accommodation of learner needs

• Instructors and students reported that it was easy for students to get the content and make up assignments when they missed class; this was very helpful for students who were sick, athletes who travel, and commuters caught in traffic.
  • Panopto has been amazing for my classes because I travel for my team, and it allows for me to continue furthering my education.
  • It's very useful, especially for lecture-heavy classes. As a commuter, it can be difficult to make it to office hours, so being able to re-watch parts of the lecture is very helpful.
• English as a secondary language (ESL) students, students with disabilities, and students with extra responsibilities outside of school benefit from the use of Panopto.
  • Panopto allows me to not be held back because of my anxiety or learning difficulties. I believe Panopto recordings should be mandatory for all courses.
  • The features in Panopto such as fast forwarding, slowing the playback speed really helps with my learning as an international student who cannot process a second language as fast.
  • Panopto makes classes more accessible to students with attendance-impacting disabilities, to student parents, to EVERYONE! Making Panopto available means acknowledging that sometimes things happen in life outside of our control, and it shouldn't impact our learning. 
  • Panopto definitely helps me because i cannot write fast and I have ADHD so I frequently miss out on things that my prof says. it definitely levels the playing field with those with disabilities.

 

Opportunities to review material

• Students report that Panopto is helpful for reviewing for exams and provides opportunities to review difficult or confusing material.
  • Panopto is incredibly useful. Many professors speak very quickly and it's difficult if not impossible to take notes and listen simultaneously. Panopto provides the opportunity to refer back to parts of the lecture that were unclear and fill in or take notes. It would be very detrimental to not have panopto recordings as a resource.
  • In class lecture is a bit of a fire hose of information so being able to go back and re-listen is extremely helpful!"
  • I think panopto is such a useful tool! When I had difficult classes it was so helpful to be able to rewatch a lecture to strengthen my understanding on a topic.
• Students appreciate being able to review a lecture as many times as they want and how fast they want.
  • Panopto is the best. If I miss class, I can watch the lecture and stay caught up with the material. For lectures I do attend, I still like to review the panopto so I can ensure that my notes are accurate. Some professors speak fast or go through material too quickly so being able to pause the lecture is very helpful in terms of learning and reviewing.
  • It helps students gather small bits of information that can easily go through one ear and out the other during lecture. Panopto allows students to work at their own pace.

Responses from students

The majority of student respondents indicated that Panopto has a positive impact on their learning.

• 96% of students agreed that Panopto contributed to their learning.
• 94% felt more secure about learning.
• 88% agreed that viewing recordings improved their grades.
• 17% thought that Panopto had no impact on their learning, of those, 47% (29 students) did not use Panopto, but 21% (13) later agreed that "Panopto clearly helps me achieve my education goals, but other aspects of my program are more important to me" or "Panopto is of great value, and it would be a significant detriment if it was not part of my education program."

When asked if there is anything else students wanted to share about their experiences using Panopto, 237 students wrote in responses.

• Nearly half commented that Panopto was very helpful to them.
  • They liked to be able to review lectures, particularly if they missed something the first time around.
  • They appreciated being able to make up for missed classes, particularly when the UW was closed for snow days.
  • It made lectures less stressful knowing they could re-watch them and take notes later.
  • They liked being able to watch lectures at their own speed.
• Fifty wrote they wanted more instructors to use Panopto.
• Others had feature requests (28 comments) and identified issues which could be resolved with more training for both themselves and their instructors (23).
• Twenty five commented on Panopto's benefits for students with special needs.
• Eleven reported technical issues, seven complained about Panopto, and six wanted to download their lectures.

Successes

• A majority (91%) of instructors would recommend Panopto to a colleague, and an additional six percent would recommend it under certain conditions.
• One instructor summarized the benefits, "Almost everyone still comes to class. If students miss class, it takes me less time to get them up to speed and they are less stressed about missing that class session. I have more robust questions in office hours because all of the basic questions are typically answered by them re-watching the lecture. The class is able to dive deeper into a topic when I post small videos before the class."

Communications

The survey for instructors included questions about their preferences for receiving communications about Panopto from UW-IT. Seventy-six percent said that their preferred method is to receive an email, and 28% said they view messages in the Canvas banner. Only 14% selected viewing the Panopto banner, and 12% reviewed the IT Connect Updates page.

Remaining Challenges

Although instructors and students find Panopto to be a valuable teaching and learning tool, work remains to narrow the gap between Panopto's potential and instructor and student usage of the tool. Although we consistently publicize Panopto's functionality to instructors and publish documentation on viewer functionality, many students and instructors are still not aware of several of Panopto's features.

Increase usage

Several students wrote that they wished instructors would use Panopto more often and would use it better. "Please encourage more instructors to use Panopto, especially for upper level classes. Sure instructors don't offer Panopto as an incentive for people to attend classes, but with a smaller class size, it's easy for them to track attendance (and they can also do quiz in PollEverywhere for this purpose). But these upper level classes usually have more challenging content, and instructors' use of Panopto would allow students to review materials so much more easily." "In my opinion having panopto is very nice because it allows for you to catch up on work where you cannot attend lecture, especially for those with course conflicts. It also helps those who are disabled be able to watch lectures from anywhere. For those who don't use panopto it does not affect them. I feel like panopto can only benefit the student. The only reason panopto could be a detriment is if the student decides to skip classes because of panopto and then doesn't take the time to fully absorb the information provided by panopto (perhaps watching on 2x speed without fulling [sic] understanding). This, however, is completely the student's choice and is completely avoidable by a [sic] academically responsible student." "Being able to experience a lecture repeatedly, at any time, and at your own speed has immeasurable value. Unfortunately, not all of my instructors have known how to use panopto or have stated they are not willing to be on camera."

Increase awareness about Panopto features

Most instructors were unaware of most of the features available to viewers.

• Over 80% did not know that viewers can comment or ask questions, take time-stamped notes, share notes, search for text, or take quizzes.
• Over 65% did not know viewers can create bookmarks, and return to the bookmarked location.

However, over 65% were aware that viewers can download recordings when permitted, view recordings on a mobile device, and vary the playback speed. Students were also unaware of several of Panopto features.

• 64% did not know they could download recordings. However, this ability is turned off by default; instructors must turn on this functionality.
• 57% were unaware they could search for text in a presentation.
• Many students were unaware they could comment or ask questions within a recording (37%), share notes with classmates (46%), and bookmark locations within a recording (47%).

Address concerns about Panopto's impact on class attendance

• Several instructors were concerned about Panopto's potential to negatively affect classroom attendance.
  • As noted above, the effect is primarily negative. This was the first time I've had students seem to expect that I would use it, and I did have one student who needed recorded lectures as an accommodation. I regularly had 5-8 students miss every session once recordings started. I have never had these types of absences in a dozen years of teaching this course.
  • Attendance is way down. When it's a huge class I understand for most people there seems little advantage to actually turning up, but it's disappointing at grad school level - which is all I teach - to have lectures basically become audio books.

• Some instructors were more positive about the impact on attendance and the ability to catch up when class is missed.
  • The most positive comments I get are from students who come to class regularly but like to use the recording as a study mechanism. I don't think that it decreases attendance, but it is nice to have a way for students who do miss to catch up. 
  • Students really appreciate the Panopto when they miss class.
  • Several students commented that they really appreciated having the Panopto recordings of the classes. It helped students review material, and learn what we went over in class if they had to miss. This was especially useful during the snow days we had in winter quarter, when many students had to miss class.

• Some instructors developed creative ways to address attendance.
  • Initially attendance went down. When I then implemented Poll Everywhere, attendance came back up... 
  • Positive - at first I was worried about students not attending due to Panopto, but this is not the case. I have regular activities in class and assign a portion of the overall grade to participation, so there is another small motivation I use instead for getting students to attend. But again many times students have something come up and can't attend, or they want to review for a quiz/test. Panopto really helps. I also was able to use in during Winter quarter to record my lecture from home, so we didn't miss a beat despite UW being closed.

• Twenty-eight percent of students responded that they were less likely to attend class. However, many of them described staying home when they were sick or missing class because of sports.
• Most students saw Panopto as a safety net that allowed them to catch up when they had a legitimate reason for missing class. Panopto is very important because it allows for us students to regrasp topics and/or watch a lecture we may have missed (which could be for a legit reason).

Most common issues and needs

  Students and instructors both identified areas for improvement or needs that were not being met. We have given feedback to Panopto, and some of the issues are currently being addressed.

• Part of my dissatisfaction with Panopto is that I haven't had time to really dig into it and learn all of the relevant features. My lack of time is part of the problem. But I do find that Panopto in general isn't as intuitive as I would have hoped.
• When asked if there is anything they would like to be able to do with Panopto that they have not been able to do:
  • Seventeen instructors mentioned that they would like editing videos to be easier. Several had tried and found it cumbersome and unintuitive. The UW, in collaboration with several other universities, are working with Panopto to improve the recording editor.
  • Twenty-three instructors mentioned issues where training or a more intuitive interface could help. Instructors wanted to share recordings with only a few students, use discussion boards in Panopto, use white boards, stream recordings, use an action camera, record in the field, etc. Panopto is planning to release a more simplified recorder which has a more intuitive interface and may be easier to use in the field.
  • Twelve wrote about Panopto improvements they would like to see. Some want to bulk change availability settings; pause, rewind, and re-record a section rather than editing it; easily move recordings each quarter; see alerts for annotations and comments; zoom into white board on a recording; improve auto-captioning; and add arrows or text to emphasize a point. Panopto is planning to address issues with the editor and potentially add some new editing functionality.

Issues

Students were asked to comment on any major problems they have had with Panopto. Two hundred and ten students wrote about the following challenges in using Panopto: When asked to describe major problems they had with Panopto, 209 students responded.

• Sixty-eight students reported technical issues ranging from accessing recordings to playback issues. It's a great idea in every way, but it's incredibly buggy and is challenging to use. Maybe UW should switch to a better provider to record class lectures?
• Many (56 comments) described their frustration when using Panopto on mobile devices. Panopto is in the process of releasing a new mobile viewer, and UW students participated in usability studies to help Panopto improve the tool.
• Issues with audio (26 comments), finding content (15), and accessing Panopto (8) came up several times.

Problems with recordings

• Poor audio and video quality (15% of the comments referred to such issues).

Issues with mobile devices

• Seventy-one percent have not used or did not know about the mobile app, and 60% have not viewed recordings on mobile devices. However, 55 (27%) of the comments were about problems with watching recordings on mobile devices.
• Eighteen percent did not know they could view recordings on a mobile device.

Problems navigating content

• Recordings are hard to find (3% of comments; down from 12% in 2016). 14% reported major problems finding recordings, and fewer than one percent had major issues.
• It is difficult to find content within recordings (4% of comments; down from 18% in 2016). Minor problems were reported by 24%, and 2% reported major problems with finding content.
• Many students (75%) have not used search or didn't know they could use search to find text in slides. A small number (4%) reported problems with search.

Needs

• More classes should use Panopto (21% of comments).
• Students would like their instructors to be better trained on using Panopto; specifically using microphones, uploading recordings, making it easy to find recordings, etc.

Expectations

Many students like Panopto as a study tool. Some view it as mandatory - they have an expectation that if a lecture isn't recorded due to a technical glitch like the camera being broken or the microphone not having batteries, that I will redo my entire lecture.

Recommendations

1. The UW Panopto team has shared the survey responses, anonymously, with Panopto. We will continue to work with them to improve functionality. They are using our feedback for their mobile devices update and are going to conduct usability tests with UW students to improve mobile device functionality. We are also collaborating with Panopto to improve/fix many other issues mentioned in the survey.
2. Encourage instructors to use Panopto more often.
3. Increase student and instructor awareness of Panopto functionality and how to use it.
   1. Increase publicity about instructor workshops.
   2. Remind instructors again about the viewing functionality documentation.

 

Summary

Most instructors and students were satisfied with Panopto. Students see the benefit of Panopto and wish more instructors used it in their courses. Some instructors hesitate to use Panopto because they are concerned about a negative impact on attendance, but most students who reported missing class had legitimate reasons and appreciated the safety net that recorded lecture provides. Panopto has made a positive impact on teaching and learning at the UW. Instructors are able to cover more content, and students are able to review and learn information that they may not have caught the first time. As challenges are addressed and needs discovered and met, the gap between Panopto's potential and its actual usage at the UW will continue to narrow.