User Testing

ImAc testing approach and results

ImAc has taken a user-centered approach, which means that users have been involved during different stages in the project and their input has been the basis for the different requirements and project developments. The slogan “Nothing about us without us” has been a reference within ImAc with the ultimate aim of making immersive environments accessible for all.

Pilots section

Results section

Appendix – Timeline and list of pilot actions

Appendix B – Glossary of terms and abbreviations

First steps: defining users and user needs

At the beginning of the project, four key areas were identified in which developments would take place longitudinally in the life of the project: content management, content production, content delivery, and content presentation.

Two main types of users were identified related to these areas:

1. home users, who would be interacting with the player and using the different access services, namely: subtitling, sign language, audio description, and audio subtitling.
2. professional users, who would be using the tools to manage and create the access services.

Based on the previous elements different user scenarios were defined, both for home users and professional users. In order to understand user needs at the very beginning of the project a round of focus groups were performed with professionals and home users.

1. A focus group on audio description and audio subtitling in the UK.
2. A focus group on audio description and audio subtitling in Spain.
3. A focus group on audio description and audio subtitling in Poland.
4. A focus group on sign language and subtitling in Germany.
5. A focus group on sign language and subtitling in Spain.

The results are included in Deliverable.2.1 and provided valuable input that informed the project initial stages. Professional users considered which new tools would be necessary to produce access services for 360° videos and pointed out that both a 2D and a 360° preview would be necessary in the editors. While sign language producers considered that sign language interpretation for 360° videos would be produced in a very similar way as for 2D content, professional audio describers saw a challenge in the new format. Subtitling professionals reflected on ways to guide users to the current speaker and mentioned an arrow, compass/wind rose or positioning the subtitles near the speaker in the current field of view as solutions.

Home users stressed that both subtitles and sign language videos should always be visible in the field of view, while the specific position should be customisable. It was important to them to have a clear speaker identification: sign language users suggested the use of arrows or “forced perspective” (automatic change of the field of view towards the speaker), while subtitle users mentioned the use of different colours. The latter user group generally expressed that subtitles should be prepared according to current standards and adapted to the challenges of immersive environments if necessary. They also pointed out that information on important sounds was needed and suggested alternatives to the traditional textual information. Audio description users highlighted the benefit of using immersive audio for both the main content as well as the access services and the need for voice interaction with the user interface. The challenges of differentiating between main and secondary action were put forward. In this regard, it was important to the users that secondary narratives should only be audio described if actively triggered.

Pilot Actions

Defining pilot actions: methodological approach

Based on this input, a series of actions within the pilots were planned addressing:

1. Actions on the professional tools with professional users
2. Actions on the player interface and the access services presentations with home users.

The methodological approach was mainly qualitative, aiming at gathering detailed user feedback during development on three main aspects: usability, presence/immersion, and preferences. We wanted to make sure that the tools and player that we were developing would be easy-to-use for the users and we also wanted to make sure that our suggested presentation modes for the access services did not break the immersion. See the appendix for a full list of pilot actions and timings.

Pilot actions with professional users

Concerning the professional tools, two iterations were performed (except for the sign language editor): a first one on an initial prototype version of the tool, and a second one on a version in which user requirements were implemented:

• Accessibility content manager: first iteration in 2018 and second iteration in 2019.
• Subtitling editor: first iteration in 2018 and second iteration in 2019.
• Audio description editor: first iteration in 2018 and second iteration in 2019.
• Sign language editor: one iteration in 2020.

Additionally, a new method for setting the angle was tested in February 2020 in a focus group.

Pilot actions with end users

Testing on the access services with home users focused on two aspects:

1. presentation modes, i.e. the specific features of the various access services.
2. player interaction, i.e. the interface characteristics.

The services were distributed in the different pilots as follows:

1. The German pilot would focus on subtitling and sign language.
2. The Spanish pilot would focus mainly on subtitling.
3. The cross-national pilot would focus on audio description and audio subtitling.

Both the Spanish and German pilots had two iterations: pilot phase 1 and pilot phase 2.

In pilot phase 1, the pilot actions were developed in closed environments with a limited number of users in order to obtain qualitative feedback during development. In pilot phase 2, a wider number of users were addressed in an open environment. Both pilot phases were preceded by a series of pre-pilot actions.

Regarding the cross-national pilot, in Spain and in the UK, it only had one iteration, preceded in this case by two pre-pilot actions.

Apart from the already planned actions, testing on easy-to-read subtitle was also added as an extra action.

Results

All the previous pilot actions have provided input which is thoroughly explained in Deliverable.2.1 for pre-pilot 1 actions and in Deliverable 5.4 for all the other actions.

The tools

During the first iteration usability values were obtained through SUS (System Usability Scale) scores. As expected the values were not very high at the prototype stage, but what was especially valuable were the suggestions made by participants through the final questionnaire.

Concerning the Accessibility Content Manager, participants made suggestions regarding the folder structure, the interface interaction, the mouse usage, and certain specific functionalities to be added or improved. You can read more about the Accessibility Content Manager on Deliverable 3.2.

As far as the audio description editor is concerned, professionals assessed very positively that the whole process of producing an audio description, including recording, takes place in one single software. They welcomed the “set the angle” option as key innovation. The interface was described as “very clear”, “simple” and “easy to use”, and it was highly appreciated that it is available online. Testing also allowed to identify some problems in the shortcuts, recording and preview modules, which have been improved in the updated version of the tool. Additional features were also suggested by participants such as a sound waveform. You can read more in our audio description editor factsheet and also on Deliverable 4.2.

Concerning the subtitling editor, participants assessed very positively that it was cloud-based and intuitive. They reacted positively to the “set the angle” option and the clear interface. Some improvements were suggested concerning the shortcuts, the viewing modes, and the default settings. Participants also highlighted the need for a sound wave and made some further suggested improvements that have allowed to develop an improved version that is currently being tested. You can read more in our subtitling editor factsheet and also on Deliverable 4.1. ImAc also tested a new setting the angle mode with VR glasses. You can read more on Deliverable 5.4.

Finally, the sign language editor tests allowed to learn more about the usability of the tool and suggested improvements. You can read more in our sign language editor factsheet and also on Deliverable 4.3.

The player interface

As far as the player interface is concerned, the different tests have allowed us to improve it. Tests have been performed on the traditional menu and on the accessibility-enhanced menu, as well as on the voice interaction. Two devices have also been considered: tablet and HMD (Head Mounted Display).

Usability scores in all cases have been positive, ranging from 68.6 to 86.25. In pre-pilot 2 actions, for instance, participants using the traditional menu on a tablet in Spain assessed the usability of the interface with A+ (86.25) on the SUS scores, and participants in Germany gave a B grade (74.79) when interacting with the interface through an HMD.

Beyond numerical data, the different tests have provided crucial qualitative input on what worked better for the different users and what they liked and disliked. Persons with different abilities have given indications on functionalities such as the indicator, the setting “area”, the personalisation options, the pointer features, the activation/deactivation options, the language settings, the menu opening options, and the icon identification.

In the open pilot, the percentage of persons who would recommend the player to a friend was above 80% and most participants found it easy to use.

In the cross-national pilot, with persons with sight loss, both the voice interaction and the enhanced accessibility menu have received excellent assessments, with values ranging from B+ (77.37) to A+ (93.30).

You can learn more in our ImAc player factsheet and also on Deliverable 3.5.

The services

Pre-pilot and pilot actions have also provided invaluable input to develop the various access services in immersive environments.

For subtitling, testing allowed as to identify the preferred comfort field of view. We also learnt that the two presentation modes developed in the project to guide users to the speaker (arrow and radar) had a similar effect in terms of immersion, but most users preferred the arrow because it was simpler to understand. The option to use sided text was not welcome. Testing also allowed to identify that most users preferred a textual representation of sounds rather than icons.

Regarding subtitle positioning, most participants prefer the always-visible subtitles as opposed to the fixed positioned subtitles. We have a more in-depth page on subtitling 360 degree videos.

An innovative test on easy-to-read subtitles put forward that many users, especially the elderly, welcome a simplified version of the subtitles as compared to the standard one.

For audio description, two aspects seemed critical in creating immersive access content: sound treatment, and scripting. A pre-pilot action was performed with different sound treatment options, but the results were not as positive as expected: many users did not perceive the different sound options as it had been initially planned. This setback actually provided interesting food for thought and compelled us to take another approach based on scripting, which has been successfully tested in the cross-national pilots. This new approach also considers some users suggestions made at the beginning concerning the inclusion of different audio description levels. We have a more in-depth page on audio describing 360 degree videos.

For audio subtitling, a first approach was made in which an audio introduction was combined with a text-to-speech voice for an opera content, using also different audio treatments. The audio treatments were again not clearly perceived by the users, who provided other ideas. Building up on the users’ ideas we have now decided to use a human voice and compare the audio subtitling of an opera content with a combination of audio subtitling with audio description. Results have proven that this approach was welcome by users.

For sign language, tests have provided indications about the comfort field of view and about the guiding mechanisms, which considered using an arrow and positioning the signer to the left or the right. A clear preference during the first tests was not identified. Regarding forced perspective, it was welcomed by testers but they requested a slower movement. Additional evaluations also allowed to identify that users seemed to slightly prefer a continuous presentation of sign language, rather than a non-continuous, because it is easier for them to concentrate on the video content. A rather unexpected and at the same time highly interesting result was that some users requested both sign language and subtitle to be used simultaneously, the reasons being that the colour of the subtitles help them identify the speaker and complement the information provided by the sign language. Information on the representation of the speaker below the sign language video through emoji or text was also obtained, and most users seemed to prefer a textual representation. 

Open pilots

Pilot phase 2 included an open pilot in Germany and an open pilot in Spain. The ImAc player with accessible content was published on RBB and CCMA’s website in order to see how users actually interacted with the immersive content. The focus was on subtitles and, additionally in the German pilot, on German Sign Language. In total, the 360º videos with access services were watched up to 1,360 times, reaching 428 visualisations in the German pilot, 788 in the Spanish pilot, and the 144 remaining ones from the General ImAc branches. More than 800 users were reached.