User Testing

ImAc testing approach and results

ImAc has taken a user-centered approach, which means that users have been involved during different stages in the project and their input has been the basis for the different requirements and project developments. The slogan “Nothing about us without us” has been a reference within ImAc with the ultimate aim of making immersive environments accessible for all.

Pilots section

Results section

Appendix – Timeline and list of pilot actions

Appendix B – Glossary of terms and abbreviations

First steps: defining users and user needs

At the beginning of the project, four key areas were identified in which developments would take place longitudinally in the life of the project: content management, content production, content delivery, and content presentation.

Two main types of users were identified related to these areas:

  1. home users, who would be interacting with the player and using the different access services, namely: subtitling, sign language, audio description, and audio subtitling.
  2. professional users, who would be using the tools to manage and create the access services.

Based on the previous elements different user scenarios were defined, both for home users and professional users. In order to understand user needs at the very beginning of the project a round of focus groups were performed with professionals and home users.

  1. A focus group on audio description and audio subtitling in the UK.
  2. A focus group on audio description and audio subtitling in Spain.
  3. A focus group on audio description and audio subtitling in Poland.
  4. A focus group on sign language and subtitling in Germany.
  5. A focus group on sign language and subtitling in Spain.

The results are included in Deliverable.2.1 and provided valuable input that informed the project initial stages. Professional users considered which new tools would be necessary to produce access services for 360° videos and pointed out that both a 2D and a 360° preview would be necessary in the editors. While sign language producers considered that sign language interpretation for 360° videos would be produced in a very similar way as for 2D content, professional audio describers saw a challenge in the new format. Subtitling professionals reflected on ways to guide users to the current speaker and mentioned an arrow, compass/wind rose or positioning the subtitles near the speaker in the current field of view as solutions.

Home users stressed that both subtitles and sign language videos should always be visible in the field of view, while the specific position should be customisable. It was important to them to have a clear speaker identification: sign language users suggested the use of arrows or “forced perspective” (automatic change of the field of view towards the speaker), while subtitle users mentioned the use of different colours. The latter user group generally expressed that subtitles should be prepared according to current standards and adapted to the challenges of immersive environments if necessary. They also pointed out that information on important sounds was needed and suggested alternatives to the traditional textual information. Audio description users highlighted the benefit of using immersive audio for both the main content as well as the access services and the need for voice interaction with the user interface. The challenges of differentiating between main and secondary action were put forward. In this regard, it was important to the users that secondary narratives should only be audio described if actively triggered.

Pilot Actions

Defining pilot actions: methodological approach

Based on this input, a series of actions within the pilots were planned addressing:

  1. Actions on the professional tools with professional users
  2. Actions on the player interface and the access services presentations with home users.

The methodological approach was mainly qualitative, aiming at gathering detailed user feedback during development on three main aspects: usability, presence/immersion, and preferences. We wanted to make sure that the tools and player that we were developing would be easy-to-use for the users and we also wanted to make sure that our suggested presentation modes for the access services did not break the immersion. See the appendix for a full list of pilot actions and timings.

Pilot actions with professional users

Concerning the professional tools, two iterations were performed (except for the sign language editor): a first one on an initial prototype version of the tool, and a second one on a version in which user requirements were implemented:

  • Accessibility content manager: first iteration in 2018 and second iteration in 2019.
  • Subtitling editor: first iteration in 2018 and second iteration in 2019.
  • Audio description editor: first iteration in 2018 and second iteration in 2019.
  • Sign language editor: one iteration in 2020.

Additionally, a new method for setting the angle was tested in February 2020 in a focus group.

Pilot actions with end users

Testing on the access services with home users focused on two aspects:

  1. presentation modes, i.e. the specific features of the various access services.
  2. player interaction, i.e. the interface characteristics.

The services were distributed in the different pilots as follows:

  1. The German pilot would focus on subtitling and sign language.
  2. The Spanish pilot would focus mainly on subtitling.
  3. The cross-national pilot would focus on audio description and audio subtitling.

Both the Spanish and German pilots had two iterations: pilot phase 1 and pilot phase 2.

In pilot phase 1, the pilot actions were developed in closed environments with a limited number of users in order to obtain qualitative feedback during development. In pilot phase 2, a wider number of users were addressed in an open environment. Both pilot phases were preceded by a series of pre-pilot actions.

Regarding the cross-national pilot, in Spain and in the UK, it only had one iteration, preceded in this case by two pre-pilot actions.

Apart from the already planned actions, testing on easy-to-read subtitle was also added as an extra action.

Results

All the previous pilot actions have provided input which is thoroughly explained in Deliverable.2.1 for pre-pilot 1 actions and in Deliverable 5.4 for all the other actions.

The tools

During the first iteration usability values were obtained through SUS (System Usability Scale) scores. As expected the values were not very high at the prototype stage, but what was especially valuable were the suggestions made by participants through the final questionnaire.

Concerning the Accessibility Content Manager, participants made suggestions regarding the folder structure, the interface interaction, the mouse usage, and certain specific functionalities to be added or improved. You can read more about the Accessibility Content Manager on Deliverable 3.2.

As far as the audio description editor is concerned, professionals assessed very positively that the whole process of producing an audio description, including recording, takes place in one single software. They welcomed the “set the angle” option as key innovation. The interface was described as “very clear”, “simple” and “easy to use”, and it was highly appreciated that it is available online. Testing also allowed to identify some problems in the shortcuts, recording and preview modules, which have been improved in the updated version of the tool. Additional features were also suggested by participants such as a sound waveform. You can read more in our audio description editor factsheet and also on Deliverable 4.2.

Concerning the subtitling editor, participants assessed very positively that it was cloud-based and intuitive. They reacted positively to the “set the angle” option and the clear interface. Some improvements were suggested concerning the shortcuts, the viewing modes, and the default settings. Participants also highlighted the need for a sound wave and made some further suggested improvements that have allowed to develop an improved version that is currently being tested. You can read more in our subtitling editor factsheet and also on Deliverable 4.1. ImAc also tested a new setting the angle mode with VR glasses. You can read more on Deliverable 5.4.

Finally, the sign language editor tests allowed to learn more about the usability of the tool and suggested improvements. You can read more in our sign language editor factsheet and also on Deliverable 4.3.

The player interface

As far as the player interface is concerned, the different tests have allowed us to improve it. Tests have been performed on the traditional menu and on the accessibility-enhanced menu, as well as on the voice interaction. Two devices have also been considered: tablet and HMD (Head Mounted Display).

Usability scores in all cases have been positive, ranging from 68.6 to 86.25. In pre-pilot 2 actions, for instance, participants using the traditional menu on a tablet in Spain assessed the usability of the interface with A+ (86.25) on the SUS scores, and participants in Germany gave a B grade (74.79) when interacting with the interface through an HMD.

Beyond numerical data, the different tests have provided crucial qualitative input on what worked better for the different users and what they liked and disliked. Persons with different abilities have given indications on functionalities such as the indicator, the setting “area”, the personalisation options, the pointer features, the activation/deactivation options, the language settings, the menu opening options, and the icon identification.

In the open pilot, the percentage of persons who would recommend the player to a friend was above 80% and most participants found it easy to use.

In the cross-national pilot, with persons with sight loss, both the voice interaction and the enhanced accessibility menu have received excellent assessments, with values ranging from B+ (77.37) to A+ (93.30).

You can learn more in our ImAc player factsheet and also on Deliverable 3.5.

The services

Pre-pilot and pilot actions have also provided invaluable input to develop the various access services in immersive environments.

For subtitling, testing allowed as to identify the preferred comfort field of view. We also learnt that the two presentation modes developed in the project to guide users to the speaker (arrow and radar) had a similar effect in terms of immersion, but most users preferred the arrow because it was simpler to understand. The option to use sided text was not welcome. Testing also allowed to identify that most users preferred a textual representation of sounds rather than icons.

Regarding subtitle positioning, most participants prefer the always-visible subtitles as opposed to the fixed positioned subtitles. We have a more in-depth page on subtitling 360 degree videos.

An innovative test on easy-to-read subtitles put forward that many users, especially the elderly, welcome a simplified version of the subtitles as compared to the standard one.

For audio description, two aspects seemed critical in creating immersive access content: sound treatment, and scripting. A pre-pilot action was performed with different sound treatment options, but the results were not as positive as expected: many users did not perceive the different sound options as it had been initially planned. This setback actually provided interesting food for thought and compelled us to take another approach based on scripting, which has been successfully tested in the cross-national pilots. This new approach also considers some users suggestions made at the beginning concerning the inclusion of different audio description levels. We have a more in-depth page on audio describing 360 degree videos.

For audio subtitling, a first approach was made in which an audio introduction was combined with a text-to-speech voice for an opera content, using also different audio treatments. The audio treatments were again not clearly perceived by the users, who provided other ideas. Building up on the users’ ideas we have now decided to use a human voice and compare the audio subtitling of an opera content with a combination of audio subtitling with audio description. Results have proven that this approach was welcome by users.

For sign language, tests have provided indications about the comfort field of view and about the guiding mechanisms, which considered using an arrow and positioning the signer to the left or the right. A clear preference during the first tests was not identified. Regarding forced perspective, it was welcomed by testers but they requested a slower movement. Additional evaluations also allowed to identify that users seemed to slightly prefer a continuous presentation of sign language, rather than a non-continuous, because it is easier for them to concentrate on the video content. A rather unexpected and at the same time highly interesting result was that some users requested both sign language and subtitle to be used simultaneously, the reasons being that the colour of the subtitles help them identify the speaker and complement the information provided by the sign language. Information on the representation of the speaker below the sign language video through emoji or text was also obtained, and most users seemed to prefer a textual representation. 

Open pilots

Pilot phase 2 included an open pilot in Germany and an open pilot in Spain. The ImAc player with accessible content was published on RBB and CCMA’s website in order to see how users actually interacted with the immersive content. The focus was on subtitles and, additionally in the German pilot, on German Sign Language. In total, the 360º videos with access services were watched up to 1,360 times, reaching 428 visualisations in the German pilot, 788 in the Spanish pilot, and the 144 remaining ones from the General ImAc branches. More than 800 users were reached.

Appendix A – Timeline and list of pilot actions

PILOT ACTION NAME

COUNTRY

PILOT ACTION TYPE

DESCRIPTIONS

ACTION OBJECTIVE

DATE OF EXECUTION

PRP1-ST-Presentation modes

Germany (RBB)
Spain (CCMA and UAB)

Closed

Comfort "Field of View"
Guiding to speaker (sided text, arrow, and compass)

To get feed-back on the features to be implemented and narrow them down if necessary.

April and May 2018

PRP1-SL-Presentation modes

Germany (RBB)

Closed

Comfort "Field of View"
Guiding to speaker (sided SL video, arrow) and forced perspective

To get feed-back on the features to be implemented and narrow them down if necessary

April 2018

PRCN1-AD-Presentation modes

United Kingdom (RNIB)
Spain (UAB)

Closed

Three versions of audio description with different sound and scripting strategies

To get feed-back on the features to be implemented and narrow them down if necessary

April and May 2018

P1-ST-Presentation modes

Germany (RBB)
Spain (CCMA)

Closed

Guiding to speaker (arrow, radar)

Gathering feedback for the second iteration

September and October 2018

P1-ST-Player

Germany (RBB)
Spain (CCMA)

Closed

Player usability: accessing subtitling service with traditional menu and HMD

Gathering feedback on player interface

September and October 2018

P1-ACM

Germany (RBB)
Spain (CCMA, UAB)

Closed

ACM usability and user preferences

Gathering feedback from professional content managers.

July 2018

P1-ST-Editor

Spain (UAB)

Closed

Subtitling editor usability and user preferences

Gathering feedback from professional subtitlers.

July 2018

P1-AD-Editor

Spain (UAB)

Closed

AD editor usability and user preferences

Gathering feedback from professional audiodescribers.

September to October 2018

PRP2-ST-Presentation modes

Spain (UAB)

Closed

Presentation modes:
always visible vs fixed positioned
arrow vs autopositioning

Comparing current solutions with ImAc developments.
Identify preferred solutions by users and most immersive options. Head tracking analysis.

November 2018

PRP2-ST-Presentation modes

Spain (UAB)

Closed

Presentation modes:
always visible vs fixed positioned
arrow vs radar

Comparing current solutions with ImAc developments.
Identify preferred solutions by users and most immersive options. Head tracking analysis.

March to April 2019

PRP2-ST-Presentation modes

Germany (RBB)
Spain (CCMA)

Closed

Subtitle positioning (always visible with arrow versus fixed positioned with arrow) and non-speech sound representation (emojis versus text)

Gathering feed-back from users on developments

May to June 2019

PRP2-ST-Player

Germany (RBB)
Spain (CCMA)

Closed

Player usability: accessing subtitling service with tablet and HMD

Features evaluation

May to June 2019

PRP2-E2R-Presentation modes

Spain (UAB)

Closed

Preferences between SDH and E2R subtitling

Gathering user feed-back on E2R subtitles

May to June 2019

PRP2-SL-Presentation modes

Germany (RBB)

Closed

Display of signer video (continuous versus non-continuous). SL with ST versus SL only. Speaker representation (emoji versus text).

User preferences on different presentation modes

May to June 2019

PRCN2-AD-Presentation modes

United Kingdom (RNIB)
Spain (UAB)

Closed

Presentation modes: classic, dynamic, static.

Testing the methodology for the cross-national and the features to be tested

May to June 2019

PRCN2-AST-Presentation modes

United Kingdom (RNIB)
Spain (UAB)

Closed

Presentation modes: classic, dynamic, both with TTS.

Testing the methodology for the cross-national and the features to be tested

May to June 2019

PRCN2-AD-Player

United Kingdom (RNIB)
Spain (UAB)

Closed

Player usability: voice interaction and enhanced menu

Testing the methodology for the cross-national and the features to be tested

May to June 2019

P2-ACM

Germany (RBB)
Spain (CCMA)

Open

ACM usability and user preferences

Gathering feed-back on improved version including user requirements.

August to Setember 2019

P2-ST-Editor

Germany (RBB)
Spain (CCMA)

Open

Subtitling editor usability and user preferences

Gathering feed-back on improved version including user requirements.

August to Setember 2019

P2-SL-Editor

Germany (RBB)

Closed

SL editor usability and user preferences

Gathering feedback from professional SL interpreters.

August to September 2019

P2-AD-Editor

Spain (UAB)

Closed

AD editor usability and user preferences

Gathering feed-back on improved version including user requirements.

September 2019

P2-ST-HbbTV+CS

Spain (CCMA)

Open

ST presentation modes, usability, tracking of personalisation settings
Use of Companion Screen device.

Gathering feed-back on the ImAc experience using a Companion Screen device.

October 2019 to January 2020

P2-ST-SA/S

Spain (CCMA)

Open

ST presentation modes, usability, tracking of personalisation settings

Gathering feed-back on the ImAc experience.

October 2019 to January 2020

P2-ST-PC+CS

Spain (CCMA)

Open

ST presentation modes, usability, tracking of personalisation settings.
Use of Companion Screen device.

Gathering feed-back on the ImAc experience using a Companion Screen device.

October 2019 to January 2020

P2-ST-HMD/S

Spain (CCMA)

Semi-open

ST presentation modes, usability, tracking of personalisation settings

Gathering feed-back on the ImAc experience.

October 2019 to January 2020

P2-ST-SA/G

Germany (RBB)

Open

ST presentation modes, usability, tracking of personalisation settings

Final feedback on the ImAc experience

October 2019 to January 2020

P2-ST-HMD/G

Germany (RBB)

Closed

Workshop: isolated presentation modes and personalisation

Final feedback on the ImAc experience

October 2019 to January 2020

P2-SL-SA/G

Germany (RBB)

Open

SL presentation modes, usability, tracking of personalisation settings

Final feedback on the ImAc experience

October 2019 to January 2020

P2-SL-HMD/G

Germany (RBB)

Closed

Workshop: isolated presentation modes and personalisation

Final feedback on the ImAc experience

October 2019 to January 2020

CN-AD-Presentation modes

United Kingdom (RNIB)
Spain (UAB)

Semi-open

AD presentation modes based on scripting features

Final feedback on the ImAc experience

October 2019 to December 2019

CN-AST-presentation modes

United Kingdom (RNIB)
Spain (UAB)

Semi-open

AST presentation modes

Final feedback on the ImAc experience

October 2019 to December 2019

CN-Player

United Kingdom (RNIB)
Spain (UAB)

Semi-open

Player usability: voice interaction and enhanced menu

Final feedback on the ImAc experience

October 2019 to December 2019

 

Appendix B – Glossary of terms and abbreviations

ACM      Accessibility Content Manager
AD          Audio description
AST        Audio subtitles
CN          Cross-national Pilot
CS           Companion Screen
E2R        Easy to read
HbbTV   Hybrid broadcast broadband TV
HoH       Hard of Hearing
HMD      Head Mounted Display
P1            Pilot 1
P2           Pilot 2
PC           Personal Computer
PRCN1   Pre-Pilot Cross-national 1
PRCN2  Pre-Pilot Cross-national 2
PRP1      Pre-Pilot 1
PRP2      Pre-Pilot 2
SA           Stand Alone
SL           Sign language
ST           Subtitle
/G           Germany
/S            Spain