Earlier this year we began our work on a new platform specifically created for #passion4digital community. Since then, the platform has went through many changes, improvements, Guerrilla usability testing and even challenges.
The work conducted for this platform was also used by me, a final year Salford Business School student for my dissertation project as part of the BSc (Hons) Business Information Technology programme. The end goal of his research was to identify the factors involved in the user engagement with educational online communities such as #passion4digital for which I used Guerrilla Usability Testing (also knows an Guerrilla UX testing).
Why is User Centered Development a must?
From the early stages of my research, I identified a necessity for involving the users in the development process. There was little data available about niche online communities such as #passion4digital with regards to their requirements and behaviours. Therefore, it was essential to work closely with users to ensure that the platform was developed to meet their needs and expectations.
The first users’ involvement in this project took place when a survey invitation was sent to #passion4digital community. The valuable responses contributed to a better understanding of the community’s expectations from #passion4digital platform and the required system’s features were identified. But this was just the beginning of involving the users in the development process, and the next step was performing user experience testing that contributed to system’s improvements.
How can Guerrilla usability testing help?
Gov.uk refers to guerrilla usability testing as a flexible, low cost method that can help validate the effectiveness of a design to its target audience. It works well for systems that need to be developed in a short amount of time, and can provide unexpected results that contradict the assumptions about the user.
“Surprisingly, up to 85% of core usability problems can be found by observing just 5 people using your application. Conducting quick usability tests at a cafe is very effective, cheap, and doesn’t require any special tools.” see NNG Group
See how Google do it:
The advantages of Guerrilla usability testing include:
- Minimum resources used such as: costs, time and people;
- No specialist technology required (low cost software and hardware);
- It can be performed in various locations (university, home, office etc.);
- It works well with agile methodology approach;
- It helps uncover unexpected issues or highlights unnecessary features;
- It is great for testing the system at the early stages of binary options development.
Apart from the benefits mentioned earlier it is essential to emphasize that anyone can conduct guerrilla testing on their website/app as long as certain elements are considered: participants, tasks, process and analysis of findings.
What data capturing tools can be used?
As mentioned earlier, one of the advantages of performing guerrilla usability testing is the fact that low cost software and hardware is needed. A personal laptop, tablet or smartphone can be used to test the system (i.e. website or an app). Then, it would be really useful to capture the session, and review the recording at a later stage.
Video recording hardware and software:
This project used a HueHD camera which is affordable, and also comes with a video recording software. There are many other cameras which can be used, or even a smartphone with a stand as long as this allows you to record video of the users and their actions on the scren and sound – just in case they are thinking aloud.
Video screen capture tools
In terms of software, video screen capture tools can be used as well. Tools such as QuickTime Player for Mac, or TinyTake for Windows are free and can capture every task performed during testing. Apart from that, a basic text editor or spreadsheet tool can be used for analysing the data collected and documenting the sessions.
Oh, and don’t forget about pen and paper, they are still useful for your back up and noting down your observatins of the user and particualr time so that you can re-visit these sesions later on. Dependent on the technique you want to use for moderating your guerrilla usability test, using tecniques such as Retrospective Think Aloud (RTA) or Concurrent Think Aloud (CTA) you might want to have an additional voice recorder or use same video or smart phone to capture that moderating data.
Who can participate?
It might be convenient asking friends or family to perform the testing, but there is a high risk that the results are highly subjective. Therefore, the target audience should include existing/potential users.
For #passion4digital testing the users selected were students with an interest in digital technology, thus potential members of #passion4digital community. Having explained the purpose of the research and observing them using the system in a familiar environment (university, home) minimised the subjectivity of the data.
What tasks can be tested
It would be ideal to be able to test every single page and every single function of the website. However, due to time constraints and available participants not every element of the system can be tested by the user.
The tasks that were considered to be performed on a regular basis were tested first on #passion4digital platform. Different users performed different tasks, and after applying the changes on the platform, the tasks were retested by new users to validate the changes or uncover new issues.
How can the testing be performed
Since the testing is an observation process it is important that the tester sits back and takes notes while observing the user performing the task assigned. This way the data collected will reflect better how the system is used without the intervention of any other party, in this case the researcher.
In order to gather as much data as possible during #passion4digital testing, notes were taken, but the sessions were also filmed. A camera was used to capture both the screen and the keyboard, and a screen capture software to record how the website was used during testing. Those methods contributed to collecting comprehensive data that could be used in the analysis stage.
How can user testing data be analysed?
Having collected valuable data from recordings and notes, it is important to analyse it and make use of it to improve the system.
For #passion4digital testing, the notes taken mentioned any errors or challenges faced by the user in performing a task. The recordings were afterwards analysed to identify whether it was a system error, or a user error, which is why the keyboard was filmed as well. As a result, a list of improvements to be made was generated and implemented in the subsequent development stage.
What’s more, a marking framework was established in order to evaluate each task tested which included four main criteria:
- Learnability – the ease of new users to learn how to use the system and accomplish basic tasks
- Efficiency – how quickly the tasks are performed, once the user familiarised with the system
- Effectiveness – how accurate are the tasks performed and whether or not they are fully completed
- Satisfaction – how the user feels about using the system.
A mark from 1 (very low) to 5 (very high) was allocated to every criteria and the overall score of each task was the sum of those marks. This helped evaluating whether or not the improvements made after each testing were successful or not.
Guerrilla usability testing adds value to a system
There are many forms recommended by the industry in which guerrilla testing can be performed. It probably depends on the complexity of the project which form of guerrilla testing is performed. What matters more is that selecting relevant participants and gathering objective data is an important step in developing a system which is more user friendly.
The developed #passion4digital platform is aiming to support its community in knowledge transfer and creating a personal web space. While users contributed to improving the platform’s experience, the user’s engagement with the platform is expected to increase.
So, after all this, – is Guerrilla usability testing a feasible approach? YES for me!
Have you done some Guerrilla usability testing and feel that this post could benefit from other advice? Please share your comments below and help us to develop a resource on guerrilla usability testing to help others! Looking forward to yoru comments below!