A Web Directions event Respond 2017 Australia's Front End design conference

Session Details

Shefik Bey

Think aloud user testing - challenging established status quo

Shefik Bey Co-founder

At Loop11, our team has been discussing the potential influence and impact of the think aloud method of user testing on the natural usage behaviour of participants. Does the method distract or change the way in which users naturally interact? Are we introducing bias into the testing process by making respondents verbalise their inner monologue?

There have been previous research studies that do cite an impact on behaviour, however with the growth of rapid online testing and the availability of quantitative metrics, the relative merits and drawbacks of employing this approach have become even more pertinent.

Our team formed its own hypothesis, that the think aloud method would extend task completion times and influence the depth to which participants explore navigation and content. We were also concerned that the approach would lead to a skew towards more vocal (possibly extroverted) attitudinal behavioural types and under-representation of others.

To put the approach to the test, our team designed a comparative testing scenario to pit think aloud testing against testing carried out without the need to verbalise thoughts.

When running the same set of tasks and questions, on the same website, but using two different usability testing tools, one prioritising think aloud, we recorded significant difference in the testing results - though not exactly in ways we had expected!

Surprisingly, for the most part we found no real discernible difference in time on task. However, an interesting result from the three studies was that the greater the level of anonymity, the lower the NPS but the higher the SUS. Does this mean that if a user knows they are being recorded, and their responses will be rated, then they are more likely to award a higher NPS score? Within this limited sample, it would appear so.

How test participants are sourced also impacts results. Providing incentives for performing tasks in a certain manner (very articulate and deliberate) leads to further skewing of results. Rating of participants perpetuates this phenomenon. Participants should be incentivised for completing the study in a natural manner, not what pleases the viewing experience.

Shefik Bey

Shefik is a leading digital innovator who has practiced and influenced the growth of User Experience (UX) in Australia and around the world for the past 20 years.

He has a proven track record delivering digital strategies that drive high profile clients to success, across a multitude of industry sectors.

In 2001 he established pioneer UX consultancy U1 Group, and the company now heralds more than 30 staff across Melbourne and Sydney.

Don't miss your chance to see Shefik Bey and many other inspiring speakers at Respond 17.

Tickets start at $999, for an in-depth day with the world's leading Front End design experts.

Register Sydney Register Melbourne

Australia's Front End design conference, now in Sydney and Melbourne.

Register Now

What is Respond?

Australia's Front End design conference, now in Sydney and Melbourne

Now in its fourth year, and curated by John Allsopp, cited by Ethan Marcotte as one of the originators of Responsive design, Respond goes beyond the technology, to address the myriad challenges we face in delivering modern, responsive Web experiences.

Featuring a mix of renowned international and local speakers, Respond is one of the World's leading events for the Web industry.

Register Sydney Register Melbourne More details

Praise for past Web Directions events

Web Directions is the must-attend event of the year for anyone serious about web development.

Phil Whitehouse, General Manager, DT Sydney

I’ve been admiring the Web Directions events for years, and was honored to be part… What a fantastic event!

Ethan Marcotte, inventor of "responsive Web design"

Out of any conference, Web Directions is far and away our favourite

Dave Greiner, founder, Campaign Monitor

our audience watches intently

About Us

Co-founded and now run by John Allsopp, Web Directions has for over a decade brought together leading developers, engineers, visual, IxD, UX and product designers, Art and Creative Directors, indeed everyone involved in producing web and digital products to learn from one another, and the World's leading experts across this vast field.

We spend our lives thinking about what comes next, keeping up with trends in technology, practices and processes, and filtering the hype, to make sure you don't miss trends that matter, and don't waste time on hype that doesn't.

We promise attending one of our events will leave you significantly better versed in the challenges you face day to day, and in solutions for addressing them.

In 2017, we'll be organising Respond in Sydney, Melbourne and Brisbane Transform in Canberra, Code also in Sydney, Melbourne and Brisbane, and Direction in Sydney.

vignettes form our events, social, speakers and more. Includes Hannah Donovan skylarking.

John Allsopp

John Allsopp has been working on the Web for over 20 years. He's been responsible for innovative developer tools such as Style Master, X-Ray and many more. He's spoken at numerous conferences around the World and delivered dozens of workshops in that time as well.

His writing includes two books, including Developing With Web Standards and countless articles and tutorials in print and online publications.

His "A Dao of Web Design" published in 2000 is cited by Ethan Marcotte as a key influence in the development of Responsive Web Design, who's rightly acclaimed article in 2010 begins by quoting John in detail, and by Jeremy Keith as "a manifesto for anyone working on the Web".

Code of Conduct

For over a decade, we've worked hard to create inclusive, fun, inspring and safe events for the Web Industry.

As part of our commitment to these values, we've adopted a code of conduct for all involved: ourselves, our speakers, our partners and our audience.

If you have any concern or feedback, please don't hesitate to contact us.