Customer Testing
Customer Testing can be a useful aid to evaluate ideas, prototypes and new features and improve our understanding of the customer, their motivations and what aspects of a new product or service is desirable to them.
Testing may be structured and formal with pre-arranged testing objectives defined and groups of representative customer testers selected and paid to evaluate a new feature or prototype. Facilities may be used to demonstrate the new features or prototype with recordings of the testers’ responses and feedback made for analysis. There are many customer testing agencies who specialise in this area recruiting testers that represent specific customer segments and can host testing events if an organisation does not have its own facilities.
Unstructured testing at the other end of the spectrum is also possible with minimum cost “bucket tests” where random volunteers are asked in the street for their impromptu feedback and views on new features or a demonstrated prototype. This might be done on the street or in a shopping mall for example for products or services that are targeted for the general public.
Performing customer testing in the “wild” in the open market can also be possible with end users who can be directed to use new features in either an “opt in” explicit approach or unknowingly use the new feature. Feedback may be collected with explicit surveys or implicit analytics of users’ behaviour when using the new features. This approach, although needing some infrastructure investment is the most representative of customer views, motivations and how they will use the new feature, as it is as close as possible to generally releasing the new feature.
There is an approximation with customer testing, in that, for structured testing for example, the results may be obtained in a “clinical” environment and with a small sample of the customer demographic, which may or may not represent how a new product or feature will be actually received in the open market. As a tool, customer testing is invaluable to gain deeper insights into the motivations and desires of the customers, but there are also limitations that should also be considered.
Testing Groups
Testing groups are recruited individuals within a structured testing approach that are intended to be representative of the target customer segment or demographic that the product or service is intended for.
Understanding the customer segment that the product or service is intended for will help to recruit a suitable testing group that can provide some deeper insights into how the product or feature will be received.
Not understanding the target customer segment and using the “wrong” testing group may provide testing feedback that is not necessarily representative of the target market and may well steer a product or service in the wrong direction or introduce distraction and unexpected data.
Testing Objectives
Testing Objectives can help to set the framing and positioning of the customer testing effort to answer specific questions, assumptions or a hypothesis that we want to prove or disprove.
Being quite explicit about the objectives for the customer testing objective can help to narrow down the testing questions and focus on specific feedback for individual features in a prototype rather than be obscured with unrelated comments and suggestions.
A|B Testing
This approach hypothesises that one feature set is more desirable over another and to evaluate the hypothesis both feature sets are released to separate customer testing groups or as pilots to end users and customers, and the test results captured.
The feedback obtained should highlight which of the feature sets are most desirable, had the expected outcomes and the intended customer goals been achieved when using the new feature.
Some infrastructural investment is needed to host multiple instances of a product or service and to redirect customers or end users to one instance or another in order to directly compare the impacts and obtain feedback data for either case.
Feedback data is usually captured through analytics if using an implicit approach where the end user is unaware they are using a new feature, or explicit feedback surveys and customer interviews if the customer is fully aware of using the new feature and may have “opted in” to evaluate it.
This approach allows for a range of ideas to be tested at once and evaluated by end users. It opens the doors to set based thinking to consider a range of concurrent experiments to evaluate which ideas resonate strongly with customers, and which ones should be discarded.
Responding to Feedback
Collected feedback can be in many forms from survey responses, customer interview responses or detailed analytics data that record customer movements and behaviour. Some feedback may be explicit and directive e.g. “…change the colour of the button to red…” or it may be implicit where the reaction of the user is of most interest, for example, can a user intuitively use the new features without the need for prompts or instructions.
Once received, the feedback should be analysed, and a deeper understanding of the customer needs derived, which can then be used to prove, disprove and refine the questions, assumptions or hypotheses that we were trying to evaluate.
Based on the new evidential data we can then derive new questions, assumptions and hypotheses to evaluate with new experiments and Sprint Goals.