How can we create a Beta Testers program in an environment where there wasn’t such a culture
Until then it didn’t exist official tests with the users, and the company didn’t have this culture for new features, when they were released the company have just expected the feedback from Customer Success. So a lot of problems arised and the stakeholders blamed the IT team.
So, I was waiting the best opportunity to change this scenario, and I was thinking how to do this, because I was the only designer in my team and I needed to influence the stakeholders for this change. Then, with the intention of explaining about the importance of the tests inside the development process that I invited our Product Owner to participate in an Usability Testing course with me. It worked! She understood the importance and helped us to implement and make the process feasibility with the directors increasing the deadline of launching, because the directors until then, didn't understand the importance of the tests.
We were waiting the best opportunity to change this scenario, and as we were planning the new release of a big feature, we took the opportunity to create the first Beta Testers Program.
About the Feature
The idea was to provide in our software, a CRM for the Real Estate Market, launching a Chat to streamline communication and business within the professionals from the Real Estate Market community. Making it possible, for example, the creation of groups within many different agencies, incorporators, brokers and etc.
Understanding how was the process:
1. Concept and goals
To create the program, we first thought which goals we wanted to reach:
- To find errors that have gone through our tests
- To know if our Chat is intuitive
- To know the learning curve of the using the Chat
- To approach and get the loyalty from this customer
- To decrease the rate of calls due to the consequence of errors or usability difficulties after the release of a new Feature
After we researched methodologies to be applied and we found many ways to achieve the proposed goals. And we chose two:
Continuous User Diary
A research methodology and test that can be applied when it is necessary to understand the behaviours of our customers about using a product or service during their day by day routine. This methodology would be ideal for us, because many of our customers aren’t close to us and just the broker using the Chat on its daily routine it would be able for us to know if the engagement and efficiency of the tool really works.
On the perception test, the user through pre-established tasks, simulate his behavior using the tool, while the designers are watching the test. This test is more effective if performed in person, so we can follow the first reaction of the user with the tool.
2. Subscription and selection
We sent a e-mail to all directors of the agencies, of these we received a large number of subscribers that were selected with criteria like higher user rate on the software.
- 1.821 open the e-mail
- 205 registered to be a Beta Tester
- 58 agencies
3. Feed and Script
We created script of tests for all Beta Testers, with littles missions for them to perform as face-to-face tests.
During the period of the testes further the phone contact with the users explaining the course of the program, we also feed with e-mails for each step with orientations and guidelines about the nexts steps.
In order to know what this user is accustomed to use and whichs tools his usually uses in for his day by day , we asked for this users which communication tool in real time is most use.
Now we know which tools they were used to and which of the tools our tool would compete, so we could predict behavior and feedbacks.
As soon as we released the Chat, we started receiving feedbacks. The customers were answering our forms and sending e-mails to us.
But we knew that just these e-mails weren't enough, because of this, we kept the phone contacts with them. We also followed the use of the database. Unfortunately, we couldn't install any screen recording software. But, this wasn’t an issue because we were able to do face-to-face testing.
We have created presentations with the results showing the performance of the tests, which gave credibility and safety to the stakeholders.
During the user tests period, we improved the experience of using our Chat with the analysis that we had done, everything in just two weeks. And after many challenges, the tool was finally launched. Due to the the tests, we avoided many complaints. About two months after the release of the chat, users exchanged about 90,000 messages among them.