I get this request on a semi-regular basis and given the work I’ve done with Microsoft, Oracle and other technical partners over the years, I’ve been involved in a number of early-release software programs. So, let’s talk about what people mean when they talk about Beta Test Programs. The first thing to know is that the perspectives of what the vendor is hoping for and wants are often different than what the users want.
First of all, what’s a beta test and who came up with that clever name?
Beta is the second letter in the latin alphabet with “alpha” being the first. Programmers, clever as we are, used the terms to refer to first and second levels of testing. Alpha testing is supposed to be the kind of testing done at the programmer’s station or at worst within the development department prior to release. Beta testing was supposed to be testing at the user level where real production data was run through the program.
In an enterprise application, neither of these terms is particularly relevant. Quality Assurance (QA) of software is a spectrum of testing that ends up with a product in the hands of an end user. The numbers of levels of testing is often unique to the particular situation.
What does the software publisher hope for from a beta tester?
Ideally, a software publisher would give a release candidate of software to a client and they would use it in full production in parallel with their own operations for a period of time and therein lies a problem. If we’re talking about an application that would be used for an hour or so a day, such testing sounds just fine. If we’re talking about an enterprise application that is used across the user-based, then such testing is almost impossible. The very people who the publisher would need to do the testing already have full-time jobs. They are processing all the enterprise data that comes in every day or every week and generating reports, activating processes or exporting and importing data just to keep up. To really run data through the new version would mean essentially duplicating that entire exercise.
Our TimeControl system is a timesheet. Surely we could get people to duplicate their work for a timesheet? You’d think so, right? Our experience is that it’s close to impossible. Perhaps if our product was a video game we would have a chance but to ask people to sit down to their timesheet once is a big enough challenge. Asking them to do it twice, even for a short number of weeks is almost always too much. Then once the data would be in, we’d have to have the supervisors, administrators and project managers duplicate their effort. It is not as easy as it sounds.
To make a beta test work, we need to not only duplicate the effort of entering the data, we need to have sufficient commitment from management for the extra time and effort and a commitment to go through the process of managing two timesheets for a time. The response from management is almost always negative.
Of course, there’s no point in having the new timesheet if you’re not going to try out the new features so not only does the publisher need the existing functionality reviewed, but a real effort on the new functionality including configuring, updating any relevant processes and more. Again, that’s a big ask.
What is the user hoping for by being a beta tester?
They’re aligned right?
No.
Mostly beta testers are interested in peeking at just-finished functionality to give it a try and see if it will fulfill their business requirements now. Almost everyone who contacts us to ask if they can be a beta tester isn’t interested in using the software in parallel with the current version. They just want to start using it with the new functionality. We’ve actually tried this in the past and found it’s terribly unsuccessful. The expectations of these users is that the software is fully functional and stable. The antithesis of a version that needs testing.
So they’re not so interested in just reporting issues to be resolved, they have an expectation that the technical support team will be instantly responding with an upgrade to their “beta” version because despite all our recommendations, they’re actually using it in production. In the end, the delays of supporting these people delays the release of the production version more than if we’d never had the beta testers at all.
I can remember working on one of the early release versions of Microsoft Project and having users horrified that when the early release version was replaced by the actual version, there was no path to upgrade the data from beta to production. There was outrage from the users that the work done in the “beta” versions could be saved and updated to the actual version despite warning after warning, the absence of any promise that this would actually be possible and the early release agreement itself. Users had put production data into the release candidate, not in parallel but instead of the current product. Mostly they had to redo that work once the new version was released. So they weren’t really interested in beta testing at all. They just wanted a fully functional, working copy of the next release before anyone else.
Who should and shouldn’t be a beta tester?
On a regular basis we get requests from prospective users offering to be a beta tester. They’ve not read this article of course. They just want a feature that isn’t yet available but might be soon. They may be very smart. They may be very experienced but they’re not experienced with our product. They’re still prospective customers. Prospective clients are a terrible choice as a beta tester.
Ideally, if a client really is willing to assist with the QA process, the testers will first have the capacity to run an enterprise product in parallel with real data and sufficient resources to manage the entire enterprise process including data entry, approval, onboarding, reporting, exporting etc. over a number of periods.
We almost never have clients who are willing to do this and, when we do, it’s almost always because of a particular environment where our technical staff are already embedded or working on some functionality specifically for that client.
So, if I can’t beta test, what can I do?
There are so many ways clients can be supportive of the evolution of the software.
Firstly, they can communicate their desires for features in upcoming versions. When this happens and the desires match up with features already being considered, we will often include dialog from the client as part of the design process. Then we might even give an advance look at how the function will work or, in some cases, weave that function into a specialized version of our product for the client to work with.
Next, clients can be proactive in using the new version in a test or staging environment in a timely fashion as soon as it is available. Early feedback from clients is expected and welcomed and, if there is some issue that pops up at that time, it’s much easier to talk about from the calm staging situation than to say that you’ve just upgraded your production environment with the new version and the whole company has stopped.
More important than all of that is something that our own staff have to be conscious of as much as our clients and prospective clients. Instead of focusing on the availability of a feature, work with the software vendor on your business problem. We recently had this very situation in working with a large software company. We needed help on a feature that is clearly not working as documented. I’ll spare you the long details but the final result was “that feature isn’t going to work like that”. It wasn’t helpful. But in debriefing with my own staff, I discovered that we’d never told the software vendor what we really needed. Instead of talking about the business problem of “I need to get timesheet actuals to your product in the area where timesheet actuals are stored”, we said “this feature, documented on page 465 isn’t working and either fix it or tell me how to code around it.” We fell victim to the very thing we ask of our clients “Please tell me your business problem before I start describing the solution”
So try thinking about what you need to accomplish in terms of a business solution rather than the in terms of the feature you hope will fix that problem in some way.
