Suggestion: QA Testing
|Opened on:||Wednesday, December 17, 2008 - 15:43|
|Last modified:||Saturday, March 21, 2009 - 04:55|
I have not a clue as to the current testing capabiliies and procedure. The following is a suggestion as to an approach to improving the quality of releases and quality of the test environment Adequate testing is a hard problem and an automated approach preserves the sanity of the developers and testers while improving the development environment and the product at the same time.
You may already have portions of the following suggestion in place. If so cool! If not the please consider it. It is well worth the effort to create such an environment.
I think it would be well worth the effort to compose an automated test that exercises the various user operations during folding.
Manual testing is laborious and the person responsble will either do a good job and end up in a padded room or do a quick job an end up releasing a buggy product. Automated testing is easier on the sanity of the developers/qa personnel
Granted maintaining the event stream needed for testing is a difficult job if the permitted operation are constantly changing, however it is well worth the effort as many bugs will be easily detected and the drudgery on repeated manual testing is avoided.
As an example the current release crashes if one clears a band connected to a sidechain during a local wiggle. If there was a recorded event stream and contained
- attach band to sidechain
- local wiggle
- clear bands
then running the test stream would have detected the crash.
Having a very rudemantary protein as the test bed and creating multiple scripts for testing various feature would also provide test scripts available to developers testing changes before checking in the changes to the production source. If fact running a defined subset before checking in changes is an excellent quality control procedure.
If a version of foldit was instrumented to record the event is a format permitting playback the test script creation would be simple and the test suite would grow as the developers/qa tester exercised the code and composed a collection of simple tests.
Note this is also an excellent way to construct reproducers for bugs. The act of fixing a bug often produces some very valuable test scripts that should be added to the test suite.
The end result will be improved quality of product releases and a happier user community. A user community aggravated by crashes and hangs will redirect the energies elsewhere and new users will dismiss the opportunity at the outset.