Suggestion: QA Testing

Case number:699969-728769
Opened by:jas0501
Opened on:Wednesday, December 17, 2008 - 15:43
Last modified:Saturday, March 21, 2009 - 04:55

I have not a clue as to the current testing capabiliies and procedure. The following is a suggestion as to an approach to improving the quality of releases and quality of the test environment Adequate testing is a hard problem and an automated approach preserves the sanity of the developers and testers while improving the development environment and the product at the same time.

You may already have portions of the following suggestion in place. If so cool! If not the please consider it. It is well worth the effort to create such an environment.

I think it would be well worth the effort to compose an automated test that exercises the various user operations during folding.

Manual testing is laborious and the person responsble will either do a good job and end up in a padded room or do a quick job an end up releasing a buggy product. Automated testing is easier on the sanity of the developers/qa personnel

Granted maintaining the event stream needed for testing is a difficult job if the permitted operation are constantly changing, however it is well worth the effort as many bugs will be easily detected and the drudgery on repeated manual testing is avoided.

As an example the current release crashes if one clears a band connected to a sidechain during a local wiggle. If there was a recorded event stream and contained
- attach band to sidechain
- local wiggle
- clear bands
- stop

then running the test stream would have detected the crash.

Having a very rudemantary protein as the test bed and creating multiple scripts for testing various feature would also provide test scripts available to developers testing changes before checking in the changes to the production source. If fact running a defined subset before checking in changes is an excellent quality control procedure.

If a version of foldit was instrumented to record the event is a format permitting playback the test script creation would be simple and the test suite would grow as the developers/qa tester exercised the code and composed a collection of simple tests.

Note this is also an excellent way to construct reproducers for bugs. The act of fixing a bug often produces some very valuable test scripts that should be added to the test suite.
The end result will be improved quality of product releases and a happier user community. A user community aggravated by crashes and hangs will redirect the energies elsewhere and new users will dismiss the opportunity at the outset.

(Wed, 12/17/2008 - 15:43  |  3 comments)

Joined: 11/10/2007
Groups: Window Group

Thanks for the suggestions. We have several low level automated tests, and we are working on testing higher level features, as you suggest.

Joined: 05/19/2008
Groups: None

...that can emulate user actions, at least the engine itself could be stress-tested easily. ;)

admin's picture
User offline. Last seen 1 day 21 hours ago. Offline
Joined: 11/10/2007
Groups: vi users
Topic: Game: Other » General
Status: Open » Closed

I think the combination of our automated tests for low-level cases and of the folders on the developer preview for high-level cases has improved our release quality. Hence, I'm closing this case. Don't hesitate to re-open it if you disagree with me.


Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons