The NewChapter roll-out has created some frustration among players. The scientists and the developers want to reassure everyone that we are listening to your frustrations, and are working hard to address them. You can expect the playability of the game to improve considerably in the next little while. This blog post is intended to help people to understand the reasons for some of these temporary problems, and to thank you for your patience as we resolve the issues, guided by your feedback.

What is Rosetta, and how is it related to the current Foldit problems?

     Rosetta is the software developed by the Baker lab for protein design and structure prediction. It has core modules for scoring the energy of a protein in a given conformation, and a number of automated protocols that use these core modules to do specific things (e.g. predict a structure from a sequence, design a sequence to fold into a desired structure, design a binding interface, etc.). For reasons that are being described in the series of blog posts on the physics of Foldit, the scientists are constantly improving Rosetta’s core modules that control how protein energies are calculated. As we improve these core modules, our strategies for predicting structures and for designing proteins often have to change. This can be frustrating for us, since it means revisiting and revising established protocols, but it ultimately results in greater predictive power and in new design capabilities.

     Foldit uses the core modules from Rosetta to score protein energetics, but allows the ingenuity of intelligent human gamers to replace the limited artificial intelligence of the automated protocols. This helps science in two ways: players can often make better predictions than the automated protocols, and by studying how players are doing what they’re doing, we (the scientists) can make the automated protocols better. Of course, unlike automated protocols, players need to have a layer in between them and the core Rosetta modules: the actual game engine, which includes the graphical user interface and the specific tools that players use to manipulate protein structures. This was created by developers in the University of Washington Center for Game Science.

     Periodically, as Rosetta's ability to score proteins gets better, it is important to update Foldit with the latest Rosetta core modules. In the last year, there have been some major improvements made to Rosetta’s energy function, and the NewChapter release was intended to carry these over into Foldit. The process of bringing Rosetta’s updated core modules into Foldit can create problems, however.

So what are the problems?

     The problems that have arisen as a result of switching over to the new scoring function fall into a few categories:

1. Bugs: These are actual errors in the program (crashes, stability problems, ballooning log files, etc.). These come from the fact that the special tools that have been written to allow players to interface with Rosetta’s core (the game interface and the protein manipulation tools that you use) have to be compatible with Rosetta’s core, and the changes to the core can break the connection to the game interface. Most of the outright bugs were corrected long before the NewChapter release. In some cases, it was necessary to change the tools a fair bit to make them compatible with the updated core – hence the new Wiggle, for example. Some intermittent bugs (particularly related to stability) are harder to diagnose without a large user base reporting issues, and these are being addressed now as the developers learn about them.

2. Slower or altered performance: The new Rosetta core computes protein energetics more accurately than before, with fewer artifacts or false low-energy states, but this is a slightly more computationally expensive calculation. This means that, in the absence of any other changes, game play would be slower and less enjoyable. To try to correct for this, the tools have been revised a bit (for example, a particular tool might refresh the protein display after fewer cycles of computation, now, allowing reasonable interactivity despite slower per-cycle computation). Large-scale player feedback is necessary, though, to establish the right balance between responsiveness and the rate of convergence to a solution. We might not have the optimal balance yet, but can improve this as we hear from you.

3. Tools that are less functional: In some cases, a tool is functional (i.e. when you click on it, it does something that looks like what it is supposed to do), but no longer has the particular behavior that players have come to expect, and is possibly less useful than it was before. Many issues were raised about the rebuild tool finding fewer useful structures, for example, particularly when building helices. The causes for this can be complex issues related to the interplay between the tool developed by the game developers and the core modules developed by the scientists. These are problems that the game developers can and will resolve, but this relies on hearing from many gamers about exactly what “less functional” means to them. Where outright bugs are usually easy to recognize, it’s not always easy for developers who might not know players’ preferred strategies to recognize reduced functionality in a tool.

4. Scripts or strategies that no longer work as before: Although this can be frustrating, this is not one of the problems that the developers can address, but is rather a challenge for the players’ ingenuity. From a scientific perspective, it parallels the fact that Rosetta developers often have to revise automated protocols to accommodate changes to the core Rosetta modules that are intended to improve accuracy or precision. From a gaming perspective, you can think of it this way: we have changed the rules of the game slightly, just as the rules of chess would change if the board were one row wider or if a new piece were introduced. Players are now invited to use their intelligence and skill to revise existing strategies, or to develop new strategies, to maximize their scores given the slightly altered gaming environment.

Why can’t we just introduce the new scoring function, but leave the old tools as they were?

     As mentioned above, the new scoring function takes slightly longer to score a protein. This means that the game’s responsiveness would take a big hit if we left everything else as it was before. Some of the game’s tools would also not interface correctly with the new Rosetta core modules, necessitating the revised versions that have been introduced. Ultimately, the NewChapter changes represent the minimum set of changes needed to accommodate the updated scoring function while keeping gameplay intact. That is to say, we're trying not to change your gameplay experience if we can avoid it; we're just forced to make these changes to tools, since without them, the changes would be even larger. The developers welcome feedback on ways in which gameplay can be improved given the altered scoring function, however, and are addressing the issues currently raised.

Why can’t we roll back to the old game while we wait for the game developers fix all of the new problems?

     In order for Foldit to be a useful scientific tool (particularly one that we can compare to our other tool, Rosetta), we need it to have the latest and most scientifically accurate core functionality, particularly with regards to scoring the energies of proteins. This makes the roll-out of NewChapter critical. The NewChapter changes were tested by the developers to find most of the outright bugs, and more bugs were identified and fixed during the pre-release. Some subtle, somewhat rarer bugs could not be identified until larger pools of people were playing, but these are now being addressed. Playability issues, however, can only be found by experts in the game: the players themselves. As mentioned before, Foldit players contribute to science not just by making useful predictions about proteins, but also by teaching us (the scientists) how to teach our computers to make useful predictions about proteins. In part, it is by learning from the community what constitutes a playable game that we achieve the second goal; in return, we will do our utmost to make the game as playable as possible for you. Once the playability issues are addressed, the NewChapter changes will ultimately allow players to make more reliable predictions about protein structures, as well as new designs that are more likely to fold, making NewChapter an essential step forward for Foldit. Rolling back, however, would mean that we would have no means of evaluating playability and addressing the issues that are being raised now. This would mean that we would end up putting you, the players, through the same frustrations again at whatever time we chose for the subsequent roll-out -- i.e. the problems can't be fixed without the community playing the game and giving feedback about playability.

     We recognize that a large change to the core of the game creates many problems that can be frustrating for players, and so we greatly appreciate your patience and your feedback as we work to make the game as functional, playable, and enjoyable as possible. Thank you for continuing to play Foldit, and for helping to advance science not only with your winning solutions, but also with your winning gameplay strategies!

( Posted by  v_mulligan 124 528  |  Sat, 02/08/2014 - 00:29  |  5 comments )