CASP11 Results!

It's been several months since CASP11 ended in August of 2014, and in December assessors presented their full analysis of CASP predictions at the CASP11 meeting in Mexico. You can see results for all teams and assessor presentations on the CASP website, but we'd like to focus for a minute on Foldit's performance.

Congratulations first to the Anthropic_Dreams team, whose top model ranked first for target TR759, and also to Beta Folders, whose model ranked first for the FOLDIT team on target TR829!

There were other "would-be" blue ribbons, in which Foldit players produced first-rate models that we failed to select as our best. Although they did not select it as their top model, the GoScience team submitted the best overall prediction for target TR769! Likewise, WeFold used a Foldit model to develop the best overall prediction for target TR837!

Overall, we were outperformed most notably in the Refinement category by the FEIG team, which uses a vast amount of supercomputing power to explicitly simulate protein dynamics; and in the Contact-Assisted category by the LEE team, which was able to take advantage of "ambiguous contacts" that were not addressed in our Foldit puzzles.

Note: These rankings are calculated using GDT_TS, which is just one metric for evaluating model quality. The CASP website explains some other metrics that might be used to evaluate models.

Refinement
With a bit of closer examination, we've concluded our troubles in the Refinement category can be divided into several distinct cases:

In the first case, Foldit players were very good at exploring solutions close to the native, but the solutions scored poorly and were not submitted as CASP predictions. Looking at the energy plot for target TR769 below, we can see that players found solutions with GDT_TS as high as 0.94(!), although their energies were less favorable than solutions near the starting model.

Energy plot for TR769. Every red dot represents a Foldit solution plotted against GDT_TS (where a value closer to 1.0 indicates a model closer to the native structure) and Rosetta energy (where a very negative value corresponds to a very high Foldit score). The blue dots represent the five solutions that were submitted as CASP predictions by the FOLDIT team. The vertical black bar represents the GDT_TS of the starting model.

In the second case, which was more common, the "energy funnel" looked good—meaning that models with better GDT_TS had more favorable energies—but Foldit players simply weren't able to explore solutions close to the native structure. In the energy plot for target TR782, for example, we can see that the best-scoring solutions at the bottom of the funnel were also the most similar to the native. Unfortunately, most Foldit players tended to move the protein away from the native conformation.


Energy plot for TR782. Every red dot represents a Foldit solution plotted against GDT_TS (where a value closer to 1.0 indicates a model closer to the native structure) and Rosetta energy (where a very negative value corresponds to a very high Foldit score). The blue dots represent the five solutions that were submitted as CASP predictions by the FOLDIT team. The vertical black bar represents the GDT_TS of the starting model.

Lastly, we saw a few targets that were both difficult to explore and difficult to score. For target TR803, solutions appeared to score better and better as they diverged from the native structure, and most Foldit players spent time moving away from the native.


Energy plot for TR803. Every red dot represents a Foldit solution plotted against GDT_TS (where a value closer to 1.0 indicates a model closer to the native structure) and Rosetta energy (where a very negative value corresponds to a very high Foldit score). The blue dots represent the five solutions that were submitted as CASP predictions by the FOLDIT team. The vertical black bar represents the GDT_TS of the starting model.

Contact-Assisted
In the Contact-Assisted category, we were happy to find that Foldit players could use predicted contacts to make huge improvements in their solutions. In most cases, we posted an initial "Ts" puzzle with a limited set of simulated contacts, and then followed it up with a more complete set of "Tc" contacts. In every instance, more contacts resulted in better predictions.

For example, compare Foldit solutions for Ts/Tc827 with T0827, which was posted without contacts under the guise of 1005: De-novo Freestyle 44. Not only did additional contacts result in further exploration toward the native structure, but the complete contacts also reshaped the energy funnel to strongly favor solutions closer to the native!


Energy plot for T0827, Ts827, Tc827. Every red dot represents a Foldit solution plotted against GDT_TS (where a value closer to 1.0 indicates a model closer to the native structure) and Rosetta energy (where a very negative value corresponds to a very high Foldit score). The blue dots represent the five solutions that were submitted as CASP predictions by the FOLDIT team. The vertical black bar represents the GDT_TS of the starting model.

In the future, we'll be working to see how we can improve scoring in cases like TR769, and how to encourage more exploration for targets like TR782. We're encouraged by the progress Foldit players have made in the use of predicted contacts, and are looking forward to applying this method in future non-CASP efforts. A big thanks to all of our players for their tireless contribution to structural biology research!

Check out the full set of Foldit energy plots for the Refinement and Contact-Assisted categories.

( Posted by  bkoep 81 1003  |  Wed, 01/28/2015 - 01:43  |  4 comments )
6
spmm's picture
User offline. Last seen 17 weeks 5 days ago. Offline
Joined: 08/05/2010
Groups: Void Crushers
congratulations!

There is a 'Finder Table' for CASP 11 in the new CASP wiki section, quickly find a target and see which puzzles it appeared in and vica versa:

http://foldit.wikia.com/wiki/CASP11-FinderTable

Joined: 05/19/2009
Groups: Contenders
Congratulations to all teams,

Congratulations to all teams, CASP was hard work !

Joined: 08/09/2010
evaluating solutions

Perhaps for future CASP we could have a function to evaluate a solution (I suppose it would be too slow at runtime instead of the current energy score), with similar params than CASP uses, to know if we are doing well

bkoep's picture
User offline. Last seen 13 hours 2 min ago. Offline
Joined: 11/15/2012
Groups: None
Native required

If I understand you correctly, BitSpawn, the problem with using CASP quality metrics (like GDT_HA) is that they all measure "similarity to native." And, of course, the native structure is unknown to us while the target is open!

There are perhaps other inconsistencies that we could work on. For example, we learned this year that the CASP organizers defined "contacts" a little differently than we did, so we could change the way we score contact-assisted puzzles.

Get Started: Download
  Windows    OSX    Linux  
Windows
(Vista/7/8)
OSX
(10.7 or later)
Linux
(64-bit)

Are you new to Foldit? Click here.

Are you a student? Click here.

Are you an educator? Click here.
Search
Only search fold.it
Recommend Foldit
User login
Soloists
Evolvers
Groups
Topics
Top New Users
Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Microsoft, Adobe, RosettaCommons