CASP8 Results

We've got some very exciting news!  The table below shows our placement in CASP8 puzzles.  Click on the  Target column to go to the respective CASP8 page showing results. The rank column shows Foldit solution rank according to the default CASP8 sorting criteria (GDT_TS).   Entries column shows the number of submissions from biochemistry labs worldwide.  The  puzzle column shows the corresponding foldit puzzle.

Looking at the numbers, it appears that Foldit players did amazingly well!  You placed in top 3 in a number of puzzles and it seems that we apparently won one of the puzzles.  Congratulations, Foldit players!  You're amzing.

Looking beyond numbers, the preliminary conclusion by biochemist is that foldit players are on par, but not better than protein folding experts at trying to solve the same problem with all tools available to them.  It also appears that foldit outperformed all fully automated server submissions.  Hopefully over time foldit can do even better, but being able to produce solutions of same quality as experts means that the top science research can now also be done outside of labs by game players, significantly speeding up the process of scientific advancement!  As developers, we are truly inspired by this.

Of course, we didn't do well on all puzzles, but even this has been very useful to us as we're evolving the game further to significantly improve over these findings. 

Some caveats that should be mentioned:

  • there is a considerable amount of discussion over what is the best way to score the quality of the solution, and GDT_TS is just one of them that CASP8 sorts their solutions by.  The ranking would be slightly different with different metrics, but the overall conclusion should not change.
  • it is possible that the specific starting point including all the constraint bands played a significant role in the success of foldit solutions, and we may have gotten a particularly good one from the Baker lab.  To understand this better, we'll be introducing new puzzles (more on that tomorrow).
  • foldit did not have the sequence allignment tool that is fundamental for protein prediction.  Scientists use sequence alighment a lot to find the best score.  For next CASP challenge we will make sure that these tools are also part of the game, which should significantly enable the gameplay and the resulting proteins.
( Posted by  zoran 103 3077  |  Wed, 12/17/2008 - 21:41  |  9 comments )
0
csohad's picture
User offline. Last seen 10 weeks 1 day ago. Offline
Joined: 06/04/2008
Groups: None
Thanks for the update. great news.

How well did we do in comparison to the Rosetta project (which share a similar algorithm for predicting structure)?

Joined: 12/14/2008
that's really cool!

I think this is amazing. In half of the puzzles, Foldit is in the first 10%.

It is also interesting that both of the baddest scores are on the TR469, while at the same time beeing on place 2 and 3. I'm new, so I don't know if there are special circumstances.

I would like to know if there is a *single* biochemist that was better than Foldit?

But either way, one is sure: It's easier to "crop" Foldit players than biochemists so there is a lot of improvements possible. More people means it's likely for better results even if its only because of luck. (I did my 115 (<15): Exocyst 1st result (at the moment at least) in about 3 hours I think and it was my second puzzle ever, the first I did more than just looking how what works.)

Joined: 05/26/2008
About the results

I think that results are very promising, and am happy to see this.

Some things I think should be kept in mind though.

1) Foldit was still in it's infancy, and some of the tools we have now either were not available for these puzzles or were brand new, and nobody had experience using them.

2) For most of us, this is something we all do a couple hours a day and not as a full time job. I know I spent a lot more time on puzzles during CASP, but with more free time I'm sure those puzzles could have been significantly improved.

3) Experience, looking back at what I was doing during this competition, compared to what I'm doing now, I don't see that period as any more than a step in a learning experience. Towards the end it got better, but overall it was pretty unrefined. I know others feel this way as well.

zoran's picture
User offline. Last seen 3 years 43 weeks ago. Offline
Joined: 11/10/2007
Groups: Window Group
About the results

Pletsch,

i'm glad you also thing that things might be much better now.  In fact you'll have an opportunity to prove this over the next month.  see my next blog post.

In truth, the game design and development has also been a continuous learning experience.  Unlike designing the game when you can freely choose the outcome, and really just maximize the players enjoyment, when developing a game that needs to achieve a specific scientific discovery, we're really doing research on how to most enable the power of collective human problem solving ability, and still try to be engaging and fun in the process.  we're a lot more succesful at this than we were 3 months ago, and i'm sure we can be a lot better still: the human brain power keeps surprising us.  The best is yet to come.

spvincent's picture
User offline. Last seen 14 hours 44 min ago. Offline
Joined: 12/07/2007
Groups: Contenders
Great news

Great news: I fear that this will only encourage me to waste even more time playing foldit than I already do. A couple of questions:

This thing called GDT_TS (Global Distance Total Test Score it appears) that's used in CASP8 to determine the closeness to the experimentally-derived structure: is it broadly similar to Rosettta@Home's RMSD metric? An improvement on it perhaps?

The proteins we've been folding so far are pretty small by the standards of those that occur in nature: how do you see the problem scaling with increasing protein size? It seems it ought to be exponential with the size of the protein, at least as far as computational methods are concerned. I think that humans might well do better. Any thoughts on this?

zoran's picture
User offline. Last seen 3 years 43 weeks ago. Offline
Joined: 11/10/2007
Groups: Window Group
some answers

Spvincent, i sure how you don't see it as time wasted, as evidenced by the results ;)

with respect to the metrics, i believe they're all some modifications of the euclidean distance between every pair of atoms.  i'm a computer scientists so I'll let the biochemists answer this specifically, but from what i gather the raw distance metric has number of problems.  One of them, is that some labs "compress" the entire protein slightly which turns out to improve the distance metric since everything is that much closer.  this, however, introduces a number of unrealistic collisions between atoms as well, despite the fact that the distance measure is better.  Apparently, at every CASP meeting people spend a good deal of time arguing about the most appropriate metrics.

regarding the size of the proteins.  in nature there are many small and many large proteins, we've certianly been focusing on the smaller ones.  it turns out that purely computational methods are pretty bad at very large protiens to the point that predictions are usually not even in the ballpark.  It is quite likely that people could improve on this, and we'll tackle this problem soon.  The issues of course are that visualization is more tricky, and our manipulation tools might be slow for direct game interaction.

Protein design, however, is something that we think people will really good at, and currently no automated methods even exist for designing the protein (picking the sidechains so that the eventual fold achieves a desired function).  Design will also have a more immediate implications to dealing with diseases.  I will write about our plans to release a first design game in a blog entry soon.

xiando's picture
User offline. Last seen 1 year 43 weeks ago. Offline
Joined: 10/19/2008
Groups: Oma Gawd
Several Spams by lanlan00 deleted

I deleted several spam messages from username lanlan00.

Might be a good idea to monitor his/her account or otherwise block it from posting

Madde's picture
User offline. Last seen 2 weeks 21 hours ago. Offline
Joined: 05/29/2008
Groups: Void Crushers
Incorrect links

There are some errors in the table:

The links of the puzzles TS389_5, TS389_1 and TS389_3 are leading all to CASP8 T0389 Refinement Start 1

The links of the puzzles TS461_1 and TS461_1 are leading both to CASP8 T0461 Start 3 Re-run

The links of the puzzles TS469_1 and TS469_3 are leading both to CASP8 T0469 Start 1 and there are no links for the puzzles TS469_5 and TS469_4

zoran's picture
User offline. Last seen 3 years 43 weeks ago. Offline
Joined: 11/10/2007
Groups: Window Group
links

we'll check again, but i don't believe these are errors.  the CASP8 submissions all came out of the different solutions to the same foldit puzzle.

Get Started: Download
  Windows    OSX    Linux  
Windows
(Vista/7/8)
OSX
(10.7 or later)
Linux
(64-bit)

Are you new to Foldit? Click here.

Are you a student? Click here.

Are you an educator? Click here.
Search
Only search fold.it
Recommend Foldit
User login
Soloists
Evolvers
Groups
Topics
Top New Users
Sitemap

Supported by: UW Center for Game Science, UW Department of Computer Science and Engineering, UW Baker Lab, DARPA, NSF, HHMI, Microsoft, and Adobe