CASP8 Results
We've got some very exciting news! The table below shows our placement in CASP8 puzzles. Click on the Target column to go to the respective CASP8 page showing results. The rank column shows Foldit solution rank according to the default CASP8 sorting criteria (GDT_TS). Entries column shows the number of submissions from biochemistry labs worldwide. The puzzle column shows the corresponding foldit puzzle.
| Target | Rank | Entries | Puzzle | Group / User |
|---|---|---|---|---|
| TR389 | 3 | 71 | TS389_5 | Another Hour Another Point |
| TR389 | 7 | 71 | TS389_1 | Another Hour Another Point |
| TR389 | 21 | 71 | TS389_3 | Freedom Folders |
| TR461 | 2 | 83 | TS461_1 | Another Hour Another Point |
| TR461 | 12 | 83 | TS461_3 | Freedom Folders |
| TR461 | 19 | 83 | TS461_5 | Another Hour Another Point |
| TR469 | 2 | 74 | TS469_5 | Another Hour Another Point |
| TR469 | 3 | 74 | TS469_4 | Another Hour Another Point |
| TR469 | 45 | 74 | TS469_1 | Catalunya |
| TR469 | 50 | 74 | TS469_3 | caglar (solo) |
| TR488 | 1 | 77 | TS488_1 | Another Hour Another Point |
| TR488 | 3 | 77 | TS488_3 | Another Hour Another Point |
| TR488 | 26 | 77 | TS488_5 | Another Hour Another Point |
| TS492 | 2 | 527 | TS492_3 | Another Hour Another Point |
| TS492 | 96 | 527 | TS492_5 | Another Hour Another Point |
Looking at the numbers, it appears that Foldit players did amazingly well! You placed in top 3 in a number of puzzles and it seems that we apparently won one of the puzzles. Congratulations, Foldit players! You're amzing.
Looking beyond numbers, the preliminary conclusion by biochemist is that foldit players are on par, but not better than protein folding experts at trying to solve the same problem with all tools available to them. It also appears that foldit outperformed all fully automated server submissions. Hopefully over time foldit can do even better, but being able to produce solutions of same quality as experts means that the top science research can now also be done outside of labs by game players, significantly speeding up the process of scientific advancement! As developers, we are truly inspired by this.
Of course, we didn't do well on all puzzles, but even this has been very useful to us as we're evolving the game further to significantly improve over these findings.
Some caveats that should be mentioned:
- there is a considerable amount of discussion over what is the best way to score the quality of the solution, and GDT_TS is just one of them that CASP8 sorts their solutions by. The ranking would be slightly different with different metrics, but the overall conclusion should not change.
- it is possible that the specific starting point including all the constraint bands played a significant role in the success of foldit solutions, and we may have gotten a particularly good one from the Baker lab. To understand this better, we'll be introducing new puzzles (more on that tomorrow).
- foldit did not have the sequence allignment tool that is fundamental for protein prediction. Scientists use sequence alighment a lot to find the best score. For next CASP challenge we will make sure that these tools are also part of the game, which should significantly enable the gameplay and the resulting proteins.
zoran 70 1803 |
Wed, 12/17/2008 - 21:41 |
9 comments
)
I think this is amazing. In half of the puzzles, Foldit is in the first 10%.
It is also interesting that both of the baddest scores are on the TR469, while at the same time beeing on place 2 and 3. I'm new, so I don't know if there are special circumstances.
I would like to know if there is a *single* biochemist that was better than Foldit?
But either way, one is sure: It's easier to "crop" Foldit players than biochemists so there is a lot of improvements possible. More people means it's likely for better results even if its only because of luck. (I did my 115 (<15): Exocyst 1st result (at the moment at least) in about 3 hours I think and it was my second puzzle ever, the first I did more than just looking how what works.)
I think that results are very promising, and am happy to see this.
Some things I think should be kept in mind though.
1) Foldit was still in it's infancy, and some of the tools we have now either were not available for these puzzles or were brand new, and nobody had experience using them.
2) For most of us, this is something we all do a couple hours a day and not as a full time job. I know I spent a lot more time on puzzles during CASP, but with more free time I'm sure those puzzles could have been significantly improved.
3) Experience, looking back at what I was doing during this competition, compared to what I'm doing now, I don't see that period as any more than a step in a learning experience. Towards the end it got better, but overall it was pretty unrefined. I know others feel this way as well.
Pletsch,
i'm glad you also thing that things might be much better now. In fact you'll have an opportunity to prove this over the next month. see my next blog post.
In truth, the game design and development has also been a continuous learning experience. Unlike designing the game when you can freely choose the outcome, and really just maximize the players enjoyment, when developing a game that needs to achieve a specific scientific discovery, we're really doing research on how to most enable the power of collective human problem solving ability, and still try to be engaging and fun in the process. we're a lot more succesful at this than we were 3 months ago, and i'm sure we can be a lot better still: the human brain power keeps surprising us. The best is yet to come.
Great news: I fear that this will only encourage me to waste even more time playing foldit than I already do. A couple of questions:
This thing called GDT_TS (Global Distance Total Test Score it appears) that's used in CASP8 to determine the closeness to the experimentally-derived structure: is it broadly similar to Rosettta@Home's RMSD metric? An improvement on it perhaps?
The proteins we've been folding so far are pretty small by the standards of those that occur in nature: how do you see the problem scaling with increasing protein size? It seems it ought to be exponential with the size of the protein, at least as far as computational methods are concerned. I think that humans might well do better. Any thoughts on this?
Spvincent, i sure how you don't see it as time wasted, as evidenced by the results ;)
with respect to the metrics, i believe they're all some modifications of the euclidean distance between every pair of atoms. i'm a computer scientists so I'll let the biochemists answer this specifically, but from what i gather the raw distance metric has number of problems. One of them, is that some labs "compress" the entire protein slightly which turns out to improve the distance metric since everything is that much closer. this, however, introduces a number of unrealistic collisions between atoms as well, despite the fact that the distance measure is better. Apparently, at every CASP meeting people spend a good deal of time arguing about the most appropriate metrics.
regarding the size of the proteins. in nature there are many small and many large proteins, we've certianly been focusing on the smaller ones. it turns out that purely computational methods are pretty bad at very large protiens to the point that predictions are usually not even in the ballpark. It is quite likely that people could improve on this, and we'll tackle this problem soon. The issues of course are that visualization is more tricky, and our manipulation tools might be slow for direct game interaction.
Protein design, however, is something that we think people will really good at, and currently no automated methods even exist for designing the protein (picking the sidechains so that the eventual fold achieves a desired function). Design will also have a more immediate implications to dealing with diseases. I will write about our plans to release a first design game in a blog entry soon.
I deleted several spam messages from username lanlan00.
Might be a good idea to monitor his/her account or otherwise block it from posting
There are some errors in the table:
The links of the puzzles TS389_5, TS389_1 and TS389_3 are leading all to CASP8 T0389 Refinement Start 1
The links of the puzzles TS461_1 and TS461_1 are leading both to CASP8 T0461 Start 3 Re-run
The links of the puzzles TS469_1 and TS469_3 are leading both to CASP8 T0469 Start 1 and there are no links for the puzzles TS469_5 and TS469_4
we'll check again, but i don't believe these are errors. the CASP8 submissions all came out of the different solutions to the same foldit puzzle.
How well did we do in comparison to the Rosetta project (which share a similar algorithm for predicting structure)?