Partition Tournament Final Results
Our Protein Design Partition Tournament concluded yesterday. In the final week, we ran four regular puzzles for the tournament front-runners, so that the greater Foldit community could help us explore the energy landscapes for those designs. Below, we present the tournament results and discuss the final outcomes.
We will host a Science Chat next Tuesday, October 23 at 21:00 GMT. We'll be happy to discuss any questions about the tournament, as well as any other recent Foldit activity! Leave your questions in the comment section below!
The final rankings for the tournament are as follows:
The champion of the tournament is Galaxie, whose design resisted challengers more effectively than any other design. No challenger was able to find a decoy with energy comparable to the design structure; the highest scoring decoy has a probability of only 10-19 (one in ten billion billion)!
Galaxie's winning design
Congratulations, Galaxie! Galaxie will receive the brand new Partition Tournament achievement (coming soon!). Galaxie was also an exceptional challenger in Phase Two, and found the highest-scoring decoy state for 7 different targets!
Two other players will receive the Partition Tournament achievement, for their outstanding contributions as challengers in Phase Two of the tournament:
The most prolific challenger was robgee, who found 20 decoy states for 14 different targets—more than any other challenger!
The most venturous challenger was Mike Cassidy, who found a decoy state for Partition Puzzle (B): MicElephant with an RMSD of 13.5 Å (off the charts—literally!).
Recall that the initial motivation for this tournament concerned our protein design strategy. In a typical Foldit design puzzle, players optimize the absolute energy (the Foldit score) of their design. However, the success of a protein design depends not on its absolute energy, but on its energy landscape. In theory, a design with a mediocre Foldit score can still have an excellent energy landscape, and might be expected to fold up well in the lab. Inversely, a high-scoring Foldit design could have a problematic energy landscape (with other high-scoring decoys), and would be expected to fold poorly. The Partition Tournament was set up to evaluate Foldit players' designs based on their energy landscapes, instead of their absolute energy.
Note that, of the five top-scoring tournament submissions selected for Phase Two of the tournament (from matosfran, actiasluna, phi16, Galaxie, and fiendish_ghoul) all five performed very well in the tournament. None of these designs were overtaken by a high-scoring decoy. This is encouraging, because it suggests that our normal strategy (i.e. optimizing for absolute energy) is an effective shortcut for finding favorable energy landscapes.
However we also see that many of the lower-scoring submissions still performed well in the tournament, and maintained high partition scores. These designs were able to resist challenges from other Foldit players, and seem to have favorable energy landscapes. This indicates that Foldit players may have protein design skills that are not captured by our energy function. If we focus solely on optimizing absolute energy, then we're probably going to miss out on some perfectly good protein designs.
These are exciting results, and we're very keen to keep exploring protein design with Foldit players! Nevertheless, we should recognize a significant limitation of the tournament results.
Most challengers tended to stay very close to the designed structure. In many of the energy landscapes below, we see the majority of solutions (usually a dense cluster of black dots) right around the 2.5 Å RMSD cutoff. The average decoy had an RMSD of only 4.0 Å.
This was also clear in our weekly tournament updates (see here and here), where we looked more closely at some players' high-scoring decoys. In most cases, the decoys were largely identical to the design structure, and only small, localized regions of the design were refolded by challengers.
These low-RMSD decoys only inform us about how these designs could partially unfold. The challengers have revealed how certain regions of the protein might unfold locally. However, it doesn't tell us much about the stability of the protein core, and how the protein might be able to refold globally. For this we would want to see many more solutions that are very different from the design structure, and that explore the energy landscape distant from the designed structure.
If we were to repeat this tournament, we would need to find ways to encourage more explorational challenges. We could imagine applying a bonus that scales with RMSD (as has already been suggested by some of you), by concealing the designed structure from challengers, or some other means to encourage broader exploration of the energy landscape.
Many of the tournament submissions are excellent designs, and these tournament results are promising, but unfortunately we can't be confident enough to test these in the lab until we have a better picture of the global energy landscape. For that, we'll submit these designs to Rosetta@home for ab initio structure prediction, which is very effective for finding decoys that are globally-refolded.
In summary, the results from the Protein Design Partition Tournament support the idea that our regular design strategy—though not perfect—is still effective for finding designs with favorable energy landscapes.
Foldit players were very effective in this tournament at finding locally-unfolded decoy states. In the weekly updates we saw exactly how this happened, and proposed ways to avoid designs that can unfold locally. Specifically, designers should focus on building a substantial, deeply-integrated core that involves all regions of the protein, and avoid long stretches of completely polar residues. Otherwise, parts of your protein may be able to unfold locally while maintaining a high score.
Partition Puzzles Summary
Below are the final energy landscapes and partition functions for all the Partition Puzzles from Phase Two of the tournament. For an explanation of these plots, see this previous blog post. Thanks to all those who participated in the tournament!
( Posted by bkoep 73 547 | Wed, 10/17/2018 - 22:48 | 3 comments )
Electron Density results and challenge!
We wanted to give you an update on the latest Cryo-EM puzzles.
In Puzzle 1554 we gave you 5 starting models to work with, and these were the results:
For all the energy plots below:
Each green dot represents a Foldit solution plotted against GDT_TS (where a value closer to 1.0 indicates a model closer to what we believe is the native structure) and Rosetta energy (where a very negative value corresponds to a very high Foldit score).
So the further to the right you are, the closer you are to the correct fold... and the lower you are, the better your Foldit score.
We don't know why we were surprised, as Foldit players never cease to amaze us with your incredible results!
This time, however, we really do have a challenge for you... because we've never posted a 221-residue density puzzle before, but these were the results for Puzzle 1579 without any experimental data:
Clearly the starting models we provided you with were nowhere near the native (they were actually 5 different CASP13 server models. You can read more about this at the very bottom if you like).
We realize how big 221-residues is for a Foldit puzzle already, which is why we are giving you over 2 weeks to work on it with electron density.
We know this is a big ask, but we also know that if anyone can do this: it's you!
Best of luck, and keep up the great folding!
For those interested in the background details for these puzzles:
As we mentioned in Puzzle 1554, the recent puzzles are part of a large protein complex with multiple subunits, which has recently been the target of some cryo-electron microscopy (cryo-EM) experiments.
These complexes were actually targets in CASP13 this past summer, but the experimentalists were kind enough to provide us with their cryo-EM data once CASP was over.
The first subunit (from Puzzle 1554) was part of this CASP13 target and the recent 221-residue subunit (Puzzle 1579) was part of this CASP13 target. You can see how large these subunits are, which is why we tackled the 149-residue protein first, and trimmed the 229-residue one for the most recent puzzle.
Most interestingly, Puzzle 1579 has no known homologs (or related proteins that have already been solved), which explains why the CASP servers had so much trouble with their predictions.( Posted by beta_helix 73 1765 | Tue, 10/16/2018 - 17:27 | 6 comments )
Partition Tournament Update: Final Week
We are now in the final week of the Protein Design Partition Tournament! We apologize for the delayed update; some unrelated technical problems kept us from running our weekly analysis yesterday. Thank you for your patience!
Today we were able to analyze the results from all current partition puzzles, plus the two regular puzzles that were posted last week: 1580: Partition Puzzle: fiendish_ghoul and 1582: Partition Puzzle: wisky. As you can see in the summary results below, the regular puzzles resulted in many additional solutions, and more thorough exploration of the energy landscapes for the designs by wisky and fiendish_ghoul.
As of 18:00 GMT October 10, the rankings are as follows:
In the final week, we will post regular puzzles for the top four designs in the tournament. Like last week's puzzles, each of these regular puzzles will be available for four days, and players will be able to load solutions from the original Partition Puzzles. Yesterday, we released the first of these, with 1583: Partition Puzzle: LociOiling. Today, we are releasing 1584: Partition Puzzle: actiasluna, and will follow up the next two days with puzzles for matosfran and Galaxie.
Do your best to find high-scoring decoys in these final puzzles! This is your last chance to explore the energy landscapes for these designs before we calculate the final partition functions to determine the tournament winner!
On Tuesday, October 16 at 23:00 GMT, the last of these puzzles will close, along with all remaining Partition Puzzles. On Wednesday, October 17 we will post the final results of the tournament.
Partition Puzzles Summary
Below are the energy landscapes and partition functions for all remaining Partition Puzzles, as of 18:00 GMT October 10. For an explanation of these plots, see this previous blog post.
( Posted by bkoep 73 547 | Wed, 10/10/2018 - 21:54 | 1 comment )
Partition Tournament Update: More Challenges
We are now two weeks into Phase Two of the Protein Design Partition Tournament! In the Partition Puzzles, challengers continue to find low energy decoys, revealing more of the energy landscape for each design. The rankings have changed significantly since last week, and we expect them to keep shifting over the next two weeks of the tournament!
As of 18:00 GMT October 1, the rankings are as follows:
This week we will start reposting some of the Partition Puzzles as regular Foldit puzzles. We will focus on puzzles that still have a high partition score, but that have not seen very much activity from challengers. We hope that more people will attempt the regular puzzles, and explore more of the energy landscape for these designs. Any players that have participated in a Partition Puzzle will be able to load their work if it is reposted as a regular puzzle.
Unfortunately, another design has fallen out of competition. Partition Puzzle: alcor29 has dropped to a score of 0.00, after challengers found a decoy structure with energy lower than the design, by about 3 kcal/mol (+30 Foldit points). We will be closing this puzzle early, so that challengers can focus on the remaining Partition Puzzles.
Taking a closer look at the decoys from this puzzle, we see that the α-helix and β-hairpin in the foreground can pack in different orientations with the larger β-sheet in the back. We loaded the original design into Foldit with a Core Existence Objective (which was not available in the original design puzzle), and we can see that this design falls short of the typical "30% core requirement" that we like to see in protein designs. In fact, only about 18% of residues in this design contribute to the protein core. The implication is that there are not very many interactions between the different parts of this protein, allowing them to refold in different orientations with minimal consequences.
This result highlights that a substantial, tightly-packed core is essential for a well-folded protein. Good core packing, with interactions between all parts of the protein, helps to ensure that there is one (and only one) stable configuration for the protein!
Partition Puzzles Summary
Below are the energy landscapes and partition functions for the remaining 16 Partition Puzzles, as of 18:00 GMT October 1. For an explanation of these plots, see this previous blog post.
Keep up the good work! There has been a lot of progress since last week's results, but there are surely more high-scoring decoys to be discovered! Check back next Tuesday, October 9 for the next tournament update!
( Posted by bkoep 73 547 | Mon, 10/01/2018 - 23:17 | 1 comment )
Partition Tournament Update: Initial Results
Phase Two of the Protein Design Partition Tournament has been underway for one week, now. Thank you, to all the Foldit players that have been participating! All 20 Partition Puzzles have seen initial challenges, and already we’re seeing that some designs are more resistant than others.
To recap: the Partition Puzzles will be evaluated by their partition functions, which we explained in a previous blog post. In order to calculate the partition function for a puzzle, we cluster all of players solutions to identify decoy states. The relative energy of those decoy states tells us how the ensemble will be distributed over those states.
In order to rank the tournament designs, we’re using a "Partition Score” to summarize how well each design resists challenges from other players. This score measures how much of the partition function has been claimed by challengers, on a logarithmic scale, where a higher score indicates a more stable protein. For example, a Partition Score of 12.0 means that challengers have claimed 10-12 (one trillionth) of the partition function; likewise, we could say the design is about 1012 times more likely to fold into the designed structure than all of the decoys combined.
As of 18:00 GMT September 24, the rankings are as follows:
Three puzzles have a Partition Score of 0.00, meaning that challengers have found decoys that score even better than the original design. We are closing these puzzles early, so that challengers can focus on the other Partition Puzzles. Below, we'll take a closer look at the solutions from these puzzles, and what features might have led to their demise. If you want to skip ahead, the summary updates for all Partition Puzzles are at the bottom of this blog post.
Protein design pitfalls
First, in Partition Puzzle: MicElephant, we see that challengers have found a decoy with an energy that is lower than the design by about 6 kcal/mol (+60 Foldit points).
Taking a closer look at the decoy structure, we see that challengers have refolded one of the β-strands into a helix. In the original design, this strand was designed with all polar residues. Without any hydrophobic residues to be buried, this section of the protein will happily flop around in solution. You might think that it would be bad to break the ladder-like hydrogen bonds that were designed between this strand and its partner—however, these surface residues can compensate by making new hydrogen bonds with the surrounding water.
Next, in Partition Puzzle: ManVsYard, challengers have found a decoy that is lower in energy by about 4 kcal/mol (+40 Foldit points), as well as a second decoy with an energy that is about the same as the designed structure.
The low-scoring decoy exposes the short helix that was packed at the top of the β-strands. Again, we see that this section was designed with entirely polar residues, which are happy to interact with the surrounding water instead of the protein. In the second decoy, challengers have completely refolded a β-strand into an α-helix. This is a little surprising, since the designed residues in this section form an alternating pattern of polar and non-polar residues (which normally favors β-strands). However, this strand may not have been packed tightly enough in the protein core; it is able to make equally favorable interactions in the protein core when refolded as a helix.
Finally, in Partition Puzzle: manu8170, challengers found two decoys that are lower in energy by about 5 kcal/mol (+50 Foldit points).
The decoys differ from the design structure in a couple regions. First we see that the terminal helix can be unwound, and is "fraying" at the end. In the original design, these end residues (again, all polar residues) were floating off in solvent, and did not make very strong interactions with the rest of the protein; these residues can be unfolded relatively easy. We also see that this design has long loops that are made mostly of polar residues, and do not form regular secondary structure. Such polar, unstructured loops are expected to be flexible and disordered in solution; challengers were able to refold the loops while keeping the rest of the structure intact.
The clear message from these early results is that long stretches of polar residues can be easily unfolded. The burial of non-polar residues is the main driving force of protein folding, and it's important that every region of a protein contributes to its core!
Partition Puzzles Summary (September 24)
Below are the energy landscapes and partition functions for the remaining 17 Partition Puzzles, as of 18:00 GMT September 24. For an explanation of these plots, see this previous blog post.
Some puzzles have seen very little activity, and their rankings may be artificially boosted due to a lack of challengers. There is still plenty of opportunity to find high-scoring decoys in your opponent's Partition Puzzles!bkoep 73 547 | Mon, 09/24/2018 - 21:18 | 6 comments )