Scoring A Twice Released Protein

Case number:671071-995549
Topic:Game: Other
Opened by:karstenw
Opened on:Tuesday, July 16, 2013 - 01:16
Last modified:Wednesday, July 17, 2013 - 17:19

This solution is assuming we continue to get a protein released twice, once as a handfold and again as regular.
It is intended to be a rewarding gaming solution for nearly all players.

The suggestion:
A protein being released twice should only take a player's single best rank to apply towards the Global Rank.

Who benefits?
Strong Handfolders- They are able to have their talents rewarded even if they prefer not to use scripts, or struggle using scripts.

Strong Scriptors- They are able to have their talents rewarded even if they prefer not to do handfold-only, or struggle with handfolding.

Power Folders (people who are good at running both releases of the protein)- In the current system, they get two ranking scores, for instance r4 and r6. That is an average combined score of r5 in global total scoring. In this proposed system they get to drop their lowest score and would get a score of r4, a small bonus for employing a broad skill set.

Casual Folders- when fitting folding into their busy schedule, they can wait for the regular protein release, and run some scripts while they go shopping or to work. When playing as a hobbyist on their laptop, they can enjoy a day of handfolding without feeling left behind if they cannot dedicate their poor computer to the high demands of script work.

Who Does not benefit?
A folder who believes that handfold only does not warrant any scoring reward simply based on methodology.
A folder who believes that scripts should not be used, or rewarded as they are not providing direct manual effort by the folder.

(Tue, 07/16/2013 - 01:16  |  14 comments)

Joined: 12/06/2008
Groups: Contenders

What if I happen to be both a strong hand folder and a strong user of scripts? Should I be penalized by not being rewarded for my efforts in both modes?

I'm not seeing the added value of this suggestion. Those who prefer to play only one mode will get the exact rank scoring your way as scoring using the current methods. The ones who play both modes lose a potentially high rank, if they score well in both modes.

gitwut's picture
User offline. Last seen 3 years 31 weeks ago. Offline
Joined: 05/18/2012
Groups: Contenders

I would like to see if statistics support the need for hand folding only puzzles. It should be relatively easy for the devs to provide us with statistics for round one vs. round two results obtained so far.

I don't think hand folding only rounds are necessary. Hand folding is used to some degree in most design puzzles. It's very unlikely that anyone will come up with a high ranking result on a De'novo, freestyle or symmetry puzzle without initial hand folding.

While I'm not a great hand folder (rank 20 and rising), I'm currently at rank 8 Global soloist, so it actually benefits me to keep things as they are. I just don't see the point in the tedious efforts of hand folding beyond the initial design phase.

Joined: 04/15/2012
Groups: Beta Folders
gitwut's picture
User offline. Last seen 3 years 31 weeks ago. Offline
Joined: 05/18/2012
Groups: Contenders

Sorry for being dense, but I'm not sure what you mean. I've read the article before and again just now, but I don't see where it contradicts my suppositions.

Joined: 04/15/2012
Groups: Beta Folders

That's fine. In the end, it shows in the hand folding round, the lowest energy states are generally closest to the native, as opposed to the regular round, where that was not always true.

"This leads me to Hand-Folding:

(Pic 1)

The above figure is an RMSD plot for Puzzle 689b: Hand-Folding CASP10 T0711 Repost. It was a 33 residue freestyle puzzle, with templates, and had 3 disulfide bridges (which you totally got perfectly!). The rosetta energy (y-axis) is what you see in the game as it corresponds to the Foldit score (except that it is negative on this plot). On the x-axis is the RMSD representing how far from the native each Foldit solution is (RMSD = 0 is a perfect match, RMSD = 33 is completely wrong).

You can see above that the lowest Rosetta Energy—the top-scoring Foldit solution—the one that is easy to pick out by score, is actually one of the closest to the native.

Now compare that RMSD plot to the one for Puzzle 694: CASP T0711 Disulfide Repost Round 2:

(Pic 2)

This is the same plot as above, except for Round 2 of this puzzle (when Lua scripts and sharing was allowed) where you can see that if we now selected the best Foldit score (lowest Rosetta energy) it would not be the solution that is closest to the native. What we want is for the lowest points to be as far to the left as possible."

Seems like it's fairly obvious HF round's highest score is closer to the native than the LUA round.

Joined: 04/15/2012
Groups: Beta Folders

In relative.

gitwut's picture
User offline. Last seen 3 years 31 weeks ago. Offline
Joined: 05/18/2012
Groups: Contenders

Thank you for the explanation. So you meant that the devs provided statistics. My confusion was with both the interpretation of we (as in Foldit, Beta Folders, or devs) and what it was we did.

I can see where a case for hand folding could be made for that pair of puzzles, but is it sound to generalize beyond that 1 set?

I was hoping to see results based on a broader and more varied puzzle population (larger proteins, more varied SS, etc.). Granted that may not be possible yet, as I have no idea if the natives are known for any of the other puzzle pairs we've done so far. I'm not even sure how many pairs we've done, though it seems like a lot.

Joined: 04/15/2012
Groups: Beta Folders

I follow. Sorry about the confusion, I see I wasn't very clear. And I agree, it would be nice to see it on other types, but much of that really comes with time. I bet we may see some with CASP, but I can't say for sure.

marie_s's picture
User offline. Last seen 2 years 17 weeks ago. Offline
Joined: 05/18/2008
Groups: None

I don't give too much importance to global score for myself but your suggestion is unfair.

Diversity is one asset of this game : diversity of problems, of strategies, of players, of puzzles, of difficulties, of tastes, of fun...
If some players say hand folding is a pleasure for them, trust them.

So the talent of hand folders have to be rewarded as the creativity and precision on symmetry puzzles and other talents...

Joined: 09/21/2011

REVISION: Who Benefits- folders who believe that "power folders" with similar ranks on both releases should not receive double points for running the same protein twice (hey, they're describing me!).

Who Does Not Benefit -power folders who believe that a protein that has been worked on only once, should only be rewarded once while a protein that has been worked on twice should receive an award for both solutions. This case is relevant when both solutions have a similar rank. If both solutions have a dissimilar rank, the benefit of dropping the lowest score may offset the loss. It should be noted that a similar rank is usually an affect of downloading the solution from the first release to be used for the second release. With moderate scripting a folder should be able to maintain a similar rank with the duplicate protein.

note: some folders like myself will be split. I would not benefit because I am one of the people who might get r2 on the handfold and invariable get around r2 on the regular thus doubling my points in the current system. But I don't feel right about it. I'd be using the same solution that I already was rewarded for in the first round to give me a big fat advantage over everyone else and most of the points in the second round load instantly when i carry my solution over from the first round, with only maybe a 2-5 percent increase from the scripts. Plus I can score way more points then others just by sitting here and playing it out through both releases which would be rewarding me for stamina, rather than a good solution.

Joined: 09/21/2011

marie, this does reward handfolders (handsomely in fact). you really need to reread this.

spmm's picture
User offline. Last seen 7 weeks 5 days ago. Offline
Joined: 08/05/2010
Groups: Void Crushers

Many puzzles in foldit are repeated, as for example (endlessly :D) in CASP, and others more recently; are you suggesting that this 'method' should apply to all repeated puzzles? which would be a logical assumption.

Hand folding also prevents copying - not entirely, as people can screen share, but that is very different to copying using a 3 dimensional guide and structure copier scripts to remove the tedious work of resetting any structure changes manually. That makes big points available for people on teams who may not have time to spend on some puzzles and wish to maintain their global rank.

I agree with boots I cannot see any value in this suggestion.

Joined: 09/21/2011

the other puzzles that are repeated rarely allow you to load a previous solution and those puzzles have already been refined with scripts when they are repeated, there would be no reason for this method to be used on those puzzles. I would also restate....this suggestion does not take away handfolding puzzles. before the double release, handfolding was not even rewarded at all, for any reason. in the current situation handfolding is rewarded, not once, but twice as those solutions are being carried over and instantly receiving all the points from the previous round then dressing it up with a few scripts and pretending you have a new solution with points awarded all over again. i think these particular variables that are easy to overlook have an important impact on this suggestion. if you "see no value" then perhaps you need to "see" the list of benefits. you may still consider those benefits offset by the downside of not receiving double points, that is your call, but now i'm just being picky about wording ;).

Joined: 04/15/2012
Groups: Beta Folders

But it's not meant as a new solution. It's really meant for you to do the Hand folding round (so Devs get varied solutions that are not pushed into false local minimums within the first day, in theory), then you load them in to refine your solution. If hand folding gets better points, and that's your only goal (which personally, it shouldn't matter that much as our goal here is science), then do the hand folding.


Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, Boehringer Ingelheim, RosettaCommons