Scoring Idea

Case number:845833-986285
Topic:Game: Display
Opened by:steveB
Opened on:Thursday, June 11, 2009 - 19:11
Last modified:Thursday, July 19, 2012 - 08:14


My guess is that at present, with all of the 'Quest to Native' and refinment puzzles, that you guys (the puzzle setters) are trying to work out why we (the solvers) are getting high scores but NOT getting structures close to the original protein's.

My educated guess, if its worth anything, is that the problem is score based. I am trying to get high scores, and by manipulating the protein I head in the direction of the highest score, not the highest score which has the correct structure.

Would it be possible to add a score multiplier to the system so that the scoring was shown in 2 parts.... Part 1 would be the protein's score, just as now. Part 2 would be a proximity score to the native, however you currently work it out (which you must be able to do looking at puzzle 156) rebased to a scale of 0 -> 1.

As an example, if my protein score is 9876.543 and is WAY OFF the native in backbone shape my score would read as follows

9876.543 x 0.62 = 6123.456

However if I had a lower scoring protein but which was closer to the native in backbone structure I would have a score as follows :

9321.123 x 0.89 = 8295.799

The scoring system would be intuitive to players, as you could see the proximity multiplier drop or rise after doing a rebuild, giving a very good clue as to the direction in which you are going.

The player's published puzzle scores could remain the same so that the system and scores are compatible with the past scoring systems, but the in-game score display would help you, and us, achieve the goal of folding to the native.

(Thu, 06/11/2009 - 19:11  |  10 comments)

steveB's picture
User offline. Last seen 1 year 17 weeks ago. Offline
Joined: 01/08/2009
Groups: Void Crushers

Just to clarify, the multiplier score WOULD be displyed on the in-game screen so that the player can see the proximity to the native.

Joined: 11/20/2008
Groups: None

That's a very interesting idea, even though it can only be used on a limited amount of puzzles (as in the end, the goal is to correctly fold puzzles for which the native isn't known, right?).

With interesting, I do not necessarily mean it would be a helpful tool for the entire community, based on the fact that everyone has "the answer" in the Quest to Native puzzles right in front of them, yet it is still the same group of people floating to the top.
Some get there by spending a lot of time working on the puzzles (me for example, lol), others have supernatural folding powers (heheh) and end up on top in a heartbeat.

So, to me the interesting part would be to see a chronological graph of those multiplier scores for individual folders.
My educated guess would be that the "workhorses" stay as close to the native as possible, gradually working their way up, while the "superfolders" take big losses in order to advance much faster in the end. (That guess is based upon the fact that simply trying to reproduce the native guide isn't the answer; turn all the sidechains the way they face on the guide and your puzzle is bound to explode).

Anyway, I like your idea, and even when implementing it in the game is not an option, I do hope these statistics are available behind the scenes and we could get a peek at them at some point.

Joined: 12/14/2008

I would say that is 1) hard to programm and 2) away from the goal.

A user controlled score mechanism (score selected residues e.g.) is more effective I think.

mat747's picture
User offline. Last seen 1 year 22 weeks ago. Offline
Joined: 10/15/2008
Groups: Void Crushers

I like idea.
The “score multiplier” could work like the coloured bar icon that is displayed below your saved solution score, but in a numerical form with the score being relative to the known structure.

Joined: 06/06/2009
Groups: None

This sounds like a good idea, but there's just one problem. On the open-ended puzzles we dont have a native solution to compare with. What we are attempting to do with the "Quest to Native" puzzles is develop a sort of intuitive feel of how the puzzle should go. The scoring system sounds pretty good for puzzles we have solutions for, though.

Thats a really neat idea. We don't have the infra-structure to get that done right now, but as soon as we start implementing ways to compare player stats thats definitely a idea we'll try to incorporate. THANKS! (hehe, superpower folding abilities)

We are currently using something similar to that, but the idea of allowing scoring to the user: "A user controlled score mechanism (score selected residues e.g.)" is really not something we can implement

The same as what I said to steveB. A good idea but impossible to implement on open-ended puzzles.

Appreciate the suggestions!!!!

Joined: 05/19/2009
Groups: Go Science

Having those solutions that get more points than the native might be a good way to find the errors in the scoring algorithm.
The key should lie in trying to somehow identify where the scoring algorithm makes errors.

@user scored algorithm:
Crtl+i could be extend to more than one residue and just sum up the numbers of the residue that are marked (especially with the new interface).

mat747's picture
User offline. Last seen 1 year 22 weeks ago. Offline
Joined: 10/15/2008
Groups: Void Crushers

Just to clarify
Aloofmoney thank you for the reply to by comment.
Just to clarify, my suggestion was only for Quest to native puzzles with known structure.

vakobo's picture
User offline. Last seen 1 year 15 weeks ago. Offline
Joined: 06/05/2008
Groups: Russian team

For non Quest Puzzles the multiplier could be =1.

I have no idea why the message above was perceived as spam.
That is annoying.

Joined: 06/06/2009
Groups: None

Thats a pretty good suggestion. We're going to try to pull that in our program.

I agree that this would work well for the native puzzles. We'll try to incorporatte it.

We're not entirely sure on how we're going to do the scoring of the non-native puzzles, but the suggestion is duly noted for when we do.

Thanks for all the feedback guys. We appreciate this.

Joined: 06/17/2010
Status: Open » Closed

We have scoring relative to guide in view options.


Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, Boehringer Ingelheim, RosettaCommons