Puzzle 1465 "SCORING BUG"

Case number:699969-2004641
Topic:General
Opened by:ZeroLeak7
Status:Open
Type:Bug
Opened on:Friday, January 5, 2018 - 14:55
Last modified:Wednesday, August 15, 2018 - 18:10

Hello,

We have a problem with puzzle 1465 a player https://fold.it/portal/user/317303 from my Group has found a glitch or bug.

He had a score of 10181 and it popped up to 10913 as he pulled an wiggled the protein.

(Fri, 01/05/2018 - 14:55  |  19 comments)


LociOiling's picture
User offline. Last seen 1 hour 10 min ago. Offline
Joined: 12/27/2012
Groups: Beta Folders

I see that Maerlyn138 has taken a big lead on 1465. The high score was 10,475 the last time we revisited this protein, so a score of 10,913 indicates there may indeed be a problem.

I'd suggest sharing the the high-scoring solution, then having someone else in the group try to open it. You may find it opens with a much lower score, while still requiring the high score plus two points to evo.

Alternately, it's just possible that you actually have a much better solution. The native for this one is seen in PDB 1RKI. It's a protein from Pyrobaculum aerophilum, which grows best around the boiling point for water. Maybe growing this protein in good old E. coli isn't quite the same.

1RKI does look a little sketchy. It's a dimer, with two copies of the protein. To me, it's hard to see what's holding the two copies together.

We're only working with one copy, but even so, it has issues. There are two groups of sheets. The outer group has three sheets. A long 8-segment loop section crosses over the middle sheet on the outside of the protein. This seems a little unusual.

So here's hoping we have a better solution. We don't really need another scoring bug.

Hanto's picture
User offline. Last seen 15 hours 26 min ago. Offline
Joined: 05/10/2008
Groups: None

Maerlyn, do NOT stop. keep going!!!!!

Hanto's picture
User offline. Last seen 15 hours 26 min ago. Offline
Joined: 05/10/2008
Groups: None

I should ask tho, were you using TRACKs when this apparent bug occurred on the client in question?

Hanto's picture
User offline. Last seen 15 hours 26 min ago. Offline
Joined: 05/10/2008
Groups: None

Or were you using multiple foldit executables inside a single folder/directory?

georg137's picture
User offline. Last seen 1 day 4 hours ago. Offline
Joined: 08/07/2010
Groups: Contenders

There is definitely a foldit scoring bug. I watched as my client score jumped over 600 points, to 9991, catapulting it from 7th to 1st on the soloist leader-board for puzzle 1559. When I try to retrieve the 1991 configuration it only retrieves the actual configuration at 9242 points. So my client appears to be number 1 on the leader-board, which is undeserved. That has to be disheartening for the actual top scorer(s), more than 300 points below my client's fictitious score.

The bug occurred when tapping quick enable/disable band keystrokes on the keyboard. At the time the client had only one foldit executable running in a single foldit directory on the single default track.

In the unlikely case that the client actually achieved a 700 point gain in a microsecond, then there is still a bug because the configuration is not retrievable.

georg137's picture
User offline. Last seen 1 day 4 hours ago. Offline
Joined: 08/07/2010
Groups: Contenders

Typo in line 2... 1991 should be 9991.

LociOiling's picture
User offline. Last seen 1 hour 10 min ago. Offline
Joined: 12/27/2012
Groups: Beta Folders

Georg137, are you using devprev or main?

It seems like devprev fixes some scoring glitches, but others still remain.

On puzzle 1557, there were a couple of cases where it appeared my "very best" solution (absolutebest pose) received credit, even when the "credit best" was a couple of points lower. I haven't seen any big scoring glitches with the current devprev however.

General tip if you can't find your solution: in open/share solutions, check "show auto and quick saves", which shows your absolutebest, creditbest, and quicksave slots. Everything is by track, so make sure you're on the right track. When there's a glitch, you may see a high-scoring solution, but it will load with a lower score.

georg137's picture
User offline. Last seen 1 day 4 hours ago. Offline
Joined: 08/07/2010
Groups: Contenders

Thanks for the tip. The scoring bug occurred while running with main. The server continues to display the top fictitious score on the leader-board for my client, while the client best is currently far below that number, sadly.

spvincent's picture
User offline. Last seen 6 hours 44 min ago. Offline
Joined: 12/07/2007
Groups: Contenders

I'm seeing what may be a similar issue on 1560.

My score suddenly jumped to 11182, apparently as a result of the filter score being added to the real score when running a script (Local Mutate 2.0). Track t1.

spvincent's picture
User offline. Last seen 6 hours 44 min ago. Offline
Joined: 12/07/2007
Groups: Contenders

Correction: the score with filters being recorded as the actual score

phi16's picture
User offline. Last seen 2 hours 5 min ago. Offline
Joined: 12/23/2008

I just jumped from 11216 on 1560, my actual score as r1, to 11549, my score without adding -333, the penalty for having 1 bad residue and 14 extra residues. There is definitely a bug in there somewhere.

georg137's picture
User offline. Last seen 1 day 4 hours ago. Offline
Joined: 08/07/2010
Groups: Contenders

Well, it now appears that active puzzles 1559 and 1560 are displaying fictitious scores for pole positions on the leader-boards.

It is not unfair to ask - if two of the three active puzzle scores are compromised, then what confidence should foldit contributors have in scoring accuracy? The same question also applies to solutions selected for further development, and to contributor ratings which are calculated from scores.

Possibly analogous to a situation where voters lose confidence in electronic balloting.

LociOiling's picture
User offline. Last seen 1 hour 10 min ago. Offline
Joined: 12/27/2012
Groups: Beta Folders

This has been a long-term problem. Part of the problem seems to be bonus-related or maybe penalty-related is a better term. For a long time, recentbest tended to overlook the penalty for added segments. This seems to have been fixed (?) as of 20180725-d14dae34c3-win_x86-devprev.

Now we're seeing a similar problem, where absolutebest gets pushed out to the server as if it were creditbest.

Not sure if penalties account for all the differences. On 1559, georg137 has a lead of over 300 points, and that puzzle has only disulfide bonus, not penalty. Others are seeing similar unexplained jumps on 1560.

Here are some examples from puzzle 1557, where it appears absolutebest was pushed out to the server instead of creditbest.



On the first one, absolutebest is only a couple of points ahead. I didn't catch this one in time to save the absolutebest pose.



The next one was kind of similar.



I was able to save the "very best" pose for the second case and open it on a different client. It opened at 10,610 instead of 10,620. The 10,620 score seems to overlook the net penalty, -10 points.

LociOiling's picture
User offline. Last seen 1 hour 10 min ago. Offline
Joined: 12/27/2012
Groups: Beta Folders

PS, I actually did end up with a score of 10620.191 with no scoring glitch on my 1557 solo. (The saved solution opens with the same score, and the penalty/bonus math seems correct.) So actiasluna was not robbed at the end, despite where things stood with 5 hours 10 minutes remaining.

LociOiling's picture
User offline. Last seen 1 hour 10 min ago. Offline
Joined: 12/27/2012
Groups: Beta Folders

Somewhere Murphy is laughing, the bug bit on me on puzzle 1560. All of a sudden both absolutebest and recentbest incorrectly show a score of 10595, 330 points higher than the creditbest score. The net penalty just happens to be 330 points. What's worse is the 10595 has been pushed out to the scoreboards.

Actually, 330 free points is not so bad, we should upgrade this bug to a feature...but seriously, willing to have this invalid high zapped. There's a slight chance I can make up the points, but it'll be tough.

A little image magic was applied to get all the bests on the screen.

LociOiling's picture
User offline. Last seen 1 hour 10 min ago. Offline
Joined: 12/27/2012
Groups: Beta Folders

Better image...

LociOiling's picture
User offline. Last seen 1 hour 10 min ago. Offline
Joined: 12/27/2012
Groups: Beta Folders

OK, I have managed to catch the bug both solo and evo on 1560. I'm currently #1 on both boards, but the scores are inflated by 330 points.

georg137's picture
User offline. Last seen 1 day 4 hours ago. Offline
Joined: 08/07/2010
Groups: Contenders

My confidence is scoring is zero. This bug has affected scores on almost every recent puzzle. My 1561 score is now fictitious, too. Looking around, one has to wonder how many other scores are fictitious.

jflat06's picture
User offline. Last seen 4 days 5 hours ago. Offline
Joined: 09/29/2010
Groups: Window Group

Thanks for the feedback everyone - we're continuing to look into this bug.

In the future, this may not be much of an issue regardless, as we will be transitioning to positive-only filter bonuses in puzzles posted from here on out. In this case, your best score will always (or nearly always) be with the filters on, so there shouldn't be any scoring mishaps.

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons