pose scoring incosistent on puzzle 1260

Case number:671071-2002626
Topic:Game: Other
Opened by:LociOiling
Status:Closed
Type:Bug
Opened on:Saturday, July 23, 2016 - 05:32
Last modified:Tuesday, July 31, 2018 - 21:17

On puzzle 1260, I noticed a couple of usually reliable recipes were unexpectedly accepting losses.

In the course of trying to debug the problems, I noticed that my test solution had some unusual features in the scoring department. The pose has been shared with scientists as "scoring inconsistencies".

In this solution, the values returned by recentbest.GetEnergyScore and absolutebest.GetEnergy score do not match the values returned by current.GetEnergyScore when these poses are loaded by recentbest.Restore and absolutebest.Restore.

The recipe Score Checker 1.0 reports the GetEnergyScore values with and without filters for current, recentbest, creditbest, and absolutebest. For the latter three poses, the recipe calls the Restore function and compares the resulting current.GetEnergyScore to the one reported for the pose.

(The recipe also checks to see if absolutebest is indeed the best. A red herring, no issues after absolutebest.Restore is called.)

The recipe output for the solution in question in shown below. In this example, recentbest.GetEnergyScore reports 10316, which unfortunately is above my best real score. Calling recentbest.Restore reveals the score to be 10016, including a filter bonus of 510. It's interesting that the discrepancy is 300 points, since this is also the max filter bonus of 810 minus 510. This solution has never reached the 810 bonus, so it's not clear that the 300 points is meaningful

The same discrepancy appears for absolutebest.

When I share the solution and open it on another client, recentbest and absolutebest still show discrepancies, but now the score they both report is 10296.23, and they load as 10096.23. On the first client, the filter bonus for these poses was 510, but it's 610 on the new client.

This seems a bit familiar, perhaps related to cases where a client reports a high score that can't be loaded or shared.

Score Checker v.1.0
Puzzle: 1260: 85 Residue Monomer Design: Ideal Loop Filter
Track: default
--
start score = 10096.23
--
basic scores
pose "current", with filters = 10096.23, without filters = 9486.23, filter bonus = 610
pose "current", conditions met = true
pose "recentbest", with filters = 10316.116, without filters = 10316.116, filter bonus = 0
pose "recentbest", conditions met = false
restored pose "recentbest", with filters = 10016.116, without filters = 9506.116, filter bonus = 510
pose "creditbest", with filters = 10107.526, without filters = 10107.526, filter bonus = 0
pose "creditbest", conditions met = true
restored pose "creditbest", with filters = 10107.526, without filters = 9497.526, filter bonus = 610
pose "absolutebest", with filters = 10316.116, without filters = 10316.116, filter bonus = 0
pose "absolutebest", conditions met = false
restored pose "absolutebest", with filters = 10016.116, without filters = 9506.116, filter bonus = 510
--
Restore() results
pose "recentbest", GetEnergyScore = 10316.116, does not match restored score with filters = 10016.116, or without filters = 9506.116
pose "absolutebest", GetEnergyScore = 10316.116, does not match restored score with filters = 10016.116, or without filters = 9506.116
Restore() mismatches = 2
--
absolutebest results
absolutebest mismatches = 0
---
Score Checker v.1.0 complete
Puzzle: 1260: 85 Residue Monomer Design: Ideal Loop Filter
Track: default
(Sat, 07/23/2016 - 05:32  |  3 comments)


LociOiling's picture
User offline. Last seen 53 min 16 sec ago. Offline
Joined: 12/27/2012
Groups: Beta Folders

The source of the original problem I was looking at may be related.

At least one of the recipes that was accepting losses on 1260 relies on recentbest.Restore to reset after runs which produced no gain. The recipe already saved the best score in a quicksave slot. So I replaced the recentbest.Restore with a quicksave.Load. Early results are encouraging.

The moral may be that recentbest is a bit wobbly under certain circumstances. Explicit quicksaves may be a more reliable tool for keeping track of your best score.

Susume's picture
User offline. Last seen 1 week 6 days ago. Offline
Joined: 10/02/2011

Puzzle 1265 is also scoring inconsistently. When I restore credit best I get 9796, but when I restore very best on the same track I get 9698. This is with filters turned on. Both poses score lower with filters turned off. I have not run any scripts that accept losses. I did run a script that turns all filters on and off while running.

Shared with scientists: "1265 track 2 calls this credit best" and "1265 track 2 calls this very best".

20160728-c4d88693d5-win_x86-devprev

LociOiling's picture
User offline. Last seen 53 min 16 sec ago. Offline
Joined: 12/27/2012
Groups: Beta Folders
Status: Open » Closed

Old, seems like things are better in 20180725-d14dae34c3-win_x86-devprev, closing. (For now....)

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons