MAIN; DEVPREV; scoring discrepancy between platforms

Case number:845799-2009760
Opened by:Bletchley Park
Opened on:Saturday, May 30, 2020 - 21:02
Last modified:Saturday, May 30, 2020 - 21:02

Following the 0527 devprev scoring issue I tested wiggle behavior on all platforms for the current clients.

Below are my findings, indicating wiggle differences that go beyond 2 points between platforms, most notably between Mac and the two others, windows and linux. For this test I took puzzle 1845, reset the puzzle and used the indicated wiggle power settings, wiggled until no full points were increasing.

platform low -> med -> high straight med straight hwp straight awp version:
mac 8328 -> 8517 -> 9151 8521 9336 8327 20190504-b55d4614bb-osx-x86
win 8365 -> 8509 -> 9153 8482 9338 8365 20200409-d067627d3b-win_x86-devprev
lin 8366 -> 8520 -> 9094 8473 9336 8366 20200413-d067627d3b-linux_x64-devprev

low->med->high is reset puzzle, lwp until no more gain, then select mwp and wiggle further until no mor egain, then select hwp setting and wiggle until no more gain.
straight med = reset puzzle, mwp setting, wiggle unti no more gain.
straight hep = reset puzzle, hwp setting, wiggle unti no more gain.
straight awp = reset puzzle, awp setting, wiggle unti no more gain.

These tests were repeated several times and scores were nearly identical every time, differing only in a few decimals depending on whether or not I let them go on for a few rounds longer.

My observation is that when wiggling on a different platform at the same setting, the outcomes are different in the backbone (I used one as a guide for the other), thus changing decisions that scripts make based on score improvement. If this happens in a design puzzle it may alter the choice of sidechains or their orientation.

I would expect all clients, regardless of the platform to score equal for given settings.

(Sat, 05/30/2020 - 21:02  |  0 comments)


Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, Boehringer Ingelheim, RosettaCommons