20210913-39cb7ecfb1-win_x86 Shake not consistent, scoring anomaly

Case number:845813-2012076
Topic:Game: Tools
Opened by:Bletchley Park
Opened on:Sunday, September 19, 2021 - 14:02
Last modified:Tuesday, September 21, 2021 - 14:32

In trying to pinpoint large scoring differences I stumbled upon something I'd like you to explain.
- shared solution puzzle 2043, 'raw2'

Load this solution
unfeeze everything
do SHAKE for 3 iterations

The score should be 5593.something

Repeat the above steps at least 5 times. You will find DIFFERENT scores after some of the SHAKEs.

If SHAKE involves randomness, this is causing significant (59 points on this step alone ! >1%) scoring differences between identical starting positions. When extrapolated to a full puzzle round this can amass to hundreds of points difference.

I'd say this is a bug.

(Sun, 09/19/2021 - 14:02  |  1 comment)

rmoretti's picture
User offline. Last seen 5 days 22 hours ago. Offline
Joined: 01/15/2010
Groups: Foldit Staff

The algorithm behind Shake is Metropolis Monte Carlo Simulated Annealing. This is an algorithm where randomness is fundamental to how it works. The algorithm just would not work without some sort of random number generation.

Granted, we could theoretically make it deterministic by resetting the seed of the pseudorandom number generator to a fixed state before each launch of the shake tool. This wouldn't improve anything, though. You'd just arbitrarily pick one of the possible results. There's no way to guarantee that it would be the best result for that shake. And small changes to the protein (e.g. subtle, even imperceptible, backbone shifts) would result in a completely different trajectory, so you'd still be potentially looking at the same sort of significant score differences, unless you used *exactly* the same structure.

Given that, I'm not sure why you'd classify this as a bug. If you're going back in time it's typically because you want to branch off in a different direction, and having a different random result in shake is part of that. It's not deliberately worse, it's just different, and that can help send you down a different (hopefully better!) path.

If you do want to exactly recapitulate a specific post-shake structure, I'd probably suggest using the undo graph to load the post-shake structure directly, or using the save/load structure feature to store it long term, rather than loading things pre-shake and then counting on shake to get the same results.


Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, Boehringer Ingelheim, RosettaCommons