Major changes in shake/wiggle - scripts stop working?

Case number:845813-991740
Topic:Game: Tools
Opened by:Rav3n_pl
Status:Closed
Type:Question
Opened on:Friday, February 3, 2012 - 18:52
Last modified:Wednesday, February 15, 2012 - 01:08

As a player who use lot of scripts I noticed that after last update/s scripts are not working as good as before.
Last few puzzles are totally failed for me.
My DRW rebuilder is not working at all on bit better scored proteins ranked under #100 (as it works b4 on top10 scoring ones). I have 0 (ZERO) gain form many hours of run. It was never happen before.
Banding scripts are affected too, but they still works (more or less, tending to less).

Also on some (not so rare) occasions I noticed kind of "wiggle lock" when script is running. Despite of strange pose after rebuild or banding protein is doing almost nothing when wiggle after shake. It is more visible on big CASP puzzles.

Analyzing points make by shake function on new (current) and old (after 1st sliders update) client our team member Auntdeen found that shake function makes now much more points that before (on same setting, same puzzle, same starting pose).

Please check/verify this.

(Fri, 02/03/2012 - 18:52  |  31 comments)


Joined: 04/19/2009

It appears that the old setting of sa = 0 no longer exists.

509 model 1 (nothing else done, shake from reset):

old client (from before 1-6-2012 update - beta):
sa0 = 8303
sa1 = 8553
sa2 = 8415
sa3 = 8433
sa4 = 8429

new client:
sa0 = 8430
sa1 = 8561
sa2 = 8520
sa3 = 8299
sa4 = 8412

508 zhang 4:

old client:
sa0 = 609
sa1 = 804
sa2 = 652
sa3 = 618
sa4 = 606

new client:
sa0 = 615
sa1 = 801
sa2 = 671
sa3 = 630
sa4 = 619

508 quark 4 was almost identical on all sa. However! From an identical wiggle sidechains from reset to an identical score, I then did a shake on both at sa0.......

old client = 4399
new client = 4430

Joined: 04/19/2009

I've tested now with new beta client, and there are differences from both of the others:

508 zhang 4:

new beta client:
sa0 = 609
sa1 = 804
sa2 = 652
sa3 = 618
sa4 = 606

508 quark 4 results now have major differences, where there was little before:

old client:
sa0 = 3667
sa1 = 3847
sa2 = 3667
sa3 = 3668
sa4 = 3668

new client:
sa0 = 3667
sa1 = 3885
sa2 = 3667
sa3 = 3668
sa4 = 3668

new beta client:
sa0 = 3668
sa1 = 3795
sa2 = 3795
sa3 = 3839
sa4 = 3836

jflat06's picture
User offline. Last seen 3 days 17 min ago. Offline
Joined: 09/29/2010
Groups: Window Group

So it looks like the newer version of the shake slider is providing better results, at least outside of scripts?

Raven - what are the common components of the scripts that are now having issues?

Joined: 04/19/2009

IMHO, I don't think that they are providing "better" - the suspicion is that they are prematurely stiffening proteins.

Since the last update, quite a few top folders are having poor scores in all puzzles. While we all expect to do poorly on different puzzles, it's highly unusual that so many would be doing so badly on all puzzles at once - in fact, in the 3 years I've been here, I've never seen this before.

This could possibly indicate a software problem rather than the techniques, abilities and/or script usage of people from varied teams going bad all at once. What made me start to investigate this was when on 3 different puzzles, using 4-5 different rebuilder scripts, I was unable to have any gain at all - in early to midgame - not even to .00000000.

That's what made me roll back to main from beta client to test wiggle and shake, and then roll back to an older, not updated (1-6-2012) copy to test. We thought it might be wiggle, but that tested accurately. But shake gives highly varied results (and it seems that none of the current settings reflect what sa=0 used to), on some puzzles/models.

Joined: 06/17/2010

Thing that is very odd that rebuilders and sidechain tweakers not finding ANYTHING.
I run full Acid Ttweaker and get NOTHING. On solution about #100!
I never hit wall so fast.

beta_helix's picture
User offline. Last seen 5 days 23 hours ago. Offline
Joined: 05/09/2008
Groups: None

assigning...

spvincent's picture
User offline. Last seen 1 hour 57 min ago. Offline
Joined: 12/07/2007
Groups: Contenders

A couple of thoughts. It seems as if using the alignment tool in recent puzzles results in structures that are much cleaner than those that occurred previously. In particular the backbone scores of newly-threaded structures are much better than they used to be with few if any segments having really bad backbone scores. So there's not much low-hanging fruit out there for the rebuild scripts.

Also there's the possibility that something inside FoldIt has changed. There's mention in the recent developer update info about a bug fix in wiggle which is a possibility. Assuming it's not that it could be related to the recent Shacc/Wacc accuracy settings: maybe these are something scripts need to manipulate now they can. For example the settings could be reduced when using low values of CI or when many bands are present to save on time and increased when doing an operation that could result in a score improvement. I'm guessing there are quite a few scripts that could benefit in this way.

jflat06's picture
User offline. Last seen 3 days 17 min ago. Offline
Joined: 09/29/2010
Groups: Window Group

This feedback was posted before the wiggle fix went out to beta, so I don't think that's it. I'm looking over the changes to shake that added the sliders, so hopefully I'll find something.

jflat06's picture
User offline. Last seen 3 days 17 min ago. Offline
Joined: 09/29/2010
Groups: Window Group

Do you know exactly which update these problems started occuring?

spvincent's picture
User offline. Last seen 1 hour 57 min ago. Offline
Joined: 12/07/2007
Groups: Contenders

Not really: it's probably something AD or Rav could better answer.

I do see, after playing with 511, that my comments made above regarding better backbone scores are completely wrong.

Joined: 06/17/2010

Looks like it is connected to 1st or 2nd update after slides arrived, around 20. Jan
Attaching my looong zipped log.txt where rebuiler+lws was run over 24hours and did... 5pts

Joined: 06/17/2010

Fresh puzzle 513.
Run of my SBS script that rester puzzles and stablilize every multistart position.

Old main
New dev
start points: 9594.9644708673
saved at: 9679.4034533246
saved at: 9681.6467225955

start points: 9602.8544393181
saved at: 9695.3820016485
saved at: 9698.1791054123

start points: 9637.5448427113
saved at: 9749.4977979257
saved at: 9742.4501999622

start points: 9647.7987836987
saved at: 9738.4009415988
saved at: 9732.6759649219

start points: 9594.6390099051
saved at: 9710.388023537
saved at: 9710.6694110998

jflat06's picture
User offline. Last seen 3 days 17 min ago. Offline
Joined: 09/29/2010
Groups: Window Group

Which client is "Old main"? Does "Old main" have sliders in it?

kenjung's picture
User offline. Last seen 10 years 14 weeks ago. Offline
Joined: 11/11/2010
Groups: None

SBS script uses shake, right?

I think you mentioned you ran the new dev tests with ShAcc 0. There was an error with the default of ShAcc, where it should have been 2. Ca you please try re running the test with ShAcc 2?

If SBS script doesn't use shake, then disregard that request.

Joined: 06/17/2010

Loaded same pose to old main client (no sliders) and current dev.
Run script that are fuzing thingy 6 times from same pose and take best one, then repeat.

Current dev client log:
Starting repeatable Fuzes... Score: 8276.397
Fuze 1/6
Gained another 372.657 pts.
Fuze 2/6
Fuze 3/6
Gained another 147.414 pts.
Fuze 4/6
Fuze 5/6
Fuze 6/6
Loop gain: 520.071
Starting repeatable Fuzes... Score: 8796.469
Fuze 1/6
Gained another 92.757 pts.
Fuze 2/6
Fuze 3/6
Fuze 4/6
Fuze 5/6
Fuze 6/6
Loop gain: 92.757
Starting repeatable Fuzes... Score: 8889.226
Fuze 1/6
Fuze 2/6
Gained another 39.359 pts.
Fuze 3/6
Fuze 4/6
Fuze 5/6
Fuze 6/6
Loop gain: 39.359
Starting repeatable Fuzes... Score: 8928.586
Fuze 1/6
Gained another 3.672 pts.
Fuze 2/6
Fuze 3/6
Fuze 4/6
Fuze 5/6
Fuze 6/6
Gained another 3.886 pts.
Loop gain: 7.558
End score: 8936.145

Old main (no sliders) client:
Starting repeatable Fuzes... Score: 8276.397
Fuze 1/6
Gained another 424.025 pts.
Fuze 2/6
Fuze 3/6
Gained another 96.03 pts.
Fuze 4/6
Fuze 5/6
Fuze 6/6
Loop gain: 520.056
Starting repeatable Fuzes... Score: 8796.453
Fuze 1/6
Gained another 94.613 pts.
Fuze 2/6
Fuze 3/6
Fuze 4/6
Fuze 5/6
Fuze 6/6
Loop gain: 94.613
Starting repeatable Fuzes... Score: 8891.066
Fuze 1/6
Gained another 15.17 pts.
Fuze 2/6
Gained another 7.614 pts.
Fuze 3/6
Fuze 4/6
Fuze 5/6
Fuze 6/6
Loop gain: 22.784
Starting repeatable Fuzes... Score: 8913.851
Fuze 1/6
Gained another 14.378 pts.
Fuze 2/6
Fuze 3/6
Fuze 4/6
Gained another 3.733 pts.
Fuze 5/6
Fuze 6/6
Gained another 2.062 pts.
Loop gain: 20.174
Starting repeatable Fuzes... Score: 8934.025
Fuze 1/6
Fuze 2/6
Gained another 2.517 pts.
Fuze 3/6
Gained another 1.096 pts.
Fuze 4/6
Fuze 5/6
Fuze 6/6
Loop gain: 3.614
End score: 8937.639

Looks like old client found almost same score but it "catch" score more often.

Joined: 06/17/2010

Main client (3 sliders)

Starting repeatable Fuzes... Score: 8276.397
Fuze 1/6
Gained another 463.504 pts.
Fuze 2/6
Fuze 3/6
Gained another 37.578 pts.
Fuze 4/6
Fuze 5/6
Fuze 6/6
Loop gain: 501.082
Starting repeatable Fuzes... Score: 8777.48
Fuze 1/6
Gained another 84.714 pts.
Fuze 2/6
Fuze 3/6
Fuze 4/6
Fuze 5/6
Fuze 6/6
Loop gain: 84.714
Starting repeatable Fuzes... Score: 8862.194
Fuze 1/6
Gained another 9.056 pts.
Fuze 2/6
Gained another 4.231 pts.
Fuze 3/6
Fuze 4/6
Gained another 3.321 pts.
Fuze 5/6
Gained another 4.141 pts.
Fuze 6/6
Gained another 8.518 pts.
Loop gain: 29.268
Starting repeatable Fuzes... Score: 8891.463
Fuze 1/6
Gained another 29.372 pts.
Fuze 2/6
Fuze 3/6
Fuze 4/6
Fuze 5/6
Fuze 6/6
Loop gain: 29.372
Starting repeatable Fuzes... Score: 8920.835
Fuze 1/6
Fuze 2/6
Gained another 3.081 pts.
Fuze 3/6
Gained another 5.112 pts.
Fuze 4/6
Fuze 5/6
Gained another 18.498 pts.
Fuze 6/6
Loop gain: 26.692
Starting repeatable Fuzes... Score: 8947.528
Fuze 1/6
Gained another 10.185 pts.
Fuze 2/6
Fuze 3/6
Fuze 4/6
Gained another 3.709 pts.
Fuze 5/6
Fuze 6/6
Gained another 2.355 pts.
Loop gain: 16.25
End score: 8963.778

Main client 5 sliders: same as current dev client.

marie_s's picture
User offline. Last seen 2 years 6 weeks ago. Offline
Joined: 05/18/2008
Groups: None

Contest testing puzzle
wiggle global makes the score to go to a very negative stat then climb again with these options of behavior

CI=0.98
wacc=7
sacc=2
others = 0.993

http://fold.it/portal/files/chatimg/irc_33153_1328987803.png

Joined: 04/19/2009

Although I could not get the BSSA & ESSA sliders to exactly hit the .993 (one at .991 & other at .994), I was still able to duplicate what marie did...

Score went down to about -110k to -120k, then blinked back to "normal" - letting the wiggle go on without stopping, it has gone down and up like that 4 times before finally stabilizing.

Joined: 06/17/2010

Made video of that phenomen.
Lower CI and wiggle: look at score and sidechains...
http://youtu.be/h3rU3dXsJnY

Joined: 10/11/2011
Groups: None

I am doing puzzle 514.
I started using doom 2.4 and just happened to look at client, it had gone negative millions ( or whatever it is) and was all red, didnt explode, then in blink of an eye it jumped back to best score.
I didnt get to take a screenshot as it took me a few seconds for it to register that it was weird

Joined: 05/03/2009
Groups: Contenders

Hmmm

Followed marie's settings and had all chains showing - this 'bug' seems to be just related to the BUR and EXP slider update, where the sidechains do the 'zombie rotating dance'. Is this just the new normal that we need to accomodate?

To replicate, merely follow the settings that marie posted above, but shift 'a' to see all sidechains, and watch what happens as they do their creepy rotations...

In terms of gameplay, it falls under the category of 'different', and may not necessarily be 'bad'
But - in it's current form, it massively increases the time needed to achieve it's goals. As AuntDeen said above, it'll settle, but you'll need to make a cup of tea, buy groceries, change career, live in Ontario, have 4 children, and then, and only then, stop wiggling. Sadly, the deadline will have passed.

CFC

beta_helix's picture
User offline. Last seen 5 days 23 hours ago. Offline
Joined: 05/09/2008
Groups: None
Status: Open » Open

Thank you for posting the exact settings and video, now that this is reproducible we should be able to get to the bottom of this!

CFC's picture
User offline. Last seen 6 years 1 day ago. Offline
Joined: 11/03/2010
Groups: Contenders

Regarding reproducibility - It may be more weighted by clash. If I wait long enough, I can wiggle at clash .98 on any current puzzle to induce the weirdness. It's quicker though, if you wiggle at clash .98, Shacc 2 and Wacc 7.

alwen's picture
User offline. Last seen 6 weeks 6 days ago. Offline
Joined: 10/03/2011

I am running FoldIt on a netbook, so things don't go as fast for me.

Using Marie's settings, I was able to reproduce the sudden point drop. Because my machine doesn't run that fast, I don't get the "flash" effect - I see the score drop by about 600 point jumps until it goes negative. Sidechains start to clash, and they hardly move at all as the score drops and drops.

http://fold.it/portal/files/chatimg/irc_343528_1329071307.png

http://fold.it/portal/files/chatimg/irc_343528_1329071568.png

If I let Wiggle All run, the score keeps dropping until the clashing sidechains lurch out of each other's way, then the score leaps back positive.

I wish I could get video of this. It keeps doing the same thing: the backbone moves, but the sidechains don't get out of each other's way. The other end of the protein did the same thing (this is the third deep score dip so far):

http://fold.it/portal/files/chatimg/irc_343528_1329072323.png

Joined: 09/21/2011
Groups: Void Crushers

Another weird score behavior. Wiggle backbone/all in selection mode.

Followed tealight suggestion to put it here. (previous http://fold.it/portal/node/991826 Closed that one)

Puzzle 511.
Description
After a total rebuild I ended up with a score of about -17000.
WA 3, SA sliders on default, nothing banded or selected. CI 1.
Did a long shake, score jumped suddenly from about -10000 to 7 or 8000.
When it reached about 9810 I switched to wiggle sidechains until the score did not change much.
Then Wiggle backbone, score jumping up and down sometimes more than a few thousends of points.
The colour of the score also changed when it went down so much.
But still gaining points.
Again a shake, followed by Wiggle all.
Again weird behavior in the score.
(Normally: wiggle all/backbone score goes up or does not change)

Saved the version just after the rebuild. Name: After rebuild. Also shared it with my group.

Reconstructing it from the undo gave the same behavior.
Reconstructing it from the saved version DID NOT.

Hope this helps for finding the cause of the problems.

Joined: 09/21/2011
Groups: Void Crushers

Now I have a reproducible one.

Puzzle 511. Use my autosaved credit best solution (shared with myself)

SA 3, WA 3.57, BSSA 0.980, ESSA 0.950

Run Rav3n_pl Fuses 1.2
When it gets to Fuze 2, WA 0.7 the score suddenly starts dropping a lot to nearly -100000.
And then it jumps back up to normal regions 11.000+

Hope this helps.

Joined: 09/21/2011
Groups: Void Crushers

Sorry Fuze 2 CI 0.7.

Joined: 06/17/2010

Temporary solution: for me setting BSA and ESA to 0.5 on SA=2 makes best results. Some scripts finally are working.

Joined: 09/21/2011
Groups: Void Crushers

This evening I have seen the score jumping at least 4 times. It's real.

Joined: 11/10/2007
Groups: Window Group
Status: Open » Closed

We were able to reproduce the issue from Marie Suchard's steps and it should be fixed. The fix is in the Developer Preview now.

Joined: 09/04/2008
Groups: test_group1
Status: Closed » Closed

This fix has been posted in an update. Thanks for helping track it down!

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, Boehringer Ingelheim, RosettaCommons