[NewChapter] rebuild is very slow and sometimes not finding anything

Case number:954892-996667
Topic:Developer Preview
Opened by:Timo van der Laan
Status:Open
Type:DevPrev Issue
Opened on:Thursday, January 16, 2014 - 11:23
Last modified:Monday, February 17, 2014 - 16:44

Shared with the scientists on 663 a pose where rebuild did not find an alternative.
I havent seen that happening on puzzles before when running in loop mode.
Also I noticed that rebuild is taking a very long time to find anything when I continued with DRW231 with timing.

(Thu, 01/16/2014 - 11:23  |  25 comments)


Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

I have also seen that rebuild is slow to find poses, but on the other hand the poses it finds score better. Unlike main/devprev, newchapter never stopped on poses with negative score (at least among the very small number of poses found in the first minute).

Average number of poses found by rebuild in 1 minute on ED puzzle: devprev 15.8, newchapter 3.2
Average SCORE of best and worst poses: devprev 7633 to -45832, newchapter 12391 to 11485
Average CHANGE in score for best and worst poses: devprev -2455 to -55920, newchapter -297 to -1203

I have not experimented yet to see how often these better scoring poses result in higher score than before the rebuild.

Joined: 09/21/2011
Groups: Void Crushers

Bump this one. Rebuild is finding too little poses almost none. CI makes no difference.

spvincent's picture
User offline. Last seen 11 hours 7 min ago. Offline
Joined: 12/07/2007
Groups: Contenders

Rebuild is definitely behaving strangely. See the attached screenshot, where despite the protein scoring highly, residues 11 and 12 are stretched in a suspicious way after a rebuild/wiggle.

Joined: 04/15/2012
Groups: Beta Folders

Something I found. Rebuild seems to get faster after you leave it on for a while.

Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

I ran enhanced DRW on 5 segment areas of spider toxin (after reset and blue fuse) in devprev and newchapter. In devprev, 180 out of 180 calls to RebuildSelected(1) were successful (produced a new pose). In newchapter, which I ran longer, 189 calls to RebuildSelected succeeded, and 1383 failed to produce a new pose. These were a mix of RebuildSelected(1), RebuildSelected(2) and RebuildSelected(3), since DRW tries all of those if it is not getting a pose back. This is an awfully high failure rate. Is there some threshold that can be adjusted to let more poses through?

Joined: 09/21/2011
Groups: Void Crushers

Question about this: does the new rebuild restricts what it delivers on idealized positions? If so please change that as those problems can be wiggled/fuzed away and more positions of the protein can be valuable even if the initial pose has idealize problems

Joined: 06/24/2008
Groups: Void Crushers

We have all noticed that rebuild is not returning the same number of poses; was this expected??

Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

I am concerned that lower scoring rebuilds which would be more daring and might be preferable to a human folder are being excluded. In the 38 rebuilds I could obtain on various areas of the ED puzzle, the mean loss of points due to the rebuild was only 38 points. The mean subsequent gain from shake was only 1.13 points, meaning rebuild candidates that initially scored low due to clashing but could be significantly improved by shake were entirely excluded by the rebuild function. Some of our best rebuilds normally come from this class. Excluding rebuilds based on overall score, even if the threshold is changed to let more of them through, is a mistake. Prioritizing based on backbone score alone, or some combination of subscores that excludes sidechain and clashing, might be sensible, but only testing could tell for sure. During hand-building, I think it's important to give the human eye the first right of refusal. If you can tell us your rationale in excluding possible rebuilds, the players may have some useful input on how to achieve that goal without cramping our style.

spvincent's picture
User offline. Last seen 11 hours 7 min ago. Offline
Joined: 12/07/2007
Groups: Contenders

I fully agree with Susume. FoldIt should not make the assumption, as seems to be the case in newchapter, that high-scoring rebuilds are the only ones of interest. Eyeballing rebuilds and selecting promising-looking ones, irrespective of their score, is an effective technique and one which can no longer be used as things currently stand. Also, it severely limits the ability of scripts to work with rebuilds based on criteria other than best score after rebuild.

Joined: 06/24/2008
Groups: Void Crushers

More tests of DRW on 663 comparing Devprev and NC wiggle low, medium and two crashed in high:

Joined: 11/10/2007
Groups: Window Group

There will be an update to newchapter shortly that should allow rebuild to find more potential solutions. Let us know if this helps out with this issue.

Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

Attached is some data from newchapter rebuild after the update - it is finding more solutions now, as many as devprev was, but at CI 1.0 they are still only high-scoring solutions. Solutions with bad sidechain clashes that could be cleared by shake are still excluded (at CI 1.0).

Newchapter found more good rebuilds (ones with net increase in score after shake and wiggle) than devprev on TR663, and fewer on the ED puzzle. On both puzzles, the overall gain from rebuilds was much less for newchapter than devprev, because the gain from each good rebuild was less.

Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

Note: If you are having trouble with rebuild, be sure to shake first, rebuild a while, then shake again and rebuild some more. Rebuild will reject poses with bad sidechain clashes that it used to accept, so it needs help from shake.

If there is a cut where you are rebuilding, disable cut bands before rebuild. They will pull like superstrong rubber bands if you don't.

Also, try different CI levels. Rebuild at zero CI will put the backbone through itself repeatedly and makes horrible helices.

spvincent's picture
User offline. Last seen 11 hours 7 min ago. Offline
Joined: 12/07/2007
Groups: Contenders

I certainly think we need to go back to the previous rebuild. I thought after the last scientist chat that this was going to happen but this doesn't seem to be the case.

NickyCGS's picture
User offline. Last seen 4 years 22 weeks ago. Offline
Joined: 06/27/2013
Groups: Repro-men

Thanks for all the feedback and wanted to give an update. Reproduced/verified this yesterday and we brought it up during the meeting and it is being looked at now.

Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

Got the update of Feb 3 to devprev, which was expected to fix some of the rebuild issues. It is not better.

Rebuild has gone back to being extremely slow to find poses. On segs 54-58 of the ED puzzle (beginner), in 2 minutes of rebuild it found only 4 poses.

It is putting the backbone through itself even at CI 1.0. This is a bug. In my experience, rebuild should pay attention to backbone clashes according to clashing importance, putting the backbone through itself only at CI 0. It should ignore sidechain clashes, since those can often be resolved by shake.

It is completely deforming an already well-formed helix, mashing it into a jumble, again crossing the backbone despite CI 1.0.

Joined: 12/06/2008
Groups: Contenders

I suggest all players refuse the most recent update until this issue is resolved, and also to always keep a backup directory with the last working version of the game in it, in case an update turns out to be a huge flop. We're offered no options to "downgrade" back to a former version of the game.

Joined: 04/19/2009

I accepted the Feb 3 devprev update and opened 841 for the first time.

I tried using rebuild for the little helix at the end, and it did rebuild - although it ended up embedded in the sheet next to it (I changed no Behavior from default). But a local shake and local wiggle brought it out of the sheet and it looked normal.

Then I tried to rebuild the large helix. First I cut both ends, unchecked Enable Cut Bands, and selected Rebuild from the wheel. One end immediately homed in on the sheet next to it, and rebuild did nothing - just small differences, nothing visually curling.

I moved the helix far away from the rest of the protein... The helix still tried to home in on that sheet, and again, nothing much happening.

Then I tried the new NC method of rebuilding a helix - tweaked it until it finally curled up. Did a local shake and wiggle... Completely red and distorted.

Finally I tried the old school way of banding both ends to create some slack for the helix, and did rebuild. Ten minutes later, it's still a distorted mess.

Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

To reproduce the problems, just put all settings to default values, reset the ED beginner puzzle, right click the long helix and hit rebuild.

jflat06's picture
User offline. Last seen 4 days 9 hours ago. Offline
Joined: 09/29/2010
Groups: Window Group

Can you re-verify the behavior and tell me the build ID of the client in question?

jflat06's picture
User offline. Last seen 4 days 9 hours ago. Offline
Joined: 09/29/2010
Groups: Window Group

Also, when you are trying to Rebuild, are you attempting to do it while the helix is stretched out, with cut bands enabled?

Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

When I open devprev today I have build 20140203-a31a342de8-win_x86-devprev. I did not check the build number when I was having errors last night, but I did get a binary update last night. It's possible I did not get a full update, since I had a non-updated client running in the background, and possibly it was using some file and didn't get a new copy. This is on win 7 64.

I am not having the same errors today as yesterday. The backbone is not going through itself, and existing helices are not being severely deformed. The rate of finding rebuild has not improved much; it's about 3 per minute on my machine.

Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

I was rebuilding an already-existing helix. Auntdeen was trying to build a helix from an extended chain, something which many many people have had trouble with since newchapter went live. People have come up with a variety of techniques to make it work, but those techniques were not working for auntdeen last night.

Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

Here is some data comparing the Rebuild tool in the latest devprev release of newchapter (Feb 13) to earlier newchapter releases and to pre-newchapter devprev. I was only able to test one puzzle in common across these releases, but it looks like the latest Rebuild is finding poses at least as good as the pre-newchapter software, and better than the earlier newchapter releases. I have also noticed that it finds rebuild poses faster than the earlier newchapter releases.

Susume's picture
User offline. Last seen 8 hours 43 min ago. Offline
Joined: 10/02/2011

I tried increasing the number of rebuilds tried on each area from 10 to 20 (data on 10 rebuilds is attached to previous post). This increased the number of rebuilds with a positive score increase, from 41 to 80. It increased the number of areas of the protein improved by the script from 12 to 19 (out of 31 tried). The average improvement from each rebuild did not change, but with more areas being improved it did add up. The overall score increase went from 962 points (with 10 rebuilds per area) to 1181 points (with 20 rebuilds per area). There were still some areas where 20 rebuilds did not find an increase even though a previous test run of 10 rebuilds had found one. Still, as expected, trying more rebuilds results in better gains from a script run, because of better coverage of the areas where score increases are possible.

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons