[NewChapter] performance (speed) issues

Case number:954892-996631
Topic:Developer Preview
Opened by:Susume
Opened on:Sunday, January 12, 2014 - 21:30
Last modified:Saturday, January 18, 2014 - 05:17

This thread is to collect all problems with speed/performance in the newchapter release into a single-issue thread. See previous discussion of speed issues here: http://fold.it/portal/node/996588

(Sun, 01/12/2014 - 21:30  |  15 comments)

Joined: 09/21/2011
Groups: Void Crushers

Performance measurements results (done after the latest update on 12/1/2014).
Method of measurement: running TvdL DRW 231 with timing (http://fold.it/portal/recipe/47727) on the ED puzzle in main (beginner puzzle) and newchapter, after an initial shake, 1 iteration.
Settings of DRW: only length 3, more options: 1 iteration. Selected to work on: 81-83. Running both minimized and in foreground.

Newchapter foreground:
Local shake during stabilizing: mean 10 seconds
Global wiggleall(6) during stabilizing: mean 180 seconds
Global shake during fuze: mean 55 seconds
Global wiggleall(2) during fuze: mean 60 seconds
Global wiggleall(6) during fuze: mean 180 seconds
Remark: there was very little variation in running times in the wiggle durations.

Newchapter minimized:
All shakes about 30% faster, all wiggles about 8% faster, same remark as on foreground.

Main foreground:
Local shake during stabilizing: 10 seconds
Global wiggleall(6) during stabilizing: 106-170, mean 140 seconds
Global shake during fuze: mean 50 seconds
Global wiggleall(2) during fuze: 6-68, mean 50 seconds
Global wiggleall(6) during fuze: 31-42, mean 35 seconds

Main minized:
Same kind of numbers but about 20% faster.

Other observations: During the newchapter wiggle all iterations take about the same time and the score is still changing even in iteration 6.

There is a major change in wiggle behavior, it is now time sliced, not depending on when the algoritm says it is enough.
This is a major game change. Wiggle will not settle after a few iterations.

Final remarks:
I will repeat these measurements on a much faster computer.
These measurements are on the ED puzzle, on ED there are 2 changes. I want to be able to do these kind of comparisons on other types of puzzles. To make that really possible other puzzle types should be available in main/devprev as well as in newchapter AND there should be a way to share from main/devprev to newchapter so beginning positions can be really equal.
I will comment later on what I see as consequences for the game of this major wiggle behavior change.

Susume's picture
User offline. Last seen 1 week 1 day ago. Offline
Joined: 10/02/2011

I have been comparing Blue Fuse v 1.1 in various contexts on the Staph Aureus ED puzzle (158 segments). The average time for the recipe to run on my machine in devprev (less precise) was 178 seconds; in newchapter the average was 396 seconds, a factor of 2.22.

Running the recipe unmodified in newchapter left points on the table, in that wiggle was still getting points when it cut off after the 8 iterations coded in the recipe. I modified it so the last wiggle would loop in blocks of 2 wiggles (you should always wiggle an even number of times) until it stopped gaining whole points. This version had an average run time of 582 seconds, or 3.27 times as long as the original in devprev. The looping version gained 75% more points on average than the original (on the one specific pose I was using - results may vary by how stable the protein is before starting).

I compared running this modified recipe when my CPU was at about 50% vs. 90-100%. While there was a noticeable speed difference, I did not see a significant difference in points.

The recipes we have now will run and gain points in newchapter; they will not be completely broken. If we don't modify the wiggles in the recipes to loop until they stop gaining, the missed gains will not necessarily be permanently lost; rather they will be postponed until later iterations of the recipe or until later recipes. It will become more common to simply run out of time for recipes, and less common to stop because you have found all the points you can.

Joined: 09/24/2012
Groups: Go Science

"(you should always wiggle an even number of times)".
I thought each step was self-sufficient. If they need 2 steps, why don't they do these 2 steps in one?

Susume's picture
User offline. Last seen 1 week 1 day ago. Offline
Joined: 10/02/2011

jflat told us this some time ago: "<@jflat06> what's actually happening is we idealize the backbone every other step."

I believe this was in a conversation about why iteration 2 had started giving far more points than iteration 1. The only place I know of where it is documented is in the details page for the structure.WiggleAll function in the wiki: http://foldit.wikia.com/wiki/Foldit_Lua_Function_structure.WiggleAll

If you are just wiggling with bands or wiggling at a low CI to lower the score, it's fine to wiggle just once, but if you are trying to get all the points out of a position it's better to wiggle an even number. This seems to be true in newchapter as well, where the odd numbered iterations go by much faster than the even numbered ones.

Joined: 09/21/2011
Groups: Void Crushers

Just have read the discussion in veterans.
It might be better to revert the wiggle to full iterations, this will have speed implications but will not break algoritms.
And add new seperate wiggle functions that do wiggle in chunks like it does now in newchapter.
Also pls do a big effort to optimize the score function as this will help not only wiggle but also mutate and shake.
These are my first thoughts after reading, maybe more when I get back from work.

Joined: 04/15/2012
Groups: Beta Folders

Yeah, perhaps "deprecate" the old function and replace it with new algorithm & changes that had been planned on being added?

jeff101's picture
User offline. Last seen 1 day 23 min ago. Offline
Joined: 04/20/2012
Groups: Go Science

I heard last night that newchapter has been in Rosetta@Home several months already (perhaps since Oct 10, 2013). Have you noticed a drop in speed with Rosetta@Home due to newchapter? If so, by how much?

dekim's picture
User offline. Last seen 8 years 31 weeks ago. Offline
Joined: 05/10/2008
Groups: None

Hi, I'm a developer for R@h. I haven't noticed a drop in throughput but if there was, it would probably be pretty small.

Joined: 09/21/2011
Groups: Void Crushers

Tested puzzle 663.
Loading the puzzle takes a long time
Initial fuzing to get it a bit stable took 1/2 hour.
Following that 1 mini round of DRW231 with timing took 3/4 hour.
I could load saved old versions. Repaired the worst parts by hand and then ran a DRW test version where I put back in tail recursion for wiggle. It takes, when the score is not too bad, about 10 rounds of wiggle to get to the stable state, if bad then 30 is not enough. Each round of wiggle taking 35 seconds.
I only selected 2 slots to be stabelized (the best score and the best pairwise score) but not succeeded getting to the fuze state. Only when the gain is getting very low a double iteration of wiggle is getting fast.

Susume's picture
User offline. Last seen 1 week 1 day ago. Offline
Joined: 10/02/2011

Wiggling to stability may be a target we simply can't hit any more. Attached is a log of one very long wiggle in newchapter. The score stabilizes at about 30 iterations (7 minutes on my machine), and a loop that tests for changes of >= 1 pt would give up there, but after 90 iterations it suddenly gains 47 more points. It gradually adds about 50 more points, then stabilizes again sometime after 112 iterations. Then after 432 iterations (18 minutes) it suddenly adds 40 more points. I gave up logging after 33 minutes, when it was still slowly gaining fractions.

brow42's picture
User offline. Last seen 3 weeks 5 days ago. Offline
Joined: 09/19/2011
Groups: None

Okay, now I *really* want wiggle sliders back. Maybe we can shortcut some of that by increasing wiggle accuracy over time.

wisky's picture
User offline. Last seen 1 year 47 weeks ago. Offline
Joined: 07/13/2011

Having a fairly powerful computer (4.1ghz proc), I decided to test just how long this wiggle will actually run before hitting stability. I used the CASP refinement puzzle.

Iterations 1-8700 (0:00-85:00) (roughly 600 points)
Starting from a score of 9177, it wiggled up to my old high score of 9650 within 200 iterations, acting very much like current Foldit's wiggle. After this, it started displaying the behavior mentioned below
As mentioned before, wiggle will appear to freeze for a good 100-200 iterations, after which it would jump up between 10-15 points in 2-5 iterations, then will crawl for the next 100ish iterations, gaining 5-10 points very steadily, then this process will repeat. Most of the point gains came from jumps. Jumps happened every 100-300 iterations. Final jump before 8700 was 14 points

Iterations 8700-12500 (85:00-113:00) (roughly 75 points)
At this point, the behavior is very similar, though most of the point gains came from crawling, with only 1-3 points per 100 iterations during crawl. The jumps also started happening less often, every 300-500 iterations, and only gained 2-4 points per jump, no jumps over 4 points.

Iterations 12500-14500 (113:00-)
I had stopped watching during this block, and only gained 5-10 points during this whole time. When I brought the client back up at iteration 14500 it had stopped moving. I let it run until iteration 16720 (161:00), and since there had been no gains for over 2000 iterations, I stopped it.

Results: around 2 hours of wiggle, and 678 points later, I now have a protein sitting at 9855.

Joined: 11/10/2007
Groups: Window Group

We are trying a "Wiggle Power" option to give some control over runtime and stability. See the feedback http://fold.it/portal/node/996664 for more info.

Joined: 06/24/2008
Groups: Void Crushers

Here are some wiggle number I did last night on 663:

New Download 17-Jan

Minutes 5 1702 8
10 4170 760 Stablized
15 4170 2134

WIGGLE HIGH Score iters
Minutes 5 -4133 3
10 -1995 4
15 221 6
20 2090 8

WIGGLE LOW Score iters
Minutes 5 6792 12
10 7894 21
15 7855.6 508 Stablized at 13:20

Joined: 06/24/2008
Groups: Void Crushers

New Download 17-Jan

Minutes 5.......1702.....8
........10......4170....760; Stabilized
........15......4170 2134

WIGGLE HIGH Score iters
Minutes 5......-4133...3
20 2090 8

WIGGLE LOW Score iters
Minutes 5.......6792....12
........15......7855.6..508; Stablized at 13:20


Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, Boehringer Ingelheim, RosettaCommons