1052 crash when using idelize recipe

Case number:845829-2000404
Topic:Crash/Hang
Opened by:Rav3n_pl
Status:Open
Type:Bug
Opened on:Sunday, February 15, 2015 - 14:02
Last modified:Sunday, March 15, 2015 - 07:34

When using http://fold.it/portal/recipe/100680 client crash on last iteration
End of log file:
[code]
FOLD_TREE EDGE 244 81 4 EDGE 244 245 7 EDGE 244 246 8 EDGE 245 162 5 EDGE 246 243 6 EDGE 162 161 2 EDGE 243 242 3 EDGE 161 82 -1 EDGE 242 163 -1 EDGE 81 80 1 EDGE 80 1 -1

ERROR: asymmetry introduced by ActionIdealize
ERROR:: Exit from: ..\..\..\source\src\interactive\application\actions\cart\PoseLoopThreadActionCart.cc line: 453

UNHANDLED EXCEPTION
1: RaiseException +88 bytes (no line)
2: library_main +38313903 bytes (no line)
3: library_main +36386022 bytes (no line)
4: library_main +7053464 bytes (no line)
5: library_main +402593 bytes (no line)
6: library_main +4221173 bytes (no line)
7: library_main +4221627 bytes (no line)
8: BaseThreadInitThunk +18 bytes (no line)
9: RtlInitializeExceptionChain +99 bytes (no line)
10: RtlInitializeExceptionChain +54 bytes (no line)
[/code]
Win7, latest main.

(Sun, 02/15/2015 - 14:02  |  10 comments)


jeff101's picture
User offline. Last seen 20 hours 28 min ago. Offline
Joined: 04/20/2012
Groups: Go Science

This fall there was a devprev BETA Symmetry Debugging puzzle that was very similar to 1052 (http://fold.it/portal/node/1998494). Some of my old log files from that puzzle ended similarly, as below, making me think the bug Rav3n_pl reports above has the same root cause:

Here is the end of a log file (logSYMh94) from a crash during an idealize step:

ASYMMETRY INTRODUCED
 atomno= 6 rsd= 1 
 atomno= 7 rsd= 1 
 atomno= 9 rsd= 1 
 atomno= 10 rsd= 1 
 atomno= 13 rsd= 1 
 atomno= 15 rsd= 1 
 atomno= 16 rsd= 1 
 atomno= 17 rsd= 1 
FOLD_TREE  EDGE 199 66 4  EDGE 199 200 7  EDGE 199 201 8  EDGE 200 132 5  EDGE 201 198 6  EDGE 132 131 2  EDGE 198 197 3  EDGE 131 67 -1  EDGE 197 133 -1  EDGE 66 65 1  EDGE 65 1 -1 


ERROR: asymmetry introduced by ActionIdealize
ERROR:: Exit from: ..\..\..\source\src\interactive\application\actions\cart\PoseLoopThreadActionCart.cc line: 453

UNHANDLED EXCEPTION
  1: RaiseException +88 bytes (no line)
  2: library_main +38290543 bytes (no line)
  3: library_main +36362886 bytes (no line)
  4: library_main +7038056 bytes (no line)
  5: library_main +402177 bytes (no line)
  6: library_main +4209525 bytes (no line)
  7: library_main +4209979 bytes (no line)
  8: BaseThreadInitThunk +18 bytes (no line)
  9: RtlInitializeExceptionChain +99 bytes (no line)
 10: RtlInitializeExceptionChain +54 bytes (no line)

Here is the end of a log file (logSYMg74) from a crash during a wiggle step:

ASYMMETRY INTRODUCED
 atomno= 1 rsd= 1 
 atomno= 5 rsd= 1 
 atomno= 6 rsd= 1 
 atomno= 7 rsd= 1 
 atomno= 8 rsd= 1 
 atomno= 9 rsd= 1 
 atomno= 10 rsd= 1 
 atomno= 12 rsd= 1 
 atomno= 13 rsd= 1 
 atomno= 14 rsd= 1 
 atomno= 15 rsd= 1 
 atomno= 16 rsd= 1 
 atomno= 17 rsd= 1 
FOLD_TREE  EDGE 199 66 4  EDGE 199 200 7  EDGE 199 201 8  EDGE 200 132 5  EDGE 201 198 6  EDGE 132 99 2  EDGE 198 165 3  EDGE 99 67 -1  EDGE 99 131 -1  EDGE 165 133 -1  EDGE 165 197 -1  EDGE 66 33 1  EDGE 33 1 -1  EDGE 33 65 -1 


ERROR: asymmetry introduced by ActionGlobalMinimize
ERROR:: Exit from: ..\..\..\source\src\interactive\application\actions\cart\PoseLoopThreadActionCart.cc line: 453

UNHANDLED EXCEPTION
  1: RaiseException +88 bytes (no line)
  2: library_main +38290543 bytes (no line)
  3: library_main +36362886 bytes (no line)
  4: library_main +7038056 bytes (no line)
  5: library_main +402177 bytes (no line)
  6: library_main +4209525 bytes (no line)
  7: library_main +4209979 bytes (no line)
  8: BaseThreadInitThunk +18 bytes (no line)
  9: RtlInitializeExceptionChain +99 bytes (no line)
 10: RtlInitializeExceptionChain +54 bytes (no line)
jeff101's picture
User offline. Last seen 20 hours 28 min ago. Offline
Joined: 04/20/2012
Groups: Go Science

Random Idealize 1.8 crashes Foldit on symmetry puzzles.

Random Idealize 1.8 (http://fold.it/portal/recipe/100243) has crashed on many symmetry puzzles,
including devprev BETA Symmetry Debugging (http://fold.it/portal/node/1998494 a trimer),
devprev Symmetric Dimer of Dimers (https://fold.it/portal/node/996795),
1052 (trimer), 1043 (hexamer), 1037 (pentamer), and 1027b (tetramer).

Below is an example of how to force a crash on puzzle 1052:

If you use the Original Interface and
reset the puzzle and reset the secondary structure
(use the "Reset Structures" not the "Auto Structures" button)
and set the clashing importance to 1 and the wiggle power to low,
the score should read -663893.498 F -5950 (the F means from filters).
Using no bands and no frozen residues, start Random Idealize 1.8.
Click the "Fast" button in the lower left corner, then from top to
bottom enter 1 5 33 62 30 30, leave 8 boxes blank, enter 3, put an
X in the next box, and then enter 2 0's. The screen should appear
as in figure 1052g52.png (attached).

Next press "OK".

jeff101's picture
User offline. Last seen 20 hours 28 min ago. Offline
Joined: 04/20/2012
Groups: Go Science

After a few minutes (5 iterations), the screen looks like in figure 1052g52b.png.
The GUI should list the score -530401.750 F -5950 while the
Recipe Output (scriptlog1052g52.txt attached) ends with:

Segments 33-62 (iiiiiiiiiiiiiiiiiiiiiiiiiiiiii=LLLLLLLLLLLLLLLLLLLLLLLLLLLLLL) 9449826961: 	-530401.781	(+0.001, +133491.718) [4] 48
   Idealize selected: 	-530401.781	(+0.001) 48
   Wiggle selected: 	-530401.781	(0.000) 48
Saved/Loaded -530401.781/-530401.781 to/from slot 13, changing score by 0.000e+000.
Idealizing 33-62 (iiiiiiiiiiiiiiiiiiiiiiiiiiiiii=LLLLLLLLLLLLLLLLLLLLLLLLLLLLLL) 0388657023 from score -530401.781:
Done Idealizing.
Saved/Loaded -530401.751/-530401.751 to/from slot 13, changing score by 0.000e+000.
Wiggling 33-62 (iiiiiiiiiiiiiiiiiiiiiiiiiiiiii=LLLLLLLLLLLLLLLLLLLLLLLLLLLLLL) 0388657023 from score -530401.751:
Done Wiggling.
Segments 33-62 (iiiiiiiiiiiiiiiiiiiiiiiiiiiiii=LLLLLLLLLLLLLLLLLLLLLLLLLLLLLL) 0388657023: 	-530401.751	(+0.030, +133491.748) [5] 60
   Idealize selected: 	-530401.751	(+0.030) 60
   Wiggle selected: 	-530401.751	(0.000) 60
Loop 1: 	-530401.751	(+133491.748, +133491.748) [5] 60
Elapsed time: 0h 04m 49s at 03/14/15 15:28:50

If you would have instead run Random Idealize 1.8 with a 60 in place of the 5,
Foldit would have crashed after iteration 5 with Recipe Output ending like below:

Segments 33-62 (iiiiiiiiiiiiiiiiiiiiiiiiiiiiii=LLLLLLLLLLLLLLLLLLLLLLLLLLLLLL) 9449826961: 	-530401.781	(+0.001, +133491.718) [4] 48
   Idealize selected: 	-530401.781	(+0.001) 48
   Wiggle selected: 	-530401.781	(0.000) 48
Saved/Loaded -530401.781/-530401.781 to/from slot 13, changing score by 0.000e+000.
Idealizing 33-62 (iiiiiiiiiiiiiiiiiiiiiiiiiiiiii=LLLLLLLLLLLLLLLLLLLLLLLLLLLLLL) 0388657023 from score -530401.781:
Done Idealizing.
Saved/Loaded -530401.751/-530401.751 to/from slot 13, changing score by 0.000e+000.
Wiggling 33-62 (iiiiiiiiiiiiiiiiiiiiiiiiiiiiii=LLLLLLLLLLLLLLLLLLLLLLLLLLLLLL) 0388657023 from score -530401.751:
Done Wiggling.
Segments 33-62 (iiiiiiiiiiiiiiiiiiiiiiiiiiiiii=LLLLLLLLLLLLLLLLLLLLLLLLLLLLLL) 0388657023: 	-530401.751	(+0.030, +133491.748) [5] 60
   Idealize selected: 	-530401.751	(+0.030) 60
   Wiggle selected: 	-530401.751	(0.000) 60
Saved/Loaded -530401.751/-530401.751 to/from slot 13, changing score by 0.000e+000.
Idealizing 33-62 (iiiiiiiiiiiiiiiiiiiiiiiiiiiiii=LLLLLLLLLLLLLLLLLLLLLLLLLLLLLL) 0134737553 from score -530401.751:
jeff101's picture
User offline. Last seen 20 hours 28 min ago. Offline
Joined: 04/20/2012
Groups: Go Science

Anyway, with the GUI listing the score -530401.750 F -5950,
go into Menu -> Open/Share Solutions and
check the "Show auto and quick saves" box there.
Look for "Quicksaved Soloist Solution 3" and load it.
This should contain the score -530401.750 F -5950 but
instead it loads with the score -530401.781 F -5950.
This score difference seems to be a symptom of the bug affecting
symmetry puzzles.

Next use the Recipe ListStructDiffs1 (https://fold.it/portal/recipe/100351).
From top to bottom enter 99 3 and X, as in 1052g53.png attached.
Then press "OK".
ListStructDiffs1 will compare the scores and subscores for
slot 3 loaded manually vs slot 3 loaded via LUA.
Its output (scriptlog1052g53.txt) is attached below.
Below are some key results from it:

Total score 1 differs by -530401.78108211 - -530401.75034317 = -0.03073893848341.
Total for Packing differs by 4492.6309975696 - 4492.6791760865 = -0.0481785168231.
Total for Hiding differs by -14458.406180883 - -14458.43141848 = 0.025237597043088.
Total for Other differs by 93.24811238913 - 93.240540938288 = 0.0075714508425477.

Segment i2 has ss L,L and the subscore differences below:
   Packing = 53.33743518335 - 53.385613700173 = -0.048178516823029.
   Hiding = -190.80997162899 - -190.83520922603 = 0.02523759703945.
   Other = -1.8499099771961 - -1.856825616336 = 0.0069156391398661.
Segment i2 has score 1 = -6677.3595530635, -6677.3435277829 (differ by -0.016025280662689).

Oddly, even though Random Idealize 1.8 did all its wiggling and idealizing on segments 33-62,
the largest score differences were in segment 2, where the 3 monomers were close together.

jeff101's picture
User offline. Last seen 20 hours 28 min ago. Offline
Joined: 04/20/2012
Groups: Go Science

Next, switch to the Selection Interface and select amino acids 33-62
so they turn blue as in 1052g53a.png. If you start with the score -530401.781 F -5950
and press button 2, the score does not change and Foldit does not crash.
If you then press button 5, again the score does not change and Foldit does not crash.

Next, use the Undo Graph to restore the most recent structure
giving the score -530401.750 F -5950
and select amino acids 33-62 again so they turn blue.
If you now press button 2 (idealize peptide bonds),
Foldit will crash, giving an image like 1052g53b.png (attached) and
a log.txt file ending like below (see log1052g53.txt attached):

 atomno= 1 rsd= 32 
 atomno= 2 rsd= 32 
 atomno= 5 rsd= 32 
 atomno= 6 rsd= 32 
 atomno= 7 rsd= 32 
 atomno= 8 rsd= 32 
 atomno= 9 rsd= 32 
 atomno= 10 rsd= 32 
 atomno= 12 rsd= 32 
 atomno= 13 rsd= 32 
 atomno= 14 rsd= 32 
 atomno= 16 rsd= 32 
 atomno= 17 rsd= 32 
 atomno= 18 rsd= 32 
 atomno= 19 rsd= 32 
FOLD_TREE  EDGE 244 81 4  EDGE 244 245 7  EDGE 244 246 8  EDGE 245 162 5  EDGE 246 243 6  EDGE 162 161 2  EDGE 243 242 3  EDGE 161 82 -1  EDGE 242 163 -1  EDGE 81 80 1  EDGE 80 1 -1 


ERROR: asymmetry introduced by ActionIdealize
ERROR:: Exit from: ..\..\..\source\src\interactive\application\actions\cart\PoseLoopThreadActionCart.cc line: 453

UNHANDLED EXCEPTION
  1: RaiseException +88 bytes (no line)
  2: library_main +38326863 bytes (no line)
  3: library_main +36399030 bytes (no line)
  4: library_main +7067241 bytes (no line)
  5: library_main +402705 bytes (no line)
  6: library_main +4233461 bytes (no line)
  7: library_main +4233915 bytes (no line)
  8: BaseThreadInitThunk +18 bytes (no line)
  9: RtlInitializeExceptionChain +99 bytes (no line)
 10: RtlInitializeExceptionChain +54 bytes (no line)
Sending SOPs:


It also gave the debug1052g53.txt file attached.

jeff101's picture
User offline. Last seen 20 hours 28 min ago. Offline
Joined: 04/20/2012
Groups: Go Science

I posted the above to encourage others to explore the symmetry bug.
The runs 1052g52 and 1052g53 above are interesting because
they crashed after only [5] iterations from a very simple structure
(all isoleucines and all loops). I did try some runs based on 1052's
initial structure but with all amino acids mutated to alanine, for example.
I was able to make all 20 such structures crash Foldit using Random Idealize 1.8.

If you can find a case on 1052 or any other symmetry puzzle
that crashes after fewer iterations, please let me know.

I haven't used ListStructDiffs1 much or
tried to interpret each crash's log.txt file.

jeff101's picture
User offline. Last seen 20 hours 28 min ago. Offline
Joined: 04/20/2012
Groups: Go Science

If you can replicate the above runs on your machine, please let me know.
I haven't tried them on Linux or a Mac, for example.

Another way to run Random Idealize 1.8 is shown in 1052e9.png below.
If you replace the loop length with 60, it will crash after iteration [13]
Idealizing 4-8 (iiiii=LLLLL) 0032736950 from score -478605.765
and give results as in scriptlog1052e8.txt log1052e8.txt and debug1052e8.txt below.
If you leave the loop length set to 13, it ends as in scriptlog1052e9.txt,
and its slot 3 loads via LUA as -478605.764 F -5950
but loads manually as -478605.636 F -5950.

Next, in the Selection Interface with segments 4-8 selected,
if you press button 2 with the score -478605.636 F -5950,
the score becomes -478609.088 F -5950 and Foldit does not crash.
Similarly, if you press button 2 with the score -478605.764 F -5950,
Foldit crashes as in 1052e15.png below.

jeff101's picture
User offline. Last seen 20 hours 28 min ago. Offline
Joined: 04/20/2012
Groups: Go Science

The way shown for run 1052e8 above is good for seeing if a crash is possible in your puzzle.
It won't always crash Foldit so quickly, but it does tend to crash Foldit by [60] iterations
if a crash is possible. Also, using different values for rand_seed or rand_so_far can give
different results.

Once you know a crash is possible, it helps to do runs like 1052g52 and 1052g53
to examine more closely what causes the crashes and find ways to make these crashes occur sooner.
Spot checks have shown similar runs with
(Min Segments)=(Max Segments)=[(Max segment #)-(Min segment #)+1]
to be independent of rand_seed and rand_so_far.
I think this should make such runs easier to replicate on different machines and operating systems.

egran48's picture
User offline. Last seen 2 years 49 weeks ago. Offline
Joined: 03/31/2014
Groups: Go Science

Thanks for your efforts Jeff, it will be a few years before this makes sense to me but I really appreciate you maths folks who can write these recipes :)

mirp's picture
User offline. Last seen 1 year 7 weeks ago. Offline
Joined: 07/24/2012
Groups: Go Science

Thank you for your efforts Jeff.

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons