Client crashes on scripts with rebuild

Case number:845829-987417
Topic:Crash/Hang
Opened by:Bletchley Park
Status:Closed
Type:Bug
Opened on:Thursday, March 4, 2010 - 11:29
Last modified:Wednesday, June 1, 2011 - 16:40

Unfortunately the client still has issues which cause it to crash unexpectedly, in particular on scripts that involve rebuilds. I had to restart at least 5 times this morning. I noticed these messages in the log:

Error: Unknown enum value for Puzzle::Type: .
Error: Unknown enum value for Puzzle::Type: .

core.optimization.LineMinimizer: Inaccurate G! step= 3.8147e-008 Deriv= -698.84 Finite Diff= 1.98572e+008

and a way before that:
SRVR_THRD error: Puzzle is closed (..\..\src\game\application\novice\boinc\Boinc.cc:349)

Is there anything I can do to make debugging easier for you ?

When filing bug reports I also get a message on the server saying that e-mail could not be sent, this may be a server issue.

Regards, BP.

(Thu, 03/04/2010 - 11:29  |  40 comments)


Joined: 11/10/2007
Groups: Window Group

What platform are you on? What scripts crash particularly often? What is your version (from menu -> general options)?

Joined: 05/19/2009
Groups: Contenders

Windows XP 32 bit, service pack 2, scripts with rebuilds in them, like overnight after a while. Some of my own scripts use rebuilds. My version is... : 21fdf490a555793ca0d1d72157357b6d.

We (group members) have also noticed that wiggling seemingly produces no result after some rebuilds.

Joined: 11/10/2007
Groups: Window Group

What view options do you have on? Which interface are you using?

Joined: 05/19/2009
Groups: Contenders

No view options selected, color: score, view protein: cartoon thin, view sidechains: don't show.
no music or sound effects, advanced interface, old-style gui.

Brick's picture
User offline. Last seen 2 years 19 weeks ago. Offline
Joined: 07/15/2008
Groups: Beta Folders

Same issue here, identical setup to Bletchley Park. I'm not at my Fold.it PC right now, so I can't give you any log snippets.

Scripts (lua-style) crash the client anywhere from almost immediately to a hour or so after starting them.

Old-style recipes don't seem to cause the problem.

Also, I still have the issue of the in-game IRC connection getting broken or corrupted somehow during long-running lua script execution. Symptom of this is an error on closing the game, and no incoming or outgoing messages are possible in chat.

Joined: 05/19/2009
Groups: Contenders

observation:

I had an unhanled win32 exception and was still able to move the protein, operate the user interface (change clashing importance for example) scroll the scores and open the chat windows, I posted a chat and stopped the shake that caused the exception. The text 'stopping' appeared, but then did not return to 'command mode'. The 'clock' kept running until I stopped the shake.

Joined: 05/19/2009
Groups: Contenders

observation:

While simply wiggling the client crashed to a halt, no screen updates, no control whatsoever. Complete termination.

Joined: 05/19/2009
Groups: Contenders

observation: crashed in module game_library.dll offset:007d706 while wiggling.

Joined: 05/26/2008
Groups: Hold My Beer

Windows 7 Enterprise
Overnight Script
df72aaf937e5773555800cc52329f0206

Joined: 05/26/2008
Groups: Hold My Beer

using the old interface here, have tried several view options, from from showing everything to showing nothing, problem persists.

Brick's picture
User offline. Last seen 2 years 19 weeks ago. Offline
Joined: 07/15/2008
Groups: Beta Folders

This from a few minutes of 'Acid Tweaker v0.5' on puzzle 266

delta_score: -9028.1
Playing sound: 0
Tool on_action_complete called
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 113 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 28824 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 57 rotamers at 9 positions.
core.pack.pack_rotamers: IG: 12800 bytes

UNHANDLED EXCEPTION
1: no symbol (no line)
2: no symbol (no line)
3: no symbol (no line)
4: no symbol (no line)
5: no symbol (no line)
6: no symbol (no line)
7: no symbol (no line)

This script and others have worked well in the past - only have problem now with the latest couple of releases of the client.

Joined: 05/19/2009
Groups: Contenders

exception at 0x107cd1c1 referencing 0x00000079 read.
exception at 0x0124d1c1 referencing 0x00030008 read The routine starts at 0x0124d190.

Thread Start: Thread ID: 0x00000C8C. Process foldit.exe (0x784)
Process Start: C:\Program Files\foldit\foldit.exe. Base Address: 0x00400000. Process

foldit.exe (0x784)
Module Load: UNKNOWN. No Debug Info. Base Address: 0x00400000. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000434. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x000009F4. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000CB8. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000BC0. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000B0C. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000CF8. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x000003FC. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x000009F0. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000158. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000B2C. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000C90. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000C94. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000340. Process foldit.exe (0x784)
Module Load: ntdll.dll. No Debug Info. Base Address: 0x77F80000. Process foldit.exe (0x784)
Module Load: KERNEL32.dll. No Debug Info. Base Address: 0x7C570000. Process foldit.exe

(0x784)
Module Load: USER32.dll. No Debug Info. Base Address: 0x77E10000. Process foldit.exe (0x784)
Module Load: GDI32.dll. No Debug Info. Base Address: 0x77F40000. Process foldit.exe (0x784)
Module Load: SHELL32.dll. No Debug Info. Base Address: 0x7CF30000. Process foldit.exe

(0x784)
Module Load: MSVCRT.dll. No Debug Info. Base Address: 0x78000000. Process foldit.exe (0x784)
Module Load: COMCTL32.dll. No Debug Info. Base Address: 0x71710000. Process foldit.exe

(0x784)
Module Load: game_library.dll. No Debug Info. Base Address: 0x10000000. Process foldit.exe

(0x784)

more libraries not shown.
Maybe this helps in debugging. Client crashes within minutes immediately after terminating lua script while shaking.

Joined: 11/10/2007
Groups: Window Group

I am able to reproduce this crash on occasion, I'm looking into it.

Joined: 05/19/2009
Groups: Contenders

What is the status on this please ? The client crashes within 2 minutes on a quake for puzzle 273, it just exits, no error message, no windows warning, nothing, there is simply no client anymore.

The situation is pretty unworkable now, I am open to suggestions and possibly installing a debugger if you have one available so I can provide you with meaningful data. I'm on XP32 bit.

Joined: 11/10/2007
Groups: Window Group

I am still looking into it, though having some trouble reproducing it. Does it happen if you play offline? What about if you delete your options.txt and have all the default options set?

Brick's picture
User offline. Last seen 2 years 19 weeks ago. Offline
Joined: 07/15/2008
Groups: Beta Folders

I can confirm it happens without ever playing offline. Notice that 3 of these crashes have the same fault address 0x007cdcf1

All of these happened while running lua-style scripts, as opposed to the older recipes.

===================================
Event Type: Error
Event Source: Application Hang
Event Category: (101)
Event ID: 1002
Date: 3/16/2010
Time: 5:41:33 PM
User: N/A
Computer: BB-XP-2008
Description:
Hanging application foldit.exe, version 0.0.0.0, hang module hungapp, version 0.0.0.0, hang address 0x00000000.
======================================
Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/16/2010
Time: 5:46:49 PM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007cdcf1.
==========================================
Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/17/2010
Time: 12:49:29 AM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007cdcf1.
=======================================
Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/17/2010
Time: 12:51:15 AM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007cdcf1.
=====================================
Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/18/2010
Time: 5:35:34 AM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007df41a.

Brick's picture
User offline. Last seen 2 years 19 weeks ago. Offline
Joined: 07/15/2008
Groups: Beta Folders

Taking your suggestion to try without an options.txt, I re-started fold.it with no options.txt and no log.txt.

Opened an existing saved best puzzle solution and immediately ran the "Overnight" script. In less than 30 minutes, the game crashed. There was no human interaction during the time the script was running. I wasn't watching it so I don't know what process it might have been working on.

I think we can assume that the options.txt isn't the problem.

This issue is getting very old and tiresome ....

==============================
Log entries:

UNHANDLED EXCEPTION
1: no symbol (no line)
2: no symbol (no line)
3: no symbol (no line)
4: no symbol (no line)
5: no symbol (no line)

===============================
Event log:

Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/18/2010
Time: 8:05:19 PM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007eb677.

Brick's picture
User offline. Last seen 2 years 19 weeks ago. Offline
Joined: 07/15/2008
Groups: Beta Folders

Moving along... I tried playing offline, same puzzle, same script.

Almost immediate crash

===================
Event Log:

Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/18/2010
Time: 8:22:18 PM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007cdcf1.

================================
log.txt

UNHANDLED EXCEPTION
1: no symbol (no line)

vakobo's picture
User offline. Last seen 1 year 25 weeks ago. Offline
Joined: 06/05/2008
Groups: Russian team

When i run Overnight-like script it often crashes after Restore recent best and rebuild begins.

steveB's picture
User offline. Last seen 1 year 27 weeks ago. Offline
Joined: 01/08/2009
Groups: Void Crushers

I am having the same problems, running service pack 2 on windows XP.

Could I make a suggestion ?

Could you make the foldit core programme do a clean shutdown routine when the user closes it and write to a small data file when it closes correctly. Then, whenever the programme starts, get it to read the contents of the data file, and if the previous run of foldit crashed, send out a notification across the internet to you guys ? That way you will be able to see how big this problem is. I had at least 20 crashes a couple of days ago but have not got the patience to report them.

Im sure you understand this, but in case its badly written I will explain it in another way

Data file contents : "Closed correctly"
I start the programme and it checks the data file, sends no message and ammends the data file to
Data file contents : "Programme Running"
I get a crash, and restart. The programme reads the data file and sends out an error message to you guys.
Data file contents : "Programme Running"
I continue playing and close the programme down when finished. Programme ammends data file to
Data file contents : "Closed correctly"

That way you get all the problems logged automatically

Joined: 05/19/2009
Groups: Contenders

The game options were not the issue before the updates appeared, I have not changed my game optins since. The big problems started right after the update a while ago. Were these client updates compiled with a different data alignment / packing option for the compiler ? This would be a common mistake.

To reproduce the error:

Load 273.
1. Run Overnight and cancel the script while it is shaking
2. Run Overnight and cancel while it is rebuilding
3. Run Overnight and cancel while it is wiggling.
4. just run Overnight for 15 minutes or so.

That should reproduce the issue with little effort. Repeat a few times if needed.

Also, if you cannot reproduce the issue, please post here sooner. It is counterproductive to wait over 10 days and find out that you cannot reproduce the issue that is hampering many of your top folders performance. In the end it bites the project because we might have come up with better solutions had we had the working tools to investigate.

Again I offer to run a debugger or examine the source if it is open. I have over 20 professional years of software development experience and specialize in x86/SSE assembly language.

steveB's picture
User offline. Last seen 1 year 27 weeks ago. Offline
Joined: 01/08/2009
Groups: Void Crushers

Attached is a screen shot of a recent crash on my machine.

There are two fantom elongated tubes on the screen which may help in diagnosing the problem

steveB's picture
User offline. Last seen 1 year 27 weeks ago. Offline
Joined: 01/08/2009
Groups: Void Crushers

Attachment....

Joined: 11/10/2007
Groups: Window Group

I have posted a debug version of the game to hopefully gather more information. To use it, you'll need to be on Windows, and follow the procedure for switching to the developer preview, but instead of "beta" use "main_debug". Hopefully there will be more useful information at the end of the log file with this version.

steveB's picture
User offline. Last seen 1 year 27 weeks ago. Offline
Joined: 01/08/2009
Groups: Void Crushers

Debug version does not work.

"There are no puzzles available in this group" appears on the startup screen.

Joined: 05/19/2009
Groups: Contenders

steveB, edit the options.txt file again and reset it to group main, then start the debug client and press the skip update button when it starts, that way you will keep the debug version, yet are able to run puzzles.

Joined: 11/10/2007
Groups: Window Group

The puzzles should appear in game_debug now.

Brick's picture
User offline. Last seen 2 years 19 weeks ago. Offline
Joined: 07/15/2008
Groups: Beta Folders

First try was running the debug version on a larger puzzle ran a script for at least 4 hours, but visually, fold.it was running *very* slow. I decided to cancel the script, which it did without crashing.

I started the same script ( the famous "overnight_script") on one of the smaller puzzles and let it run - well - overnight.
It's still running this morning without crashing, but fold.it is still obviously executing much slower.

Should I assume this slowness is somehow related to having the debug code in place?

I'm going to let this run all day to see if it can catch a crash.

Joined: 11/10/2007
Groups: Window Group

Thanks for helping out. Yes, I would expect the debug version to run slower than the release.

Brick's picture
User offline. Last seen 2 years 19 weeks ago. Offline
Joined: 07/15/2008
Groups: Beta Folders

I ran the debug version for over 24 hours on the "Overnight_script" without a crash.

I finally exited the script - no crash - and set my fold.it options.txt back to "main".

After restarting fold.it and doing a couple of manual global shakes and wiggles, I started the "overnight_script" on puzzle 274. It ran no longer than 20 minutes and crashed.

Is it possible there is some sort of timing or race condition that's causing the crash that doesn't come into play when the main_debug version is running, perhaps because the additional slowness of the script execution in debug mode prevents it? Something to look for...

Brick

Joined: 05/19/2009
Groups: Contenders

The debug version is too slow to be practical on this testing. I noticed that events that normally take place in a second or 2 now take 10 minutes. I have the impression that the entire OpenGL is handled in software. Horribly slow. So no, that debug version did not crash with me either. 20 minutes real-world translates to 6000 minutes in debug.

Joined: 11/10/2007
Groups: Window Group

Brick, did you get any information from the log when it crashed?

Brick's picture
User offline. Last seen 2 years 19 weeks ago. Offline
Joined: 07/15/2008
Groups: Beta Folders

Sorry, no. I was back in "main" mode mode so I didn't think to save the log.

I'd give _debug another try, but now I can't seem to get any puzzles showing in-game. 277 doesn't show up for main or main_debug.

Joined: 05/19/2009
Groups: Contenders

Here is a snippet where an exception occurred. From the logic you may be able to deduct which routine it is. Start is at 0124DCC0. Occurred while interrupting a shake in Lua.

steveB's picture
User offline. Last seen 1 year 27 weeks ago. Offline
Joined: 01/08/2009
Groups: Void Crushers

Is it possible that the bug is coming from the chat client code ? I had a stable system for hours tonight and the moment I typed the first character into the chat window the system crashed.

Joined: 11/10/2007
Groups: Window Group

I think I may have tracked down the source of the crashing when canceling a script, and at least one cause of the crashing while running the scripts. I've posted a new update for main_debug that addresses these (and should be faster than the last one). Please let me know if this is any more stable.

Joined: 05/09/2008
Groups: Void Crushers

I've had some stability running scripts by unchecking all the view options. There are still crashes but not as many as before. The scripts run longer.

Joined: 05/19/2009
Groups: Contenders

It seems more stable however when adding a script from the webpage I got this entirely new error:

Microsoft Visual C++ runtime library

Program: c:\program files\foldit\foldit.exe

R6025
- pure virtual function call

See also: http://support.microsoft.com/kb/125749

Joined: 06/17/2010
Status: Open » Closed

Client updated. Closing.

cjmarsh's picture
User offline. Last seen 4 years 31 weeks ago. Offline
Joined: 05/21/2011
Groups: Go Science

You may be interested to see some replays from Dirt 3. I believe the same thing is happening in the multi-player sessions - quantum entanglement theory is being experimented on in real time in the form of driving curves, or in the case of foldit, the trajectories to the native fold.

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, Boehringer Ingelheim, RosettaCommons