Client crashes on scripts with rebuild
Case number: | 845829-987417 |
Topic: | Crash/Hang |
Opened by: | Bletchley Park |
Status: | Closed |
Type: | Bug |
Opened on: | Thursday, March 4, 2010 - 11:29 |
Last modified: | Wednesday, June 1, 2011 - 16:40 |
Unfortunately the client still has issues which cause it to crash unexpectedly, in particular on scripts that involve rebuilds. I had to restart at least 5 times this morning. I noticed these messages in the log:
Error: Unknown enum value for Puzzle::Type: .
Error: Unknown enum value for Puzzle::Type: .
core.optimization.LineMinimizer: Inaccurate G! step= 3.8147e-008 Deriv= -698.84 Finite Diff= 1.98572e+008
and a way before that:
SRVR_THRD error: Puzzle is closed (..\..\src\game\application\novice\boinc\Boinc.cc:349)
Is there anything I can do to make debugging easier for you ?
When filing bug reports I also get a message on the server saying that e-mail could not be sent, this may be a server issue.
Regards, BP.
Windows XP 32 bit, service pack 2, scripts with rebuilds in them, like overnight after a while. Some of my own scripts use rebuilds. My version is... : 21fdf490a555793ca0d1d72157357b6d.
We (group members) have also noticed that wiggling seemingly produces no result after some rebuilds.
What view options do you have on? Which interface are you using?
No view options selected, color: score, view protein: cartoon thin, view sidechains: don't show.
no music or sound effects, advanced interface, old-style gui.
Same issue here, identical setup to Bletchley Park. I'm not at my Fold.it PC right now, so I can't give you any log snippets.
Scripts (lua-style) crash the client anywhere from almost immediately to a hour or so after starting them.
Old-style recipes don't seem to cause the problem.
Also, I still have the issue of the in-game IRC connection getting broken or corrupted somehow during long-running lua script execution. Symptom of this is an error on closing the game, and no incoming or outgoing messages are possible in chat.
observation:
I had an unhanled win32 exception and was still able to move the protein, operate the user interface (change clashing importance for example) scroll the scores and open the chat windows, I posted a chat and stopped the shake that caused the exception. The text 'stopping' appeared, but then did not return to 'command mode'. The 'clock' kept running until I stopped the shake.
observation:
While simply wiggling the client crashed to a halt, no screen updates, no control whatsoever. Complete termination.
observation: crashed in module game_library.dll offset:007d706 while wiggling.
Windows 7 Enterprise
Overnight Script
df72aaf937e5773555800cc52329f0206
using the old interface here, have tried several view options, from from showing everything to showing nothing, problem persists.
This from a few minutes of 'Acid Tweaker v0.5' on puzzle 266
delta_score: -9028.1
Playing sound: 0
Tool on_action_complete called
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 113 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 28824 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 57 rotamers at 9 positions.
core.pack.pack_rotamers: IG: 12800 bytes
UNHANDLED EXCEPTION
1: no symbol (no line)
2: no symbol (no line)
3: no symbol (no line)
4: no symbol (no line)
5: no symbol (no line)
6: no symbol (no line)
7: no symbol (no line)
This script and others have worked well in the past - only have problem now with the latest couple of releases of the client.
exception at 0x107cd1c1 referencing 0x00000079 read.
exception at 0x0124d1c1 referencing 0x00030008 read The routine starts at 0x0124d190.
Thread Start: Thread ID: 0x00000C8C. Process foldit.exe (0x784)
Process Start: C:\Program Files\foldit\foldit.exe. Base Address: 0x00400000. Process
foldit.exe (0x784)
Module Load: UNKNOWN. No Debug Info. Base Address: 0x00400000. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000434. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x000009F4. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000CB8. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000BC0. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000B0C. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000CF8. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x000003FC. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x000009F0. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000158. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000B2C. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000C90. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000C94. Process foldit.exe (0x784)
Thread Start: Thread ID: 0x00000340. Process foldit.exe (0x784)
Module Load: ntdll.dll. No Debug Info. Base Address: 0x77F80000. Process foldit.exe (0x784)
Module Load: KERNEL32.dll. No Debug Info. Base Address: 0x7C570000. Process foldit.exe
(0x784)
Module Load: USER32.dll. No Debug Info. Base Address: 0x77E10000. Process foldit.exe (0x784)
Module Load: GDI32.dll. No Debug Info. Base Address: 0x77F40000. Process foldit.exe (0x784)
Module Load: SHELL32.dll. No Debug Info. Base Address: 0x7CF30000. Process foldit.exe
(0x784)
Module Load: MSVCRT.dll. No Debug Info. Base Address: 0x78000000. Process foldit.exe (0x784)
Module Load: COMCTL32.dll. No Debug Info. Base Address: 0x71710000. Process foldit.exe
(0x784)
Module Load: game_library.dll. No Debug Info. Base Address: 0x10000000. Process foldit.exe
(0x784)
more libraries not shown.
Maybe this helps in debugging. Client crashes within minutes immediately after terminating lua script while shaking.
I am able to reproduce this crash on occasion, I'm looking into it.
What is the status on this please ? The client crashes within 2 minutes on a quake for puzzle 273, it just exits, no error message, no windows warning, nothing, there is simply no client anymore.
The situation is pretty unworkable now, I am open to suggestions and possibly installing a debugger if you have one available so I can provide you with meaningful data. I'm on XP32 bit.
I am still looking into it, though having some trouble reproducing it. Does it happen if you play offline? What about if you delete your options.txt and have all the default options set?
I can confirm it happens without ever playing offline. Notice that 3 of these crashes have the same fault address 0x007cdcf1
All of these happened while running lua-style scripts, as opposed to the older recipes.
===================================
Event Type: Error
Event Source: Application Hang
Event Category: (101)
Event ID: 1002
Date: 3/16/2010
Time: 5:41:33 PM
User: N/A
Computer: BB-XP-2008
Description:
Hanging application foldit.exe, version 0.0.0.0, hang module hungapp, version 0.0.0.0, hang address 0x00000000.
======================================
Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/16/2010
Time: 5:46:49 PM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007cdcf1.
==========================================
Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/17/2010
Time: 12:49:29 AM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007cdcf1.
=======================================
Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/17/2010
Time: 12:51:15 AM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007cdcf1.
=====================================
Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/18/2010
Time: 5:35:34 AM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007df41a.
Taking your suggestion to try without an options.txt, I re-started fold.it with no options.txt and no log.txt.
Opened an existing saved best puzzle solution and immediately ran the "Overnight" script. In less than 30 minutes, the game crashed. There was no human interaction during the time the script was running. I wasn't watching it so I don't know what process it might have been working on.
I think we can assume that the options.txt isn't the problem.
This issue is getting very old and tiresome ....
==============================
Log entries:
UNHANDLED EXCEPTION
1: no symbol (no line)
2: no symbol (no line)
3: no symbol (no line)
4: no symbol (no line)
5: no symbol (no line)
===============================
Event log:
Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/18/2010
Time: 8:05:19 PM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007eb677.
Moving along... I tried playing offline, same puzzle, same script.
Almost immediate crash
===================
Event Log:
Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date: 3/18/2010
Time: 8:22:18 PM
User: N/A
Computer: BB-XP-2008
Description:
Faulting application foldit.exe, version 0.0.0.0, faulting module game_library.dll, version 0.0.0.0, fault address 0x007cdcf1.
================================
log.txt
UNHANDLED EXCEPTION
1: no symbol (no line)
When i run Overnight-like script it often crashes after Restore recent best and rebuild begins.
I am having the same problems, running service pack 2 on windows XP.
Could I make a suggestion ?
Could you make the foldit core programme do a clean shutdown routine when the user closes it and write to a small data file when it closes correctly. Then, whenever the programme starts, get it to read the contents of the data file, and if the previous run of foldit crashed, send out a notification across the internet to you guys ? That way you will be able to see how big this problem is. I had at least 20 crashes a couple of days ago but have not got the patience to report them.
Im sure you understand this, but in case its badly written I will explain it in another way
Data file contents : "Closed correctly"
I start the programme and it checks the data file, sends no message and ammends the data file to
Data file contents : "Programme Running"
I get a crash, and restart. The programme reads the data file and sends out an error message to you guys.
Data file contents : "Programme Running"
I continue playing and close the programme down when finished. Programme ammends data file to
Data file contents : "Closed correctly"
That way you get all the problems logged automatically
The game options were not the issue before the updates appeared, I have not changed my game optins since. The big problems started right after the update a while ago. Were these client updates compiled with a different data alignment / packing option for the compiler ? This would be a common mistake.
To reproduce the error:
Load 273.
1. Run Overnight and cancel the script while it is shaking
2. Run Overnight and cancel while it is rebuilding
3. Run Overnight and cancel while it is wiggling.
4. just run Overnight for 15 minutes or so.
That should reproduce the issue with little effort. Repeat a few times if needed.
Also, if you cannot reproduce the issue, please post here sooner. It is counterproductive to wait over 10 days and find out that you cannot reproduce the issue that is hampering many of your top folders performance. In the end it bites the project because we might have come up with better solutions had we had the working tools to investigate.
Again I offer to run a debugger or examine the source if it is open. I have over 20 professional years of software development experience and specialize in x86/SSE assembly language.
Attached is a screen shot of a recent crash on my machine.
There are two fantom elongated tubes on the screen which may help in diagnosing the problem
Attachment....
I have posted a debug version of the game to hopefully gather more information. To use it, you'll need to be on Windows, and follow the procedure for switching to the developer preview, but instead of "beta" use "main_debug". Hopefully there will be more useful information at the end of the log file with this version.
Debug version does not work.
"There are no puzzles available in this group" appears on the startup screen.
steveB, edit the options.txt file again and reset it to group main, then start the debug client and press the skip update button when it starts, that way you will keep the debug version, yet are able to run puzzles.
The puzzles should appear in game_debug now.
First try was running the debug version on a larger puzzle ran a script for at least 4 hours, but visually, fold.it was running *very* slow. I decided to cancel the script, which it did without crashing.
I started the same script ( the famous "overnight_script") on one of the smaller puzzles and let it run - well - overnight.
It's still running this morning without crashing, but fold.it is still obviously executing much slower.
Should I assume this slowness is somehow related to having the debug code in place?
I'm going to let this run all day to see if it can catch a crash.
Thanks for helping out. Yes, I would expect the debug version to run slower than the release.
I ran the debug version for over 24 hours on the "Overnight_script" without a crash.
I finally exited the script - no crash - and set my fold.it options.txt back to "main".
After restarting fold.it and doing a couple of manual global shakes and wiggles, I started the "overnight_script" on puzzle 274. It ran no longer than 20 minutes and crashed.
Is it possible there is some sort of timing or race condition that's causing the crash that doesn't come into play when the main_debug version is running, perhaps because the additional slowness of the script execution in debug mode prevents it? Something to look for...
Brick
The debug version is too slow to be practical on this testing. I noticed that events that normally take place in a second or 2 now take 10 minutes. I have the impression that the entire OpenGL is handled in software. Horribly slow. So no, that debug version did not crash with me either. 20 minutes real-world translates to 6000 minutes in debug.
Brick, did you get any information from the log when it crashed?
Sorry, no. I was back in "main" mode mode so I didn't think to save the log.
I'd give _debug another try, but now I can't seem to get any puzzles showing in-game. 277 doesn't show up for main or main_debug.
Here is a snippet where an exception occurred. From the logic you may be able to deduct which routine it is. Start is at 0124DCC0. Occurred while interrupting a shake in Lua.
Is it possible that the bug is coming from the chat client code ? I had a stable system for hours tonight and the moment I typed the first character into the chat window the system crashed.
I think I may have tracked down the source of the crashing when canceling a script, and at least one cause of the crashing while running the scripts. I've posted a new update for main_debug that addresses these (and should be faster than the last one). Please let me know if this is any more stable.
I've had some stability running scripts by unchecking all the view options. There are still crashes but not as many as before. The scripts run longer.
It seems more stable however when adding a script from the webpage I got this entirely new error:
Microsoft Visual C++ runtime library
Program: c:\program files\foldit\foldit.exe
R6025
- pure virtual function call
See also: http://support.microsoft.com/kb/125749
Status: Open » Closed |
Client updated. Closing.
You may be interested to see some replays from Dirt 3. I believe the same thing is happening in the multi-player sessions - quantum entanglement theory is being experimented on in real time in the form of driving curves, or in the case of foldit, the trajectories to the native fold.
What platform are you on? What scripts crash particularly often? What is your version (from menu -> general options)?