Yet another kind of hang

Case number:845829-995010
Topic:Crash/Hang
Opened by:Timo van der Laan
Status:Closed
Type:Bug
Opened on:Tuesday, April 23, 2013 - 22:06
Last modified:Monday, May 6, 2013 - 21:45

From the log: (windows 7 32 bit)
core.scoring.NeighborList: Minimization stats: 12 score/deriv cals, 0 narrow-from-wide updates, 0 full updates.
delta_score: 0.0653467
Playing sound: 4
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 126 rotamers at 7 positions.
core.pack.pack_rotamers: IG: 22424 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 218 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 66300 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 218 rotamers at 8 positions.
core.pack.pack_rotamers: IG: 75444 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 405 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 267148 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 318 rotamers at 9 positions.
core.pack.pack_rotamers: IG: 158104 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 249 rotamers at 6 positions.
core.pack.pack_rotamers: IG: 83560 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 35 rotamers at 4 positions.
core.pack.pack_rotamers: IG: 4248 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 253 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 100068 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 170 rotamers at 7 positions.
core.pack.pack_rotamers: IG: 46904 bytes
delta_score: 1664.4
Playing sound: 8
Autosaved puzzles/0000994977/0000308854/q5/autosave.ir_solution.
Autosaved puzzles/0000994977/0000308854/q5/autosave-best.ir_solution.
Autosaved puzzles/0000994977/0000308854/q5/autosave-recentbest.ir_solution.
Autosaved puzzles/0000994977/0000308854/q5/autosave-creditbest.ir_solution.

UNHANDLED EXCEPTION
1: library_main +3854501 bytes (no line)
2: library_main +3855718 bytes (no line)
3: library_main +2406953 bytes (no line)
4: library_main +2407254 bytes (no line)
5: library_main +896593 bytes (no line)
6: library_main +1046846 bytes (no line)
7: library_main +1094184 bytes (no line)
8: library_main +1144085 bytes (no line)
9: library_main +1255401 bytes (no line)
10: library_main +3842797 bytes (no line)
11: library_main +4765158 bytes (no line)
12: library_main +4730141 bytes (no line)
13: library_main +4733871 bytes (no line)
14: library_main +17328826 bytes (no line)
15: library_main +17330528 bytes (no line)
16: library_main +18068222 bytes (no line)
17: library_main +18078469 bytes (no line)
18: library_main +17333191 bytes (no line)
19: library_main +17328069 bytes (no line)
20: library_main +4771115 bytes (no line)
21: library_main +4285374 bytes (no line)
22: library_main +4288518 bytes (no line)
23: BaseThreadInitThunk +18 bytes (no line)
24: RtlInitializeExceptionChain +239 bytes (no line)
25: RtlInitializeExceptionChain +194 bytes (no line)

(Tue, 04/23/2013 - 22:06  |  22 comments)


Joined: 05/11/2012

What were you doing at the time?

Joined: 09/21/2011
Groups: Void Crushers

Just running my DRW. Probably inside qstab. Shake (0.1) and Wiggle all (CI 1.0), score computing and using recentbest. Puzzle 701.

Madde's picture
User offline. Last seen 1 year 4 weeks ago. Offline
Joined: 05/29/2008
Groups: Void Crushers

My client crashed last night while running Timo's DRW on puzzle 704.
(Win7 64 - 2013-04-18-54935M-win_x86-devprev)

Last lines of log.txt:

core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 28 rotamers at 3 positions.
core.pack.pack_rotamers: IG: 2724 bytes
delta_score: 58.502
Playing sound: 5
core.fragment.picking_old.vall.eval.IdentityEval: ss = LLL | aa = SSE
core.fragment.picking_old.vall.VallLibrarian: best fragment: score = 0.000369381
core.fragment.picking_old.vall.VallLibrarian: worst fragment: score = 1.00052
Unhandled exception in thread 02E3B0B8 (class game::application::novice::BoincThread): couldn't connect to server: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:98)

ERROR: Unhandled exception: couldn't connect to server: Error contacting server.
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly
CB: 0.00377471 1
TB: 13.5134 11.2665
CB: 0.000925891 1
TB: 11.4368 11.2665
CB: 0.00148015 1
TB: 13.2533 11.2665
CB: 0.119052 1
TB: 21.5009 11.2665
CB: 0.00480801 1
TB: 12.6156 11.2665
CB: 0.00895984 1
TB: 13.6597 11.2665
CB: 0.00423218 1
TB: 12.5033 11.2665
CB: 0.00811043 1
TB: 12.4718 11.2665
CB: 0.000866957 1
TB: 20.8049 11.2665
CB: 0.0834339 1
TB: 20.3214 11.2665
CB: 0.00543372 1
TB: 13.8357 11.2665
CB: 0.000547543 1
TB: 11.7531 11.2665
CB: 0.0158337 1
TB: 17.5193 11.2665
CB: 0.000167563 1
TB: 11.2912 11.2665
CB: 0.00473409 1
TB: 22.2685 11.2665
CB: 0.00121871 1
TB: 14.0446 21.2665
delta_score: -54.8563
Playing sound: 3
delta_score: 0
Playing sound: 4

Joined: 05/11/2012

Assigning for BoincThread exception issue (which may or may not be the only issue).

Hanto's picture
User offline. Last seen 2 hours 45 min ago. Offline
Joined: 05/10/2008
Groups: None

Having similar problem, just took out 6 clients of which only 4 were 704, other two were 701, all devprev win7 x64. Voting up

Madde's picture
User offline. Last seen 1 year 4 weeks ago. Offline
Joined: 05/29/2008
Groups: Void Crushers

Another crash log from me:
(Win7 64 - 2013-04-18-54935M-win_x86-devprev)

SRVR_THRD starting solution data upload...
Autosaved puzzles/0000995004/0000042629/default/autosave.ir_solution.
Autosaved puzzles/0000995004/0000042629/default/autosave-best.ir_solution.
Autosaved puzzles/0000995004/0000042629/default/autosave-recentbest.ir_solution.
Autosaved puzzles/0000995004/0000042629/default/autosave-creditbest.ir_solution.
SRVR_THRD finished solution data upload. took 3.51132 seconds.
Unhandled exception in thread 02F6D800 (class game::application::novice::BoincThread): Error: database returned invalid result at time 1366848145 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:469)

ERROR: Unhandled exception: Error: database returned invalid result at time 1366848145
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly
delta_score: 0.0403983
Playing sound: 4

RicGray's picture
User offline. Last seen 2 years 24 weeks ago. Offline
Joined: 11/15/2012
Groups: ricg test group
Status: Open » Closed

Added more debugging information to the log to have a better idea of what is going on when this occurs.

MurloW's picture
User offline. Last seen 1 week 4 days ago. Offline
Joined: 11/21/2012

Had two clients running overnight, 701 and 704.
Both crashed leaving behind on the screen only a windows error message each.
Win 7 x64, main latest update.

I did not have an internet connection at the time, but have never had crash issues due to this.
If that is currently the case though, well that'd suck. :/
But I'm hoping not, because I have a lot of these from halfway through log down to the end:

CB: 0.000253392 1
TB: 15.8127 20.2671
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
delta_score: -1797.51
Playing sound: 0
delta_score: 0
Playing sound: 4
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_user3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_group3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph

Bottom lines:

core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 79 rotamers at 8 positions.
core.pack.pack_rotamers: IG: 15928 bytes
Error contacting server using URL: https://fold.it/comm/achievements_report
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_user3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_group3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
SRVR_THRD getting notifications...
Error contacting server using URL: https://fold.it/comm/notification_list
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzles25_main_csv.txt
SRVR_THRD sending play event log...
Error contacting server using URL: https://fold.it/comm/log
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 80 rotamers at 8 positions.
core.pack.pack_rotamers: IG: 17136 bytes
delta_score: 1.34605e-010
Playing sound: 4
delta_score: 524.049
Playing sound: 6
delta_score: 1.14007
Playing sound: 4
Error contacting server using URL: https://fold.it/comm/achievements_report
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_user3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_group3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
SRVR_THRD getting notifications...
delta_score: 0.0440358
Playing sound: 4
Error contacting server using URL: https://fold.it/comm/notification_list
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzles25_main_csv.txt
Error contacting server using URL: https://fold.it/comm/foldit_ops
Unhandled exception in thread 02893608 (class game::application::novice::BoincThread): couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)

ERROR: Unhandled exception: couldn't resolve host name: Error contacting server.
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly

Joined: 06/04/2012

While running TDRW on one client and Acid Tweaker on the other

MurloW's picture
User offline. Last seen 1 week 4 days ago. Offline
Joined: 11/21/2012

More.
Tdrw on 706 crashed client,
at the exact same moment several team members' clients went down as well on different puzzles.

core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 274 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 119676 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 118 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 29692 bytes
Unhandled exception in thread 02903608 (class game::application::novice::BoincThread): Error: database returned invalid result at time 1367103871 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)

ERROR: Unhandled exception: Error: database returned invalid result at time 1367103871
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly
812 bytes

Joined: 09/21/2011
Groups: Void Crushers
Status: Closed » Open

Reopened because of lots of crash reports.

Susume's picture
User offline. Last seen 9 hours 44 min ago. Offline
Joined: 10/02/2011
Status: Open » Closed

lost one client in past hour, on 701, running my own version of Rav3n_pl GAB v0.6.1
also lost one client at same time as Murlow and several teammates ~2.5 hours ago, 701, TvdL DRW
win 7 64

most recent error in log:

Unhandled exception in thread 02C97E60 (class game::application::novice::BoincThread): Error: database returned invalid result at time 1367103807 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)
ERROR: Unhandled exception: Error: database returned invalid result at time 1367103807
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly

Susume's picture
User offline. Last seen 9 hours 44 min ago. Offline
Joined: 10/02/2011
Status: Closed » Open

Oops, Timo and I crossed and status went to done again. Changing to Ongoing and priority 1.

Joined: 09/21/2011
Groups: Void Crushers
Status: Open » Open

Changed it to open so it is visible on the list.

Joined: 04/19/2009

This isn't just windows, one of my mac clients has also crashed:

ERROR: Unhandled exception: Error: database returned invalid result at time 1367107230
ERROR:: Exit from: /Users/folditbuilder/foldit/mini-interactive-main-release/rosetta_source/xcode_4/../src/interactive/application/Thread.cc line: 210
ERROR: Rosetta exit unexpectedly

And, many are reporting crashes. It seems that there were a lot that crashed about the same time (just after 7pm EDT). There have been smaller groups of crashes and individual ones, also.

My crash was in a secondary (not a chat enabled) client.

Joined: 11/05/2010
Groups: None

Nuff said, as far as I can remember none of my crashes of 6-8 clients at a time have been on primary ( chat-enabled ) clients. I can only assume it's because I only very rarely do any processing on a chat-enabled client.

MurloW's picture
User offline. Last seen 1 week 4 days ago. Offline
Joined: 11/21/2012

delta_score: -8.41838
Playing sound: 4
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_user3_csv/994977_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_group3_csv/994977_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm/foldit_ops
Unhandled exception in thread 02963608 (class game::application::novice::BoincThread): couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)

ERROR: Unhandled exception: couldn't resolve host name: Error contacting server.
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly

All 3 clients went down overnight. This is becoming a daily routine.
I use only main especially to avoid these circumstances.
I'm starting to believe karstenw was on to something with his folding hiatus,
I won't keep playing like this.

RicGray's picture
User offline. Last seen 2 years 24 weeks ago. Offline
Joined: 11/15/2012
Groups: ricg test group

Found the issue with the extra debugging information added. A fix for this issue will go out today.

MurloW's picture
User offline. Last seen 1 week 4 days ago. Offline
Joined: 11/21/2012

Wonderful, thank you very much. =)

RicGray's picture
User offline. Last seen 2 years 24 weeks ago. Offline
Joined: 11/15/2012
Groups: ricg test group
Status: Open » Closed

There was an exception that was occurring on the background server communication thread that was not being caught.

MurloW's picture
User offline. Last seen 1 week 4 days ago. Offline
Joined: 11/21/2012
Status: Closed » Open

More Boinc silliness ?
(If I'm mistaken and this is unrelated, apologies, I'll post new feedback.)

I came back to my pc in the morning, had 2 clients still running, WITH server connection, but not IRC.
I tried opening a third client, but it wouldn't open (hang after click on login).
So I forced that client shut, and closed two other clients properly. (Cancel recipe, save and exit, worked fine although they took a while to actually go away.)

Went to look at log.txt and these are the ending lines.:

delta_score: 0
Playing sound: 4
SRVR_THRD error: exiting...Error: database returned invalid result at time 1367485830
(..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)
Autosaved puzzles/0000995058/0000467884/05/autosave.ir_solution.
Autosaved puzzles/0000995058/0000467884/05/autosave-recentbest.ir_solution.
ViewOptionsWidget::refresh()
Option file found. Sending...
closing boinc...
SRVR_THRD starting solution data upload...
SRVR_THRD finished solution data upload. took 10.1071 seconds.
Boinc error: Error: database returned invalid result at time 1367485901 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)
Couldn't quit: Illegal operation for this state (..\..\..\rosetta_source\src\game\util\IRC.cc:465)
done.

There are more errors, starting with this:

Playing sound: 4
SRVR_THRD error: Error: database returned invalid result at time 1367485684 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)
delta_score: 0.161132
Playing sound: 4

then every 100 lines or so:

core.pack.pack_rotamers: IG: 26968 bytes
SRVR_THRD starting solution data upload...
Error contacting server using URL: https://fold.it/comm/user_structure_upload
SRVR_THRD error: failed sending data to the peer: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph

delta_score: 7018.74
Playing sound: 8
SRVR_THRD error: Error: database returned invalid result at time 1367485765 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)
delta_score: 899.122
Playing sound: 7

Also, my log is looking pretty strange of itself, at least I think so.
It's taking up 97MB and I don't think it's supposed to be a solid minimum 90% spaces. As in, the spaces between words, " ".

RicGray's picture
User offline. Last seen 2 years 24 weeks ago. Offline
Joined: 11/15/2012
Groups: ricg test group
Status: Open » Closed

Thurday-Friday our web servers where having and issue due to a client sending bad data.

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons