Yet another kind of hang

Case number:845829-995010
Topic:Crash/Hang
Opened by:Timo van der Laan
Status:Closed
Type:Bug
Opened on:Tuesday, April 23, 2013 - 22:06
Last modified:Monday, May 6, 2013 - 21:45

From the log: (windows 7 32 bit)
core.scoring.NeighborList: Minimization stats: 12 score/deriv cals, 0 narrow-from-wide updates, 0 full updates.
delta_score: 0.0653467
Playing sound: 4
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 126 rotamers at 7 positions.
core.pack.pack_rotamers: IG: 22424 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 218 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 66300 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 218 rotamers at 8 positions.
core.pack.pack_rotamers: IG: 75444 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 405 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 267148 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 318 rotamers at 9 positions.
core.pack.pack_rotamers: IG: 158104 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 249 rotamers at 6 positions.
core.pack.pack_rotamers: IG: 83560 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 35 rotamers at 4 positions.
core.pack.pack_rotamers: IG: 4248 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 253 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 100068 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 170 rotamers at 7 positions.
core.pack.pack_rotamers: IG: 46904 bytes
delta_score: 1664.4
Playing sound: 8
Autosaved puzzles/0000994977/0000308854/q5/autosave.ir_solution.
Autosaved puzzles/0000994977/0000308854/q5/autosave-best.ir_solution.
Autosaved puzzles/0000994977/0000308854/q5/autosave-recentbest.ir_solution.
Autosaved puzzles/0000994977/0000308854/q5/autosave-creditbest.ir_solution.

UNHANDLED EXCEPTION
1: library_main +3854501 bytes (no line)
2: library_main +3855718 bytes (no line)
3: library_main +2406953 bytes (no line)
4: library_main +2407254 bytes (no line)
5: library_main +896593 bytes (no line)
6: library_main +1046846 bytes (no line)
7: library_main +1094184 bytes (no line)
8: library_main +1144085 bytes (no line)
9: library_main +1255401 bytes (no line)
10: library_main +3842797 bytes (no line)
11: library_main +4765158 bytes (no line)
12: library_main +4730141 bytes (no line)
13: library_main +4733871 bytes (no line)
14: library_main +17328826 bytes (no line)
15: library_main +17330528 bytes (no line)
16: library_main +18068222 bytes (no line)
17: library_main +18078469 bytes (no line)
18: library_main +17333191 bytes (no line)
19: library_main +17328069 bytes (no line)
20: library_main +4771115 bytes (no line)
21: library_main +4285374 bytes (no line)
22: library_main +4288518 bytes (no line)
23: BaseThreadInitThunk +18 bytes (no line)
24: RtlInitializeExceptionChain +239 bytes (no line)
25: RtlInitializeExceptionChain +194 bytes (no line)

(Tue, 04/23/2013 - 22:06  |  22 comments)


tamirh's picture
User offline. Last seen 6 years 10 weeks ago. Offline
Joined: 05/11/2012

What were you doing at the time?

Joined: 09/21/2011
Groups: Void Crushers

Just running my DRW. Probably inside qstab. Shake (0.1) and Wiggle all (CI 1.0), score computing and using recentbest. Puzzle 701.

Madde's picture
User offline. Last seen 11 weeks 3 days ago. Offline
Joined: 05/29/2008
Groups: Void Crushers

My client crashed last night while running Timo's DRW on puzzle 704.
(Win7 64 - 2013-04-18-54935M-win_x86-devprev)

Last lines of log.txt:

core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 28 rotamers at 3 positions.
core.pack.pack_rotamers: IG: 2724 bytes
delta_score: 58.502
Playing sound: 5
core.fragment.picking_old.vall.eval.IdentityEval: ss = LLL | aa = SSE
core.fragment.picking_old.vall.VallLibrarian: best fragment: score = 0.000369381
core.fragment.picking_old.vall.VallLibrarian: worst fragment: score = 1.00052
Unhandled exception in thread 02E3B0B8 (class game::application::novice::BoincThread): couldn't connect to server: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:98)

ERROR: Unhandled exception: couldn't connect to server: Error contacting server.
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly
CB: 0.00377471 1
TB: 13.5134 11.2665
CB: 0.000925891 1
TB: 11.4368 11.2665
CB: 0.00148015 1
TB: 13.2533 11.2665
CB: 0.119052 1
TB: 21.5009 11.2665
CB: 0.00480801 1
TB: 12.6156 11.2665
CB: 0.00895984 1
TB: 13.6597 11.2665
CB: 0.00423218 1
TB: 12.5033 11.2665
CB: 0.00811043 1
TB: 12.4718 11.2665
CB: 0.000866957 1
TB: 20.8049 11.2665
CB: 0.0834339 1
TB: 20.3214 11.2665
CB: 0.00543372 1
TB: 13.8357 11.2665
CB: 0.000547543 1
TB: 11.7531 11.2665
CB: 0.0158337 1
TB: 17.5193 11.2665
CB: 0.000167563 1
TB: 11.2912 11.2665
CB: 0.00473409 1
TB: 22.2685 11.2665
CB: 0.00121871 1
TB: 14.0446 21.2665
delta_score: -54.8563
Playing sound: 3
delta_score: 0
Playing sound: 4

tamirh's picture
User offline. Last seen 6 years 10 weeks ago. Offline
Joined: 05/11/2012

Assigning for BoincThread exception issue (which may or may not be the only issue).

Hanto's picture
User offline. Last seen 2 days 6 hours ago. Offline
Joined: 05/10/2008
Groups: None

Having similar problem, just took out 6 clients of which only 4 were 704, other two were 701, all devprev win7 x64. Voting up

Madde's picture
User offline. Last seen 11 weeks 3 days ago. Offline
Joined: 05/29/2008
Groups: Void Crushers

Another crash log from me:
(Win7 64 - 2013-04-18-54935M-win_x86-devprev)

SRVR_THRD starting solution data upload...
Autosaved puzzles/0000995004/0000042629/default/autosave.ir_solution.
Autosaved puzzles/0000995004/0000042629/default/autosave-best.ir_solution.
Autosaved puzzles/0000995004/0000042629/default/autosave-recentbest.ir_solution.
Autosaved puzzles/0000995004/0000042629/default/autosave-creditbest.ir_solution.
SRVR_THRD finished solution data upload. took 3.51132 seconds.
Unhandled exception in thread 02F6D800 (class game::application::novice::BoincThread): Error: database returned invalid result at time 1366848145 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:469)

ERROR: Unhandled exception: Error: database returned invalid result at time 1366848145
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly
delta_score: 0.0403983
Playing sound: 4

RicGray's picture
User offline. Last seen 1 year 31 weeks ago. Offline
Joined: 11/15/2012
Groups: ricg test group
Status: Open » Closed

Added more debugging information to the log to have a better idea of what is going on when this occurs.

MurloW's picture
User offline. Last seen 1 day 8 hours ago. Offline
Joined: 11/21/2012

Had two clients running overnight, 701 and 704.
Both crashed leaving behind on the screen only a windows error message each.
Win 7 x64, main latest update.

I did not have an internet connection at the time, but have never had crash issues due to this.
If that is currently the case though, well that'd suck. :/
But I'm hoping not, because I have a lot of these from halfway through log down to the end:

CB: 0.000253392 1
TB: 15.8127 20.2671
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
delta_score: -1797.51
Playing sound: 0
delta_score: 0
Playing sound: 4
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_user3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_group3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph

Bottom lines:

core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 79 rotamers at 8 positions.
core.pack.pack_rotamers: IG: 15928 bytes
Error contacting server using URL: https://fold.it/comm/achievements_report
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_user3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_group3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
SRVR_THRD getting notifications...
Error contacting server using URL: https://fold.it/comm/notification_list
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzles25_main_csv.txt
SRVR_THRD sending play event log...
Error contacting server using URL: https://fold.it/comm/log
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 80 rotamers at 8 positions.
core.pack.pack_rotamers: IG: 17136 bytes
delta_score: 1.34605e-010
Playing sound: 4
delta_score: 524.049
Playing sound: 6
delta_score: 1.14007
Playing sound: 4
Error contacting server using URL: https://fold.it/comm/achievements_report
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_user3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_group3_csv/995004_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
SRVR_THRD getting notifications...
delta_score: 0.0440358
Playing sound: 4
Error contacting server using URL: https://fold.it/comm/notification_list
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzles25_main_csv.txt
Error contacting server using URL: https://fold.it/comm/foldit_ops
Unhandled exception in thread 02893608 (class game::application::novice::BoincThread): couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)

ERROR: Unhandled exception: couldn't resolve host name: Error contacting server.
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly

Joined: 06/04/2012

While running TDRW on one client and Acid Tweaker on the other

MurloW's picture
User offline. Last seen 1 day 8 hours ago. Offline
Joined: 11/21/2012

More.
Tdrw on 706 crashed client,
at the exact same moment several team members' clients went down as well on different puzzles.

core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 274 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 119676 bytes
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph
core.pack.pack_rotamers: built 118 rotamers at 10 positions.
core.pack.pack_rotamers: IG: 29692 bytes
Unhandled exception in thread 02903608 (class game::application::novice::BoincThread): Error: database returned invalid result at time 1367103871 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)

ERROR: Unhandled exception: Error: database returned invalid result at time 1367103871
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly
812 bytes

Joined: 09/21/2011
Groups: Void Crushers
Status: Closed » Open

Reopened because of lots of crash reports.

Susume's picture
User offline. Last seen 1 day 9 hours ago. Offline
Joined: 10/02/2011
Status: Open » Closed

lost one client in past hour, on 701, running my own version of Rav3n_pl GAB v0.6.1
also lost one client at same time as Murlow and several teammates ~2.5 hours ago, 701, TvdL DRW
win 7 64

most recent error in log:

Unhandled exception in thread 02C97E60 (class game::application::novice::BoincThread): Error: database returned invalid result at time 1367103807 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)
ERROR: Unhandled exception: Error: database returned invalid result at time 1367103807
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly

Susume's picture
User offline. Last seen 1 day 9 hours ago. Offline
Joined: 10/02/2011
Status: Closed » Open

Oops, Timo and I crossed and status went to done again. Changing to Ongoing and priority 1.

Joined: 09/21/2011
Groups: Void Crushers
Status: Open » Open

Changed it to open so it is visible on the list.

Joined: 04/19/2009

This isn't just windows, one of my mac clients has also crashed:

ERROR: Unhandled exception: Error: database returned invalid result at time 1367107230
ERROR:: Exit from: /Users/folditbuilder/foldit/mini-interactive-main-release/rosetta_source/xcode_4/../src/interactive/application/Thread.cc line: 210
ERROR: Rosetta exit unexpectedly

And, many are reporting crashes. It seems that there were a lot that crashed about the same time (just after 7pm EDT). There have been smaller groups of crashes and individual ones, also.

My crash was in a secondary (not a chat enabled) client.

Joined: 11/05/2010
Groups: None

Nuff said, as far as I can remember none of my crashes of 6-8 clients at a time have been on primary ( chat-enabled ) clients. I can only assume it's because I only very rarely do any processing on a chat-enabled client.

MurloW's picture
User offline. Last seen 1 day 8 hours ago. Offline
Joined: 11/21/2012

delta_score: -8.41838
Playing sound: 4
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
Error contacting server using URL: https://fold.it/portal/comm/whoami_csv
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_user3_csv/994977_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm_cache/puzzle_group3_csv/994977_diff.txt
SRVR_THRD error: couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
Error contacting server using URL: https://fold.it/comm/foldit_ops
Unhandled exception in thread 02963608 (class game::application::novice::BoincThread): couldn't resolve host name: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)

ERROR: Unhandled exception: couldn't resolve host name: Error contacting server.
ERROR:: Exit from: ..\..\..\rosetta_source\src\interactive\application\Thread.cc line: 210
ERROR: Rosetta exit unexpectedly

All 3 clients went down overnight. This is becoming a daily routine.
I use only main especially to avoid these circumstances.
I'm starting to believe karstenw was on to something with his folding hiatus,
I won't keep playing like this.

RicGray's picture
User offline. Last seen 1 year 31 weeks ago. Offline
Joined: 11/15/2012
Groups: ricg test group

Found the issue with the extra debugging information added. A fix for this issue will go out today.

MurloW's picture
User offline. Last seen 1 day 8 hours ago. Offline
Joined: 11/21/2012

Wonderful, thank you very much. =)

RicGray's picture
User offline. Last seen 1 year 31 weeks ago. Offline
Joined: 11/15/2012
Groups: ricg test group
Status: Open » Closed

There was an exception that was occurring on the background server communication thread that was not being caught.

MurloW's picture
User offline. Last seen 1 day 8 hours ago. Offline
Joined: 11/21/2012
Status: Closed » Open

More Boinc silliness ?
(If I'm mistaken and this is unrelated, apologies, I'll post new feedback.)

I came back to my pc in the morning, had 2 clients still running, WITH server connection, but not IRC.
I tried opening a third client, but it wouldn't open (hang after click on login).
So I forced that client shut, and closed two other clients properly. (Cancel recipe, save and exit, worked fine although they took a while to actually go away.)

Went to look at log.txt and these are the ending lines.:

delta_score: 0
Playing sound: 4
SRVR_THRD error: exiting...Error: database returned invalid result at time 1367485830
(..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)
Autosaved puzzles/0000995058/0000467884/05/autosave.ir_solution.
Autosaved puzzles/0000995058/0000467884/05/autosave-recentbest.ir_solution.
ViewOptionsWidget::refresh()
Option file found. Sending...
closing boinc...
SRVR_THRD starting solution data upload...
SRVR_THRD finished solution data upload. took 10.1071 seconds.
Boinc error: Error: database returned invalid result at time 1367485901 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)
Couldn't quit: Illegal operation for this state (..\..\..\rosetta_source\src\game\util\IRC.cc:465)
done.

There are more errors, starting with this:

Playing sound: 4
SRVR_THRD error: Error: database returned invalid result at time 1367485684 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)
delta_score: 0.161132
Playing sound: 4

then every 100 lines or so:

core.pack.pack_rotamers: IG: 26968 bytes
SRVR_THRD starting solution data upload...
Error contacting server using URL: https://fold.it/comm/user_structure_upload
SRVR_THRD error: failed sending data to the peer: Error contacting server. (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:458)
core.pack.interaction_graph.interaction_graph_factory: Instantiating DensePDInteractionGraph

delta_score: 7018.74
Playing sound: 8
SRVR_THRD error: Error: database returned invalid result at time 1367485765 (..\..\..\rosetta_source\src\game\application\novice\boinc\Boinc.cc:479)
delta_score: 899.122
Playing sound: 7

Also, my log is looking pretty strange of itself, at least I think so.
It's taking up 97MB and I don't think it's supposed to be a solid minimum 90% spaces. As in, the spaces between words, " ".

RicGray's picture
User offline. Last seen 1 year 31 weeks ago. Offline
Joined: 11/15/2012
Groups: ricg test group
Status: Open » Closed

Thurday-Friday our web servers where having and issue due to a client sending bad data.

Sitemap

Developed by: UW Center for Game Science, UW Institute for Protein Design, Northeastern University, Vanderbilt University Meiler Lab, UC Davis
Supported by: DARPA, NSF, NIH, HHMI, Amazon, Microsoft, Adobe, RosettaCommons