梦流星
2017-08-06 10:29:31 UTC
Will that result in a supervisorctl crash ?? what should I do before I send a startProcess command.
I have met the supervisord crash problem, here my app log and supervisor log
I wonder if I misunderstand some concepts or using tips of the supervisor?
What should I do to investigate into this problem?
My supervisor log:
2017-08-04 14:50:57,345 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 74580 2017-08-04 14:50:57,566 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:50:58,569 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 74608 2017-08-04 14:50:58,612 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:50:59,867 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 74638 2017-08-04 14:51:00,078 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:51:02,177 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 75438 2017-08-04 14:51:02,221 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:51:03,224 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 75508 2017-08-04 14:51:03,224 INFO gave up: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d entered FATAL state, too many start retries too quickly 2017-08-04 14:51:03,458 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:51:04,459 INFO gave up: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 entered FATAL state, too many start retries too quickly 2017-08-04 14:51:05,316 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 75655 2017-08-04 14:51:05,363 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:51:06,145 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 75724 2017-08-04 14:51:06,166 INFO stopped: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (terminated by SIGTERM) 2017-08-04 14:51:07,170 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 75784 2017-08-04 14:51:07,216 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:51:09,220 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 75837 2017-08-04 14:51:09,269 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:51:11,210 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 75902 2017-08-04 14:51:11,428 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:51:12,431 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 75924 2017-08-04 14:51:12,433 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 75925 2017-08-04 14:51:12,480 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:51:12,487 INFO gave up: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d entered FATAL state, too many start retries too quickly 2017-08-04 14:51:12,647 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:51:14,900 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 75974 2017-08-04 14:51:15,105 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:51:17,199 INFO success: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
My app log: E0804 14:51:00.298438 38880 process_manager.go:321] error: "ALREADY_STARTED: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d" code: 60 W0804 14:51:00.298461 38880 matrix_instance.go:360] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process W0804 14:51:00.298478 38880 matrix_instance.go:291] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process I0804 14:51:00.310288 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:01.074691 38880 process_manager.go:321] error: "ALREADY_STARTED: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2" code: 60 W0804 14:51:01.074759 38880 matrix_instance.go:360] task c1f7b4725d44f0e3de1437e3df7e7da2 start error: can not start process W0804 14:51:01.074778 38880 matrix_instance.go:291] task c1f7b4725d44f0e3de1437e3df7e7da2 start error: can not start process E0804 14:51:01.082880 38880 process_manager.go:331] error: "FAILED: attempted to kill mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d with sig SIGTERM but it wasn't running" code: 30 E0804 14:51:01.084197 38880 process_manager.go:336] error: "STILL_RUNNING" code: 91 E0804 14:51:01.172538 38880 process_manager.go:331] error: "FAILED: attempted to kill vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 with sig SIGTERM but it wasn't running" code: 30 E0804 14:51:01.173965 38880 process_manager.go:336] error: "STILL_RUNNING" code: 91 I0804 14:51:04.492483 38880 matrix_instance.go:355] matrix instance try to start process I0804 14:51:05.330212 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:06.159047 38880 process_manager.go:331] error: "FAILED: attempted to kill mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d with sig SIGTERM but it wasn't running" code: 30 E0804 14:51:06.160440 38880 process_manager.go:336] error: "STILL_RUNNING" code: 91 I0804 14:51:09.608146 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:10.408757 38880 process_manager.go:321] error: "ALREADY_STARTED: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d" code: 60 W0804 14:51:10.408779 38880 matrix_instance.go:360] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process W0804 14:51:10.408799 38880 matrix_instance.go:291] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process I0804 14:51:10.420027 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:11.223602 38880 process_manager.go:331] error: "FAILED: attempted to kill mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d with sig SIGTERM but it wasn't running" code: 30 E0804 14:51:11.225005 38880 process_manager.go:336] error: "STILL_RUNNING" code: 91 I0804 14:51:14.492659 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:15.393642 38880 process_manager.go:321] request error: bad status code - 500 W0804 14:51:15.393663 38880 matrix_instance.go:360] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process W0804 14:51:15.393681 38880 matrix_instance.go:291] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process I0804 14:51:15.407221 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:16.185026 38880 process_manager.go:321] connection is shut down W0804 14:51:16.185061 38880 matrix_instance.go:360] task c1f7b4725d44f0e3de1437e3df7e7da2 start error: can not start process W0804 14:51:16.185084 38880 matrix_instance.go:291] task c1f7b4725d44f0e3de1437e3df7e7da2 start error: can not start process E0804 14:51:16.193922 38880 process_manager.go:331] connection is shut down E0804 14:51:16.193967 38880 process_manager.go:336] connection is shut down E0804 14:51:16.198475 38880 process_manager.go:331] connection is shut down E0804 14:51:16.198492 38880 process_manager.go:336] connection is shut down
I have met the supervisord crash problem, here my app log and supervisor log
I wonder if I misunderstand some concepts or using tips of the supervisor?
What should I do to investigate into this problem?
My supervisor log:
2017-08-04 14:50:57,345 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 74580 2017-08-04 14:50:57,566 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:50:58,569 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 74608 2017-08-04 14:50:58,612 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:50:59,867 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 74638 2017-08-04 14:51:00,078 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:51:02,177 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 75438 2017-08-04 14:51:02,221 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:51:03,224 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 75508 2017-08-04 14:51:03,224 INFO gave up: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d entered FATAL state, too many start retries too quickly 2017-08-04 14:51:03,458 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:51:04,459 INFO gave up: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 entered FATAL state, too many start retries too quickly 2017-08-04 14:51:05,316 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 75655 2017-08-04 14:51:05,363 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:51:06,145 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 75724 2017-08-04 14:51:06,166 INFO stopped: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (terminated by SIGTERM) 2017-08-04 14:51:07,170 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 75784 2017-08-04 14:51:07,216 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:51:09,220 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 75837 2017-08-04 14:51:09,269 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:51:11,210 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 75902 2017-08-04 14:51:11,428 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:51:12,431 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 75924 2017-08-04 14:51:12,433 INFO spawned: 'mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d' with pid 75925 2017-08-04 14:51:12,480 INFO exited: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d (exit status 1; not expected) 2017-08-04 14:51:12,487 INFO gave up: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d entered FATAL state, too many start retries too quickly 2017-08-04 14:51:12,647 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:51:14,900 INFO spawned: 'vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2' with pid 75974 2017-08-04 14:51:15,105 INFO exited: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 (exit status 134; not expected) 2017-08-04 14:51:17,199 INFO success: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
My app log: E0804 14:51:00.298438 38880 process_manager.go:321] error: "ALREADY_STARTED: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d" code: 60 W0804 14:51:00.298461 38880 matrix_instance.go:360] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process W0804 14:51:00.298478 38880 matrix_instance.go:291] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process I0804 14:51:00.310288 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:01.074691 38880 process_manager.go:321] error: "ALREADY_STARTED: vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2" code: 60 W0804 14:51:01.074759 38880 matrix_instance.go:360] task c1f7b4725d44f0e3de1437e3df7e7da2 start error: can not start process W0804 14:51:01.074778 38880 matrix_instance.go:291] task c1f7b4725d44f0e3de1437e3df7e7da2 start error: can not start process E0804 14:51:01.082880 38880 process_manager.go:331] error: "FAILED: attempted to kill mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d with sig SIGTERM but it wasn't running" code: 30 E0804 14:51:01.084197 38880 process_manager.go:336] error: "STILL_RUNNING" code: 91 E0804 14:51:01.172538 38880 process_manager.go:331] error: "FAILED: attempted to kill vsd-209f9d4165b98922dc6c35666ae04e5c-c1f7b4725d44f0e3de1437e3df7e7da2 with sig SIGTERM but it wasn't running" code: 30 E0804 14:51:01.173965 38880 process_manager.go:336] error: "STILL_RUNNING" code: 91 I0804 14:51:04.492483 38880 matrix_instance.go:355] matrix instance try to start process I0804 14:51:05.330212 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:06.159047 38880 process_manager.go:331] error: "FAILED: attempted to kill mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d with sig SIGTERM but it wasn't running" code: 30 E0804 14:51:06.160440 38880 process_manager.go:336] error: "STILL_RUNNING" code: 91 I0804 14:51:09.608146 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:10.408757 38880 process_manager.go:321] error: "ALREADY_STARTED: mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d" code: 60 W0804 14:51:10.408779 38880 matrix_instance.go:360] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process W0804 14:51:10.408799 38880 matrix_instance.go:291] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process I0804 14:51:10.420027 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:11.223602 38880 process_manager.go:331] error: "FAILED: attempted to kill mock-209f9d4165b98922dc6c35666ae04e5c-9d95038ae0714aa103ac456d11c6cb0d with sig SIGTERM but it wasn't running" code: 30 E0804 14:51:11.225005 38880 process_manager.go:336] error: "STILL_RUNNING" code: 91 I0804 14:51:14.492659 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:15.393642 38880 process_manager.go:321] request error: bad status code - 500 W0804 14:51:15.393663 38880 matrix_instance.go:360] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process W0804 14:51:15.393681 38880 matrix_instance.go:291] task 9d95038ae0714aa103ac456d11c6cb0d start error: can not start process I0804 14:51:15.407221 38880 matrix_instance.go:355] matrix instance try to start process E0804 14:51:16.185026 38880 process_manager.go:321] connection is shut down W0804 14:51:16.185061 38880 matrix_instance.go:360] task c1f7b4725d44f0e3de1437e3df7e7da2 start error: can not start process W0804 14:51:16.185084 38880 matrix_instance.go:291] task c1f7b4725d44f0e3de1437e3df7e7da2 start error: can not start process E0804 14:51:16.193922 38880 process_manager.go:331] connection is shut down E0804 14:51:16.193967 38880 process_manager.go:336] connection is shut down E0804 14:51:16.198475 38880 process_manager.go:331] connection is shut down E0804 14:51:16.198492 38880 process_manager.go:336] connection is shut down