beshoo
2017-03-14 15:52:31 UTC
I have a long CURL processes.
In past , (5 days before now) Supervisor was able to to run the worker .
Today i note that worker is run for 15 sec then Supervisor restart it ....
here is a part of LONG log :
Please Advice
=============================================================
2017-03-14 11:23:36,752 INFO exited: 01 (terminated by SIGKILL; not
expected)
2017-03-14 11:23:36,754 INFO spawned: '01' with pid 686253
2017-03-14 11:23:36,792 INFO success: 01 entered RUNNING state, process has
stayed up for > than 0 seconds (startsecs)
2017-03-14 11:23:40,757 INFO exited: 11 (terminated by SIGKILL; not
expected)
2017-03-14 11:23:40,760 INFO spawned: '11' with pid 686272
2017-03-14 11:23:40,798 INFO success: 11 entered RUNNING state, process has
stayed up for > than 0 seconds (startsecs)
2017-03-14 11:23:44,762 INFO exited: 01 (terminated by SIGKILL; not
expected)
2017-03-14 11:23:44,766 INFO spawned: '01' with pid 686288
2017-03-14 11:23:44,804 INFO success: 01 entered RUNNING state, process has
stayed up for > than 0 seconds (startsecs)
2017-03-14 11:23:48,768 INFO exited: 11 (terminated by SIGKILL; not
expected)
2017-03-14 11:23:48,770 INFO spawned: '11' with pid 686297
2017-03-14 11:23:48,771 INFO success: 11 entered RUNNING state, process has
stayed up for > than 0 seconds (startsecs)
2017-03-14 11:23:52,773 INFO exited: 01 (terminated by SIGKILL; not
expected)
=============================================================
here is Supervisor conf :
=============================================================
***@server [/output]# cat /etc/supervisord.conf
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
;serverurl = http://localhost:9001
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet
socket
;username=chris ; should be same as http_username if set
;password=123 ; should be same as http_password if set
;prompt=mysupervisor ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history ; use readline history if available
; The below sample program section shows all possible program subsection
values,
; create one or more 'real' program: sections to be able to control them
under
; supervisor.
;[program:theprogramname]
;command=/bin/cat ; the program (relative uses PATH, can take
args)
;process_name=%(program_name)s ; process_name expr (default
%(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;directory=/tmp ; directory to cwd to before exec (def no
cwd)
;umask=022 ; umask for process (default None)
;priority=999 ; the relative start priority (default 999)
;autostart=true ; start at supervisord start (default: true)
;startsecs=1 ; # of secs prog must stay up to be running
(def. 1)
;startretries=3 ; max # of serial start failures when
starting (default 3)
;autorestart=unexpected ; when to restart if exited after running
(def: unexpected)
;exitcodes=0,2 ; 'expected' exit codes used with
autorestart (default 0,2)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default
10)
;stopasgroup=false ; send stop signal to the UNIX process group
(default false)
;killasgroup=false ; SIGKILL the UNIX process group (def false)
;user=chrism ; setuid to this UNIX account to run the
program
;redirect_stderr=true ; redirect proc stderr to stdout (default
false)
;stdout_logfile=/a/path ; stdout log path, NONE for none; default
AUTO
;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default
50MB)
;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
;stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default
0)
;stdout_events_enabled=false ; emit events on stdout writes (default
false)
;stderr_logfile=/a/path ; stderr log path, NONE for none; default
AUTO
;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default
50MB)
;stderr_logfile_backups=10 ; # of stderr logfile backups (default 10)
;stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default
0)
;stderr_events_enabled=false ; emit events on stderr writes (default
false)
;environment=A="1",B="2" ; process environment additions (def no adds)
;serverurl=AUTO ; override serverurl computation (childutils)
[supervisord]
logfile=/output/supervisord.log ; (main log file;default
$CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default
50MB)
logfile_backups=10 ; (num of main logfile rotation
backups;default 10)
loglevel=info ; (log level;default info; others:
debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default
1024)
minprocs=200 ; (min. avail process descriptors;default 200)
;umask=022 ; (process file creation umask;default 022)
user=root ; (default is current user, required if root)
;identifier=supervisor ; (supervisord identifier, default is
'supervisor')
;directory=/tmp ; (default is not to cd during start)
;nocleanup=true ; (don't clean up tempfiles at start;default
false)
;childlogdir=/tmp ; ('AUTO' child log dir, default $TEMP)
;environment=KEY="value" ; (key value pairs to add to environment)
;strip_ansi=false ; (strip ansi escape codes in logs; def. false)
;http_port=/tmp/supervisor.sock;
[unix_http_server]
file = /tmp/supervisor.sock
chmod = 0777
chown= root:root
[rpcinterface:supervisor]
supervisor.rpcinterface_factory =
supervisor.rpcinterface:make_main_rpcinterface
[include]
files = /etc/supervisor/*.conf
=============================================================
And here is the worker conf include file:
[program:gearmanworker]
command=/usr/local/bin/php /home/myinstaranker/www/panel/boot/worker.php
%(process_num)02d
process_name=%(process_num)02d
numprocs=30
startsecs = 0
directory=/output
autostart=true
autorestart=true
user=root
stdout_logfile=/output/%(program_name)s_%(process_num)02d
priority=1
=============================================================
supervisord -v
***@server [/etc/supervisor]# supervisord -v
3.3.1
=============================================================
***@server [/etc/supervisor]# uname -a
Linux server.net 2.6.32-673.26.1.lve1.4.22.el6.x86_64 #1 SMP Fri Feb 24
08:35:09 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
***@server [/etc/supervisor]#
=============================================================
Server Ram
***@server [/etc/supervisor]# free -m
total used free shared buffers cached
Mem: 64242 63849 393 1 321 60491
-/+ buffers/cache: 3035 61207
Swap: 1952 29 1923
***@server [/etc/supervisor]#
=============================================================
Server 24 CUP Info:
processor :0 -> 23
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
stepping : 7
microcode : 1808
cpu MHz : 2000.123
cache size : 15360 KB
physical id : 1
siblings : 12
core id : 5
cpu cores : 6
apicid : 43
initial apicid : 43
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2
ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt
tsc_deadline_timer aes xsave avx lahf_lm ida arat epb pln pts dtherm
tpr_shadow vnmi flexpriority ept vpid xsaveopt
bogomips : 3999.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
In past , (5 days before now) Supervisor was able to to run the worker .
Today i note that worker is run for 15 sec then Supervisor restart it ....
here is a part of LONG log :
Please Advice
=============================================================
2017-03-14 11:23:36,752 INFO exited: 01 (terminated by SIGKILL; not
expected)
2017-03-14 11:23:36,754 INFO spawned: '01' with pid 686253
2017-03-14 11:23:36,792 INFO success: 01 entered RUNNING state, process has
stayed up for > than 0 seconds (startsecs)
2017-03-14 11:23:40,757 INFO exited: 11 (terminated by SIGKILL; not
expected)
2017-03-14 11:23:40,760 INFO spawned: '11' with pid 686272
2017-03-14 11:23:40,798 INFO success: 11 entered RUNNING state, process has
stayed up for > than 0 seconds (startsecs)
2017-03-14 11:23:44,762 INFO exited: 01 (terminated by SIGKILL; not
expected)
2017-03-14 11:23:44,766 INFO spawned: '01' with pid 686288
2017-03-14 11:23:44,804 INFO success: 01 entered RUNNING state, process has
stayed up for > than 0 seconds (startsecs)
2017-03-14 11:23:48,768 INFO exited: 11 (terminated by SIGKILL; not
expected)
2017-03-14 11:23:48,770 INFO spawned: '11' with pid 686297
2017-03-14 11:23:48,771 INFO success: 11 entered RUNNING state, process has
stayed up for > than 0 seconds (startsecs)
2017-03-14 11:23:52,773 INFO exited: 01 (terminated by SIGKILL; not
expected)
=============================================================
here is Supervisor conf :
=============================================================
***@server [/output]# cat /etc/supervisord.conf
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
;serverurl = http://localhost:9001
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet
socket
;username=chris ; should be same as http_username if set
;password=123 ; should be same as http_password if set
;prompt=mysupervisor ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history ; use readline history if available
; The below sample program section shows all possible program subsection
values,
; create one or more 'real' program: sections to be able to control them
under
; supervisor.
;[program:theprogramname]
;command=/bin/cat ; the program (relative uses PATH, can take
args)
;process_name=%(program_name)s ; process_name expr (default
%(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;directory=/tmp ; directory to cwd to before exec (def no
cwd)
;umask=022 ; umask for process (default None)
;priority=999 ; the relative start priority (default 999)
;autostart=true ; start at supervisord start (default: true)
;startsecs=1 ; # of secs prog must stay up to be running
(def. 1)
;startretries=3 ; max # of serial start failures when
starting (default 3)
;autorestart=unexpected ; when to restart if exited after running
(def: unexpected)
;exitcodes=0,2 ; 'expected' exit codes used with
autorestart (default 0,2)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default
10)
;stopasgroup=false ; send stop signal to the UNIX process group
(default false)
;killasgroup=false ; SIGKILL the UNIX process group (def false)
;user=chrism ; setuid to this UNIX account to run the
program
;redirect_stderr=true ; redirect proc stderr to stdout (default
false)
;stdout_logfile=/a/path ; stdout log path, NONE for none; default
AUTO
;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default
50MB)
;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
;stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default
0)
;stdout_events_enabled=false ; emit events on stdout writes (default
false)
;stderr_logfile=/a/path ; stderr log path, NONE for none; default
AUTO
;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default
50MB)
;stderr_logfile_backups=10 ; # of stderr logfile backups (default 10)
;stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default
0)
;stderr_events_enabled=false ; emit events on stderr writes (default
false)
;environment=A="1",B="2" ; process environment additions (def no adds)
;serverurl=AUTO ; override serverurl computation (childutils)
[supervisord]
logfile=/output/supervisord.log ; (main log file;default
$CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default
50MB)
logfile_backups=10 ; (num of main logfile rotation
backups;default 10)
loglevel=info ; (log level;default info; others:
debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default
1024)
minprocs=200 ; (min. avail process descriptors;default 200)
;umask=022 ; (process file creation umask;default 022)
user=root ; (default is current user, required if root)
;identifier=supervisor ; (supervisord identifier, default is
'supervisor')
;directory=/tmp ; (default is not to cd during start)
;nocleanup=true ; (don't clean up tempfiles at start;default
false)
;childlogdir=/tmp ; ('AUTO' child log dir, default $TEMP)
;environment=KEY="value" ; (key value pairs to add to environment)
;strip_ansi=false ; (strip ansi escape codes in logs; def. false)
;http_port=/tmp/supervisor.sock;
[unix_http_server]
file = /tmp/supervisor.sock
chmod = 0777
chown= root:root
[rpcinterface:supervisor]
supervisor.rpcinterface_factory =
supervisor.rpcinterface:make_main_rpcinterface
[include]
files = /etc/supervisor/*.conf
=============================================================
And here is the worker conf include file:
[program:gearmanworker]
command=/usr/local/bin/php /home/myinstaranker/www/panel/boot/worker.php
%(process_num)02d
process_name=%(process_num)02d
numprocs=30
startsecs = 0
directory=/output
autostart=true
autorestart=true
user=root
stdout_logfile=/output/%(program_name)s_%(process_num)02d
priority=1
=============================================================
supervisord -v
***@server [/etc/supervisor]# supervisord -v
3.3.1
=============================================================
***@server [/etc/supervisor]# uname -a
Linux server.net 2.6.32-673.26.1.lve1.4.22.el6.x86_64 #1 SMP Fri Feb 24
08:35:09 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
***@server [/etc/supervisor]#
=============================================================
Server Ram
***@server [/etc/supervisor]# free -m
total used free shared buffers cached
Mem: 64242 63849 393 1 321 60491
-/+ buffers/cache: 3035 61207
Swap: 1952 29 1923
***@server [/etc/supervisor]#
=============================================================
Server 24 CUP Info:
processor :0 -> 23
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
stepping : 7
microcode : 1808
cpu MHz : 2000.123
cache size : 15360 KB
physical id : 1
siblings : 12
core id : 5
cpu cores : 6
apicid : 43
initial apicid : 43
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology
nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2
ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt
tsc_deadline_timer aes xsave avx lahf_lm ida arat epb pln pts dtherm
tpr_shadow vnmi flexpriority ept vpid xsaveopt
bogomips : 3999.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management: