swarm でnginxのコンテナーを二台作って運用してますが、
たまにコンテナーから
curl http://localhost
すると接続できるのに
EC2のホストインスタンスでcurl http://localhost
するとtimeoutになる事象があります。
docker stack rm -> docker stack deploy を繰り返すと解消します。
おそらくswarmのnetwork周りでの不具合のような気がしますが、トラブルシューティングの方法がよくわからなず、こちらに質問させていただきます。
たまに接続できなかった時の/var/log/syslogを見ると
以下のようなエラーが出てましたが、事象が起きない時にも出てたので関係あるかよくわかりません。
Mar 27 15:10:10 ip-10-1-4-105 dd.collector[1876]: WARNING (init.py:717): No matching process 'ntpd' was found
Mar 27 15:10:10 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /run/docker/netns/ingress_sbox: [Errno 13] Permission denied: '/run/docker/netns/ingress_sbox'
Mar 27 15:10:10 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /run/docker/netns/1-w3xyvpluf9: [Errno 13] Permission denied: '/run/docker/netns/1-w3xyvpluf9'
Mar 27 15:10:10 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /var/lib/docker/overlay2/caf2ef51c59d0ea8781c67d31109f7dcdec16791344ec32038047f9bd7a79f15/merged: [Errno 13] Permission denied: '/var/lib/docker/overlay2/caf2ef51c59d0ea8781c67d31109f7dcdec16791344ec32038047f9bd7a79f15/merged'
Mar 27 15:10:10 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /run/docker/netns/1-v16ais59zb: [Errno 13] Permission denied: '/run/docker/netns/1-v16ais59zb'
Mar 27 15:10:10 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /var/lib/docker/containers/2f103e89891a4b4be08985bba11475867e0fb5f0971269721b9f917ee8ab5d48/shm: [Errno 13] Permission denied: '/var/lib/docker/containers/2f103e89891a4b4be08985bba11475867e0fb5f0971269721b9f917ee8ab5d48/shm'
Mar 27 15:10:10 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /run/docker/netns/71c7583d8dc7: [Errno 13] Permission denied: '/run/docker/netns/71c7583d8dc7'
Mar 27 15:10:10 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /var/lib/docker/overlay2/68999c875435d3f6dd1e3e147b9728df739ba506cd34d8330becc72170c8d2d6/merged: [Errno 13] Permission denied: '/var/lib/docker/overlay2/68999c875435d3f6dd1e3e147b9728df739ba506cd34d8330becc72170c8d2d6/merged'
Mar 27 15:10:10 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /var/lib/docker/containers/7dc245b5eaa85caed8f914d897b45723bcb1bdc79a30ddadc246e22ef6e59742/shm: [Errno 13] Permission denied: '/var/lib/docker/containers/7dc245b5eaa85caed8f914d897b45723bcb1bdc79a30ddadc246e22ef6e59742/shm'
Mar 27 15:10:11 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /run/docker/netns/3af015b84761: [Errno 13] Permission denied: '/run/docker/netns/3af015b84761'
Mar 27 15:10:11 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /var/lib/docker/overlay2/2357cfa2312b6942643035b1d8146a9cc1978f40b129c7e5e29baab65c377821/merged: [Errno 13] Permission denied: '/var/lib/docker/overlay2/2357cfa2312b6942643035b1d8146a9cc1978f40b129c7e5e29baab65c377821/merged'
Mar 27 15:10:11 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /var/lib/docker/containers/ef1b991403048d7f35e4df51490a2a05dc2de18128192e9faa5e685be134e2c4/shm: [Errno 13] Permission denied: '/var/lib/docker/containers/ef1b991403048d7f35e4df51490a2a05dc2de18128192e9faa5e685be134e2c4/shm'
Mar 27 15:10:11 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /run/docker/netns/fe76a5442ab5: [Errno 13] Permission denied: '/run/docker/netns/fe76a5442ab5'
Mar 27 15:10:11 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /var/lib/docker/overlay2/6980e182377d27bf0250824edce1c5cb03fb47aebc0e7a755fa0dcd93f67779d/merged: [Errno 13] Permission denied: '/var/lib/docker/overlay2/6980e182377d27bf0250824edce1c5cb03fb47aebc0e7a755fa0dcd93f67779d/merged'
Mar 27 15:10:11 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /var/lib/docker/containers/da7d3d83e8a82d8ec62fd311938075417e6416964759c93bf0eb5dfddf035e7f/shm: [Errno 13] Permission denied: '/var/lib/docker/containers/da7d3d83e8a82d8ec62fd311938075417e6416964759c93bf0eb5dfddf035e7f/shm'
Mar 27 15:10:11 ip-10-1-4-105 dd.collector[1876]: WARNING (disk.py:104): Unable to get disk metrics for /run/docker/netns/853da57c9287: [Errno 13] Permission denied: '/run/docker/netns/853da57c9287'
Mar 27 15:10:11 ip-10-1-4-105 dd.collector[1876]: CRITICAL (docker_daemon.py:274): Docker client initialization failed permanently. Docker-related features will fail.
Mar 27 15:10:11 ip-10-1-4-105 dd.collector[1876]: WARNING (init.py:717): Initialization failed. Will retry at next iteration
Mar 27 15:10:14 ip-10-1-4-105 dockerd[1245]: time="2018-03-27T15:10:14.202428684+09:00" level=info msg="Node join event for ip-10-1-4-207-f123a5016d97/10.1.4.207"
何か参考できる情報がありましたら教えていただけますか。
あなたの回答
tips
プレビュー