リモートサイトで複数のデバイスを実行しています。最近1年以上にわたって展開されたサーバーのうち、2台がSSH接続を拒否し始めました。ソフトウェアエージェントへの低特権のSSH接続を取得し、追加情報を収集しました。これらのデバイス構成は変更されておらず、他のノードは非常に似た設定でまだうまく動作します。
これは、接続しようとしたときにsshdから取得したデバッグログです。
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: inetd sockets after dupping: 3, 4
Dec 29 01:28:33 intel-corei7-64 auth.info sshd[6258]: Connection from xxx.yyy.20.1 port 34310 on xxx.yyy.100.1 port 22
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: Client protocol version 2.0; client software version OpenSSH_8.8
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: match: OpenSSH_8.8 pat OpenSSH* compat 0x04000000
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: Local version string SSH-2.0-OpenSSH_7.9
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: fd 3 setting O_NONBLOCK
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: Network child is on pid 6259
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: permanently_set_uid: 996/994 [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: SSH2_MSG_KEXINIT received [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: local server KEXINIT proposal [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha51
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: compression ctos: none [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: compression stoc: none [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: languages ctos: [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: languages stoc: [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: first_kex_follows 0 [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: reserved 0 [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: peer client KEXINIT proposal [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha51
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],sk
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected] [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: compression ctos: none,[email protected],zlib [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: compression stoc: none,[email protected],zlib [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: languages ctos: [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: languages stoc: [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: first_kex_follows 0 [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: reserved 0 [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: kex: algorithm: curve25519-sha256 [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: kex: host key algorithm: ssh-ed25519 [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: monitor_read: 6 used once, disabling now
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug2: set_newkeys: mode 1 [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: rekey after 134217728 blocks [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Dec 29 01:28:33 intel-corei7-64 auth.debug sshd[6258]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug2: set_newkeys: mode 0 [preauth]
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug1: rekey after 134217728 blocks [preauth]
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug1: KEX done [preauth]
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug1: userauth-request for user connectorio service ssh-connection method none [preauth]
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug1: attempt 0 failures 0 [preauth]
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug2: parse_server_config: config reprocess config len 381
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug2: monitor_read: 8 used once, disabling now
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug2: input_userauth_request: setting up authctxt for connectorio [preauth]
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug2: input_userauth_request: try method none [preauth]
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug2: monitor_read: 4 used once, disabling now
Dec 29 01:28:34 intel-corei7-64 auth.info sshd[6258]: Failed none for remoteuser from xxx.yyy.20.1 port 34310 ssh2
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug1: monitor_read_log: child log fd closed
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug1: do_cleanup
Dec 29 01:28:34 intel-corei7-64 auth.debug sshd[6258]: debug1: Killing privsep child 6259
私が理解していないことはFailed none for remoteuser何も言わないということです。このメッセージの意味がわかりません。これは失敗ですが、そのシステムにロードされた公開鍵はまだ有効です。
クライアント側ではログイン試行に使用されるキーがほとんどないようです。
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/zzz/.ssh/id_rsa2 RSA SHA256:... explicit agent
debug1: Will attempt key: /home/zzz/.ssh/id_rsa RSA SHA256:... agent
debug1: Will attempt key: zzz@arch RSA SHA256:... agent
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
Connection closed by xxx.yyy.100.1 port 22
ちなみに非常にsshd_config自由奔放な文も載せました。
# $OpenBSD: sshd_config,v 1.102 2018/02/16 02:32:40 djm Exp $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
Port 24
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
# Ciphers and keying
#RekeyLimit default none
# Logging
SyslogFacility AUTH
LogLevel DEBUG2
# Authentication:
#LoginGraceTime 2m
PermitRootLogin yes
#StrictModes yes
MaxAuthTries 3
#MaxSessions 10
#PubkeyAuthentication yes
# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile .ssh/authorized_keys
#AuthorizedPrincipalsFile none
#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody
# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
PermitEmptyPasswords yes
# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication yes
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM no
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#PermitUserEnvironment no
Compression no
ClientAliveInterval 15
ClientAliveCountMax 4
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none
# no default banner path
#Banner none
# override default of no subsystems
Subsystem sftp /usr/libexec/sftp-server
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server
一部の回答では、リソースの制限によりsshデーモンの接続が失われる可能性があることを示唆していますが、私のfree出力はあまり心配しません。
total used free shared buff/cache available
Mem: 3878132 973044 2371328 11600 533760 2832068
Swap: 45052 0 45052
私はアイデアがまったくないので、この問題を解決することはできません。これまでrundeckを実行し、高い特権でいくつかのコマンドを実行できますが、デバイスの更新を完了するためにSSH接続を再び機能させることはできません。どんな提案でも歓迎します！