random stuck version 0.35.0

[dshleg]

• I have read and understood docs/Issues.md.

Describe the bug
Container after start stuck at some random time until restart

To Reproduce
i can reproduce (( after downgrade to 0.34.3 configuration correctly corrected

Expected behavior
A clear and concise description of what you expected to happen.

Configuration

model: ios
model: junos
model: routeros
resolve_dns: true
interval: 3600
use_syslog: false
debug: true
run_once: false
threads: 30
use_max_threads: false
timeout: 20
retries: 3
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/
next_adds_job: false
vars:
  auth_methods: [ "publickey" ]
  ssh_keys: "/home/oxidized/.ssh/ed25519"
  # log: /home/oxidized/.config/oxidized/oxy.log
  remove_secret: false
groups:
  cisco_group:
    username: rcfg
    password: <hidden>
    model: cisco
    vars:
      #enable: enable_secret
      ios_rbac: true
      auth_methods: [ "password", "keyboard-interactive" ]
      #ssh_keys: ["/home/oxidized/.ssh/dsh.ed25519","/home/oxidized/.ssh/ed25519"]
      #ssh_proxy: bastion-nord
  cisco_enable:
    username: rcfg
    password: <hidden>
    model: cisco
    vars:
      enable: <hidden>
      ios_rbac: true
      auth_methods: [ "password", "keyboard-interactive" ]
      #ssh_keys: ["/home/oxidized/.ssh/dsh.ed25519","/home/oxidized/.ssh/ed25519"]
      #ssh_proxy: bastion-nord
  junos_group:
    username: rcfg
    model: junos
    vars:
      auth_methods: [ "publickey" ]
      remove_secret: false
  junos_old_group:
    username: rcfg
    model: junos
    vars:
      auth_methods: [ "publickey" ]
      ssh_keys: ["/home/oxidized/.ssh/rsa2k"]
      remove_secret: false
  routeros_group:
    username: rcfg
    model: routeros
    vars:
      auth_methods: [ "publickey" ]
      ssh_keys: "/home/oxidized/.ssh/rsa2k"
  vyos_group:
    username: rcfg
    model: vyos
    vars:
     auth_methods: [ "publickey" ]
     #ssh_keys: "/home/oxidized/.ssh/ed25519"

group_map: {}
models: {}
pid: "/home/oxidized/.config/oxidized/pid"
extensions:
  oxidized-web:
    load: true
    listen: 0.0.0.0
    port: 8888
    #url_prefix: prod

crash:
  directory: "/home/oxidized/.config/oxidized/crashes"
  hostnames: false
stats:
  history_size: 10
input:
  default: ssh
  debug: false
  ssh:
    secure: false
  ftp:
    passive: true
  utf8_encoded: true
output:
  default: git
  file:
    directory: "/home/oxidized/.config/oxidized/configs"
  git:
    single_repo: true
    user: oxidized
    email: depit@<hidden>.com
    repo: "/home/oxidized/.config/oxidized/configs.git"
hooks:
  push_to_remote:
    type: githubrepo
    events: [post_store]
    remote_repo: "git@gitlab.com:<hidden>/network-backup.git"
    #branch: master
    privatekey: "/home/oxidized/.ssh/gitlab"
    publickey: "/home/oxidized/.ssh/gitlab.pub"
source:
  default: csv
  csv:
    file: "/home/oxidized/.config/oxidized/router.db"
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      ip: 1
      model: 2
      group: 3
    vars_map:
      ssh_port: 4
    gpg: false
model_map:
  juniper: junos
  cisco: ios
  mikrotik: routeros
  vyos: vyos

Logs

logs without error, only stuck at random host


/usr/lib/ruby/3.3.0/openssl/pkey.rb:262:in `rescue in dsa_sign_asn1'
/usr/lib/ruby/3.3.0/openssl/pkey.rb:259:in `dsa_sign_asn1'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/transport/openssl.rb:246:in `ssh_do_sign'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/key_manager.rb:185:in `sign'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/methods/publickey.rb:60:in `authenticate_with_alg'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/methods/publickey.rb:126:in `authenticate_with'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/methods/publickey.rb:19:in `block in authenticate'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/key_manager.rb:142:in `block in each_identity'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/key_manager.rb:139:in `each'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/key_manager.rb:139:in `each_identity'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/methods/publickey.rb:18:in `authenticate'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/session.rb:88:in `block in authenticate'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/session.rb:72:in `each'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh/authentication/session.rb:72:in `authenticate'
/usr/share/rubygems-integration/all/gems/net-ssh-7.3.0/lib/net/ssh.rb:262:in `start'
/var/lib/gems/3.3.0/gems/oxidized-0.35.0/lib/oxidized/input/ssh.rb:32:in `connect'
/var/lib/gems/3.3.0/gems/oxidized-0.35.0/lib/oxidized/node.rb:76:in `run_input'
/var/lib/gems/3.3.0/gems/oxidized-0.35.0/lib/oxidized/node.rb:51:in `block in run'
/var/lib/gems/3.3.0/gems/oxidized-0.35.0/lib/oxidized/node.rb:45:in `each'
/var/lib/gems/3.3.0/gems/oxidized-0.35.0/lib/oxidized/node.rb:45:in `run'
/var/lib/gems/3.3.0/gems/oxidized-0.35.0/lib/oxidized/job.rb:15:in `block (2 levels) in initialize'
/usr/lib/ruby/3.3.0/timeout.rb:186:in `block in timeout'
/usr/lib/ruby/3.3.0/timeout.rb:41:in `handle_timeout'
/usr/lib/ruby/3.3.0/timeout.rb:195:in `timeout'
/var/lib/gems/3.3.0/gems/oxidized-0.35.0/lib/oxidized/job.rb:14:in `block in initialize'

Running environment (please complete the following information):

• OS: Linux NetBackup 6.1.0-42-amd64]
• oxidized version: [e.g. 0.35.0]
• Docker version 29.2.1, build a5c7197

[ytti]

Uff. This might be tricksy to solve without access.

I don't really understand why Oxidized version would do this, unless OpenSSL version changed too.

Do you have any DSA key in .ssh directory of Oxidized user or in ssh-agent? If so, could you try after removing those?

This really shouldn't be random and it really shouldn't depend on Oxidized, but it could depend one OpenSSL being too new for DSA and net-ssh not handling that.

[dshleg]

I understand that this is very difficult to track.
Can I enable additional debugging?

[robertcheramy]

Can I enable additional debugging?

Yes you can 😃 .

1. Find the process-id of oxidized (ps -aux | grep oxidized, this works outside the container).
2. Send a SIGTTIN to the process id (sudo kill -SIGTTIN <PID>) to dump where it hangs
3. Paste the results from the logs/output of the container here