35 replies to this topic

[Gennar1]

Today I updated my box and it's not responding any more to ssh:

ssh root@192.168.XXX.XXX
 
ssh: connect to host 192.168.XXX.XXX port 22: Connection refused

The box is responding as usual to telnet:

telnet 192.168.XXX.XXX
 
Trying 192.168.XXX.XXX...
Connected to 192.168.XXX.XXX.
Escape character is '^]'.
OpenEmbedded Linux et9x00
 
openpli 2.1 et9x00
 
et9x00 login: root
Password:

Maybe the problem is related to this update:

http://openpli.git.s...53c1b47e2932b95

Edited by Gennar1, 1 November 2011 - 14:17.

[Gennar1]

The ssh daemon is provided by the "dropbear" package.

There is something wrong in the startup script:

/etc/init.d/dropbear

as the script is no more able to start the server correctly.
dropbear works perfectly when started manually:

/usr/sbin/dropbear -E

and then it's possible to ssh into the box again.
But the startup script don't work anymore, as if you type:

root@et9x00:~# /etc/init.d/dropbear stop
Stopping Dropbear SSH server: stopped /usr/sbin/dropbear (pid 739)
[739] Nov 01 15:21:42 premature exit: Terminated by signal
dropbear.
root@et9x00:~# /etc/init.d/dropbear start
Starting Dropbear SSH server: dropbear.

then again it's not possible to log into the box through ssh.

Edited by Gennar1, 1 November 2011 - 15:23.

[blzr]

here's on VUduo no problems with ssh at all

brt@brt:~$ ssh root@192.168.2.111
root@192.168.2.111's password:
root@vuduo:~#

//and tested after stopping/starting dropbear too

[Gennar1]

The problem seems to be connected with logging to syslog.

If I start dropbear with:

/usr/sbin/dropbear -E

it works OK (-E is an option to send logs to stderr instead of syslog).
But if I start it normally:

/usr/sbin/dropbear

then it does not start! That's why the startup script is not working.
Maybe I've messed up syslog...

[Gennar1]

Problem solved: I had modified /etc/default/busybox-syslog, so when busybox was updated it complained about the modified file so it did not install correctly. As a result, syslog was not autostarted any more and hence dropbear failed to initialize because the syslog device was not existing.

To fix it I had to reflash, because reinstalling busybox killed the box (you can not opkg install remote files because there is no wget, nor local files because another binary is missing).

After reflash with a recent backup, I restored the original /etc/default/busybox-syslog and then I run opkg update again: this time everything was fine and now syslog and dropbear work again.

[Gennar1]

Well, it was not working anyway.
After a reboot, the same problem was still here.

On syslog I can see:

Nov  1 17:36:19 et9x00 authpriv.warn dropbear[518]: Failed reading '/etc/dropbear/dropbear_dss_host_key', disabling DSS
Nov  1 17:36:19 et9x00 authpriv.info dropbear[519]: premature exit: Failed to daemonize: No such device

The first line is expected (we are using rsa and not DSS), but the second line points to the error that prevents dropbear from running. A device i missing and it is "/dev/null"!

Or better, /dev/null is still there but it is replaced by a regular file instead of a device:

root@et9x00:~# ls -al /dev/null
-rw-rw-rw-	1 root	 root			 0 Jan  1  1970 /dev/null
root@et9x00:~# echo test > /dev/null
root@et9x00:~# cat /dev/null
test

To fix the problem just remove the file and replace it with a new device:

root@et9x00:~# rm /dev/null
root@et9x00:~# mknod /dev/null c 1 3

and now everything is fine again:

root@et9x00:~# ls -al /dev/null
crw-r--r--	1 root	 root		1,   3 Nov  1 17:44 /dev/null
root@et9x00:~# echo test > /dev/null
root@et9x00:~# cat /dev/null
 

and in fact dropbear and ssh work again!

Probably the update of busybox replaced erroneously /dev/null with a normal file

Edited by Gennar1, 1 November 2011 - 17:50.

[Gennar1]

Anyway, even after creating the new /dev/null device, after a reboot it is replaced again by a text file and the problem is still there.

What's wrong?

[hemispherical1]

I have the same problem. Copying freshly installed /etc/default/busybox-syslog-opkg over my edited (remote logging) /etc/default/busybox-syslog didn't help. That would "seem" to be the "fix" required, but it didn't help. /usr/sbin/dropbear -E worked as you stated above, as well as /usr/sbin/dropbear -F, but I can't get it started through /etc/init.d/dropbear at all.

I forced reinstall of busybox-syslog, (I don't seem to have any trouble downloading thru opkg & wget works fine), but that didn't help either.

--
hemi

Ps. Man I took too long to post this, I started before the previous 2 posts...

Pps. Ditto on /dev/null being a normal file?

Edited by hemispherical1, 1 November 2011 - 18:07.

[MiLo]

My guess would be that something that writes to /dev/null is started before mdev (mdev arranges for most /dev/ things to exist).

Which is unlikely, mdev starts at S04...

[Gennar1]

This sequence of commands restores the correct device in /dev/null and starts dropbear again (so SSH works fine):

root@et9x00:~# rm /dev/null
root@et9x00:~# mknod /dev/null c 1 3
root@et9x00:~# chmod 666 /dev/null
root@et9x00:~# ls -al /dev/null
crw-rw-rw-    1 root     root       1,   3 Jan  1  1970 /dev/null
root@et9x00:~# /etc/init.d/dropbear  start
Starting Dropbear SSH server: dropbear.

But after very reboot /dev/null is replaced again with the plain file, and dropbear stops working.
Having a normal file in place of /dev/null may cause big problems, as the file will grow indefinitely and will take all the free flash memory!

So this is going to be a big issue. We better fix it soon.

[Gennar1]

My guess would be that something that writes to /dev/null is started before mdev (mdev arranges for most /dev/ things to exist).

Which is unlikely, mdev starts at S04...

root@et9x00:~# ls -l /etc/rcS.d/
lrwxrwxrwx    1 root	 root		    16 Oct 19 02:02 S02banner -> ../init.d/banner
lrwxrwxrwx    1 root	 root		    18 Oct 19 02:02 S03sysfs -> ../init.d/sysfs.sh
lrwxrwxrwx    1 root	 root		    14 Oct 19 02:02 S04mdev -> ../init.d/mdev
lrwxrwxrwx    1 root	 root		    17 Oct 19 02:02 S05devices -> ../init.d/devices
lrwxrwxrwx    1 root	 root		    19 Oct 19 02:02 S10checkroot -> ../init.d/checkroot
lrwxrwxrwx    1 root	 root		    21 Oct 19 02:02 S20modutils.sh -> ../init.d/modutils.sh
lrwxrwxrwx    1 root	 root		    18 Oct 19 02:02 S21bootlogo -> ../init.d/bootlogo
lrwxrwxrwx    1 root	 root		    18 Oct 19 02:02 S22fpupdate -> ../init.d/fpupdate
lrwxrwxrwx    1 root	 root		    17 Oct 19 02:02 S30ramdisk -> ../init.d/ramdisk
lrwxrwxrwx    1 root	 root		    21 Oct 19 02:02 S35mountall.sh -> ../init.d/mountall.sh
lrwxrwxrwx    1 root	 root		    30 Oct 19 02:02 S37populate-volatile.sh -> ../init.d/populate-volatile.sh
lrwxrwxrwx    1 root	 root		    19 Oct 19 02:02 S38devpts.sh -> ../init.d/devpts.sh
lrwxrwxrwx    1 root	 root		    20 Oct 19 02:02 S39alsa-state -> ../init.d/alsa-state
lrwxrwxrwx    1 root	 root		    21 Oct 19 02:02 S39hostname.sh -> ../init.d/hostname.sh
lrwxrwxrwx    1 root	 root		    20 Oct 19 02:02 S41networking -> ../init.d/networking
lrwxrwxrwx    1 root	 root		    17 Oct 19 02:02 S43portmap -> ../init.d/portmap
lrwxrwxrwx    1 root	 root		    21 Oct 19 02:02 S45mountnfs.sh -> ../init.d/mountnfs.sh
lrwxrwxrwx    1 root	 root		    21 Oct 19 02:02 S55bootmisc.sh -> ../init.d/bootmisc.sh
lrwxrwxrwx    1 root	 root		    19 Oct 19 02:02 S99finish.sh -> ../init.d/finish.sh

[pieterg]

Probably my fault, in the mdev-mount.sh script I piped the 'mount' output to /dev/null. I must have made a typo there I guess.
That commit wasn't doing anything yet till we bumped busybox PR yesterday.

[MiLo]

But after very reboot /dev/null is replaced again with the plain file, and dropbear stops working.
Having a normal file in place of /dev/null may cause big problems, as the file will grow indefinitely and will take all the free flash memory!

Luckily /dev/ is in RAM (that's why your changes disappear at boot) and limited to only 64k, so it fills up really quick and won't kill your flash or anything. Which is still very bad, because some things need to write there...

[Gennar1]

Probably my fault, in the mdev-mount.sh script I piped the 'mount' output to /dev/null. I must have made a typo there I guess.
That commit wasn't doing anything yet till we bumped busybox PR yesterday.

Ok, so it's init.d/sysfs.sh that is calling mount at S03 before mdev is started.

[pieterg]

are you sure it's not just /etc/mdev/mdev-mount.sh instead?

[Gennar1]

But after very reboot /dev/null is replaced again with the plain file, and dropbear stops working.
Having a normal file in place of /dev/null may cause big problems, as the file will grow indefinitely and will take all the free flash memory!

Luckily /dev/ is in RAM (that's why your changes disappear at boot) and limited to only 64k, so it fills up really quick and won't kill your flash or anything. Which is still very bad, because some things need to write there...

I see, thanks.

[Gennar1]

are you sure it's not just /etc/mdev/mdev-mount.sh instead?

You must be right, but when is called mdev-mount.sh?

[MiLo]

sysfs.sh just calls "mount" directly, not the mdev wrapper.

[MiLo]

mdev-mount.sh gets called for all /dev/sdXX devices, and seems to beat /dev/null to the punch.

[theparasol]

Updated 5 mins ago, same issue here on my not modified dm800hd pvr.
Going to use telnet for the time being