5

After configuring a new Raspberry Pi Zero W, I have not been able to ssh into it over WiFi. The IP address is assigned and pingable, but I get 'connection refused' errors.

After some research, I discovered one possible solution is to replace the SSH host keys on the pi.

But it's a Catch-22, because I can't login to the pi to change the keys that I need to change to be able to log into the pi.

What can I do?

Why do I need to replace host keys?

Short answer: I don't know. I don't even know if that's the only or best solution or just one workaround. Host Keys are configured when ssh is configured. Did that configuration not run, or did it fail for some reason? One thing I did on both Pi's was added a dtoverlay=dwc2 line in config.txt to enable ethernet over USB. Could that have thwarted the boot/initialization sequence? Will run some experiments and post info if I learn anything new.

Configuration details: The way I configured my Pi's are as follows: Using Etcher.app on macOS, I flashed an SSD card with the full desktop version of Raspian, 2018-11-13-raspbian-stretch-full.img, downloaded from the Raspian site, and installed it on a Pi Zero W and a Pi 3B+, both headless, and used the headless approach to configuring WiFi and ssh (i.e. touched ssh in /boot and created wpa_supplicant.conf in /boot).

I didn't have this problem when I similarly installed Raspian lite (on my Pi 3B, about a month ago, 2018-10-09-raspbian-stretch-lite.img

Follow-up to @Milliway's comment below: This is the Reddit thread containing post that gave me the idea that regenerating host keys might fix the problem:

https://www.reddit.com/r/raspberry_pi/comments/704v5y/ssh_connection_refused_off_a_headless_setup/

clearlight
  • 259
  • 2
  • 9
  • Whilst the choice of duplicate was not the best I could have chosen I find that entering ssh connection refused into the search box throws up 168 matching questions - some of which even have accepted answers, that strikes me that The answer is out there, Neo, and it's looking for you, and it will find you if you want it to. a little research may have saved some effort all around. – SlySven Dec 21 '18 at 21:00
  • 3
    I don't think this is a duplicate, but perhaps this could be edited to clarify along the lines of the deeper question, "How can I fix the Pi's SSH keys if I suspect they are broken on a headless Pi?" so that it is no longer mistaken with the suggested dupes? – Aurora0001 Dec 21 '18 at 21:30
  • @Aurora0001 Now that is a great suggestion! – clearlight Dec 21 '18 at 21:34
  • 1
    @clearlight It'll automatically be sent to the review queues after editing (and hopefully reopened), but if nothing seems to be happening you can take it to meta after a day or two and get some input there to see what others think (and that will notify the moderators anyway) – Aurora0001 Dec 21 '18 at 21:38
  • 1
    @clearlight I've reopened the question, feel free to edit it further to clarify this a unique issue where an answer on the site does not already exist. Also I would like to commend you on a fantastic answer. – Darth Vader Dec 21 '18 at 21:44
  • @SlySven Thanks for the help. I was initially bugged but you guys were very cool about it. And I think the question is much better as a result. – clearlight Dec 21 '18 at 21:54
  • 1
    The question and answer does, IMHO look much more useful. I am sorry that in getting to this point things were a little stressful - we do get a lot of questions about networking and ssh, sometimes it isn't obvious that a new question does need a *new* answer... – SlySven Dec 21 '18 at 22:05
  • 1
    The original question was predicated on "I read somewhere that the server-side ssh server keys (host keys) can get botched in some Raspian(sic) distros" without any evidence and no reports of anyone else experiencing this "problem". ssh keys are generated on a new installation (in ALL Linux distros). This seems to be a solution to a non-existent problem. – Milliways Dec 22 '18 at 11:35
  • @Milliways: ___Existent problem!___ Here's a link to a reddit thread, proving someone had this issue a year ago and posted the suggestion to replace host keys, which worked for me after I figured out a different way to replace them. This is possibly where I read it. I'm going to keep researching this. I really wish you wouldn't jump to conclusions. When in doubt give benefit of doubt. Just because you couldn't find evidence doesn't mean there was none! reddit.com/r/raspberry_pi/comments/704v5y/… – – clearlight Dec 23 '18 at 21:17
  • The question isn't whether the problem exists for some people in some configurations, the questions are 1. What is the root cause. 2. What conditions does it manifest under? 3. What are the solutions, and the best solution, if any? – clearlight Dec 23 '18 at 21:28
  • Does the problem is reproducible? I should if you do a fresh flash from the image. What exactly is the output from ssh -vvv pi@<ip addr> when the connection is refused? – Ingo Dec 23 '18 at 22:15
  • @ingo I've already updated the host keys, but have more ssd cards and still have the downloaded image. I'll try to make some time in the next day or so to try the configuration from scratch again and see of the dtoverlay has anything to do with it. – clearlight Dec 23 '18 at 22:19
  • Just another question to isolate the problem: what I have to do to / what do I need to get the error in my home environment? – Ingo Dec 23 '18 at 22:32
  • @Ingo, I just now flashed a new 16GB micro SD card, with the same version of Raspian. This time I'm going to just touch /boot/ssh and create the wpa_supplicant.conf and see if it happens again. I'll also ensure no other Pis are on the Lan. If that works, then I will try to flash the SD again and add the dtoverlay line and see if that factors in. Might poke around first to see what's there for the keys and see if I can figure out if the ssh config actually ran. Still need to find out at what step in boot that's supposed to happen. – clearlight Dec 23 '18 at 22:48
  • @Ingo, when I did the most simple procedure and did not add the dtoverlay it just worked. I had to remove an ssh entry from known hosts and enter yes because the host key was different. The wpa_supplicant.conf line had an update_config=1 line in it, and not sure the scope of what that impacts or if that was present in the wpa_supplicant originally so I have a couple of other permutations to try. – clearlight Dec 23 '18 at 22:59
  • @ingo after adding dtoverlay=dwc2, and the corresponding entry in cmdline.txt I couldn't ssh in, but I will have to continue this whole thing another day. Sunk too much time into it already, can't work on anything I want to. I don't think what happened to me is a common scenario now, even though whatever I did to initially configure it, left me in the same boat twice and a simpler config doesn't. Nevertheless, i think the answer could help people in various ways. I will update this again when I need a break and another puzzle to chase down. – clearlight Dec 24 '18 at 00:00
  • Well done (^.^)d Removing dtoverlay=dwc2 and it works again? – Ingo Dec 24 '18 at 00:12
  • I still have to confirm that's the cause and things play out. It's a bit tedious so I'll update the bug once I'm sure what I think happened happened (within a day or so) – clearlight Dec 24 '18 at 01:55

2 Answers2

9

HOW TO REPLACE SSH HOST KEYS ON RASPBIAN W/O BOOTING INTO IT

Note: I had this happen with a recent download and install of Raspian full... it occurred on a headless pi zero W and a headless pi 3B+. This solved it in both cases. Not sure what was wrong with the keys but I suspect something may have been broken in the distribution.

If your ssh "connection refused" error is due to a problem with the Raspbian host's ssh keys (and one way to find out is to update them), you can replace them without booting into Raspbian if you have access to a running Linux host, which I will call a "helper" host. (Note: I went this route because there aren't safe/supported solutions I could find for mounting Linux ext4 partitions in 'writable' mode on macOS).

Your Linux helper host could be Raspbian running on another Pi, Linux running on PC hardware, or if you only have a Windows or Mac host, it could be guest virtual machine running Linux with VirtualBox, etc...

Assuming you have a way to mount the Raspbian media on that host, it's easy to reconfigure the SSH keys by simply overwriting them, because virtually all Linux distributions, (including Raspbian), have openssh installed, which offers the same set of tools.

PROCEDURE

Let's assume you've installed Raspbian on a 16GB micro SD card.

  1. Put that micro SD into a USB flash card reader, and insert it into a USB port on your Linux helper host. If you're using a virtual guest you'll have to configure it to intercept the USB device (micro SD card reader), otherwise your host machine may snag that USB host and the VM won't see it.

  2. On the Linux helper host, as root, run fdisk -l to find the device associated with your Raspbian media. In particular, you want to find the device associated with Raspbian Linux ext4 partition.

    For example, in the following fdisk output the Linux partition is at /dev/sb2

    $ sudo fdisk -l

Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x989b1246

Device     Boot Start      End  Sectors  Size Id Type
/dev/sdb1        8192    98045    89854 43.9M  c W95 FAT32 (LBA)
/dev/sdb2       98304 30375935 30277632 14.4G 83 Linux

  3. Mount the Linux partition, as follows (we'll assume the device is /dev/sb2 and the mount point is /mnt through the rest of the example).

    Note: Your actual device and chosen mount point may be different and you'll have to adjust this procedure accordingly.

    $ sudo mount /dev/sb2 /mnt

  4. Now you can generate new keys to replace the old keys in your micro SD's (Raspbian OS) ssh directory:

    $ sudo ssh-keygen -f /mnt/etc/ssh/ssh_host_rsa_key -N '' -t rsa 
$ sudo ssh-keygen -f /mnt/etc/ssh/ssh_host_dsa_key -N '' -t dsa
$ sudo ssh-keygen -f /mnt/etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa -b 521

  5. Now eject the micro SD card, so you can safely physically remove it:

    $ sudo eject /mnt

    Note: Eject will fail if any terminal windows are still cd'd into any Raspbian partition directories, so either close any such windows or cd to a different directory.

  6. Re-insert the micro SD into the pi and try booting. Hopefully the ssh connection refused problem will be gone. Of course there are any number of ways ssh might not be working - not installed, out of date, misconfigured, not starting, etc...

clearlight
  • 259
  • 2
  • 9
1

This can happen when you have ssh'ed into a pi, then created a new filesystem (FS) (new sd flashed), and then attempt to ssh in again because the host keys are now different than what has been stored into your known_hosts.

Thus, your host that you are attempting to ssh from notes that the host keys of the Pi have changed and correctly sees this as a security issue (man-in-the-middle attack), and will refuse to complete the connection.

So, if you know that the problem is due to a legitimate change in the host key, then what you need to do is update ~/.ssh/known_hosts by removing the old and now incorrect key.

Procedure

While you can manually delete the offend key, it is far easier and safer to use the CLI. The releveant key is identified by the host with which you connected.

Given: 

host = raspberrypi.local

Use this command:

$ ssh-keygen -R rasberrypi.local

You may need to delete it by IP.

You can then ssh in but will be warned about the new key.

Watch my YouTube video on what to do to ensure the key's fingerprint is correct.

https://www.youtube.com/watch?v=794sn9nllSM&t=2s
patrick
  • 29
  • 4