SSH keys are supposed to make your life easier. But sometimes, things don’t go as planned.
You generate a key, set it up, and yet when you try to connect, Linux still prompts you for a password. Or worse, it flat-out refuses the connection.
If your SSH key file doesn’t work, don’t panic. In this guide, we’ll walk through the most common causes and how to fix them step by step.
Table of Contents
Check the Basics: Are You Using the Right Key?
It might sound obvious, but one of the most common issues is simply pointing SSH to the wrong private key.
By default, SSH looks for keys in ~/.ssh/:
id_rsa(RSA)id_ecdsa(ECDSA)id_ed25519(Ed25519, modern default)
👉 To explicitly tell SSH which key to use:
ssh -i ~/.ssh/id_ed25519 user@server
If this works, you can add it to your ~/.ssh/config for convenience:
Host myserver
HostName server.example.com
User user
IdentityFile ~/.ssh/id_ed25519
Now, connecting is as easy as:
ssh myserver
File Permissions: Too Open = Not Allowed
SSH is picky about file permissions. If your private key or .ssh folder is world-readable, SSH will refuse to use it.
✅ Fix it with:
See also: Mastering the Linux Command Line — Your Complete Free Training Guide
chmod 600 ~/.ssh/id_ed25519
chmod 700 ~/.ssh
Check the server side too:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
If permissions are too loose, you’ll see errors like:
Permissions 0644 for 'id_rsa' are too open.
Wrong Key on the Server
Even if your local key is correct, the server must recognize it.
- Your public key (
id_ed25519.pub) should be in:~/.ssh/authorized_keys - Make sure you copied it correctly:
cat ~/.ssh/id_ed25519.pub | ssh user@server "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys" - Verify that the line in
authorized_keysexactly matches your local.pubfile.
A single missing character or extra space can break authentication.
The Server Is Ignoring Your Key
Sometimes, SSH on the server isn’t configured to allow key authentication.
Check /etc/ssh/sshd_config:
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication no # optional, if you want to force keys only
After changes, restart SSH:
sudo systemctl restart sshd
⚠️ Important: If you disable password authentication, make sure your key works first, or you may lock yourself out.
Debugging with v
When all else fails, let SSH tell you what’s happening. Use:
ssh -v user@server
or for even more detail:
ssh -vvv user@server
You’ll see messages like:
Offering public key: …Server accepts key: …Permission denied (publickey)
This helps pinpoint whether the issue is local (wrong key, permissions) or server-side (authorized_keys, sshd config).
SELinux or Other Restrictions
On hardened Linux systems, SELinux or other security frameworks can block SSH from reading keys.
To test:
getenforce
If it says Enforcing, try:
restorecon -Rv ~/.ssh
This resets SELinux labels on the .ssh folder.
The Case of Old Keys
Some servers reject old key types for security reasons. For example:
- DSA (
id_dsa) is considered obsolete. - Modern systems prefer Ed25519 or RSA (4096-bit).
👉 If your key is outdated, generate a new one:
ssh-keygen -t ed25519 -C "[email protected]"
Putting It All Together
When SSH keys don’t work, it usually boils down to:
- Wrong key file
- Incorrect permissions
- Public key not copied correctly
- Server not allowing key authentication
- Old/unsupported key format
With the steps above, you should be able to isolate the problem quickly.
Final Thoughts
SSH keys are one of the most reliable ways to secure access in Linux, but small misconfigurations can make them frustrating. The good news? Once you fix the root cause, they usually “just work” for years without further hassle.
If you find yourself troubleshooting often, consider setting up a ~/.ssh/config file to manage multiple servers and keys. It’s a small step that saves a lot of headaches down the road.
