ACTION REQUIRED: Update your Bitbucket Cloud SSH Host Keys

Hello Bitbucket Cloud users,

We recently learned that encrypted copies of Bitbucket’s SSH host keys were included in a data breach of a third-party credential management vendor. The SSH protocol uses host keys to establish the identity of a trusted server for every SSH connection, like when a git pull establishes a SSH connection to Bitbucket Cloud.

In response, Bitbucket issued two new SSH host keys today and will be replacing the current host keys on June 20, 2023. Please review this blog and complete the applicable steps outlined below as soon as possible.

Though we believe the risk of compromise is low, by rotating the host keys proactively we are mitigating future risk should the old host keys be decrypted. If we did not change the host keys it might have been possible in the future for a threat actor to potentially use the old host keys in combination with an already compromised network to trick clients into connecting to and trusting a malicious host.

I want to assure you that a threat actor cannot use the old host keys to directly access your data on Bitbucket Cloud, or to access your private SSH keys.

We understand that rotating host keys can be disruptive. Your security is always our top priority, and we believe that acting proactively is the best approach.

Please read the below for details.

Bala Sathiamurthy, CISO/Head of Security

WHAT'S CHANGING

New host keys added

On May 15, 2023 2300 UTC we added two new host keys using the ECDSA and Ed25519 algorithms:

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPIQmuzMBuKdWeF4+a2sjSSpBK0iqitSQ+5BM9KhpexuGt20JpTVM7u5BDZngncgrqDMbWdxMWWOGtZ9UgbqgZE= bitbucket.org

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIazEu89wgQZ4bqs3d63QSMzYVa0MuJ2e2gKTKqu+UUO bitbucket.org

The corresponding fingerprints are:

256 SHA256:FC73VB6C4OQLSCrjEayhMp9UMxS97caD/Yyi2bhW/J0 bitbucket.org (ECDSA)

256 SHA256:ybgmFkzwOSotHTHLJgHO0QN8L0xErw6vd0VhFA9m3SM bitbucket.org (ED25519)

RSA host key rotation

On June 20, 2023 1700 UTC we will replace our current RSA host key with the following:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQeJzhupRu0u0cdegZIa8e86EG2qOCsIsD1Xw0xSeiPDlCr7kq97NLmMbpKTX6Esc30NuoqEEHCuc7yWtwp8dI76EEEB1VqY9QJq6vk+aySyboD5QF61I/1WeTwu+deCbgKMGbUijeXhtfbxSxm6JwGrXrhBdofTsbKRUsrN1WoNgUa8uqN1Vx6WAJw1JHPhglEGGHea6QICwJOAr/6mrui/oB7pkaWKHj3z7d1IC4KWLtY47elvjbaTlkN04Kc/5LFEirorGYVbt15kAUlqGM65pk6ZBxtaO3+30LVlORZkxOh+LKL/BvbZ/iRNhItLqNyieoQj/uh/7Iv4uyH/cV/0b4WDSd3DptigWq84lJubb9t/DnZlrJazxyDCulTmKdOR7vs9gMTo+uoIrPSb8ScTtvw65+odKAlBj59dhnVp9zd7QUojOpXlL62Aw56U4oO+FALuevvMjiWeavKhJqlR7i5n9srYcrNV7ttmDw7kf/97P5zauIhxcjX+xHv4M= bitbucket.org

The corresponding RSA key fingerprint is:

3072 SHA256:46OSHA1Rmj8E8ERTC6xkNcmGOw9oFxYr0WF6zWW8l1E bitbucket.org (RSA)

DSA host key removal

On June 20, 2023 1700 UTC we will also remove our DSA host key; this key will stop working entirely.

WHAT YOU NEED TO DO

We highly recommend that you immediately switch to using the newer ECDSA or Ed25519 host keys for each of your SSH clients that connect to bitbucket.org, including other applications such as IDEs and CI/CD build systems. No action is required for your Bitbucket Pipelines builds or user-generated SSH keys.

Many SSH clients will switch to one of the new host keys automatically, but you will need to verify this for each connected SSH client. Follow the instructions below to check if your client has automatically switched to the new host keys. If your client did not automatically switch, you will need to follow steps 2 and 3 to manually update your client configuration.

1. IDENTIFY IF YOUR CLIENT IS IMPACTED

To verify which host key your SSH client is using, you can run the following command:

$ ssh git@bitbucket.org host_key_info
You are using host key with fingerprint:
ssh-ed25519 SHA256:ybgmFkzwOSotHTHLJgHO0QN8L0xErw6vd0VhFA9m3SM

See https://bitbucket.org/blog/ssh-host-key-changes for more details.

Do you see either the new ECDSA or Ed25519 host key fingerprint in the output?

Yes! Your SSH client has switched to the new host keys automatically and no further action is required for this client.

No. Proceed to Step 2.

2. CONFIGURE YOUR CLIENT TO TRUST THE NEW HOST KEYS

If neither new fingerprints appear in the output of your OpenSSH (or compatible) client, you can configure the new trusted host keys in the known_hosts file with these commands:

$ ssh-keygen -R bitbucket.org && curl https://bitbucket.org/site/ssh >> ~/.ssh/known_hosts

If you are using a different SSH client, you should consult its documentation on how to update its configuration.

3. CONFIRM SUCCESSFUL KEY ADDITION

Re-run the commands in Step 1 to confirm that your client now trusts the new host keys.

TROUBLESHOOTING SCENARIOS

I've followed the steps, but the new host key fingerprints are not appearing in my client output?

Consult your specific SSH client's documentation to ensure the following:

Your client is not configured to prefer RSA over ECDSA/Ed25519.
Your client is using the configuration in ~/.ssh/known_hosts; otherwise, you will have to manually reconfigure it to use one or both of the new host keys.

My client only supports RSA host key algorithms.

You will need to replace the RSA key on June 20, 2023.

My Bamboo builds are failing to trigger.

If your Bamboo instance is configured to use trusted keys your administrator will need to update your configuration to trust the new ECDSA/Ed25519 host keys.

I think my SSH client is broken, what do I do?

A small number of clients may have a configuration that attempts to validate the new ECDSA or Ed25519 host keys against the old RSA key, causing the connection to fail. This may include services using low-level libraries like xcrypto/ssh for Go, or libssh2 for C. You can resolve this by updating the configuration of the library to trust the new ECDSA or Ed25519 host keys.

If you still need help after following the step outlined above, you can reach out to our Support team for help by raising a Bitbucket Cloud support ticket at https://support.atlassian.com/contact/#/

I'm using port 443 to run my connection via altssh.bitbucket.org. How do I update my known_hosts file?

You can fetch the altssh.bitbucket.org fingerprints and add it to a known_hosts file by running the following:

ssh-keygen -R '[altssh.bitbucket.org]:443' && curl https://bitbucket.org/site/ssh | sed 's/bitbucket.org/[altssh.bitbucket.org]:443/' >> ~/.ssh/known_hosts

FAQ

What does a host key do?

The SSH protocol uses host keys to establish the identity of a trusted server for every SSH connection, like when a git pull establishes a SSH connection to bitbucket.org.

Why are you rotating the RSA host key?

We recently learned that encrypted copies of Bitbucket’s SSH host keys were included in a data breach of a third-party credential management vendor. Though we believe the risk of compromise is low, by rotating the host keys proactively we are mitigating future risk should the old host keys be decrypted. If we did not change the host keys it might have been possible in the future for a threat actor to potentially use the old host keys in combination with an already compromised network to trick clients into connecting to and trusting a malicious host.

A threat actor cannot use the old host keys to directly access your data on Bitbucket Cloud, or to access your private SSH keys.

We understand that rotating host keys can be disruptive. Your security is always our top priority, and we believe that acting proactively is the best approach.

Why aren't you rotating the RSA host key immediately?

We do not believe there is an imminent and significant risk to customers. Continuing to allow clients to use a known host key gives users time to carefully and securely establish trust in the new keys.

By taking a measured approach, we are giving users time to transition to the new keys safely and with as little interruption as possible to workflows.

Is any customer data impacted?

No, SSH host keys do not secure any data on Bitbucket Cloud.

Are any customer accounts or credentials impacted?

No, SSH host keys are not involved in user authentication on Bitbucket Cloud.

Are user SSH keys impacted?

No, there is no impact to user-generated SSH keys that have been uploaded to your Bitbucket Personal profile, added to a Workspace, or used in a Repository as a deploy key.

Is Bitbucket Data Center affected?

No, there is no impact to Bitbucket Data Center.

Will you be rotating host keys regularly?

We do not have plans for future rotations at this time.

Is Git over HTTPS affected?

No, only Git over SSH is affected.

You can check whether a specific repository is using SSH by checking its remotes.

If your remote starts with https then you are using Git over HTTPS and you are not affected:

$ git remote -v
origin    https://bitbucket.org/test/repository (fetch)
origin    https://bitbucket.org/test/repository (push)

If your remote starts with git@bitbucket.org then you are using Git over SSH and action may be required (see below):

$ git remote -v
origin      git@bitbucket.org:/test/repository.git (fetch)
origin      git@bitbucket.org:/test/repository.git (push)

How do I know if my SSH connections will be impacted by this?

You can check specific clients with our host key information SSH command:

$ ssh git@bitbucket.org host_key_info
You are using host key with fingerprint:
ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A

WARNING: The host key your client is using will be removed in the near future.

Please configure your client to trust a new host key.

See https://bitbucket.org/blog/ssh-host-key-changes for more details.

If your client is already using either the ECDSA or Ed25519 host key, then you're all set.

If your client is using either the RSA or DSA host key, then see the steps above to update the configuration of your client.

Do I need to make any changes for Bitbucket Pipelines?

No, the trusted host keys provided to your builds will be updated by Atlassian.

Why are you removing support for DSA?

The DSA algorithm was deprecated by OpenSSH in 2015. You should update your client configuration to use stronger ECDSA, Ed25519, or RSA encryption algorithms instead.