Matrix.org Team

Overview

We became aware today of a flaw in sydent’s validation of email addresses which can lead to a failure to correctly limit registration to a given email domain. This only affects people who run their own sydent, and are relying on allowed_local_3pid in their synapse config. We’d like to thank @fs0c131y for bringing it to our attention on Twitter this morning. We are not aware of this being exploited in the wild other than the initial report.

If you are running your own sydent, and limiting signup for your server using the allowed_local_3pids configuration option, then you need to upgrade your sydent immediately to Sydent 1.0.2.

Meanwhile, if you have been relying on the allowed_local_3pids configuration option to restrict access to your homeserver, you may wish to check your homeserver’s user_threepids table for malformed email addresses and your sydent’s database as follows:

$ sqlite3 sydent.db 
sqlite> select count(*) from global_threepid_associations where address like '%@%@%';
0

$ psql matrix
matrix=> select count(*) from user_threepids where address like '%@%@%';
 count 
-------
     0

If the queries return more than 0 results, please let us know at security@matrix.org - otherwise you are fine.

Details

A flaw existed in sydent whereby it was possible to bypass the requirement specified in synapse’s allowed_local_3pids option, which restricts that users may only register with an email address matching a specific format.

This relied on two things:

1. sydent uses python's email.utils.parseaddr function to parse the input email address before sending validation mail to it, but it turns out that if you hand parseaddr an malformed email address of form a@b.com@c.com, it silently discards the @c.com prefix without error. The result of this is that if one requested a validation token for 'a@malicious.org@important.com', the token would be sent to 'a@malicious.org', but the address 'a@malicious.org@important.com' would be marked as validated. This release fixes this behaviour by asserting that the parsed email address is the same as the input email address.
2. synapse's checking of email addresses relies on regular expressions in the home server configuration file. synapse does not validate email addresses before checking them against these regular expressions, so naive regular expressions will detect the second domain in email addresses such as the above, causing them to pass the check.

You can get sydent 1.0.2 from https://github.com/matrix-org/sydent/releases/tag/v1.0.2.

Welcome to the new blog!

Check out the new digs! We're happy with this newly deployed blog, and all the old and loveable content is right down there. If you find issues, let me know. You may remember Nad, from previous editions of Matrix Live - huge thanks to him for his work on the design and upkeep of this new deployment.

Notes from the downtimes

Github tokens

jaywink:

Due to the security incident, all GitHub access tokens for the Scalar GitHub integration were cleared. This means that if you have a GitHub bot in the channel and want to use the !github bot commands, you need to re-login to github via the integration manager menu. Note, existing webhooks are untouched and should work fine without re-authenticating.

Bridges

Half-Shot:

From the matrix.org bridge team, we are resurrecting bridges as fast as possible. Currently running are the freenode, slack, gitter and gimpnet (now hosted on gnome.org) with more to come today and next week.
We have the snoonet and oftc irc bridges back. Mozilla is coming soon hopefully this weekend too!

Whoop! Mozilla is actually up now!

Pattle new release on F-Droid

Wilko:

A big release of Pattle has just been pushed to the F-droid repo! Changes include:

• Display names are now shown
• You can now click on chats and view them!
• Messages are grouped by time and sender (see screenshots)
• Add fancy transition animation and ripples to chat messages (see video)
• Use Sentry for error reporting (only Android version and device model is sent, along with the stacktrace of the error)

Also, please note that if you have a matrix.org you probably have reinstall the app if you're logged in because of the recent matrix.org incident (because there's no logout button yet and no detection for invalidated access tokens)

There has actually been a release since, which includes:

message sending and viewing image history!

libQMatrixClient 0.5.1.2 && Quaternion 0.0.9.4

kitsune is talking about the long road to 0.0.9.4:

libQMatrixClient 0.5.1.2 has been released, with all the remaining bugfixes for Quaternion 0.0.9.4 that's coming any day soon now. The release notes are here: https://github.com/QMatrixClient/libqmatrixclient/releases/tag/0.5.1.2

Quaternion 0.0.9.4 RC3, the last one before the release that will happen in the nearest days, is out. Release notes can be found at https://github.com/QMatrixClient/Quaternion/releases/tag/0.0.9.4-rc3. Translators, you literally have hours to add your translations for 0.0.9.4!

neo

I reimplemented the matrix sdk into Neo, so it works nicer. Colors and font look nicer (base16-tomorrow, Open Sans), and there's text message sending, with localecho!
I also fixed a bug where react would recycle displayname-components across rooms, attributing them to the wrong messages

a video at https://lain.haus/_matrix/media/v1/download/lain.haus/VfshWRfaNUnpGQbdkyYczxvd

Go test Neo at: https://neo.lain.haus/neo

matrix-registration update

ZerataX:

it's been a long while, but I've finally come around to improving on matrix-registration
For those of you, who have forgotten what this project is about, it basically lets you invite people to your homeserver with tokens, e.g. https://homeserver.tld/register?token=DoubleWizardSky
This whole update was about making the project more user friendly.
I made a new default registration page that requires 0 setup and you can install the project right from pypi with pip, so you don't even need to clone the repo any longer.
check a live example here: https://chat.dmnd.sh/register
and to play around with the api you can can go over to the github page: https://zeratax.github.io/matrix-registration/demo.html?token=ColorWhiskeyExpand
channel: #matrix-registration:dmnd.sh
github: https://github.com/zeratax/matrix-registration

Video available here.

matrix-media-repo now has s3 support

TravisR:

matrix-media-repo now has s3 (and s3-like) support, making it easier to archive older media or use minimal disk space. See the new datastores option in the config and the admin docs ( https://github.com/turt2live/matrix-media-repo/blob/master/docs/admin.md#datastore-management ) for more information.

Dimension

TravisR:

Dimension has been updated to more safely handle when upstream integration managers (like Scalar) are offline. Instead of crashing or breaking in various ways, it'll report which integrations are not accessible.
As well, due to recent events, if you use matrix.org bots or bridges in Dimension then go the the admin section and log everyone out using the red button. Dimension caches upstream tokens and isn't smart enough to realize that they are no longer valid, which means they need clearing. Clients should automatically handle getting new tokens in the background.

Riot iOS

• Interactive device verification is still in progress
• Update WebRTC and Jitsi libs (in progress)

Riot Android

*We are finalizing SAS device verification feature *We are preparing a release to fix some issues

RiotX (Android)

• We can now clear the cache of the application (Riot legacy feature)
• Sync filter has been implemented
• François is working on thumbnail of video (upload and preview) in the timeline

That's all I know

See you next week, and be sure to stop by #twim:matrix.org with your updates!

Update: for the full story here, please see the post mortem.

Here's what you need to know.

TL;DR: An attacker gained access to the servers hosting Matrix.org. The intruder had access to the production databases, potentially giving them access to unencrypted message data, password hashes and access tokens. As a precaution, if you're a matrix.org user you should change your password now.

The matrix.org homeserver has been rebuilt and is running securely; bridges and other ancillary services (e.g. this blog) will follow as soon as possible. Modular.im homeservers have not been affected by this outage.

The security breach is not a Matrix issue.

The hacker exploited a vulnerability in our production infrastructure (specifically a slightly outdated version of Jenkins). Homeservers other than matrix.org are unaffected.

How does this affect me?

We have invalidated all of the active access tokens for users on Matrix.org - all users have been logged out.

Users with Matrix.org accounts should:

• Change your password now - no plaintext Matrix passwords were leaked, but weak passwords could still be cracked from the hashed passwords
• Change your NickServ password (if you're using IRC bridging) - there's no evidence bridge credentials were compromised, but if you have given the IRC bridges credentials to your NickServ account we would recommend changing this password

And as a reminder, it's good practice to:

• Review your device list regularly - make sure you recognise all of the devices connected to your account
• Always make sure you enable E2E encryption for private conversations

What user data has been accessed?

Forensics are ongoing; so far we've found no evidence of large quantities of data being downloaded. The attacker did have access to the production database, so unencrypted content (including private messages, password hashes and access tokens) may be compromised.

What has not been affected?

• Source code and packages have not been impacted based on our initial investigations. However, we will be replacing signing keys as a precaution.
• Modular.im servers are not affected, based on our initial analysis
• Identity server data does not appear to have been compromised

The target appeared to be internal credentials for onward exploits, not end user information from the matrix.org homeserver.

You might have lost access to your encrypted messages.

As we had to log out all users from matrix.org, if you do not have backups of your encryption keys you will not be able to read your encrypted conversation history. However, if you use server-side encryption key backup (the default in Riot these days) or take manual key backups, you’ll be okay.

This was a difficult choice to make. We weighed the risk of some users losing access to encrypted messages against that of all users' accounts being vulnerable to hijack via the compromised access tokens. We hope you can see why we made the decision to prioritise account integrity over access to encrypted messages, but we're sorry for the inconvenience this may have caused.

What happened?

We were using Jenkins for continuous integration (automatically testing our software). The version of Jenkins we were using had a vulnerability (CVE-2019-1003000, CVE-2019-1003001, CVE-2019-1003002) which allowed an attacker to hijack credentials (forwarded ssh keys), giving access to our production infrastructure. Thanks to @jaikeysarraf for drawing this to our attention.

Timeline

March 13th Updated 2019-04-12 11:00 UTC

• Attacker compromises Jenkins CI server

April 4th Updated 2019-04-12 11:00 UTC

• Attacker gains access to production infrastructure by hijacking a forwarded SSH agent logging into the compromised Jenkins worker

April 9th

• Jenkins vulnerability brought to our attention by @jaikeysarraf

April 10th

• Investigation identified the compromised machines and the full scope of the attack
• Jenkins was removed
• Attacker's access to compromised machines was removed

April 11th

• Matrix.org was taken offline and production infrastructure fully rebuilt
• Having fully flushed out the attacker, external communication was published informing users and advising on next steps
• Matrix.org homeserver restored, with bridges and ancillary services (e.g. this blog) following as soon as possible

Update 2019-04-12

At around 5am UTC on Apr 12, the attacker used a cloudflare API key to repoint DNS for matrix.org to a defacement website (https://github.com/matrixnotorg/matrixnotorg.github.io). The API key was known compromised in the original attack, and during the rebuild the key was theoretically replaced. However, unfortunately only personal keys were rotated, enabling the defacement. We are currently doublechecking that all compromised secrets have been rotated.

The rebuilt infrastructure itself is secure, however, and the DNS issue has been solved without further abuse. If you have already changed your password, you do not need to do so again.

The defacement confirms that encrypted password hashes were exfiltrated from the production database, so it is even more important for everyone to change their password. We will shortly be messaging and emailing all users to announce the breach and advise them to change their passwords. We will also look at ways of non-destructively forcing a password reset at next login.

The attacker has also posted github issues detailing some of their actions and suggested remediations at https://github.com/matrix-org/matrix.org/issues/created_by/matrixnotorg.

This confirms that GPG keys used for signing packages were compromised. These keys are used for signing the synapse debian repository (AD0592FE47F0DF61), and releases of Riot/Web (E019645248E8F4A1). Both keys have now been revoked. The window of compromise for the keys started from April 4th; there have been no Synapse releases since then. There has been one release of Riot/Web (1.0.7), however as the key was passphrased and based on our initial analysis of the release, we believe it to be secure.

What are we doing to prevent this in future?

Once things are back up and running we will retrospect on this incident in detail to identify the changes we need to make. We will provide a proper postmortem, including follow-up steps; meanwhile we are obviously going to take measures to improve the security of our production infrastructure, including patching services more aggressively and more regular vulnerability scans.