We have all been there during security operations. One of the parties involved in an incident or daily routine is not prepared for thinking they could be compromised.
Communications and information sharing is one of the fundamental things that you need to get right during a crisis.
As now-retired FBI director James Comey put it to 60 minutes:
There are two kinds of big companies in the United States. There are those who've been hacked by the Chinese and those who don't know they've been hacked by the Chinese.
The following question always arises: How do we maintain operational security while still being able to communicate with all parties involved?
In practical terms this requires a communications platform to:
- Be independent of the service infrastructure
- Provide traceability
- Be resistant to resourceful threat actors
- Have simple and secure identity management
- Have cross-platform compability
- Provide file-sharing capabilities and ability to give the user an opportunity to express himself
- Support video and audio exchanges
- Be under the control of the team using it (the smallest circle of trust)
- Provide both end-to-end and transport layer encryption
- Disposable server infrastructure
This could have been a bit too much to ask for a couple of years ago, but today there are at least two alternatives satisfying the above requirements: Mattermost and the Matrix ecosystem. For the remainder of this post I will focus on how to establish an ad-hoc system with the tools provided by the Matrix project.
Setting Up An Out-of-Band Channel for Incident Handling with Matrix
Getting started takes three steps:
- Establish a back-end server on Digital Ocean
- Serve the Riot front-end website
- Establish a recording capability with Matrix Recorder
For the two first points, it is clever to use an approach that can be easily reproduced and that provides exactly the same, secure-by-default configuration each time. Due to this the preferred method in this case is to manage the VPS that can be established on anything with Debian or CentOS with Ansible. There is a script available on Github, known as matrix-docker-ansible-deploy. The latter have also been endorsed by the Matrix project. Both 1 and 2 can be accomplished with
So let's get started.
For this example I created a domain on namesilo.com and pointed that to
(ns1|ns2|ns3).digitalocean.com. It would be ufortunate for the continuity of the service if a domain was taken offline or redirected somewhere, but due to the end to end encryption in Matrix it would not compromise the content of the conversations. Now that Digital Ocean has control of the primary domain, make sure to add the following before continuing:
Type Hostname Value TTL
A <domain> <ip> 600
A riot.<domain> <ip> 600
A matrix.<domain> <ip> 600
SRV _matrix._tcp.<domain> 10 0 8448 matrix.<domain> 600
This can take some time to propagate, so make sure that the DNS-infrastructure is readily resolvable before you continue deploying the services.
Make sure to grab a copy of the current
matrix-docker-ansible-deploy by running
git clone https://github.com/spantaleev/matrix-docker-ansible-deploy.git.
Create the following files:
vars.yml should look like this:
matrix_coturn_turn_static_auth_secret: "<run pwgen -s 64 1>"
matrix_synapse_macaroon_secret_key: "<run pwgen -s 64 1>"
hosts file should be formatted like the following:
Deploy and Execute
Now that your configuration files and server are ready, you can start deploying the Matrix Synapse server and start serving the Riot HTML/JS client.
First deploy the services (Riot and Matrix Synapse) by running:
ansible-playbook -i inventory/hosts setup.yml --tags=setup-main
When that completes successfully, you can start the services by:
ansible-playbook -i inventory/hosts setup.yml --tags=start
After starting the services, the Riot web interface is available on
https://riot.<domain> where metadata is protected by a Let's Encrypt certificate.
The two primary endpoints you now have exposed to the WWW is:
- The Matrix API which runs at https://matrix.
- The Riot UI which runs at https://riot.
Registration is disabled by default on the server, so new users can be added by the following command:
ansible-playbook -i inventory/hosts setup.yml --tags=register-user --extra-vars='username=<first user> password=<some password> admin=(yes|no)'
It is better to use pseudonyms on such a platform to make sure no information can be traced to a specific individual not involved in the case. Each user needs to verify his private key fingerprint with the other participants.
Vital Steps to Take as an Administrator
When using several servers, it is necessary to create an
#control channel that is a fallback if a server hosting a room goes down.
Setup Matrix Recorder
To make sure that all communications is stored for traceability make sure to install the Matrix Recorded (MR). MR should be installed locally and not on the Matrix server.
git clone https://gitlab.com/argit/matrix-recorder.git
To execute the recorder, run the following. The first time you will be asked to enter the login credentials of the user.
$ node matrix-recorder.js <case-folder>
Your homeserver (give full URL): https://matrix.<domain>
Your username at the homeserver: <username>
Your password at the homeserver: <password>
No of items to retrieve for initial sync: 1000
Messages can be viewed as HTML by running the Matrix Recorder conversion script:
node recorder-to-html.js <case-folder>
Which results in something like the following:
Access monitoring can be done in the console by e.g.
tail -f /matrix/synapse/run/homeserver.log.
The Power of Disposability
At some point you have finished the information exchange. The beauty of this setup is that is can now be safely deleted from the Digital Ocean droplet console.