chore(docs): update README to reflect manager move (#1519)

* chore(docs): simplify and update `README` to reflect manager move

* Add pointer to shadowbox README.

* Cleanup the Shadowbox `README`

* Move around the Shadowbox instructions to make Node.js and Docker equal options for running the Shadowbox.

* Address review comments.

* Revert intro sentence to what was there before.

* Address review comments.
This commit is contained in:
Sander Bruens 2024-03-12 15:09:01 -04:00 committed by GitHub
parent 78f0934f4a
commit 37d3f33faf
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 187 additions and 249 deletions

View file

@ -3,7 +3,9 @@
We'd love to accept your patches and contributions to this project. There are
just a few small guidelines you need to follow.
## Contributor License Agreement
## Before you begin
### Contributor License Agreement
Contributions to this project must be accompanied by a Contributor License
Agreement. You (or your employer) retain the copyright to your contribution,
@ -15,9 +17,33 @@ You generally only need to submit a CLA once, so if you've already submitted one
(even if it was for a different project), you probably don't need to do it
again.
## Code reviews
### Code reviews
All submissions, including submissions by project members, require review. We
use GitHub pull requests for this purpose. Consult
[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
information on using pull requests.
## Build Actions
We have a very simple build system based on package.json scripts that are called using `npm run`
and a thin wrapper for what we call build "actions".
We've defined a package.json script called `action` whose parameter is a relative path:
```sh
npm run action $ACTION
```
This command will define a `run_action()` function and call `${ACTION}.action.sh`, which must exist.
The called action script can use `run_action` to call its dependencies. The $ACTION parameter is
always resolved from the project root, regardless of the caller location.
The idea of `run_action` is to keep the build logic next to where the relevant code is.
It also defines two environmental variables:
- `ROOT_DIR`: the root directory of the project, as an absolute path.
- `BUILD_DIR`: where the build output should go, as an absolute path.
> [!TIP]
> To find all the actions in this project, run `npm run action:list`

131
README.md
View file

@ -2,126 +2,55 @@
![Build and Test](https://github.com/Jigsaw-Code/outline-server/actions/workflows/build_and_test_debug.yml/badge.svg?branch=master) [![Mattermost](https://badgen.net/badge/Mattermost/Outline%20Community/blue)](https://community.internetfreedomfestival.org/community/channels/outline-community) [![Reddit](https://badgen.net/badge/Reddit/r%2Foutlinevpn/orange)](https://www.reddit.com/r/outlinevpn/)
This repository has all the code needed to create and manage Outline servers on
DigitalOcean. An Outline server runs instances of Shadowsocks proxies and
provides an API used by the Outline Manager application.
Outline Server is the component that provides the Shadowsocks service (via [outline-ss-server](https://github.com/Jigsaw-Code/outline-ss-server/)) and a service management API. You can deploy this server directly following simple instructions in this repository, or if you prefer a ready-to-use graphical interface you can use the [Outline Manager](https://github.com/Jigsaw-Code/outline-apps/).
Go to https://getoutline.org for ready-to-use versions of the software. **To join our Outline Community, [sign up for the IFF Mattermost](https://internetfreedomfestival.org/wiki/index.php/IFF_Mattermost).**
**Components:**
## Components
- **Outline Server** ([`src/shadowbox`](src/shadowbox)): The core proxy server that runs and manages [outline-ss-server](https://github.com/Jigsaw-Code/outline-ss-server/), a Shadowsocks backend. It provides a REST API for access key management.
The system comprises the following components:
- **Metrics Server** ([`src/metrics_server`](src/metrics_server)): A REST service for optional, anonymous metrics sharing.
- **Outline Server**: a proxy server that runs a Shadowsocks instance for each
access key and a REST API to manage the access keys. The Outline Server runs
in a Docker container in the host machine.
**Join the Outline Community** by signing up for the [IFF Mattermost](https://wiki.digitalrights.community/index.php?title=IFF_Mattermost)!
See [`src/shadowbox`](src/shadowbox)
## Shadowsocks and Anti-Censorship
- **Outline Manager:** an [Electron](https://electronjs.org/) application that
can create Outline Servers on the cloud and talks to their access key
management API to manage who has access to the server.
Outline's use of Shadowsocks means it benefits from ongoing improvements that strengthen its resistance against detection and blocking.
See [`src/server_manager`](src/server_manager)
**Key Protections:**
- **Metrics Server:** a REST service that the Outline Server talks to
if the user opts-in to anonymous metrics sharing.
- **AEAD ciphers** are mandatory.
- **Probing resistance** mitigates detection techniques.
- **Protection against replayed data.**
- **Variable packet sizes** to hinder identification.
See [`src/metrics_server`](src/metrics_server)
See [Shadowsocks resistance against detection and blocking](docs/shadowsocks.md).
## Code Prerequisites
## Installation
In order to build and run the code, you need the following installed:
**Prerequisites**
- [Node](https://nodejs.org/en/download/) LTS (`lts/hydrogen`, version `18.16.0`)
- [NPM](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) (version `9.5.1`)
- Manager-specific
- [Wine](https://www.winehq.org/download), if you would like to generate binaries for Windows.
- Server-specific
- [Docker](https://docs.docker.com/engine/install/), to build the Docker image and to run the integration test.
- [docker-compose](https://docs.docker.com/compose/install/), to run the integration test.
> 💡 NOTE: if you have `nvm` installed, run `nvm use` to switch to the correct node version!
1. **Install dependencies**
Install dependencies with:
```sh
npm install
```
```sh
npm install
```
1. **Start the server**
This project uses [NPM workspaces](https://docs.npmjs.com/cli/v7/using-npm/workspaces/).
```sh
npm run action shadowbox/server/start
```
## Build System
Exploring further options:
We have a very simple build system based on package.json scripts that are called using `npm run`
and a thin wrapper for what we call build "actions".
- **Refer to the README:** Find additional configuration and usage options in the core server's [`README`](src/shadowbox/README.md).
- **Learn about the build system:** For in-depth build system information, consult the [contributing guide](CONTRIBUTING.md).
We've defined a package.json script called `action` whose parameter is a relative path:
1. **To clean up**
```shell
npm run action $ACTION
```
This command will define a `run_action()` function and call `${ACTION}.action.sh`, which must exist.
The called action script can use `run_action` to call its dependencies. The $ACTION parameter is
always resolved from the project root, regardless of the caller location.
The idea of `run_action` is to keep the build logic next to where the relevant code is.
It also defines two environmental variables:
- ROOT_DIR: the root directory of the project, as an absolute path.
- BUILD_DIR: where the build output should go, as an absolute path.
> ⚠️ To find all the actions in this project, run `npm run action:list`
### Build output
Building creates the following directories under `build/`:
- `web_app/`: The Manager web app.
- `static/`: The standalone web app static files. This is what one deploys to a web server or runs with Electron.
- `electron_app/`: The launcher desktop Electron app
- `static/`: The Manager Electron app to run with the electron command-line
- `bundled/`: The Electron app bundled to run standalone on each platform
- `packaged/`: The Electron app bundles packaged as single files for distribution
- `invite_page`: the Invite Page
- `static`: The standalone static files to be deployed
- `shadowbox`: The Proxy Server
The directories have subdirectories for intermediate output:
- `ts/`: Autogenerated Typescript files
- `js/`: The output from compiling Typescript code
- `browserified/`: The output of browserifying the JavaScript code
To clean up:
```
npm run clean
```
## Shadowsocks Resistance Against Detection and Blocking
Shadowsocks used to be blocked in some countries, and because Outline uses Shadowsocks, there has been skepticism about Outline working in those countries. In fact, people have tried Outline in the past and had their servers blocked.
However, since the second half of 2020 things have changed. The Outline team and Shadowsocks community made a number of improvements that strengthened Shadowsocks beyond the censor's current capabilities.
As shown in the research [How China Detects and Blocks Shadowsocks](https://gfw.report/talks/imc20/en/), the censor uses active probing to detect Shadowsocks servers. The probing may be triggered by packet sniffing, but that's not how the servers are detected.
Even though Shadowsocks is a standard, it leaves a lot of room for choices on how it's implemented and deployed.
First of all, you **must use AEAD ciphers**. The old stream ciphers are easy to break and manipulate, exposing you to simple detection and decryption attacks. Outline has banned all stream ciphers, since people copy old examples to set up their servers. The Outline Manager goes further and picks the cipher for you, since users don't usually know how to choose a cipher, and it generates a long random secret, so you are not vulnerable to dictionary-based attacks.
Second, you need **probing resistance**. Both shadowsocks-libev and Outline have added that. The research [Detecting Probe-resistant Proxies](https://www.ndss-symposium.org/ndss-paper/detecting-probe-resistant-proxies/) showed that, in the past, an invalid byte would trigger different behaviors whether it was inserted in positions 49, 50 or 51 of the stream, which is very telling. That behavior is now gone, and the censor can no longer rely on that.
Third, you need **protection against replayed data**. Both shadowsocks-libev and Outline have added such protection, which you may need to enable explicitly on ss-libev, but it's the default on Outline.
Fourth, Outline and clients using shadowsocks-libev now **merge the SOCKS address and the initial data** in the same initial encrypted frame, making the size of the first packet variable. Before the first packet only had the SOCKS address, with a fixed size, and that was a giveaway.
The censors used to block Shadowsocks, but Shadowsocks has evolved, and in 2021, it was ahead again in the cat and mouse game.
In 2022 China started blocking seemingly random traffic ([report](https://www.opentech.fund/news/exposing-the-great-firewalls-dynamic-blocking-of-fully-encrypted-traffic/)). While there is no evidence they could detect Shadowsocks, the protocol ended up blocked.
As a reponse, we [added a feature to the Outline Client](https://github.com/Jigsaw-Code/outline-client/pull/1454) that allows service managers to specify in the access key a prefix to be used in the Shadowsocks initialization, which can be used to bypass the blocking in China.
Shadowsocks remains our protocol of choice because it's simple, well understood and very performant. Furthermore, it has an enthusiastic community of very smart people behind it.
```sh
npm run clean
```

25
docs/shadowsocks.md Normal file
View file

@ -0,0 +1,25 @@
# Shadowsocks Resistance Against Detection and Blocking
Shadowsocks used to be blocked in some countries, and because Outline uses Shadowsocks, there has been skepticism about Outline working in those countries. In fact, people have tried Outline in the past and had their servers blocked.
However, since the second half of 2020 things have changed. The Outline team and Shadowsocks community made a number of improvements that strengthened Shadowsocks beyond the censor's current capabilities.
As shown in the research [How China Detects and Blocks Shadowsocks](https://gfw.report/talks/imc20/en/), the censor uses active probing to detect Shadowsocks servers. The probing may be triggered by packet sniffing, but that's not how the servers are detected.
Even though Shadowsocks is a standard, it leaves a lot of room for choices on how it's implemented and deployed.
First of all, you **must use AEAD ciphers**. The old stream ciphers are easy to break and manipulate, exposing you to simple detection and decryption attacks. Outline has banned all stream ciphers, since people copy old examples to set up their servers. The Outline Manager goes further and picks the cipher for you, since users don't usually know how to choose a cipher, and it generates a long random secret, so you are not vulnerable to dictionary-based attacks.
Second, you need **probing resistance**. Both shadowsocks-libev and Outline have added that. The research [Detecting Probe-resistant Proxies](https://www.ndss-symposium.org/ndss-paper/detecting-probe-resistant-proxies/) showed that, in the past, an invalid byte would trigger different behaviors whether it was inserted in positions 49, 50 or 51 of the stream, which is very telling. That behavior is now gone, and the censor can no longer rely on that.
Third, you need **protection against replayed data**. Both shadowsocks-libev and Outline have added such protection, which you may need to enable explicitly on ss-libev, but it's the default on Outline.
Fourth, Outline and clients using shadowsocks-libev now **merge the SOCKS address and the initial data** in the same initial encrypted frame, making the size of the first packet variable. Before the first packet only had the SOCKS address, with a fixed size, and that was a giveaway.
The censors used to block Shadowsocks, but Shadowsocks has evolved, and in 2021, it was ahead again in the cat and mouse game.
In 2022 China started blocking seemingly random traffic ([report](https://www.opentech.fund/news/exposing-the-great-firewalls-dynamic-blocking-of-fully-encrypted-traffic/)). While there is no evidence they could detect Shadowsocks, the protocol ended up blocked.
As a reponse, we [added a feature to the Outline Client](https://github.com/Jigsaw-Code/outline-apps/pull/1454) that allows service managers to specify in the access key a prefix to be used in the Shadowsocks initialization, which can be used to bypass the blocking in China.
Shadowsocks remains our protocol of choice because it's simple, well understood and very performant. Furthermore, it has an enthusiastic community of very smart people behind it.

View file

@ -1,203 +1,161 @@
# Outline Server
# Outline Server (Shadowbox)
The internal name for the Outline server is "Shadowbox". It is a server set up
that runs a user management API and starts Shadowsocks instances on demand.
The Outline Server, internal name "Shadowbox," is designed to streamline the setup and sharing of Shadowsocks servers. It includes a user management API and creates Shadowsocks instances when needed. It's managed by the [Outline Manager](https://github.com/Jigsaw-Code/outline-apps/) and used as proxy by the [Outline Client](https://github.com/Jigsaw-Code/outline-apps/) apps. Shadowbox is also compatible with standard Shadowsocks clients.
It aims to make it as easy as possible to set up and share a Shadowsocks
server. It's managed by the Outline Manager and used as proxy by the Outline
client apps. Shadowbox is also compatible with standard Shadowsocks clients.
## Installation
## Self-hosted installation
### Self-Hosted Installation
To install and run Shadowbox on your own server, run
1. **Run the Installation Script**
```
sudo bash -c "$(wget -qO- https://raw.githubusercontent.com/Jigsaw-Code/outline-apps/master/server_manager/install_scripts/install_server.sh)"
```
```sh
sudo bash -c "$(wget -qO- https://raw.githubusercontent.com/Jigsaw-Code/outline-apps/master/server_manager/install_scripts/install_server.sh)"
```
You can specify flags to customize the installation. For example, to use hostname `myserver.com` and the port 443 for access keys, you can run:
1. **Customize (Optional)**
```
sudo bash -c "$(wget -qO- https://raw.githubusercontent.com/Jigsaw-Code/outline-apps/master/server_manager/install_scripts/install_server.sh)" install_server.sh --hostname=myserver.com --keys-port=443
```
Add flags for hostname, port, etc. Example:
Use `sudo --preserve-env` if you need to pass environment variables. Use `bash -x` if you need to debug the installation.
```sh
sudo bash -c "$(wget -qO- https://raw.githubusercontent.com/Jigsaw-Code/outline-apps/master/server_manager/install_scripts/install_server.sh)" install_server.sh \
--hostname=myserver.com \
--keys-port=443
```
## Running from source code
- Use `sudo --preserve-env` for environment variables.
- Use `bash -x` for debugging.
### Prerequisites
### Running from Source Code
Shadowbox supports running on linux and macOS hosts.
**Prerequisites**
Besides [Node](https://nodejs.org/en/download/) you will also need:
- [Docker](https://docs.docker.com/engine/install/)
- [Node](https://nodejs.org/en/download/) LTS (`lts/hydrogen`, version `18.16.0`)
- [NPM](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) (version `9.5.1`)
1. [Docker Engine](https://docs.docker.com/engine/installation/)
> [!TIP]
> If you use `nvm`, switch to the correct Node version with `nvm use`.
### Running Shadowbox as a Node.js app
1. **Build and Run:**
Build and run the server as a Node.js app:
Shadowbox supports running on linux and macOS hosts.
```
npm run action shadowbox/server/start
```
- **Node.js App**
The output will be at `build/shadowbox/app`.
```sh
npm run action shadowbox/server/start
```
### Running Shadowbox as a Docker container
- **Docker Container**
### With docker command
```sh
npm run action shadowbox/docker/start
```
Build the image and run server:
> [!TIP]
> Some useful commands when working with Docker images and containers:
>
> - **Debug Image:**
>
> ```sh
> docker run --rm -it --entrypoint=sh localhost/outline/shadowbox
> ```
>
> - **Debug Running Container:**
>
> ```sh
> docker exec -it shadowbox sh
> ```
>
> - **Cleanup Dangling Images:**
>
> ```sh
> docker rmi $(docker images -f dangling=true -q)
> ```
```
npm run action shadowbox/docker/start
```
1. **Send a Test Request**
You should be able to successfully query the management API:
```
curl --insecure https://[::]:8081/TestApiPrefix/server
```
To build the image only:
```
npm run action shadowbox/docker/build
```
Debug image:
```
docker run --rm -it --entrypoint=sh localhost/outline/shadowbox
```
Or a running container:
```
docker exec -it shadowbox sh
```
Delete dangling images:
```
docker rmi $(docker images -f dangling=true -q)
```
```sh
curl --insecure https://[::]:8081/TestApiPrefix/server
```
## Access Keys Management API
In order to utilize the Management API, you'll need to know the apiUrl for your Outline server.
You can obtain this information from the "Settings" tab of the server page in the Outline Manager.
Alternatively, you can check the 'access.txt' file under the '/opt/outline' directory of an Outline server. An example apiUrl is: https://1.2.3.4:1234/3pQ4jf6qSr5WVeMO0XOo4z.
The Outline Server provides a REST API for access key management. If you know the `apiUrl` of your Outline Server (e.g. `https://1.2.3.4:1234/3pQ4jf6qSr5WVeMO0XOo4z`), you can directly manage the server's access keys using HTTP requests:
See [Full API Documentation](https://redocly.github.io/redoc/?url=https://raw.githubusercontent.com/Jigsaw-Code/outline-server/master/src/shadowbox/server/api.yml).
The OpenAPI specification can be found at [api.yml](./server/api.yml).
1. **Find the Server's `apiUrl`:**
### Examples
- **Deployed with the Installation Script:** Run `grep "apiUrl" /opt/outline/access.txt | cut -d: -f 2-`
Start by storing the apiURL you see see in that file, as a variable. For example:
- **Deployed with the Outline Manager:** Check the "Settings" tab.
```
API_URL=https://1.2.3.4:1234/3pQ4jf6qSr5WVeMO0XOo4z
```
- **Local Deployments from Source:** `https://[::]:8081/TestApiPrefix`
You can then perform the following operations on the server, remotely.
1. **API Examples:**
List access keys
Replace `$API_URL` with your actual `apiUrl`.
```
curl --insecure $API_URL/access-keys/
```
- **List access keys:** `curl --insecure $API_URL/access-keys/`
Create an access key
- **Create an access key:** `curl --insecure -X POST $API_URL/access-keys`
```
curl --insecure -X POST $API_URL/access-keys
```
- **Get an access key (e.g. ID 1):** `curl --insecure $API_URL/access-keys/1`
Get an access key (e.g. get access key 1)
- **Rename an access key:** `curl --insecure -X PUT -F 'name=albion' $API_URL/access-keys/2/name`
```
curl --insecure $API_URL/access-keys/1
```
- **Remove an access key:** `curl --insecure -X DELETE $API_URL/access-keys/1`
Rename an access key
(e.g. rename access key 2 to 'albion')
- **Set a data limit for all access keys:** (e.g. limit outbound data transfer access keys to 1MB over 30 days) `curl --insecure -X PUT -H "Content-Type: application/json" -d '{"limit": {"bytes": 1000}}' $API_URL/experimental/access-key-data-limit`
```
curl --insecure -X PUT -F 'name=albion' $API_URL/access-keys/2/name
```
- **Remove the access key data limit:** `curl --insecure -X DELETE $API_URL/experimental/access-key-data-limit`
Remove an access key
(e.g. remove access key 2)
- **And more...**
```
curl --insecure -X DELETE $API_URL/access-keys/2
```
1. **Further Options:**
Set a data limit for all access keys
(e.g. limit outbound data transfer access keys to 1MB over 30 days)
```
curl -v --insecure -X PUT -H "Content-Type: application/json" -d '{"limit": {"bytes": 1000}}' $API_URL/experimental/access-key-data-limit
```
Remove the access key data limit
```
curl -v --insecure -X DELETE $API_URL/experimental/access-key-data-limit
```
Consult the [OpenAPI spec](./server/api.yml) and [documentation](https://redocly.github.io/redoc/?url=https://raw.githubusercontent.com/Jigsaw-Code/outline-server/master/src/shadowbox/server/api.yml) for more options.
## Testing
### Manual
After building a docker image with some local changes,
upload it to your favorite registry
(e.g. Docker Hub, quay.io, etc.).
1. Prerequisites
Then set your `SB_IMAGE` environment variable to point to the image you just
uploaded (e.g. `export SB_IMAGE=yourdockerhubusername/shadowbox`) and
run `npm run action server_manager/electron_app/start` and your droplet should be created with your
modified image.
- A locally built Docker image containing your modifications.
- Your Docker image uploaded to a registry (e.g., Docker Hub, Quay.io).
1. Setup
- **Environment Variable:** Set `SB_IMAGE` to the uploaded image location. Example:
```sh
export SB_IMAGE=yourdockerhubusername/shadowbox
```
- **Start the Server:**
```sh
npm run action shadowbox/docker/start
```
### Automated
To run the integration test:
1. **Build and Run:**
```
npm run action shadowbox/integration_test/run
```
```sh
npm run action shadowbox/integration_test/run
```
This will set up three containers and two networks:
This does the following:
```
client <-> shadowbox <-> target
```
- Sets up three containers (`client`, `shadowbox`, `target`) and two networks.
- Creates a user on `shadowbox`.
- Connects to `target` through `shadowbox` using a Shadowsocks `client`: `client <-> shadowbox <-> target`
`client` can only access `target` via shadowbox. We create a user on `shadowbox` then connect using the Shadowsocks client.
1. **Testing Changes to the Server Config:**
To test clients that rely on fetching a docker image from Dockerhub, you can push an image to your account and modify the
client to use your image. To push your own image:
If your change includes new fields in the server config which are needed at server start-up time, then you mey need to remove the pre-existing test config:
```
npm run action shadowbox/docker/build && docker tag quay.io/outline/shadowbox $USER/shadowbox && docker push $USER/shadowbox
```
- **Delete Existing Config:** `rm /tmp/outline/persisted-state/shadowbox_server_config.json`
If you need to test an unsigned image (e.g. your dev one):
```
DOCKER_CONTENT_TRUST=0 SB_IMAGE=$USER/shadowbox npm run action shadowbox/integration_test/run
```
You can add tags if you need different versions in different clients.
### Testing Changes to the Server Config
If your change includes new fields in the server config which are needed at server
start-up time, then you mey need to remove the pre-existing test config:
```
rm /tmp/outline/persisted-state/shadowbox_server_config.json
```
This will warn about deleting a write-protected file, which is okay to ignore. You will then need to hand-edit the JSON string in src/shadowbox/docker/start.action.sh.
- **Manually Edit:** You'll need to edit the JSON string within [`src/shadowbox/docker/start.action.sh`](src/shadowbox/docker/start.action.sh).