diff --git a/AGENTS.md b/AGENTS.md index 28af622..17c8855 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -29,49 +29,78 @@ Dockerfile Docker image build | `mtp_secure` | "Secure" randomized-packet-size protocol | | `mtp_dc_pool` / `mtp_down_conn` | Pooled/multiplexed connections to Telegram DCs | | `mtp_rpc` | RPC framing protocol between proxy and Telegram | -| `mtp_config` | Periodically fetches Telegram DC configuration | +| `mtp_config` | Periodically fetches Telegram DC configuration; in split mode exposes `backend_node/0` and remote-aware `get_downstream_pool/1` / `get_default_dc/0` | | `mtp_policy` / `mtp_policy_table` | Connection limit, blacklist, and whitelist rules | | `mtp_codec` / `mtp_aes_cbc` | Codec pipeline (MTProto framing + AES-CBC encryption) | | `mtp_abridged` / `mtp_full` / `mtp_intermediate` | MTProto transport codec variants | -| `mtp_metric` | Metrics/telemetry | +| `mtp_metric` | Metrics/telemetry; `passive_metrics/0` is role-aware | | `mtp_session_storage` | Replay-attack protection (session deduplication) | +| `mtproto_proxy_sup` | Root supervisor; calls `children(Role)` — children differ by `node_role` | +| `mtproto_proxy_app` | OTP application callback; `start/2` and `config_change/3` are role-gated | ### Process architecture +**Role overview** — what starts in each `node_role`: + +```mermaid +flowchart LR + subgraph BOTH["node_role=both (single server, default)"] + direction TB + F0["Ranch listeners\nmtp_handler\nmtp_session_storage\nmtp_policy_*"] + B0["mtp_config\nmtp_dc_pool\nmtp_down_conn"] + end + + subgraph SPLIT["Split mode (two servers)"] + direction TB + subgraph FRONT["node_role=front (domestic server)"] + F1["Ranch listeners\nmtp_handler\nmtp_session_storage\nmtp_policy_*"] + end + subgraph BACK["node_role=back (foreign server)"] + B1["mtp_config\nmtp_dc_pool\nmtp_down_conn"] + end + FRONT -- "Erlang distribution\n(TLS or VPN tunnel)" --> BACK + end +``` + ``` OTP supervision tree ──────────────────────────────────────────────────────────────────── -mtproto_proxy_sup (one_for_one) - ├── mtp_config (gen_server, singleton) - ├── mtp_session_storage (gen_server, singleton) - ├── mtp_dc_pool_sup (supervisor, simple_one_for_one) - │ └── mtp_dc_pool (gen_server, one per DC id, permanent) - ├── mtp_down_conn_sup (supervisor, simple_one_for_one) - │ └── mtp_down_conn (gen_server, one per Telegram TCP conn, temporary) - └── Ranch listeners (one per configured port: mtp_ipv4, mtp_ipv6, …) - └── mtp_handler (gen_server, one per client TCP conn, transient) +The supervisor is role-parameterised via the `node_role` config key +(`front | back | both`, default `both`). Each role starts a different +subset of children: + node_role=both (default — single server) + ├── mtp_config (gen_server, singleton) + ├── mtp_session_storage (gen_server, singleton) + ├── mtp_dc_pool_sup (supervisor, simple_one_for_one) + │ └── mtp_dc_pool (gen_server, one per DC id, permanent) + ├── mtp_down_conn_sup (supervisor, simple_one_for_one) + │ └── mtp_down_conn (gen_server, one per Telegram TCP conn, temporary) + └── Ranch listeners (one per configured port: mtp_ipv4, mtp_ipv6, …) + └── mtp_handler (gen_server, one per client TCP conn, transient) -Data-plane message flow (one client connection) -──────────────────────────────────────────────────────────────────── + node_role=front (domestic server — accepts Telegram clients) + ├── mtp_session_storage + ├── mtp_policy_table + ├── mtp_policy_counter + └── Ranch listeners → mtp_handler + + node_role=back (foreign server — connects to Telegram DCs) + ├── mtp_config + ├── mtp_dc_pool_sup → mtp_dc_pool + └── mtp_down_conn_sup → mtp_down_conn + +In split mode, the front node holds `back_node` in its config and +addresses back-node processes as `{RegisteredName, BackNode}`. +Multiple front nodes can share one back node. +``` + +**Data-plane message flow** — see [`doc/handler-downstream-flow.md`](doc/handler-downstream-flow.md) +for the full sequence diagram (pool lookup, steady-state data exchange, connection release) and +[`doc/migration-flow.md`](doc/migration-flow.md) for transparent DC connection rotation. +Both diagrams show the front/back node boundary. In `both` mode all processes share the same +node and there is no distribution overhead. - Telegram client Telegram server - │ │ - │ TCP (fake-TLS / obfuscated / secure) │ - ▼ ▼ - ┌─────────────┐ gen_server:call({send,Data}) ┌──────────────┐ raw TCP ┌──────────────┐ - │ mtp_handler │ ──────────────────────────────► │ mtp_down_conn│ ────────────► │ Telegram DC │ - │ (upstream) │ │ (downstream) │ ◄──────────── │ (middle srv) │ - │ │ ◄────────────────────────────── │ │ └──────────────┘ - └──────┬──────┘ gen_server:cast({proxy_ans}) └──────────────┘ - │ - │ on first data: mtp_config:get_downstream_safe/2 → picks (pool, down_conn) - │ on disconnect: cast({return, self()}) → releases slot - ▼ - ┌─────────────┐ - │ mtp_dc_pool │ — spawns mtp_down_conn via mtp_down_conn_sup when pool is empty - │ (per DC) │ - └─────────────┘ > **Naming note:** the terms "upstream" and "downstream" in the current code are the > opposite of what one might expect: @@ -80,12 +109,17 @@ Data-plane message flow (one client connection) > This will be renamed in a future refactor. -Key interactions -──────────────────────────────────────────────────────────────────── +**Key interactions:** + +``` mtp_handler → mtp_config : get_downstream_safe/2 — resolves DC id to a (pool_pid, down_conn_pid) pair on first - upstream data packet -mtp_handler → mtp_down_conn : send/2 (sync call) — forward client data + upstream data packet. + In split mode returns {PoolName, BackNode}; + uses erpc:call to check pool existence on + the back node. +mtp_handler → mtp_down_conn : send/2 (sync call) — forward client data; + in split mode this is a cross-node gen_server call mtp_down_conn → mtp_handler : cast {proxy_ans, …} — forward Telegram reply mtp_down_conn → mtp_handler : cast {close_ext, …} — Telegram closed stream mtp_handler → mtp_dc_pool : return/2 (cast) — release slot on disconnect @@ -147,9 +181,9 @@ There are three kinds of tests, each with a clear home: |------|-------|-------------| | **EUnit** (unit) | `src/*.erl`, `-ifdef(TEST)` blocks | Pure functions with no I/O: codec encode/decode round-trips, packet parsing helpers, crypto primitives | | **PropEr** (property-based) | `test/prop_mtp_.erl` | Codec/parser properties that should hold for *arbitrary* inputs — e.g. encode→decode identity, parser accepts all valid inputs, parser never crashes on random bytes | -| **Common Test** (integration) | `test/single_dc_SUITE.erl` | End-to-end behaviour involving a real listener + fake DC: protocol negotiation, policy enforcement, error handling visible at the TCP level (alerts sent, connections closed), domain fronting, replay protection | +| **Common Test** (integration) | `test/single_dc_SUITE.erl`, `test/split_dc_SUITE.erl` | End-to-end behaviour involving a real listener + fake DC: protocol negotiation, policy enforcement, error handling visible at the TCP level (alerts sent, connections closed), domain fronting, replay protection. `split_dc_SUITE` tests the same paths in split mode (front/back on separate `peer` nodes). | -**Rule of thumb:** if the behaviour is observable only over a TCP socket or requires a running application, it belongs in `single_dc_SUITE`. If it is a property of a pure function, add a PropEr property in the matching `prop_mtp_.erl`. If it is a targeted unit case for a specific input, use EUnit. +**Rule of thumb:** if the behaviour is observable only over a TCP socket or requires a running application, it belongs in `single_dc_SUITE`. If it requires two nodes communicating over Erlang distribution, it belongs in `split_dc_SUITE`. If it is a property of a pure function, add a PropEr property in the matching `prop_mtp_.erl`. If it is a targeted unit case for a specific input, use EUnit. **What changes need new tests:** @@ -199,6 +233,18 @@ Workflow: - All configuration options are documented in `src/mtproto_proxy.app.src`. - Config can be reloaded without restart: `make update-sysconfig && systemctl reload mtproto-proxy`. +### Split-mode config keys + +| Key | Node | Meaning | +|-----|------|---------| +| `node_role` | both | `front \| back \| both` (default `both` — single-server mode) | +| `back_node` | front only | Atom name of the back node, e.g. `'back@10.0.0.2'` | +| `external_ip` | back | Public IP of the back server (used in the RPC handshake AES key) | + +`mtp_config` is **not started on a front node** — never call `mtp_config:status()` or any +`mtp_config` function from code that runs on a front node. `get_downstream_safe/2` is the +correct entry point; it is role-aware and dispatches remotely when needed. + ## Debugging ### Enabling debug logs for a single module at runtime diff --git a/Makefile b/Makefile index 1d89693..55ca271 100644 --- a/Makefile +++ b/Makefile @@ -6,6 +6,23 @@ EPMD_SERVICE:=$(DESTDIR)/etc/systemd/system/epmd.service LOGDIR:=$(DESTDIR)/var/log/mtproto-proxy USER:=mtproto-proxy +# ROLE selects which config templates are used. +# Values: both (default, single server), front (domestic), back (foreign). +# For split mode: run `make init-config ROLE=front` / `make init-config ROLE=back` +# on each server, edit the resulting config files, then run `make ROLE=front` etc. +ROLE ?= both + +ifeq ($(ROLE),front) +SYS_CONFIG_SRC := config/sys.config.front.example +VM_ARGS_SRC := config/vm.args.front.example +else ifeq ($(ROLE),back) +SYS_CONFIG_SRC := config/sys.config.back.example +VM_ARGS_SRC := config/vm.args.back.example +else +SYS_CONFIG_SRC := config/sys.config.example +VM_ARGS_SRC := config/vm.args.example +endif + all: config/prod-sys.config config/prod-vm.args $(REBAR3) as prod release @@ -19,15 +36,25 @@ test: $(REBAR3) dialyzer $(REBAR3) cover -v -config/prod-sys.config: config/sys.config.example +config/prod-sys.config: $(SYS_CONFIG_SRC) [ -f $@ ] && diff -u $@ $^ || true cp -i -b $^ $@ -config/prod-vm.args: config/vm.args.example +config/prod-vm.args: $(VM_ARGS_SRC) [ -f $@ ] && diff -u $@ $^ || true cp -i -b $^ $@ @IP=$(shell curl -s -4 -m 10 http://ip.seriyps.com || curl -s -4 -m 10 https://digitalresistance.dog/myIp) \ && sed -i s/@0\.0\.0\.0/@$${IP}/ $@ +.PHONY: init-config +init-config: + cp $(SYS_CONFIG_SRC) config/prod-sys.config + cp $(VM_ARGS_SRC) config/prod-vm.args + @IP=$$(curl -s -4 -m 10 http://ip.seriyps.com || curl -s -4 -m 10 https://digitalresistance.dog/myIp) \ + && sed -i s/@0\.0\.0\.0/@$${IP}/ config/prod-vm.args; true + @echo "" + @echo "Config created from ROLE=$(ROLE) templates." + @echo "Edit config/prod-sys.config and config/prod-vm.args, then run: make [ROLE=$(ROLE)]" + user: sudo useradd -r $(USER) || true diff --git a/README.md b/README.md index cbbcdcf..8c94307 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,10 @@ Features * Automatic telegram configuration reload (no need for restarts once per day) * IPv6 for client connections * All configuration options can be updated without service restart +* Split-mode setup: run the client-facing part (front) on a domestic server and + the Telegram-facing part (back) on a foreign server, connected via Erlang distribution + (TLS or a censorship-resistant tunnel). Bypasses ISP blocks that target direct + domestic→foreign connections. Multiple front servers can share one back server. * Small codebase compared to official one, code is covered by automated tests * A lots of metrics could be exported (optional) @@ -94,7 +98,7 @@ It's ok to provide both `-a dd -a tls` to allow both protocols. If no `-a` optio ### To run with custom config-file 1. Get the code `git clone https://github.com/seriyps/mtproto_proxy.git && cd mtproto_proxy/` -2. Copy config templates `cp config/{vm.args.example,prod-vm.args}; cp config/{sys.config.example,prod-sys.config}` +2. Copy config templates `make init-config` (or manually: `cp config/{vm.args.example,prod-vm.args}; cp config/{sys.config.example,prod-sys.config}`) 3. Edit configs. See [Settings](#settings). 4. Build `docker build -t mtproto-proxy-erl .` 5. Start `docker run -d --network=host mtproto-proxy-erl` @@ -112,8 +116,7 @@ You need at least Erlang version 25! Recommended OS is Ubuntu 24.04. sudo apt install erlang-nox erlang-dev build-essential git clone https://github.com/seriyps/mtproto_proxy.git cd mtproto_proxy/ -cp config/vm.args.example config/prod-vm.args -cp config/sys.config.example config/prod-sys.config +make init-config # copies templates and auto-detects your server's IP # configure your port, secret, ad_tag. See [Settings](#settings) below. nano config/prod-sys.config make && sudo make install @@ -158,7 +161,12 @@ cd mtproto_proxy/ ### Create config file -see [Settings](#settings). +```bash +make init-config # copies sys.config.example → prod-sys.config and vm.args.example → prod-vm.args + # also auto-detects your server's public IP +``` + +Edit `config/prod-sys.config` — see [Settings](#settings) for all options. ### Build and install @@ -810,3 +818,177 @@ Number of connections ```bash /opt/mtp_proxy/bin/mtp_proxy eval 'lists:sum([maps:get(all_connections, L) || {_, L} <- maps:to_list(ranch:info())]).' ``` + +Split-mode setup (front + back) +-------------------------------- + +### Why split mode? + +Some censors (e.g. Roskomnadzor) monitor connections from domestic IPs to foreign +servers more aggressively than domestic-to-domestic traffic. A common workaround is +to split the proxy across two servers: + +- **Front server** — domestic (or neutral) IP, accepts Telegram client connections. +- **Back server** — foreign IP, connects to Telegram data centres. + +``` +Telegram client + │ + ▼ (443 / any port) + ┌────────────┐ + │ front node │ domestic server (mtp_handler, session storage, policies) + └─────┬──────┘ + │ inter-server link (VPN or TLS) + ▼ + ┌────────────┐ + │ back node │ foreign server (DC pool connections to Telegram) + └─────┬──────┘ + │ TCP to Telegram + ▼ + Telegram DC +``` + +Multiple front servers can share one back server — just set the same `back_node` +address in each front's config. The DC pools on the back multiplex all client +connections regardless of which front they came from. + +### Prerequisites + +- Erlang/OTP 25+ installed on **both** servers (same version recommended). +- Both servers can reach each other over TCP (the inter-server port, see below). +- The back server has outbound TCP access to Telegram's infrastructure (ports + are announced dynamically in [Telegram's proxy config](https://core.telegram.org/getProxyConfig). + +### Step 1 — secure the inter-server link + +The two servers communicate using Erlang's built-in distribution protocol, which +allows full remote control of the process. **You must restrict access to this +channel** to the two proxy servers only. There are two ways to do this: + +#### Option A: Censorship-resistant tunnel (recommended if front is in Russia) + +Standard VPN protocols (WireGuard, OpenVPN) are detectable and blocked on many +Russian ISPs. Use a DPI-resistant tunnel instead: + +- **[Shadowsocks](https://shadowsocks.org/)** — widely used, low overhead +- **[VLESS/XRay](https://github.com/XTLS/Xray-core)** — highly configurable, very hard to block +- **[Hysteria2](https://github.com/apernet/hysteria)** — QUIC-based, good for lossy links + +Set up the tunnel between front and back servers and use the **tunnel interface +addresses** in the node names (`front@10.8.0.1`, `back@10.8.0.2`). No extra +Erlang config is needed once the tunnel is up. + +> If the front server is **not** in a heavily censored region, WireGuard or +> IPsec work equally well and are simpler to set up. + +#### Option B: TLS distribution (no tunnel required) + +If you prefer not to run a separate tunnel, you can secure the distribution link +with mutual-TLS certificates. Run on the **back server**: + +```bash +# Step 1 — on the back server: initialise CA and generate back cert +./scripts/gen_dist_certs.sh init /etc/mtproto-proxy/dist + +# Step 2 — repeat for each front server you add +./scripts/gen_dist_certs.sh add-node /etc/mtproto-proxy/dist front +# (use a distinct name per front, e.g. front1, front2, …) +``` +Copy to each server (paths already substituted — no editing needed): + + * back server: ca.pem back.pem back.key ssl_dist.back.conf + * front server: ca.pem front.pem front.key ssl_dist.front.conf + +Place all files in `/etc/mtproto-proxy/dist/`. +Then uncomment `-proto_dist` and `-ssl_dist_optfile` in `vm.args` on each server. + + +Reference: [Erlang TLS distribution docs](https://www.erlang.org/doc/apps/ssl/ssl_distribution.html) + +### Step 2 — configure the back server + +Run on the **back server**: + +```bash +make init-config ROLE=back +``` + +This copies `config/sys.config.back.example` → `config/prod-sys.config` and +`config/vm.args.back.example` → `config/prod-vm.args`. Edit them: + +- In `vm.args`: set the **back** server's IP: `-name back@` and choose + a strong cookie string (`-setcookie ...`). +- In `sys.config`: set `external_ip` to the back server's public IP, or leave + `ip_lookup_services` to auto-detect it. +- If using TLS distribution (Option B): uncomment the `-proto_dist` / + `-ssl_dist_optfile` lines in `vm.args`. + +### Step 3 — configure the front server + +Run on the **front server**: + +```bash +make init-config ROLE=front +``` + +This copies `config/sys.config.front.example` → `config/prod-sys.config` and +`config/vm.args.front.example` → `config/prod-vm.args`. Edit them: + +- In `vm.args`: set the **front** server's IP: `-name front@` and the + **same** cookie string as the back. +- In `sys.config`: set `back_node` to the back node name you chose above + (e.g. `'back@10.8.0.2'`), configure `ports` / `secret` / `tag` as usual. +- If using TLS distribution: uncomment `-proto_dist` / `-ssl_dist_optfile` here too. + +### Step 4 — start in order + +Always **start the back server first**. The front server connects to it on +startup; if the back is not yet up the front will log a warning and retry +automatically on the next client connection. + +```bash +# On back server: +make ROLE=back && sudo make install && systemctl start mtproto-proxy + +# On front server (after back is up): +make ROLE=front && sudo make install && systemctl start mtproto-proxy +``` + +> **Why `ROLE=` on every `make`?** After `git pull`, if a new release updates +> `config/sys.config.back.example`, `make ROLE=back` detects the change (target +> is older than its prerequisite), shows a diff, and prompts before overwriting +> your `prod-sys.config`. Plain `make` (default `ROLE=both`) compares against +> `sys.config.example` instead and silently misses changes to the back/front +> templates. + +### Step 5 — verify + +On the front server, check that the back node is visible: + +```bash +/opt/personal_mtproxy/bin/mtproto_proxy remote_console +# In the Erlang shell: +nodes(). % should list the back node +``` + +On the back server, verify DC pools are running: + +```bash +/opt/personal_mtproxy/bin/mtproto_proxy remote_console +# In the Erlang shell: +mtp_config:status(). +``` + +### Firewall rules + +| Server | Port | Allow from | +|--------|-----------------------|---------------| +| back | 4369 (EPMD) | front IP only | +| back | 9199 (dist) | front IP only | +| front | 4369 (EPMD) | back IP only | +| front | 9199 (dist) | back IP only | +| front | 443 / your proxy port | anywhere | + +If you used a fixed dist port in `vm.args` (`inet_dist_listen_min/max 9199`), +only port 9199 needs to be open; otherwise allow the full 9199–9254 range plus +EPMD (4369). **Never expose the distribution port to the public internet.** diff --git a/config/vm.args.back.example b/config/vm.args.back.example index ceee2ac..30c414f 100644 --- a/config/vm.args.back.example +++ b/config/vm.args.back.example @@ -30,15 +30,16 @@ ## Option B: TLS distribution (no tunnel required) ## - Generate certificates: scripts/gen_dist_certs.sh init /etc/mtproto-proxy/dist ## - Place ca.pem, back.pem, back.key in /etc/mtproto-proxy/dist/ -## - Deploy ssl_dist.back.conf as /etc/mtproto-proxy/dist/ssl_dist.conf +## - Place ssl_dist.back.conf in /etc/mtproto-proxy/dist/ on the back server. ## - On each front node run: scripts/gen_dist_certs.sh add-node /etc/mtproto-proxy/dist front +## - Uncomment the lines below: ## -## -proto_dist inet_tls -## -ssl_dist_optfile /etc/mtproto-proxy/dist/ssl_dist.conf +# -proto_dist inet_tls +# -ssl_dist_optfile /etc/mtproto-proxy/dist/ssl_dist.back.conf ## ## Firewall: allow TCP on the distribution port (9199 below) only between ## the front and back servers, never to the public internet. ## -## -kernel inet_dist_listen_min 9199 -## -kernel inet_dist_listen_max 9199 +# -kernel inet_dist_listen_min 9199 +# -kernel inet_dist_listen_max 9199 ## ----------------------------------------------------------------------- diff --git a/config/vm.args.front.example b/config/vm.args.front.example index 278eba8..4f2333e 100644 --- a/config/vm.args.front.example +++ b/config/vm.args.front.example @@ -28,17 +28,18 @@ ## - If front is NOT in a censored region, WireGuard/IPsec work fine too. ## ## Option B: TLS distribution (no tunnel required) -## - On back server: scripts/gen_dist_certs.sh init /etc/mtproto-proxy/dist -## - Run per front: scripts/gen_dist_certs.sh add-node /etc/mtproto-proxy/dist front +## - On back server: `scripts/gen_dist_certs.sh init /etc/mtproto-proxy/dist` +## - Run per front: `scripts/gen_dist_certs.sh add-node /etc/mtproto-proxy/dist front` ## - Place ca.pem, front.pem, front.key in /etc/mtproto-proxy/dist/ here. -## - Deploy ssl_dist.front.conf as /etc/mtproto-proxy/dist/ssl_dist.conf +## - Place ssl_dist.front.conf in /etc/mtproto-proxy/dist/ on the front server. +## - Uncomment the lines below: ## -## -proto_dist inet_tls -## -ssl_dist_optfile /etc/mtproto-proxy/dist/ssl_dist.conf +# -proto_dist inet_tls +# -ssl_dist_optfile /etc/mtproto-proxy/dist/ssl_dist.front.conf ## ## Firewall: allow TCP on the distribution port (9199 below) only between ## the front and back servers, never to the public internet. ## -## -kernel inet_dist_listen_min 9199 -## -kernel inet_dist_listen_max 9199 +# -kernel inet_dist_listen_min 9199 +# -kernel inet_dist_listen_max 9199 ## ----------------------------------------------------------------------- diff --git a/doc/handler-downstream-flow.md b/doc/handler-downstream-flow.md index 9a48dac..ecd6f51 100644 --- a/doc/handler-downstream-flow.md +++ b/doc/handler-downstream-flow.md @@ -9,19 +9,30 @@ and the steady-state data flow that follows. - `mtp_down_conn` — multiplexed TCP connection to a Telegram DC - `Telegram DC` — the upstream Telegram data-centre server +In **split mode** (`node_role = front / back`) `mtp_handler` runs on the front +node and `mtp_dc_pool` / `mtp_down_conn` run on the back node. The pool is +addressed as `{mtp_dc_pool_N, BackNode}` — Erlang distribution makes the +`gen_server:call` and all subsequent casts transparent across nodes. Multiple +front nodes can share the same back node; the pools multiplex over all +upstream connections regardless of origin. + ```mermaid sequenceDiagram participant Client as Telegram client - participant Handler as mtp_handler - participant Pool as mtp_dc_pool - participant Down as mtp_down_conn + box LightBlue "FRONT node" + participant Handler as mtp_handler + end + box LightGreen "BACK node" + participant Pool as mtp_dc_pool + participant Down as mtp_down_conn + end participant TG as Telegram DC Client->>Handler: TCP connect + Hello bytes Note over Handler: decode protocol headers
(fake-TLS / obfuscated / secure)
stage: hello → tunnel - Note over Handler: resolve pool: whereis(dc_to_pool_name(DcId))
(registered name lookup; falls back to default DC from mtp_config if not found) + Note over Handler: resolve pool:
single-node: whereis(dc_to_pool_name(DcId))
split mode: erpc:call(BackNode, erlang, whereis, [PoolName])
→ returns {PoolName, BackNode}
(falls back to default DC from mtp_config if not found) Handler->>Pool: mtp_dc_pool:get(Pool, self(), Opts) [sync] Pool-->>Down: upstream_new(Handler, Opts) [cast] Pool->>Handler: Downstream pid diff --git a/doc/migration-flow.md b/doc/migration-flow.md index 3302f67..f7faed6 100644 --- a/doc/migration-flow.md +++ b/doc/migration-flow.md @@ -11,13 +11,26 @@ surviving (or freshly-started) DC connection transparently. - `mtp_handler` — one process per connected Telegram client - `mtp_down_conn (new)` — replacement downstream spawned by the pool +**Split-mode note:** in `front/back` split mode `mtp_handler` lives on the +front node and `mtp_dc_pool` / `mtp_down_conn` live on the back node. Every +message in the diagram below that crosses the front↔back boundary (the +`migrate` cast, `upstream_new` cast, `Pool->>Handler` reply, etc.) is carried +transparently by Erlang distribution — no code changes are needed because +Erlang PIDs and `gen_server` calls work across nodes unchanged. Process +monitors also fire on node disconnection, so a back-node restart causes all +affected front-node handlers to exit cleanly. + ```mermaid sequenceDiagram participant TG as Telegram - participant OldDown as mtp_down_conn (old) - participant Pool as mtp_dc_pool - participant Handler as mtp_handler - participant NewDown as mtp_down_conn (new) + box LightGreen "BACK node" + participant OldDown as mtp_down_conn (old) + participant Pool as mtp_dc_pool + participant NewDown as mtp_down_conn (new) + end + box LightBlue "FRONT node" + participant Handler as mtp_handler + end TG->>OldDown: TCP close diff --git a/scripts/gen_dist_certs.sh b/scripts/gen_dist_certs.sh new file mode 100755 index 0000000..c11e07b --- /dev/null +++ b/scripts/gen_dist_certs.sh @@ -0,0 +1,144 @@ +#!/usr/bin/env bash +# gen_dist_certs.sh — generate TLS certificates for Erlang distribution +# +# This script is designed to be run in TWO steps: +# +# Step 1 — run on the BACK server (once, to set up the CA and back cert): +# ./scripts/gen_dist_certs.sh init +# +# Step 2 — run on the BACK server for each FRONT server you add: +# ./scripts/gen_dist_certs.sh add-node +# +# Example — back server and two front servers: +# ./scripts/gen_dist_certs.sh init /etc/mtproto-proxy/dist +# ./scripts/gen_dist_certs.sh add-node /etc/mtproto-proxy/dist front1 +# ./scripts/gen_dist_certs.sh add-node /etc/mtproto-proxy/dist front2 +# +# Output files (all in ): +# ca.pem — CA certificate (copy to every server) +# back.pem / back.key — back node cert/key (keep on back server) +# .pem / .key — per-front cert/key (copy to that front server) +# ssl_dist.back.conf — ready-to-use Erlang TLS config for back +# (copy to /etc/mtproto-proxy/dist/ on back server) +# ssl_dist..conf — ready-to-use Erlang TLS config for that front +# (copy to /etc/mtproto-proxy/dist/ on that front server) +# +# After setup: +# On back server: ca.pem back.pem back.key ssl_dist.back.conf +# On each front: ca.pem .pem .key ssl_dist..conf +# Uncomment -proto_dist and -ssl_dist_optfile in vm.args on each server. +# +# Reference: https://www.erlang.org/doc/apps/ssl/ssl_distribution.html + +set -euo pipefail + +usage() { + echo "Usage:" >&2 + echo " $0 init # Step 1: CA + back cert" >&2 + echo " $0 add-node # Step 2: add a front cert" >&2 + exit 1 +} + +[ $# -lt 2 ] && usage + +CMD="$1" +OUTDIR="$2" +DAYS=3650 # 10 years + +mkdir -p "$OUTDIR" + +# ---- shared: ensure CA exists ---- +ensure_ca() { + if [ ! -f "$OUTDIR/ca.key" ]; then + echo "==> Generating CA key and certificate..." + openssl req -new -x509 -days "$DAYS" \ + -keyout "$OUTDIR/ca.key" \ + -out "$OUTDIR/ca.pem" \ + -nodes \ + -subj "/CN=mtproto-proxy-dist-ca" + chmod 600 "$OUTDIR/ca.key" + else + echo "==> Using existing CA: $OUTDIR/ca.key" + fi +} + +# ---- shared: generate one node cert signed by the CA ---- +gen_node_cert() { + local NAME="$1" + echo "==> Generating certificate for node: $NAME" + openssl req -new \ + -keyout "$OUTDIR/$NAME.key" \ + -out "$OUTDIR/$NAME.csr" \ + -nodes \ + -subj "/CN=$NAME" + openssl x509 -req -days "$DAYS" \ + -in "$OUTDIR/$NAME.csr" \ + -CA "$OUTDIR/ca.pem" \ + -CAkey "$OUTDIR/ca.key" \ + -CAcreateserial \ + -out "$OUTDIR/$NAME.pem" + rm -f "$OUTDIR/$NAME.csr" + chmod 600 "$OUTDIR/$NAME.key" + echo " -> $OUTDIR/$NAME.pem $OUTDIR/$NAME.key" +} + +# ---- shared: write a ready-to-use ssl_dist..conf for one node ---- +write_ssl_conf() { + local NAME="$1" + local CONF="$OUTDIR/ssl_dist.$NAME.conf" + cat > "$CONF" << EOF +%% Erlang TLS distribution config for node: $NAME +%% Copy this file to /etc/mtproto-proxy/dist/ on the '$NAME' server. +[ + {server, + [{certfile, "/etc/mtproto-proxy/dist/$NAME.pem"}, + {keyfile, "/etc/mtproto-proxy/dist/$NAME.key"}, + {cacertfile, "/etc/mtproto-proxy/dist/ca.pem"}, + {verify, verify_peer}, + {fail_if_no_peer_cert, true}]}, + {client, + [{certfile, "/etc/mtproto-proxy/dist/$NAME.pem"}, + {keyfile, "/etc/mtproto-proxy/dist/$NAME.key"}, + {cacertfile, "/etc/mtproto-proxy/dist/ca.pem"}, + {verify, verify_peer}]} +]. +EOF + echo " -> $CONF" +} + +case "$CMD" in + init) + ensure_ca + gen_node_cert "back" + write_ssl_conf "back" + echo "" + echo "==> Done. Back server setup complete." + echo "" + echo "Next:" + echo " 1. On the back server, copy to /etc/mtproto-proxy/dist/:" + echo " ca.pem back.pem back.key" + echo " 2. Copy ssl_dist.back.conf to /etc/mtproto-proxy/dist/" + echo " 3. Uncomment -proto_dist / -ssl_dist_optfile in vm.args." + echo " 4. For each front server, run:" + echo " $0 add-node $OUTDIR " + ;; + add-node) + [ $# -lt 3 ] && usage + NAME="$3" + if [ ! -f "$OUTDIR/ca.key" ]; then + echo "ERROR: CA not found in $OUTDIR. Run '$0 init $OUTDIR' first." >&2 + exit 1 + fi + gen_node_cert "$NAME" + write_ssl_conf "$NAME" + echo "" + echo "==> Done. Certificate for '$NAME' generated." + echo "" + echo "Copy to the '$NAME' front server:" + echo " ca.pem $NAME.pem $NAME.key" + echo " ssl_dist.$NAME.conf (copy to /etc/mtproto-proxy/dist/ on that server)" + ;; + *) + usage + ;; +esac