mtproto_proxy/AGENTS.md

# AGENTS.md

## Project Overview

This is a high-performance **Telegram MTProto proxy** written in **Erlang/OTP**. It sits between Telegram
clients and Telegram servers, helping users bypass DPI-based censorship. It supports multiple anti-detection
protocols (fake-TLS, obfuscated/secure), connection multiplexing, replay attack protection, domain fronting,
and flexible connection policies.

## Repository Layout

```
src/              Erlang source files (OTP application)
test/             EUnit, Common Test, and PropEr test suites + benchmarks
config/           Example configs (sys.config.example, vm.args.example)
rebar.config      Build tool configuration and dependencies
Makefile          Build, test, install targets
start.sh          Foreground start script for development
Dockerfile        Docker image build
```

### Key source modules

| Module                                           | Role                                                  |
|--------------------------------------------------|-------------------------------------------------------|
| `mtp_handler`                                    | Accepts client TCP connections (Ranch listener)       |
| `mtp_obfuscated`                                 | Obfuscated MTProto protocol (client-side codec)       |
| `mtp_fake_tls`                                   | Fake-TLS protocol (mimics TLSv1.3 + HTTP/2)           |
| `mtp_secure`                                     | "Secure" randomized-packet-size protocol              |
| `mtp_dc_pool` / `mtp_down_conn`                  | Pooled/multiplexed connections to Telegram DCs        |
| `mtp_rpc`                                        | RPC framing protocol between proxy and Telegram       |
| `mtp_config`                                     | Periodically fetches Telegram DC configuration; in split mode exposes `backend_node/0` and remote-aware `get_downstream_pool/1` / `get_default_dc/0` |
| `mtp_policy` / `mtp_policy_table`                | Connection limit, blacklist, and whitelist rules      |
| `mtp_codec` / `mtp_aes_cbc`                      | Codec pipeline (MTProto framing + AES-CBC encryption) |
| `mtp_abridged` / `mtp_full` / `mtp_intermediate` | MTProto transport codec variants                      |
| `mtp_metric`                                     | Metrics/telemetry; `passive_metrics/0` is role-aware  |
| `mtp_session_storage`                            | Replay-attack protection (session deduplication)      |
| `mtproto_proxy_sup`                              | Root supervisor; calls `children(Role)` — children differ by `node_role` |
| `mtproto_proxy_app`                              | OTP application callback; `start/2` and `config_change/3` are role-gated |

### Process architecture

**Role overview** — what starts in each `node_role`:

```mermaid
flowchart LR
    subgraph BOTH["node_role=both  (single server, default)"]
        direction TB
        F0["Ranch listeners\nmtp_handler\nmtp_session_storage\nmtp_policy_*"]
        B0["mtp_config\nmtp_dc_pool\nmtp_down_conn"]
    end

    subgraph SPLIT["Split mode  (two servers)"]
        direction TB
        subgraph FRONT["node_role=front  (domestic server)"]
            F1["Ranch listeners\nmtp_handler\nmtp_session_storage\nmtp_policy_*"]
        end
        subgraph BACK["node_role=back  (foreign server)"]
            B1["mtp_config\nmtp_dc_pool\nmtp_down_conn"]
        end
        FRONT -- "Erlang distribution\n(TLS or VPN tunnel)" --> BACK
    end
```

```
OTP supervision tree
────────────────────────────────────────────────────────────────────
The supervisor is role-parameterised via the `node_role` config key
(`front | back | both`, default `both`).  Each role starts a different
subset of children:

  node_role=both  (default — single server)
  ├── mtp_config            (gen_server, singleton)
  ├── mtp_session_storage   (gen_server, singleton)
  ├── mtp_dc_pool_sup       (supervisor, simple_one_for_one)
  │    └── mtp_dc_pool      (gen_server, one per DC id, permanent)
  ├── mtp_down_conn_sup     (supervisor, simple_one_for_one)
  │    └── mtp_down_conn    (gen_server, one per Telegram TCP conn, temporary)
  └── Ranch listeners       (one per configured port: mtp_ipv4, mtp_ipv6, …)
       └── mtp_handler      (gen_server, one per client TCP conn, transient)

  node_role=front  (domestic server — accepts Telegram clients)
  ├── mtp_session_storage
  ├── mtp_policy_table
  ├── mtp_policy_counter
  └── Ranch listeners → mtp_handler

  node_role=back  (foreign server — connects to Telegram DCs)
  ├── mtp_config
  ├── mtp_dc_pool_sup → mtp_dc_pool
  └── mtp_down_conn_sup → mtp_down_conn

In split mode, the front node holds `back_node` in its config and
addresses back-node processes as `{RegisteredName, BackNode}`.
Multiple front nodes can share one back node.
```

**Data-plane message flow** — see [`doc/handler-downstream-flow.md`](doc/handler-downstream-flow.md)
for the full sequence diagram (pool lookup, steady-state data exchange, connection release) and
[`doc/migration-flow.md`](doc/migration-flow.md) for transparent DC connection rotation.
Both diagrams show the front/back node boundary. In `both` mode all processes share the same
node and there is no distribution overhead.


> **Naming note:** the terms "upstream" and "downstream" in the current code are the
> opposite of what one might expect:
> `upstream` = the client-side connection (`mtp_handler`),
> `downstream` = the Telegram-server-side connection (`mtp_down_conn`).
> This will be renamed in a future refactor.


**Key interactions:**

```
mtp_handler  → mtp_config       : get_downstream_safe/2 — resolves DC id to
                                  a (pool_pid, down_conn_pid) pair on first
                                  upstream data packet.
                                  In split mode returns {PoolName, BackNode};
                                  uses erpc:call to check pool existence on
                                  the back node.
mtp_handler  → mtp_down_conn    : send/2 (sync call) — forward client data;
                                  in split mode this is a cross-node gen_server call
mtp_down_conn → mtp_handler     : cast {proxy_ans, …} — forward Telegram reply
mtp_down_conn → mtp_handler     : cast {close_ext, …} — Telegram closed stream
mtp_handler  → mtp_dc_pool      : return/2 (cast) — release slot on disconnect
mtp_dc_pool  → mtp_down_conn    : upstream_new/upstream_closed (cast)
mtp_dc_pool  → mtp_down_conn_sup: start_conn/2 — spawn new TCP conn to Telegram
mtp_down_conn → mtp_config      : get_netloc/1, get_secret/0 — read DC address
                                  and proxy secret for RPC handshake
mtp_config   → mtp_dc_pool_sup  : start_pool/1 — create pool when new DC seen
```

## Build

Requires Erlang/OTP 25+.

```bash
# Install dependencies and compile
./rebar3 compile

# Build a production release (requires config/prod-sys.config and config/prod-vm.args)
cp config/sys.config.example config/prod-sys.config
cp config/vm.args.example config/prod-vm.args
make
```

## Running Locally (dev)

```bash
./rebar3 shell   # starts an Erlang shell with the app loaded (easiest for dev/debugging)
```

`start.sh` is the Docker container entry-point; use `rebar3 shell` for local development instead.

## Testing

Run the full test suite (xref, eunit, common test, property-based tests, dialyzer, coverage):

```bash
make test
```

Individual steps:

```bash
./rebar3 xref              # cross-reference checks (undefined calls, unused locals)
./rebar3 eunit -c          # unit tests
./rebar3 ct -c             # common tests (integration, uses test/test-sys.config)
./rebar3 proper -c -n 50   # PropEr property-based tests (50 runs each)
./rebar3 dialyzer          # type analysis
./rebar3 cover -v          # coverage report
```

Always run `make test` before committing. Fix all xref warnings and dialyzer errors — they are treated as errors.

### Test organisation — where to add new tests

There are three kinds of tests, each with a clear home:

| Kind | Files | When to add |
|------|-------|-------------|
| **EUnit** (unit) | `src/*.erl`, `-ifdef(TEST)` blocks | Pure functions with no I/O: codec encode/decode round-trips, packet parsing helpers, crypto primitives |
| **PropEr** (property-based) | `test/prop_mtp_<module>.erl` | Codec/parser properties that should hold for *arbitrary* inputs — e.g. encode→decode identity, parser accepts all valid inputs, parser never crashes on random bytes |
| **Common Test** (integration) | `test/single_dc_SUITE.erl`, `test/split_dc_SUITE.erl` | End-to-end behaviour involving a real listener + fake DC: protocol negotiation, policy enforcement, error handling visible at the TCP level (alerts sent, connections closed), domain fronting, replay protection. `split_dc_SUITE` tests the same paths in split mode (front/back on separate `peer` nodes). |

**Rule of thumb:** if the behaviour is observable only over a TCP socket or requires a running application, it belongs in `single_dc_SUITE`. If it requires two nodes communicating over Erlang distribution, it belongs in `split_dc_SUITE`. If it is a property of a pure function, add a PropEr property in the matching `prop_mtp_<module>.erl`. If it is a targeted unit case for a specific input, use EUnit.

**What changes need new tests:**

- **New codec or protocol module** → PropEr round-trip property in `prop_mtp_<module>.erl` + a CT `echo_*_case` in `single_dc_SUITE`
- **New protocol error path** → CT case that sends the triggering byte sequence over TCP and asserts the exact response (alert bytes, metric counter, connection close)
- **New policy or config option** → CT case that sets the env, exercises the path, resets env in `{post, Cfg}`
- **New parser clause or binary pattern** → PropEr property verifying the clause accepts all valid inputs and a targeted EUnit/PropEr case for boundary/malformed inputs
- **Security-critical paths** (replay detection, session storage, digest validation) → CT case; also consider PropEr for the pure crypto/comparison functions

**Naming conventions:**

- CT cases: `<description>_case/1` — auto-discovered by `all/0`
- PropEr properties: `prop_<description>/0` (or `/1` with a `doc` clause)
- Each CT case must implement `{pre, Cfg}` / `{post, Cfg}` / `Cfg when is_list(Cfg)` clauses and call `setup_single` / `stop_single` to avoid resource leaks

### Debugging CT failures

When `rebar3 ct` (or `make test`) reports failures, **do not rely on the terminal output** — it is truncated and shows only the last error. Instead, go straight to the HTML logs:

```
_build/test/logs/ct_run.<timestamp>/lib.mtproto_proxy.logs/run.<timestamp>/
```

Key files:
- `suite.log` — machine-readable summary; `=case` lines show test order, `=result failed` shows which failed
- `single_dc_suite.<test_name>.html` — full log for one test case (strip HTML tags to read: `sed 's/<[^>]*>//g'`)
- `suite.log.html` / `index.html` — human-readable in a browser

Workflow:
1. Run `make test` — note how many pass/fail
2. Check `suite.log` for `=case` ordering and `=result failed` to identify the failing test
3. Read that test's `.html` log for the full stacktrace and system reports
4. Fix, then re-run `make test`. If tests still fail spuriously, try `rm -rf _build/test && make test` to clear stale test artifacts (removing only `_build/test` is faster than a full clean build).

## Code Style

- Language: **Erlang**. Follow standard Erlang OTP conventions.
- Module names use `snake_case`; all prefixed with `mtp_` (or `mtproto_` for top-level app modules).
- Keep modules focused; each codec/protocol has its own module.
- Avoid adding dependencies — the dep list in `rebar.config` is intentionally minimal (Ranch + psq).
- Comments use `%%` (module-level) or `%` (inline). Don't over-comment obvious code.
- Codecs are implemented as layered pipelines via `mtp_codec` — follow this pattern for new protocols.

## Configuration

- Config lives in `config/prod-sys.config` (Erlang term format). Do **not** edit `src/mtproto_proxy.app.src` — it documents defaults only.
- All configuration options are documented in `src/mtproto_proxy.app.src`.
- Config can be reloaded without restart: `make update-sysconfig && systemctl reload mtproto-proxy`.

### Split-mode config keys

| Key | Node | Meaning |
|-----|------|---------|
| `node_role` | both | `front \| back \| both` (default `both` — single-server mode) |
| `back_node` | front only | Atom name of the back node, e.g. `'back@10.0.0.2'` |
| `external_ip` | back | Public IP of the back server (used in the RPC handshake AES key) |

`mtp_config` is **not started on a front node** — never call `mtp_config:status()` or any
`mtp_config` function from code that runs on a front node.  `get_downstream_safe/2` is the
correct entry point; it is role-aware and dispatches remotely when needed.

## Debugging

### Enabling debug logs for a single module at runtime

The primary log level is `info`. To see `?LOG_DEBUG` messages from one module without
flooding the log with debug output from all of OTP:

```erlang
% In the running Erlang shell (e.g. via: sudo /opt/personal_mtproxy/bin/mtproto_proxy remote_console)
logger:set_module_level(mtp_handler, debug).   % override primary gate for this module only
logger:set_handler_config(default, level, debug).  % let the file handler pass debug through
```

This works because `set_module_level` bypasses the primary level check *only* for the
named module — no other module's debug messages are affected. The handler level change
is required because the `default` file handler has its own `level => info` guard.

To revert:

```erlang
logger:unset_module_level(mtp_handler).
logger:set_handler_config(default, level, info).
```

Both settings are in-memory only and reset on restart.

## Security Considerations

- Do **not** commit real secrets, tags, or credentials into config files.
- Replay attack protection (`replay_check_session_storage`) must stay correct — the session storage logic is security-critical.
- The fake-TLS and obfuscated protocol implementations must stay byte-exact with the reference (`../MTProxy/`).
- When modifying crypto code (`mtp_aes_cbc`, `mtp_obfuscated`, `mtp_fake_tls`), verify against reference
  implementations: `../MTProxy/` (C), `../mtprotoproxy/` (Python), `../mtg/` (Go), `../telemt/` (Rust).

## Reference Implementations

*Feature comparison last verified: 2026-04-03. These projects evolve independently — re-check if significant time has passed.*

Reference implementations may or may not be checked out in sibling directories. If a directory is missing, clone it from GitHub:

| Implementation        | Sibling dir        | GitHub URL                                   |
|-----------------------|--------------------|----------------------------------------------|
| MTProxy (C, official) | `../MTProxy/`      | https://github.com/TelegramMessenger/MTProxy |
| mtprotoproxy (Python) | `../mtprotoproxy/` | https://github.com/alexbers/mtprotoproxy     |
| mtg (Go)              | `../mtg/`          | https://github.com/9seconds/mtg              |
| telemt (Rust)         | `../telemt/`       | https://github.com/telemt/telemt             |

There are two ways a proxy can connect to Telegram on the backend:

- **Middle proxy (RPC/multiplexed)**: the proxy speaks the Telegram internal RPC protocol to a Telegram
  "middle server". Many client connections are multiplexed over a small number of long-lived proxy→Telegram
  connections. Required for `ad_tag` (promoted channels) support.
- **Direct**: the proxy opens a new raw TCP connection to a Telegram DC per client connection.
  Simpler, but no `ad_tag` support and more connections to Telegram.

Client-side connection protocols (what the Telegram app uses to connect to the proxy):

| Implementation                   | Classic (no prefix) | Secure (`dd`)    | Fake-TLS (`ee`) | Domain fronting²                 | Backend mode                                        |
|----------------------------------|---------------------|------------------|-----------------|----------------------------------|-----------------------------------------------------|
| **mtproto_proxy** (this, Erlang) | ✅                  | ✅               | ✅              | ✅ (`domain_fronting` config)    | Middle proxy (multiplexed)                          |
| **MTProxy** (C, official)        | ✅                  | ✅               | ✅              | ✅ (`--domain` flag)             | Middle proxy (multiplexed)                          |
| **mtprotoproxy** (Python)        | ✅                  | ✅               | ✅              | ✅ (`TLS_DOMAIN` config)         | Both (`USE_MIDDLE_PROXY`, auto-enabled on `AD_TAG`) |
| **mtg** (Go)                     | ❌ dropped in v2    | ❌ dropped in v2 | ✅ only         | ✅ (`domain-fronting-port` flag) | Direct (per-client connection)                      |
| **telemt** (Rust)                | ✅                  | ✅               | ✅              | ✅ (TLS-fronting)                | Both (configurable: `use_middle_proxy`)             |

² **Domain fronting**: when a fake-TLS handshake fails (non-proxy client, e.g. a real browser or DPI probe),
  the proxy forwards the connection to the real host from the TLS SNI field, making the proxy indistinguishable
  from a normal HTTPS server. Without this, a failed handshake results in an abrupt close, which itself can
  be a detection signal.

Key takeaways:
- **mtproto_proxy** and **MTProxy** always use the middle proxy (multiplexed) backend.
- **mtprotoproxy** and **telemt** support both backend modes (middle proxy auto-enabled when an ad_tag is configured).
- **mtg** v2 intentionally dropped `dd`/classic support and ad_tag/middle-proxy in favour of simplicity;
  it only accepts `ee` (fake-TLS) secrets and always connects directly to Telegram DCs.