Opengram: designing a server that speaks MTProto

Telegram compatibility is not "supporting an API". It means adopting someone else's decisions wholesale: their binary protocol, their handshake cryptography, their update model. The client will not meet you halfway — it either receives exactly the bytes it expects, or it drops the connection without explaining why.

Opengram is a self-hosted messenger in Go that speaks MTProto for real. What follows isn't a protocol tour but a look at the three places where architecture decides the outcome: two front doors into one system, message ordering, and the storage layout.

#Two front doors, one core

Opengram runs in two modes, and that's the first architectural decision. Full mode — real Telegram clients connect over MTProto. Embedded mode — other apps in the fleet get 1:1 chat through plain REST+WS and a signed host-app token.

The tempting move is to build two systems. The failure mode is familiar: two implementations of "send a message" will diverge within a couple of months, and they will diverge silently.

So the boundary sits here: a gateway only handles protocol — parse bytes, hold the session, answer in the right shape. Everything that means "send a message", "open a dialog", "read history" lives in internal/messaging and is called identically by both doors. In this design the MTProto gateway is a translator, not a second server.

The side effect is pleasant: a feature is written once and exists in both modes immediately.

#Ordering is the hard part

Telegram doesn't push "just events" to a client. Every user has a pts — a monotonic state counter. The client remembers the last one it saw, and when it comes back online it asks updates.getDifference for everything it missed. That imposes hard requirements:

• ▸numbers are strictly consecutive, with no gaps;
• ▸a number, once issued, can never be reissued with different content;
• ▸the record of what happened must survive a crash landing exactly between issuing the number and writing the event.

That last one is where homegrown implementations usually break. The tempting shape is: take a counter in Redis, increment it, then write the event to the real store. The process can die between those two steps — and the user's timeline keeps a permanent hole, so the client will forever ask for a diff that doesn't exist.

Opengram's answer is deliberately boring: the number and the record of it are issued inside a single Postgres transaction.

The row lock (FOR UPDATE) serializes number issuance per user — not across the database. Writing the pending row in the same transaction guarantees a number only ever exists together with a description of what it was issued for. And ON CONFLICT DO NOTHING on the idempotency key makes retries safe: a client that resent after a timeout gets the same pts back, not a second message.

Two background drains then carry that row into Scylla (the durable update log) and Redpanda (the bus feeding search and everything else). Both are idempotent and both survive a rerun, which is exactly why they're allowed to lag or crash. That's the central trade: consistency is bought in one place — the transaction — and everything downstream is allowed to catch up.

One honest limit: the sequencer runs single-replica today. The shard-lease table exists and the schema exercises it, but no cross-replica request routing is written. That's an addition on top, not a redesign — but until it's built, claiming otherwise would be a lie.

#Why there are several stores

Different data behaves differently, and forcing it into one database usually ends with that database serving both cases badly.

The detail that matters: access to timelines is fenced behind its own service (message-data), the only component that talks to Scylla. Not for layer aesthetics — so the partitioning scheme can't be quietly violated from some incidental corner of the code. It's the kind of boundary that's cheap while a project is small and impossible to introduce later.

#A separate problem: not lying to yourself about progress

Telegram has roughly 781 callable methods — a number obtained by reflecting over the gotd/td client library, not by eyeballing. Implementing all of them isn't the goal. The real problem is different: in a project this size it is very easy to lose track of what actually works.

So every method lives in a manifest in one of three states: implemented, intentionally unsupported, planned. A transition into "implemented" doesn't count without a real e2e test against live infrastructure. The manifest once caught 188 methods still filed as "planned" when the code behind them already existed — and it just as reliably stops the number in the README from quietly turning into marketing.

A second decision of the same family: any correctly decoded method with no handler gets a well-formed protocol-level rpc_error with METHOD_NOT_IMPLEMENTED. Not a dropped connection, not an empty success. The client learns the truth and keeps working — and the logs show exactly what's missing.

#What to take from this

• ▸A protocol is a transport layer. Everything that can be lifted out of it into a shared core should be, or the second front door will inevitably drift from the first.
• ▸Atomicity is needed at exactly one point — where the sequence number is issued. Everything else can be catch-up work, provided it's idempotent.
• ▸Pick a store by the shape of the load, not by habit; but keep exactly one owner in front of each.
• ▸Measure progress with a machine. A number no test can verify grows optimistic over time.