Conversation
The bot never registered its commands, so Telegram's in-app "/" menu button
had nothing to show and the commands were only discoverable by reading
/mp_help. Register the list with setMyCommands so commands are tappable.
Lists are scoped to mirror the role model in _process_cmd rather than
exposing the whole admin surface to every user:
- default scope -> 5 public self-service commands
- admin/superadmin chats via admins.conf -> 18 command control plane
- root chat and superadmin admins -> those 18 plus the four commands
gated on the superadmin role (/mp_remove, /mp_restart, /mp_update,
/mp_lockdown)
The command tables live once, in the manager. The generated bot daemon
re-runs `mtproxymax telegram sync-commands` on boot instead of carrying its
own copy, so the menu self-heals and picks up newly added admins. Syncing is
best-effort throughout: a Telegram outage must never break setup or the poll
loop, and revoking an admin calls deleteMyCommands so a stale admin menu is
not left behind.
Also add an explicit `telegram sync-commands` subcommand for manual re-sync.
Foundation for interactive inline-keyboard menus. No user-visible change:
nothing attaches a keyboard yet, and the callback dispatcher is a stub.
Additions (all inside the generated bot daemon, which is self-contained
because the heredoc is quoted and inherits nothing from the manager):
- Bot API primitives: _tg_post_method as a single curl chokepoint, plus
tg_send_kb/tg_send_to_kb/tg_edit/tg_edit_markup/tg_answer_cb. reply_markup
travels as an ordinary urlencoded form field, so no Content-Type header and
no temp file, and the token stays out of the process list.
- Message chunking (_tg_chunk_text/_tg_send_pieces). Telegram caps a message
at 4096 units and bot messages are built by appending one line per secret,
so a large enough fleet produced a message that failed to send outright. A
400 from a malformed Markdown entity now retries without parse_mode,
costing the formatting instead of the whole message.
- callback_data codec (_cb_enc/_cb_dec/_cb_label_ok). The cap is 64 bytes and
one over-long payload makes Telegram reject the entire reply_markup, so the
encoder refuses rather than truncating — a truncated payload would decode
into a different, still-valid target.
Fixes two latent bugs in the getUpdates path:
- The no-python3 fallback extracted text and chat id in two independent
grep|tail passes and paired them by position, so a batch lost every update
but the last and could pair one update's text with another's chat id.
- callback_query updates were never parsed. The extractor read
r.get('message',{}), which is empty for a callback, so _process_cmd ran on
empty input, wrote the offset, and the callback was confirmed and never
redelivered.
The replacement awk extractor is a character scanner rather than a regex
pass, since a regex cannot tell whether a brace or quote sits inside a string
literal — a command's text can contain both. Both extractors now emit
identical records, and the tests assert that byte-for-byte, because the awk
path is the only one available without python3 (notably on Alpine, which this
project supports via OpenRC).
Also makes _tg_have_python probe by executing rather than by `command -v`:
the Windows Store ships a python3.exe alias that is on PATH but fails when
run. And drops the offset write from _process_cmd into _consume_updates, so
it advances per consumed record and a parser failure mid-batch redelivers the
tail instead of losing it.
Tapping now navigates: a hub, a paginated user list, a per-user detail card with its own actions, and a confirmation step before anything destructive. All navigation state travels in callback_data, so there is no server-side session store and an old message's buttons keep working. Security model: callback_data is entirely attacker-controlled — a user can send any payload they like — so every tap is re-authorised against the same rules _process_cmd applies. TG_CB_CAPS is the single source of truth for "what capability does this action need", and BOTH the renderer (which buttons to show) and the enforcer (whether to act) read it, so the two cannot drift. A table-driven test fails if it stops matching _process_cmd's gates. The four refusals mirror _process_cmd exactly: public actions run before any role check, an unauthenticated chatter is answered but ignored silently, a recognised-but-underprivileged role is refused loudly and audited, and an unrecognised role string fails closed to public-only (admins.conf is a plain file an operator can hand-edit). Answering the callback is structural rather than a discipline: _cb_dispatch never answers and _process_callback always does, exactly once, on every path including denial, a stale menu and an unknown namespace. Without that the client spins forever. Also attaches a role-filtered button bar to /start, /mp_status, /mp_secrets, /mp_traffic and /mp_help. Inline buttons carry their message implicitly via callback_query.message.message_id, so a reply becomes a live dashboard with no stored message id. Fixes a pagination bug found while testing: `local _i _start=$(( page * per )) _end=$(( _start + per ))` expands _start against the OUTER scope, so _end came out as per rather than start+per and every page after the first rendered empty. tests/test_telegram_reseller_rbac.sh gains stubs for the two new senders and exposes the menu block, since /start now routes through tg_send_to_kb and builds its button bar via _tg_button_bar.
This was referenced Sep 17, 2026
Two pieces of reply plumbing, both in service of making the bot usable from
buttons and readable on a phone.
Pending input
-------------
Inline buttons cannot collect typed text, so a flow that needs a value arms a
prompt for that chat and takes the next plain message from it as the answer.
State is one line per chat in relay_stats/.tg_pending; the daemon is its only
reader and only writer, so the rewrite needs no lock.
The consumption rule is the part that has to be right, because a stale or
misfiled entry would swallow a user's next real message:
- a slash command always escapes, and clears the prompt on the way out;
- re-arming replaces rather than appends;
- an expired row is dropped on read;
- the answer runs as the sender, after the role lookup.
The first consumer is the "add user" flow, which validates the label before it
reaches the CLI and hands back a connect link built from the secret it created.
No code box
-----------
Every fenced site wrapped CLI output in ```…```, which Telegram draws as a
monospace box with a copy button — the wrong shape for a status reply.
The replacement is one line per fact. Padding is not available as an alignment
mechanism: printf "%-14s" pads with letters, and letters do not have a uniform
advance width in a proportional font, so padded columns collapse outside a
fence. Block-element glyphs are a different case and stay inline.
Two fixes fall out of reading vouchers.conf instead of the CLI's padded table:
- /mp_voucher create showed the WRONG rows. It read the active list back and
did `tail -n +3`, which skipped the first voucher and re-announced an
existing code instead of the one just generated. It now snapshots the
active count and reports only the rows past it.
- /mp_fleet rendered one block per node rather than a padded table, so the
columns survive without a fence.
Adds the manage card behind a user (u:m) with a picker per limit field —
quota, connections, IPs, expiry and the monthly quota-reset day — and a
template picker that applies a saved template to that secret.
The write path is the part that matters. The obvious verb for "set the quota"
is `secret setlimits <label> <conns> <ips> <quota> <expires>`, but
secret_set_limits reads "0" as UNLIMITED rather than "leave alone", so tapping
"quota: 10G" through it would silently clear the connection and IP caps too.
Every commit goes through the per-field `secret setlimit <label> <field>
<value>` form instead, and the tests pin that a quota change never calls
setlimits at all.
Values arrive through callback_data, which is attacker-controlled, and the
label can also come back out of secrets.conf, which is hand-editable, so both
are re-validated before they reach the CLI.
Two smaller things worth naming:
- Relative expiry ("+30d") and "never" mean different CLI verbs — `secret
extend` does the calendar arithmetic, `secret setlimit expires 0` clears
the date — so one picker drives both.
- A custom value the presets do not cover arms a pending prompt whose verb is
the same one the preset commits through, so the typed value lands on
exactly the same validated path rather than a second, laxer one.
_kb_spec drops a button whose payload would exceed 64 bytes: one oversized
payload makes Telegram reject the whole keyboard, so a missing button beats a
card that will not render. The template picker names those templates in the
body instead of letting them vanish silently.
Completes the parity goal: an operator should never have to remember a command.
The hub gains Server, Templates and Tools sections, and the views behind them
cover what /mp_digest, /mp_upstreams, /mp_fleet, /mp_voucher, /mp_update,
/mp_add, /mp_broadcast, /mp_restart, /mp_update and /mp_lockdown used to be the
only route to.
Global verbs. Rotate-all, restart, update and lockdown act on the server rather
than on one secret, so they have no label. They carry "_" as a placeholder, and
that placeholder is matched ONLY for these four — a secret genuinely called "_"
is legal and keeps its enable/disable/rotate/remove verbs. All four go through a
confirmation; the confirm tap is the only thing that performs the write, and the
test pins that the first tap runs nothing.
Templates are now editable, not just applicable. templates.conf rows are
"name|conns|ips|quota|expires|notes" and the CLI's `template save` overwrites by
name, so an edit is a read-modify-write that must carry the fields it did not
touch — notes and expiry being the easiest two to lose, since they are the last
columns. The rebuild reads and writes in one call for that reason.
/help is now rendered from `telegram commands`, which prints the same TG_CMDS_*
lists that are registered with Telegram. The old view was a hand-maintained
string that had already drifted: it advertised commands that no longer existed
and missed ones that did.
Two bugs found while wiring this up, both silent:
- /mp_digest called load_ssl_config, load_speed_limits and
load_cloud_backup_config, which are manager-only. Inside the daemon they are
command-not-found under 2>/dev/null, so the digest advertised an SSL Shield
and a Cloud Backup status it could never read. The new digest view shows
only the settings the daemon genuinely loads.
- _kb_can was written where _cb_can was meant in the template editor, so the
Apply and Delete buttons never rendered at all.
…key to a QR service Two pre-existing bugs the button work exposed but did not cause. /mp_setlimit cleared limits it was not asked to touch. It called `secret setlimits <label> <conns> <ips> <quota> <expires>`, but an omitted field arrives as 0, and secret_set_limits reads 0 as UNLIMITED rather than "leave alone" — so `/mp_setlimit alice 100` also removed alice's IP cap and quota. Each provided field now goes through the per-field `secret setlimit <label> <field> <value>` verb, which touches one field and no other. Every field but the last is applied with --no-restart so a multi-field change still costs a single engine sync. The QR code handed the credential to a third party. The proxy link IS the key — server, port and secret — and it was posted to api.qrserver.com for Telegram to fetch, on every /mp_link, every /start, and every voucher redemption, including in the customer's own chat. The same URL was embedded in the exported HTML sheets and the QR sent after Telegram setup. It is now rendered on this host, by qrencode or python3-qrcode, and uploaded as a multipart file (the Bot API only fetches a URL itself, which is exactly the capability we are removing). The HTML sheets embed the image as a data: URI so they stay self-contained. When neither renderer is present the bot sends the tappable link instead. It deliberately does NOT fall back to a remote renderer: the whole point is that the key stays on the machine, and a fallback would silently reintroduce the leak on exactly the minimal hosts least likely to notice. The three URL-passing photo senders (tg_send_photo, tg_send_photo_to, telegram_send_photo) and generate_qr_url are removed rather than left in place. They existed only to serve this path, and an obviously-named helper sitting unused is how the exposure would come back. The QR test pins the behaviour by capturing curl's argv and its -K config separately: the upload must be multipart, must target sendPhoto, must carry the chat id, and the bot token must appear in the config and NOT in argv.
The periodic report sent the same three facts every interval — uptime, live connections, and two LIFETIME cumulative totals — whether or not anything had happened. Those only ever grow, so the message could never answer "how much moved today, and is that more or less than yesterday?". New rolling history store under relay_stats/history/ (5-minute samples, 7-day retention, pruned daily): global.tsv epoch|in_delta|out_delta|conns|marker users.tsv epoch|label|in_delta|out_delta (sparse) It stores DELTAS, not cumulative counters: a cumulative record goes stale the moment the engine restarts or a traffic reset rewinds the counters, forcing every reader to re-implement restart detection. With deltas a reset just yields a smaller bucket. The buckets are accumulated inside update_traffic's existing delta loop, so the restart semantics are inherited by construction. Analytics — 24h/7d/30d totals, period-over-period change, peak and average rate, top talkers, hourly sparkline, quota pressure and expiry watch. Peak uses the ACTUAL gap between samples rather than the nominal interval, so a daemon outage shows as a real gap instead of a phantom spike, and average divides by the span actually covered. Period-over-period reports "n/a" rather than a numeric sentinel, because -1 would be indistinguishable from a genuine 1% decrease. Every query is one awk pass; a six-user page costs one scan, not six. The report is now activity-aware: with no traffic in the window it collapses to a one-line heartbeat, since a dashboard of zeroes every six hours is noise. TELEGRAM_REPORT_DETAIL (auto|full|summary) overrides that. Also adds `mtproxymax telegram history [status|prune|reset]`, documents the inline menus and the new settings, and corrects the README's Telegram command table, which claimed "21 Commands" over 19 rows while omitting every public command, /reply, /mp_broadcast, /mp_fleet and /mp_voucher. New settings keys are registered in all four places, including the bot daemon's own independent load_tg_settings whitelist — omitting that one would leave the daemon silently running on defaults while `telegram status` reported the configured value. tests/test_telegram_settings_roundtrip.sh pins it. Note: the sparkline splits its glyph table on spaces rather than on "" — splitting a multibyte string into characters is locale-dependent, and under a C locale substr would slice a 3-byte bar glyph in half and emit mojibake.
…derr
history_prune() guarded on `command -v flock`, but busybox ships a flock
applet — so the guard passes, and it is specifically `-w` that busybox
lacks. The lock call then failed, the loop `continue`d, and the prune did
nothing at all on Alpine: global.tsv grew without bound and
TELEGRAM_HISTORY_RETENTION_DAYS was silently not honoured. The CLI
`telegram history prune` was unaffected because it has its own copy with
no lock, so the two paths disagreed with each other.
The comment above the function asserted that busybox "has no flock", which
is what produced the wrong guard. Corrected here.
Fall back to `-n`, which busybox does support, so the lock is still taken:
flock -w 5 8 || flock -n 8 || { exec 8>&-; continue; }
Second defect on the same lines: `exec 8>... 2>/dev/null` is
redirection-only, so that 2>/dev/null applies to the whole shell and
silences stderr for the rest of the process, not just for that line. In
the daemon every subsequent diagnostic was discarded. Both the open and
the close form are fixed.
Verified: tests/test_telegram_history.sh goes from 3 failures to 0 under a
busybox-style flock — now stubbed on PATH so the regression is caught on
every platform rather than only on Alpine — and still passes with GNU
flock. All ten test files added by this PR pass.
Per-user traffic. users.tsv has been written since the history subsystem landed, but nothing read it back per user — the user card showed only the cumulative total since the last reset, which cannot answer "how much did they move today, and is that up or down?". The new card shows 24h / 7d / 30d totals from history_users (reused rather than re-derived) and a 24-hour sparkline from a new history_user_series, with the quota and expiry beside it so the numbers can be judged without navigating back. All three windows sit on one card rather than behind a selector: three totals fit comfortably, and a selector would need a fifth field the callback grammar does not have, which would cost the origin page on the way back. A quiet user gets a flat line, not a missing row — history_user_series reports zeros rather than an empty string, and the test pins that, because an empty string would drop the sparkline silently instead of drawing it flat. Sparkline fences. The traffic view and the periodic report wrapped the sparkline in a code fence "for monospace alignment". The glyphs are block elements, which are drawn as a tiling set and share an advance width in a proportional font, so the fence bought nothing and cost a monospace box with a copy button on every report. The periodic-report test asserted the fence was PRESENT; those assertions are inverted here, and the escape check now runs on the glyph run rather than the whole line — the body carries its newlines as literal \n, so the line has backslashes in it either way. Test harness fix, which the above is what surfaced. assert_contains used `printf … | grep -qF`, and under `set -o pipefail` that fails SPURIOUSLY once the haystack exceeds the pipe buffer: grep -q exits at the first match, printf takes SIGPIPE, and the pipeline reports failure even though the needle was found. It only started biting when the daemon grew past 64 KiB, which is why it looked like the markers had gone missing. Every telegram test file now compares with `[[ … == *"$needle"* ]]`, which has no pipe to break. Also fixes the README's Telegram Bot table-of-contents anchor, which PR 4's heading rename from "(21 Commands)" to "(27 Commands)" had left dangling.
rvalitov
force-pushed
the
pr/4-analytics
branch
from
September 19, 2026 06:57
a40b84e to
87d7a94
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #139
Related PR list:
feat(telegram): traffic history, analytics and an informative periodic report
The problem
The periodic report sends the same three facts every six hours, whether or not
anything happened:
Those two totals are lifetime cumulative counters. They only ever grow, so
the message is monotonically less informative as uptime increases, and it can
never answer the obvious question: how much moved today, and is that more or
less than yesterday?
Nothing in the codebase records how usage changed over time.
relay_stats/holdsa running total plus a single "last raw values" snapshot used only for delta
arithmetic.
The rolling history store
New files under
relay_stats/history/(5-minute samples, 7-day retention):It stores deltas, not cumulative counters. A cumulative record goes stale the
moment the engine restarts or a traffic reset rewinds the counters, forcing every
reader to re-implement restart detection. With deltas a reset simply yields a
smaller bucket, and a windowed sum is
t >= start { s += $2 }.The buckets are accumulated inside
update_traffic's existing delta loop, sothe restart-handling semantics are inherited by construction rather than
re-derived — including the "negative delta means restart" correction. Ordering is
already safe: buckets are computed before
save_trafficruns, so a resetconsumed inside it cannot retroactively corrupt a bucket.
users.tsvis sparse — only non-zero deltas are written, so fifty idle users donot generate 288 no-op rows a day.
Sampling is driven from inside the existing 60-second tick on its own coarser
timer, and sits before the
TELEGRAM_ENABLEDguard, so history keepsrecording while the bot is switched off.
Marker column:
-normal,Rengine restart (detected from a decrease intelemt_uptime_seconds),Xtraffic reset.Analytics
24h / 7d / 30d totals, period-over-period change, peak and average rate, top
talkers, hourly sparkline, quota pressure and expiry watch.
Two deliberate correctness choices:
daemon outage then shows as a genuine gap rather than a phantom spike. A test
pins this with a 100-second gap that would read 3× too low under the nominal
300s assumption.
it, so a 7-day query against 3 hours of history reports honestly instead of
inflating.
n/a, not a numeric sentinel.-1would beindistinguishable from a genuine 1% decrease;
n/ais unambiguous and stillavoids
inf/nan, which would break the formatter.Every query is one awk pass. A six-user page costs one scan, not six, matching
the
_load_all_cumulative_user_statsprecedent.history_topis one awk plusone
sort, whatever the user count.The report
It is activity-aware: with no traffic in the window it collapses to a
one-line heartbeat, because a dashboard of zeroes every six hours is noise. That
branch is free —
history_summaryalready returned the totals.TELEGRAM_REPORT_DETAIL(auto|full|summary) overrides it.Quota and expiry warnings reuse the same 80/100 thresholds and the same
_iso_to_epochas the enforcement loop, so the two cannot disagree.The
TELEGRAM_INTERVALknob is reused rather than adding a second interval,since it is already documented, validated and surfaced in
telegram status; asecond one would be a UX trap. One verbosity key was added instead.
Operational
Pruning is the only read-modify-write against the store and takes a lock — on
fd 8, not the fd 9 that
save_trafficuses, because bash file descriptors areprocess-global and reusing 9 would close the traffic lock mid-critical-section.
It also uses the
command -v flockguard fromtraffic resetrather thansave_traffic's unguarded idiom, which silently persists nothing on a hostwithout
flock(busybox/Alpine).Appends need no lock: a single writer,
O_APPEND, small writes.Settings
Four new keys —
TELEGRAM_HISTORY_ENABLED,TELEGRAM_HISTORY_INTERVAL_MIN,TELEGRAM_HISTORY_RETENTION_DAYS,TELEGRAM_REPORT_DETAIL— registered in allfour required places, including the bot daemon's own independent
load_tg_settingswhitelist. Omitting that one leaves the daemon silentlyrunning on defaults while
telegram statusreports the configured value — awrong-behaviour bug with no error anywhere.
tests/test_telegram_settings_roundtrip.shpins it.
Testing
80 new assertions, 0 failures.
The history test uses hand-written fixtures, not fixtures generated by the
code under test, so the expected values are checkable by eye. It covers exact
window sums, the n/a case versus a genuine decrease, actual-gap peak
calculation, one-pass multi-label lookup including labels with no history,
hourly bucketing, top-N ordering, append format, 20 concurrent appends,
retention, and prune idempotency.
Notes
"". Splitting amultibyte string into characters is locale-dependent, and under a C locale
substrslices a 3-byte bar glyph in half and emits mojibake. The test countsbytes rather than characters for the same reason.
omitting every public command,
/reply,/mp_broadcast,/mp_fleetand/mp_voucher. It is now generated from the actual role tables and grouped byscope. This is the one place where PR 4 touches a line PR 1 also touches.
Files
mtproxymax.sh— history block, report block, the sampling tick,update_trafficbucket accumulation, four settings key registrations,telegram historyCLItests/test_telegram_history.sh— newtests/test_telegram_periodic_report.sh— newtests/test_telegram_settings_roundtrip.sh— newREADME.md— command table, inline menus, traffic history, report settings