https://gitlab.synchro.net/main/sbbs/-/issues/1225#note_10180
## Six more occurrences, a second error code, and a correction
Two users (Tracker1 and DM) reported SSH sessions dropping over the past
week, which sent me back through the logs. This is not a one-off, and the
`-32` packet-type case filed above is the rarer of two shapes the same
defect takes.
### The other error code
The common variant returns **`CRYPT_ERROR_INTERNAL` (-16), "Internal
consistency check failed"** out of the same `crypt_pop_channel_data()`
call, and kills the node the same way:
```
17:51:09 Node 1 SSH note 'Internal consistency check failed' (-16) popping
data from input_thread
17:51:12 Node 1 logout initiated
17:51:13 Node 1 thread terminated (0 node threads remain, 6547 clients
served)
```
Note the severity: cryptlib rates this **`note`**, not `ERROR`. The session
dies regardless âÇö see the mechanism section below.
### Occurrences
| When | Host | User | Error | Time into session | Activity at drop |
|---|---|---|---|---|---|
| Jun 3 07:41:21 | vert | Arqueiro | -16 | âëê45 s | after logon event |
| Jun 3 07:43:37 | vert | Arqueiro | -16 | âëê90 s | after logon event |
| Aug 11 17:51:09 | vert | Tracker1 | -16 | 70 s | idle at menu |
| Aug 11 17:53:59 | vert | Tracker1 | -16 | 2 m 28 s | idle, after a node
message |
| Aug 11 22:33:46 | git | Digital Man | -32 | mid-door | sixel door (this
issue) |
| Aug 14 02:57:55 | vert | R3vlibre | -16 | 4 m 52 s | idle |
| Aug 14 04:32:57 | vert | R3vlibre | -16 | 1 h 33 m | idle, after exiting
Utopia |
Arqueiro's June pair means this predates the past week; it is not a new
regression, just newly noticed.
### Tracker1's Aug 11 sequence
The most informative run, because he kept reconnecting:
```
17:49:52 SSH connect, TERM xterm-256color, UTF-8 -> logon 17:49:59 ->
DROPPED 17:51:09 (-16)
17:51:24 SSH connect, TERM xterm-256color, UTF-8 -> logon 17:51:31 ->
DROPPED 17:53:59 (-16)
17:54:12 SSH connect -> session
establishment failed 17:54:20
17:54:25 Telnet -> logon 17:54:33 ->
clean logout 18:05:26
18:08:01 SSH connect, TERM ansi, CP437 -> logon 18:08:03 ->
fine through 18:46+
```
So SSH is not simply broken for him: his fourth attempt, over SSH, ran
nearly 40 minutes without trouble. What changed between the sessions that
died and the one that lived was the terminal mode.
### An observation, offered with its sample size
Every one of the seven drops above was a **UTF-8** session. The two SSH
sessions in the sample that were **CP437** (Tracker1's 18:08 `ansi`
session, and a `syncterm` session on Aug 13) both survived. Counting all
SSH sessions on vert that reached terminal-type detection over Aug 11/13/14:
6 UTF-8, of which 4 dropped; 2 CP437, of which 0 dropped.
Two of two is not evidence. It is, however, cheap to act on: an attempt to
reproduce should use **UTF-8 with an xterm-family TERM over SSH**, since
that is where every observed failure lives. The failures are on the input
path, so if this holds it is presumably about what the client *sends* -- a
UTF-8 terminal emits different and more input (cursor-position and device
attribute reports, mouse encoding) than a CP437 one.
### This is not SyncTERM-specific
That was the initial suspicion and it does not survive the logs. Terminal
types by protocol on vert across Aug 11/13/14:
```
464 Telnet DUMB 5 SSH xterm-256color
35 Telnet syncterm 1 SSH xterm
20 Telnet ANSI 1 SSH syncterm
16 Telnet PETSCII 1 SSH ansi
8 Telnet RIP (+ smaller Telnet types)
```
Only **8 SSH sessions reached logon on vert in three days**, and 4 of them
were killed. The 35 `syncterm` sessions were all Telnet, so they were never
exposed to this path and say nothing either way. Meanwhile the `-32` drop
in the description above *was* a SyncTERM client. Both sides of the client
question are represented among the failures; the constant is SSH.
### Correction to the description above
The description reasons that "the corruption is inside one of the two
cryptlib endpoints." That premise needs revisiting: **SyncTERM's SSH client
is DeuceSSH** (`src/ssh/`, see the header comment in
`src/syncterm/ssh.c`), not cryptlib. So for the SyncTERM case the two
endpoints are DeuceSSH (client) and cryptlib (server), and "both endpoints
are cryptlib" no longer bounds where to look.
For what it is worth, DeuceSSH's auto-rekey is **not** an explanation:
`rekey_seconds` defaults to 0 (disabled) in `dssh_session_init`, and the
byte threshold is the per-cipher `bytes_per_key`, which is 64 GiB for every
AES module. Neither could have fired on any session here. The description's
"no rekeying" reasoning holds, for a different reason than stated.
### A correlation I tested and rejected
Each drop happens while SSH scanners are failing session establishment on
other nodes, which looks suggestive. It is not: session-establishment
failures occur in **26-55% of all logged minutes** on vert (measured across
the three days above). With seven events, co-occurrence is what chance
predicts. Not evidence, and worth recording so it is not re-derived as a
lead.
### Mechanism
`src/sbbs3/main.cpp:2319-2333`. Everything except `CRYPT_ERROR_COMPLETE`
and `CRYPT_ERROR_TIMEOUT` is fatal:
```c
/* Handle the SSH error here... */
GCES(err, sbbs->cfg.node_num, sbbs->ssh_session, "popping data");
break; /* input thread exits -> node torn down */
```
`GCES` calls `get_crypt_error_string()`, which returns a **severity** along
with the message -- that is why these log as `note` and `dbg` rather than
`ERROR`. The `break` then ignores that severity entirely, so a condition
cryptlib itself classifies as informational takes a logged-on user's
session down. The Telnet path immediately below tolerates transient
conditions (`if (rd == 0 && !socket_recvdone(sock, 0)) continue;`) in a way
the SSH path does not.
This is the amplifier rather than the cause, but it is the part that turns
a single cryptlib hiccup into a lost session, and it is separable from
whatever makes cryptlib return -16 in the first place.
Also landing here, and near-certainly benign: `CRYPT_ERROR_PARAM1` (-1) at
teardown, 354 times in the Aug 10-14 log on git and 31 in a week on vert,
always
immediately after `disconnecting client` -- the session handle is already
gone. Those cost nothing today only because the session was ending anyway.
### Relationship to #302
\\#302 ("Cryptlib 'internal consistency check' failures cause SSH and TLS
connection issues") is the same -16 condition. It was closed on the
assessment that it "doesn't appear to be fatal." On this path it is fatal.
Worth reopening or explicitly folding into this issue.
### Diagnostic gap
We do not log the peer's SSH version string anywhere, so for the vert
occurrences I can identify the TERM each client sent but not which client
sent it. Logging the peer ident on the terminal server would settle the
client question the next time this happens, and raising `SSHErrorLevel` on
vert would be cheap -- the hit rate there is high enough to catch a
recurrence quickly.
-- *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
|
