open
https://gitlab.synchro.net/main/sbbs/-/work_items/1245
## Summary
`ircd.js` polls a semaphore file once a second from the main event loop using a **blocking** filesystem call. Synchronet's JS event loop is single-threaded, so if that one `stat()` blocks, the entire IRCd stops: no PONG handling, no socket reads, no timers, no logging. When the stall outlasts the configured ping intervals, every server link and every client expires at once the moment the loop resumes, producing a network-wide netsplit from what was really just a slow filesystem.
This is not hypothetical. It took the Synchronet IRC network apart on 2026-09-19 and the affected server did not recover on its own.
## The code
`exec/ircd.js:158`, introduced in `d7cb93923c` (wave-27-oops, 2021-05-22):
```js
function config_rehash_semaphore_check() {
if(file_date(system.ctrl_dir + "ircd.rehash") > Time_Config_Read) {
Read_Config_File();
}
}
js.setInterval(config_rehash_semaphore_check, 1000 /* milliseconds */);
```
`file_date()` is a synchronous `stat()`. On any install where `ctrl_dir` is on network or otherwise high-latency storage (NFS, SMB/CIFS, a loopback CIFS mount of the local Samba server, a stalled or spun-down disk), this call can block for
as long as the filesystem takes to answer, which for SMB means up to the client's
reconnect timeout.
The sibling poll at `ircd.js:176` (`shutdown_semaphore_check`) runs at the same 1 Hz but only reads `server.terminated`, so it is not affected.
## What happened
A web-scraping storm saturated the BBS's web server. Its file I/O flows through the same Samba server that backs the install directory, and the CIFS client wedged in `smb2_reconnect`. The kernel logs nothing while a reconnect is pending, only when it finally fails, so the first kernel message appears at the **end** of the stall:
```
17:02:04.343132 kernel: CIFS: VFS: reconnect tcon failed rc = -11 17:02:09.347138 kernel: smb2_reconnect: 56922 callbacks suppressed
```
Peak suppression that day reached 5,901,653 callbacks. `rc = -11` is `-EAGAIN`.
Gap analysis of the two Synchronet daemons' own log output:
```
ircd.service 17:00:33 -> 17:02:04 GAP 91s
sbbs.service 17:00:42 -> 17:02:04 GAP 82s
```
Both went silent and both resumed at the **same second**, which is the second the CIFS client gave up and started returning errors. The IRCd's next scheduled `file_date()` after 17:00:33 blocked in uninterruptible sleep and stayed there.
91 seconds exceeds every ping threshold in the shipped config
(`Class 30` leaf-to-hub `PingFrequency=60`, `Class 40`/`Class 50` hub-to-hub `PingFrequency=90`), so when the loop resumed every timer was already expired. About 50 server links were SQUIT in the following 60 seconds.
## The second-order failure
The netsplit was survivable. What was not: roughly 69 leaf servers then reconnected at once, each requiring a full netburst (all nicks, all channels, all ban lists) from a single-threaded JS process. That tipped into congestion collapse. Six hours later the process was still pegged at 100% of a core, accepting TCP connections and **logging** that it had sent the greeting while the client actually received zero bytes, because the send queue was never being flushed. It never recovered without a restart.
## Suggested fixes
Worth separating, because they are independent and the second is arguably more valuable than the first:
**1. Stop blocking the event loop on the semaphore poll.** A JS-side change cannot make the `stat()` itself non-blocking, so the realistic options are to poll far less often, to let the sysop disable the poll, or to drive rehash from something that does not touch the filesystem on the hot path. Synchronet already
has an MQTT control channel and a signal path that could carry "rehash" without a 1 Hz `stat()`. Note this pattern is not unique to the IRCd; any Synchronet JS service that polls a file from its event loop has the same exposure.
**2. Do not detonate every timer after an event-loop stall.** The IRCd should notice that the loop itself was frozen (compare a monotonic timestamp across iterations) and grant a grace period before expiring ping timers, rather than concluding that every peer on the network simultaneously stopped responding. A stall is observably different from 50 independent peers timing out in the same second, and treating them the same is what turns a local hiccup into a network-wide split. This also protects against unrelated causes such as scheduler starvation, swap, or a long GC pause.
Fix 2 alone would have reduced this incident to a brief freeze with no split.
-- *Authored by Claude (Claude Code), on behalf of @rswindell*
---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net