You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Extend workerd with some graceful shutdown to enable 0-downtime reload. #101
Arrange so that when SIGTERM is received, workerd will stop accepting new connections, but will finish up any already-in-flight requests, then exit. This should be easy.
Figure out how best to start up a new instance of workerd using the same listen socket, while the old instance is still draining. I would guess that systemd can do this automatically, so probably this is just a matter of documenting how to configure systemd.
But I haven't figured out how to configure systemd for part 2. systemd has tons of configuration options but I don't see any that says: "On restart, start up the new process without first waiting for the old process to shut down."
Maybe it truly doesn't have this option, which is surprising since it support so many other obscure modes.
If that's the case, then what we probably need is a wrapper process that starts workerd as a child process and then, when it receives SIGHUP, sends SIGTERM to the child while also simultaneously starting a new child to replace it. If the old child doesn't shut down withing 30 seconds, the wrapper should kill it.
I'm not really sure if this should be a built-in feature of the workerd binary vs. a separate binary. Making it a built-in feature is tricky because presumably you want to be able to upgrade the workerd binary itself, but if the wrapper itself is running workerd then it seems to need to exec() the new binary to update itself, which is possible but makes it more complicated to maintain other state.
Since part 1 is done, how can we write a callback for SIGTERM? I am expecting code like this:
process.on('SIGTERM',()=>{console.info('SIGTERM signal received.');// Clean up, such as sending termination messages to websocket clients and servers});
It's unlikely that we will expose an API callable from within a worker to listen to signals since that API would only work in workerd and wouldn't be usable in the edge production environment (we are trying to avoid cases where code only works in workerd and not the edge). That said, I do wonder if there's some way we could actually provide some means of hooking signals via the config using a separate script than the actual worker. hmm... something to think about.
@kentonv In regards to part 2, I did a thorough investigation and found that there is no facility in systemd to achieve, as you put it "On restart, start up the new process without first waiting for the old process to shut down.". However, I think we can achieve zero downtime with systemd only; it's just a bit more manual than expected (but can be scripted). The idea:
Rely on socket activation. Define a .socket unit that binds to the chosen port (say 80). The workerd service will be expected to listen to the socket created by systemd.
Allow for multiple sockets on the same port. In the .socket unit, set ReusePort=True (which sets SO_REUSEPORT, see LWN article).
Define workerd service as a template unit. This way, one can start multiple concurrent services of workerd. (One would probably do that anyway to use multiple core). For the sake of the example, we'll have two services: one named workerd@blue and the other workerd@green.
Here is how a deployment would happen:
Initial state is workerd@blue is started and active.
The workerd executable is replaced with the new version (or its configuration, depending on how the unit is written; doesn't matter).
User runs systemctl start workerd@green. It gets its own port on the same socket. The kernel load-balances the connections between the two instances.
User runs systemctl stop workerd@blue. This sends a SIGTERM to the process, which gracefully shuts down.
The user can repeat steps 3 and 4 by switching blue and green to get back to the initial state; or they may not.
I have not performed any tests but for the time being, I think it's a clean-enough solution.
@yacinehmito That's pretty neat actually. I like that you can easily roll back in case of problems.
I do still think it should be pretty easy to create a wrapper around workerd which gracefully restarts workerd on SIGHUP. I'd expect this to be like 30 lines of C/C++, or maybe it could even be a shell script.
Activity
thet0ast3r commentedon Oct 13, 2022
sorry, somehow i cannot add labels or assignees to the issue.
kentonv commentedon Oct 14, 2022
Two steps here:
SIGTERMis received,workerdwill stop accepting new connections, but will finish up any already-in-flight requests, then exit. This should be easy.workerdusing the same listen socket, while the old instance is still draining. I would guess that systemd can do this automatically, so probably this is just a matter of documenting how to configure systemd.kentonv commentedon Jan 2, 2023
So part 1 is done by #244.
But I haven't figured out how to configure systemd for part 2. systemd has tons of configuration options but I don't see any that says: "On restart, start up the new process without first waiting for the old process to shut down."
Maybe it truly doesn't have this option, which is surprising since it support so many other obscure modes.
If that's the case, then what we probably need is a wrapper process that starts
workerdas a child process and then, when it receives SIGHUP, sends SIGTERM to the child while also simultaneously starting a new child to replace it. If the old child doesn't shut down withing 30 seconds, the wrapper should kill it.I'm not really sure if this should be a built-in feature of the workerd binary vs. a separate binary. Making it a built-in feature is tricky because presumably you want to be able to upgrade the workerd binary itself, but if the wrapper itself is running workerd then it seems to need to exec() the new binary to update itself, which is possible but makes it more complicated to maintain other state.
joeky888 commentedon Mar 1, 2023
Since part 1 is done, how can we write a callback for
SIGTERM? I am expecting code like this:jasnell commentedon Mar 3, 2023
It's unlikely that we will expose an API callable from within a worker to listen to signals since that API would only work in workerd and wouldn't be usable in the edge production environment (we are trying to avoid cases where code only works in workerd and not the edge). That said, I do wonder if there's some way we could actually provide some means of hooking signals via the config using a separate script than the actual worker. hmm... something to think about.
yacinehmito commentedon Aug 21, 2023
@kentonv In regards to part 2, I did a thorough investigation and found that there is no facility in systemd to achieve, as you put it "On restart, start up the new process without first waiting for the old process to shut down.". However, I think we can achieve zero downtime with systemd only; it's just a bit more manual than expected (but can be scripted). The idea:
.socketunit that binds to the chosen port (say 80). The workerd service will be expected to listen to the socket created by systemd..socketunit, setReusePort=True(which setsSO_REUSEPORT, see LWN article).workerd@blueand the otherworkerd@green.Here is how a deployment would happen:
workerd@blueis started and active.workerdexecutable is replaced with the new version (or its configuration, depending on how the unit is written; doesn't matter).systemctl start workerd@green. It gets its own port on the same socket. The kernel load-balances the connections between the two instances.systemctl stop workerd@blue. This sends aSIGTERMto the process, which gracefully shuts down.The user can repeat steps 3 and 4 by switching
blueandgreento get back to the initial state; or they may not.I have not performed any tests but for the time being, I think it's a clean-enough solution.
Once workerd supports multi-threading, it might be worth copying what HAProxy does for zero-downtime restarts. TL;DR: The old process passes its file descriptors to the new process before shutting down.
Sources:
kentonv commentedon Aug 23, 2023
@yacinehmito That's pretty neat actually. I like that you can easily roll back in case of problems.
I do still think it should be pretty easy to create a wrapper around workerd which gracefully restarts workerd on SIGHUP. I'd expect this to be like 30 lines of C/C++, or maybe it could even be a shell script.
fix(server): facet channel UaF after alarm timeout
fix(server): facet channel UaF after alarm timeout