Skip to content

Extend workerd with some graceful shutdown to enable 0-downtime reload. #101

Open

Description

@thet0ast3r

Currently, there is no way to restart workerd after updating/adding workers without having a bit of downtime.

@kentonv suggested the creation of this issue :)

Activity

  1. thet0ast3r commented on Oct 13, 2022

    @thet0ast3r
    Author

    sorry, somehow i cannot add labels or assignees to the issue.

  2. kentonv commented on Oct 14, 2022

    @kentonv
    Member

    Two steps here:

    1. Arrange so that when SIGTERM is received, workerd will stop accepting new connections, but will finish up any already-in-flight requests, then exit. This should be easy.
    2. Figure out how best to start up a new instance of workerd using the same listen socket, while the old instance is still draining. I would guess that systemd can do this automatically, so probably this is just a matter of documenting how to configure systemd.
  3. kentonv commented on Jan 2, 2023

    @kentonv
    Member

    So part 1 is done by #244.

    But I haven't figured out how to configure systemd for part 2. systemd has tons of configuration options but I don't see any that says: "On restart, start up the new process without first waiting for the old process to shut down."

    Maybe it truly doesn't have this option, which is surprising since it support so many other obscure modes.

    If that's the case, then what we probably need is a wrapper process that starts workerd as a child process and then, when it receives SIGHUP, sends SIGTERM to the child while also simultaneously starting a new child to replace it. If the old child doesn't shut down withing 30 seconds, the wrapper should kill it.

    I'm not really sure if this should be a built-in feature of the workerd binary vs. a separate binary. Making it a built-in feature is tricky because presumably you want to be able to upgrade the workerd binary itself, but if the wrapper itself is running workerd then it seems to need to exec() the new binary to update itself, which is possible but makes it more complicated to maintain other state.

  4. joeky888 commented on Mar 1, 2023

    @joeky888

    Since part 1 is done, how can we write a callback for SIGTERM? I am expecting code like this:

    process.on('SIGTERM', () => {
      console.info('SIGTERM signal received.');
      // Clean up, such as sending termination messages to websocket clients and servers
    });
  5. jasnell commented on Mar 3, 2023

    @jasnell
    Collaborator

    It's unlikely that we will expose an API callable from within a worker to listen to signals since that API would only work in workerd and wouldn't be usable in the edge production environment (we are trying to avoid cases where code only works in workerd and not the edge). That said, I do wonder if there's some way we could actually provide some means of hooking signals via the config using a separate script than the actual worker. hmm... something to think about.

  6. yacinehmito commented on Aug 21, 2023

    @yacinehmito

    @kentonv In regards to part 2, I did a thorough investigation and found that there is no facility in systemd to achieve, as you put it "On restart, start up the new process without first waiting for the old process to shut down.". However, I think we can achieve zero downtime with systemd only; it's just a bit more manual than expected (but can be scripted). The idea:

    1. Rely on socket activation. Define a .socket unit that binds to the chosen port (say 80). The workerd service will be expected to listen to the socket created by systemd.
    2. Allow for multiple sockets on the same port. In the .socket unit, set ReusePort=True (which sets SO_REUSEPORT, see LWN article).
    3. Define workerd service as a template unit. This way, one can start multiple concurrent services of workerd. (One would probably do that anyway to use multiple core). For the sake of the example, we'll have two services: one named workerd@blue and the other workerd@green.

    Here is how a deployment would happen:

    1. Initial state is workerd@blue is started and active.
    2. The workerd executable is replaced with the new version (or its configuration, depending on how the unit is written; doesn't matter).
    3. User runs systemctl start workerd@green. It gets its own port on the same socket. The kernel load-balances the connections between the two instances.
    4. User runs systemctl stop workerd@blue. This sends a SIGTERM to the process, which gracefully shuts down.

    The user can repeat steps 3 and 4 by switching blue and green to get back to the initial state; or they may not.

    I have not performed any tests but for the time being, I think it's a clean-enough solution.

    Once workerd supports multi-threading, it might be worth copying what HAProxy does for zero-downtime restarts. TL;DR: The old process passes its file descriptors to the new process before shutting down.

    Sources:

  7. kentonv commented on Aug 23, 2023

    @kentonv
    Member

    @yacinehmito That's pretty neat actually. I like that you can easily roll back in case of problems.

    I do still think it should be pretty easy to create a wrapper around workerd which gracefully restarts workerd on SIGHUP. I'd expect this to be like 30 lines of C/C++, or maybe it could even be a shell script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature requestRequest for Workers team to add a feature

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions