Split the augur backend container to separate workers into their own container

[MoralCode]

The augur backend container is still sort of a "kitchen sink" container that handles the frontend, the API, and the workers.

Can we split the workers into their own container?

The main usecase I have for this is because I would like to use the --scale parameters in docker to simulate/test a more gradual bringup process for the workers. Example:

1. configure augur to have a really low worker count for each task (like 1-2 for each of core facade and secondary
2. start augur with one worker.
3. use the --scale flag to bring up multiple instances of the worker container, each running a set of workers with these combined low limits, but spaced out over time to watch memory use patterns in grafana

im still seeing some concerning memory use patterns even when running from 8ec768d.

[SuyashJain17]

I’ll look into this and see what it would take to split the workers into a separate container.

[MoralCode]

sounds good! just note that this issue is more exploratory and seeking feedback/feasibility. We dont currently have enough information to create a pull request.

If you're able to post a short summary of your findings (mostly interested in if you see any obvious problems with doing this - i.e. super tight integration between workers and the other parts of the backend that would make this infeasible), that would be awesome

[guptapratykshh]

i have took look at this and it is definitely doable. codebase is already set up in way that makes this pretty straightforward.

the workers and api server are already loosely coupled , they talk to each other through rabbitmq rather than direct process calls, which is exactly what we want for splitting containers. even better, there is already --disable-collection flag and augur collection start command that we can leverage.

for memory testing use case, it would be perfect. we could start with one worker container running minimal workers(like 1-2 for each task type), watch memory usage in grafana, then bring up second worker container, monitor again, and keep scaling gradually.

i did not find any dealbreakers.architecture is already distributed, everything goes through the database, redis, and rabbitmq, which are designed for exactly this kind of multi container setup.

[SuyashJain17]

I did a quick feasibility pass on this.

At a high level, the workers don’t appear to be tightly coupled to the API or frontend at a process level. Most coordination seems to happen via external services (DB, Redis/broker, Celery), which suggests splitting workers into a separate container should be feasible in principle.

The main things that would need attention are:

• Configuration/env vars that are currently assumed to be shared inside a single backend container
• Startup order (ensuring DB, Redis, migrations, etc. are ready before workers start)
• Any filesystem or local-state assumptions shared between API and workers

I didn’t notice any obvious hard blockers like in-memory dependencies or worker logic that directly relies on API internals at runtime. That said, validating this would likely require some targeted experimentation (e.g. running workers-only containers and observing behavior and memory patterns).

[MoralCode]

This is a lot less urgent now that we found what was causing the memory issues with facade in #3593

[MoralCode]

there is already --disable-collection flag and augur collection start command that we can leverage.

yes, but i think the collection start CLI is part of the backend, and is coded to spawn tasks as subprocesses (i.e it was built for a manual/bare metal install)

I think a cool first step to this would be to try and package a worker (maybe pick one type) as a docker container and see what kind of plumbing we need to connect it to the rest of the stack (i.e. give it the same redis/rabbit urls so it can pick tasks from the queue)

IMO this can be done as an experiment in a new repo with a dockerfile and things. If it works (and once we get through the PR backlog) maybe it can become mature enough to be its own PR

[MoralCode]

Retitling this as it has been discussed in a maintainers call and seems like it would help with some other changes to scheduling as well (especially if the workers could be run from a separate container).

the memory patterns thing is less relevant to this issue.

This will likely involve (likely separate PRS):

• splitting docker container for the backend into two, including separating the dependencies for the workers from those used for the backend. Ensure the backend container continues to function (API serves requests everything comes up, even if flower shows no workers)
• determining which code in the augur codebase is for the workers(and therefore what should be brought into the new container). Ensuring the new worker container starts up, can connect to rabbit/celery, and can start processing tasks properly.
• Understanding the current startup procedure and making sure it is appropriately split or modified
• Updating the docker compose file in the root of the repo to use the newly split containers and ensuring the readTheDocs and any other documentation is updated as necessary.

I would not recommend this as a task for anyone who hasn't been contributing to augur mostly continuously for at least the past few months or who doesnt feel confident about their ability to do it to the expected quality standard (this is a major change and will take a LOT of scrutiny to merge, id rather you ask in advance than try something and have to redo it later).

I'd be happy to chat about this in more detail over on the CHAOSS Slack in the #wg-augur-8knot channel