Detect available docker-hosts and distribute builds across them

[tobybellwood]

In a scenario where there are N docker-hosts in N zones, the remote-controller should be able to provide a build pod with the connection details for it's closest docker-host. The build will then use that docker-host for the duration of it's build.

[shreddedbacon]

Since we know that the topology aware hints stuff doesn't really work, we can reject that. I don't think we will have an easy way to properly do topologically closest docker-hosts.

Alternatively though, we could have the remote-controller scan for kind: Service within a cluster that have a specific label (eg app.kubernetes.io/component=lagoon-remote-docker-host) to determine if there are 1 or more docker-hosts within a cluster. Or we can define custom labels to use to select docker-hosts too.

When the controller is creating the build pods, it can just round robin them amongst the known docker-hosts by injecting the docker-host record into a build, rather than the builds defaulting to docker-host.lagoon.svc, that can be a fallback.

[tobybellwood]

I still think it may be worth trying to identify the "closest" zone-based (per labels) docker-host and allocating that to a build in this scenario, but round-robining if necessary (overload or unavailablility)

[shreddedbacon]

I still think it may be worth trying to identify the "closest" zone-based (per labels) docker-host and allocating that to a build in this scenario, but round-robining if necessary (overload or unavailablility)

The main problem though, is that the controller doesn't schedule the builds, and there is no easy way for a build pod to identify which zone it is in to know which to choose from, but agree, if it is possible to solve this without the remote-controller having to handle scheduling too, then that would be nice.

[shreddedbacon]

Re scoping this to be able to inspect the docker host(s) deployed in a cluster and try to distribute builds amongst the available docker hosts.