Limit throughput of function deployments

[merlinnot]

Google Cloud functions has a quota on API calls (WRITE), which makes deployments of large number of functions impossible. This can be solved by limiting the number of requests firebase-tools makes per 100 seconds by queueing and performing operations within this limitation.

I'd be willing to work on adding support for this scenario, please let me know if such a PR would be accepted. Update: don't have capacity for this at the moment.

[ryanpbrewster]

I have encountered the same issue and considered the same fix.

[thechenky]

Hah, we have an internal bug for specifically this! Would love your help in implementing :) thanks for offering!

Internal issue reference: 126680733

[merlinnot]

I'm still going to implement it, I started with migrating some of the files I'll need to touch from JavaScript to TypeScript.

Today I received some more information from Google Cloud Support which I might end up incorporating in the PR for this issue:

The overall problem is that the deployments are very bursty (like 100+ in a single second and then nothing for 30 minutes). To make it scalable and sustainable we'd suggest changing the deployment automation so that it does one function deployment every few seconds or so. It will allow for O(10000) deployments per day. Once it's not enough there's a possibility to use more granular deployment unit by deploying several functions as separate endpoints in a single App Engine or Cloud Run service.

Request limit for one of my projects was actually reduced to 50 (the default is 80):

We've reduced GCF Write (i.e. deployment) quotas from 100 to 50 per 100s for project . The customer recently asked for the quota increase from 80 to 100 per 100s in sfstory/. The increase has been granted but since then we observed some problems caused by bursts of deployments traffic. E.g. on Jul 21st 04:17 PDT there were 115 requests in a single second, most of which failed due to overload. Hope that reducing the quota will allow traffic to level out and be eventually more successful.

My initial intention was to fire 80 requests and then fire another batch after 100 seconds and so on, with a capped exponential backoff for contention errors. Given the above, I'd go for firing 1 request every 1.25 seconds so it complies with the default limit and the suggestion quoted above.

In the future we might want to think of exposing this value (time between deployments) as a CLI option for occasions like this, but I'll make a separate issue for it once the PR with 1.25 lands in master.

Let me know what you think, if this approach would be ok with you. Maybe you have some other ideas?

[merlinnot]

As discussed some time ago, I don't have capacity at the moment to work on this feature, so I changed the title to reflect it.

[deremer]

I was just about to log the same feature request to this repo. Looks like #1858 is still a WIP? Just wanted to update this issue for future readers.

[jsakas]

I think this issue impacts a lot of users. Would love to see a fix!

If someone can outline the high level steps on what needs to change in the repo, I'd give it a shot. I have no experience contributing to this tool so some direction would probably be helpful.

[mbleigh]

Folks who are still experiencing this issue, can you please make sure you're on the latest version of the CLI and provide details (how many functions you're deploying, --debug logs of the specific error)? We would expect this to have been largely fixed due to recent changes in functions deployment.