API Gateway timeout

[nitishxp]

I recently updated my google-cloud-storage to 1.35.0 in my production system after that I am randomly getting this error
I am running my code on Cloud Functions

gcs.get_bucket(bucket_name) File "/env/local/lib/python3.7/site-packages/google/cloud/storage/client.py", line 400, in get_bucket retry=retry, File "/env/local/lib/python3.7/site-packages/google/cloud/storage/bucket.py", line 1002, in reload retry=retry, File "/env/local/lib/python3.7/site-packages/google/cloud/storage/_helpers.py", line 225, in reload retry=retry, File "/env/local/lib/python3.7/site-packages/google/cloud/storage/_http.py", line 63, in api_request return call() File "/env/local/lib/python3.7/site-packages/google/cloud/_http.py", line 438, in api_request raise exceptions.from_http_response(response) google.api_core.exceptions.GatewayTimeout: 504 GET https://storage.googleapis.com/storage/v1/b/bucket_name?projection=noAcl&prettyPrint=false:cs.get_bucket(project + '-stage') File "/env/local/lib/python3.7/site-packages/google/cloud/storage/client.py", line 400, in get_bucket retry=retry, File "/env/local/lib/python3.7/site-packages/google/cloud/storage/bucket.py", line 1002, in reload retry=retry, File "/env/local/lib/python3.7/site-packages/google/cloud/storage/_helpers.py", line 225, in reload retry=retry, File "/env/local/lib/python3.7/site-packages/google/cloud/storage/_http.py", line 63, in api_request return call() File "/env/local/lib/python3.7/site-packages/google/cloud/_http.py", line 438, in api_request raise exceptions.from_http_response(response) google.api_core.exceptions.GatewayTimeout: 504 GET https://storage.googleapis.com/storage/v1/b/bucket_name?projection=noAcl&prettyPrint=false:

[andrewsg]

It looks like the refactor in 1.35.0 accidentally changed the default retry behavior for get_bucket. Thank you for your timely report. I'll fix this but in the meantime you can enable retries on get_bucket by calling it as:

from google.cloud.storage import retry
gcs.get_bucket(bucket_name, retry=retry.DEFAULT_RETRY)

[nitishxp]

Thank you @andrewsg when I can expect the fix to be released?

[nitishxp]

@andrewsg

Now in deleting blob this issue is there can you tell me on which version should I downgrade because its my production system

bucket.delete_blob(i.name)
File "C:\Users\user\envs\env04\lib\site-packages\google\cloud\storage\bucket.py", line 1676, in delete_blob
client._connection.api_request(
File "C:\Users\user\envs\env04\lib\site-packages\google\cloud\storage_http.py", line 63, in api_request
return call()
File "C:\Users\user\envs\env04\lib\site-packages\google\cloud_http.py", line 438, in api_request
raise exceptions.from_http_response(response)
google.api_core.exceptions.GatewayTimeout: 504 DELETE

[andrewsg]

@nitishxp Since it's a potentially destructive API call, delete_blob is only retried if if_generation_match is set when the function is called. For instance if your blob is called blob, then delete_blob(if_generation_match=blob.generation). Rolling back won't help but if you add if_generation_match that should enable retries. You can also do delete_blob(retry=DEFAULT_RETRY) as above to override this behavior. Unlike with get_bucket, the if-generation-match-required default is the intended default and is not a bug.

I'll also clarify that there is not a bug in the library causing these errors to be issued by the API; the bug is only that get_bucket isn't being retried automatically. The actual 504 errors don't appear to be related to anything the client library is doing and if they persist at an unusual level I would recommend calling support -- though in most cases fixing the retry issue here will resolve it.