Explicitly passed credentials are ignored if no project id passed

[evil-shrike]

I have a code that detects current credentials and creates credentials object. And then the acquired credentials objects are being passed everywhere through application for all GCP API clients.
And it works fine with many client (BigQuery, Sheets, Drive, etc). But not with Storage.

def get_credentials(args: argparse.Namespace, scopes = _SCOPES) -> credentials.Credentials:
  if args.client_secrets_file:
    appflow = flow.InstalledAppFlow.from_client_secrets_file(
        args.client_secrets_file, scopes)
    if args.non_interactive:
      appflow.run_console()
    else:
      appflow.run_local_server()
    credentials = appflow.credentials
  elif args.service_account_file:
    credentials = service_account.Credentials.from_service_account_file(
          args.service_account_file, scopes)
  else:
    # NOTE: if you use `gcloud auth application-default login` then the scopes here will be ignored,
    #       you should specify them as parameter --scopes for the gcloud command
    credentials, project = google.auth.default(scopes = scopes)

  return credentials


credentials = auth.get_credentials(args)
storage_client = storage.Client(credentials = credentials)
# this works: storage_client = storage.Client(project = project_id, credentials = credentials)

With Storage client I'm getting error:

  File "/Users/segy/work/gmc-dsa/product-dsa/./install/cloud_env_setup.py", line 66, in backup_config
    storage_client = storage.Client(credentials = credentials)
  File "/Users/segy/work/gmc-dsa/product-dsa/.venv/lib/python3.9/site-packages/google/cloud/storage/client.py", line 124, in __init__
    super(Client, self).__init__(
  File "/Users/segy/work/gmc-dsa/product-dsa/.venv/lib/python3.9/site-packages/google/cloud/client.py", line 318, in __init__
    _ClientProjectMixin.__init__(self, project=project, credentials=credentials)
  File "/Users/segy/work/gmc-dsa/product-dsa/.venv/lib/python3.9/site-packages/google/cloud/client.py", line 266, in __init__
    project = self._determine_default(project)
  File "/Users/segy/work/gmc-dsa/product-dsa/.venv/lib/python3.9/site-packages/google/cloud/client.py", line 285, in _determine_default
    return _determine_default_project(project)
  File "/Users/segy/work/gmc-dsa/product-dsa/.venv/lib/python3.9/site-packages/google/cloud/_helpers.py", line 186, in _determine_default_project
    _, project = google.auth.default()
  File "/Users/segy/work/gmc-dsa/product-dsa/.venv/lib/python3.9/site-packages/google/auth/_default.py", line 483, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

So the client ignores the credentials object I passed and tries to find ADC.

As it's not happening with other client as mentioned I believe it's a bug.

# works fine:
bigquery_datatransfer_v1.DataTransferServiceClient(credentials=credentials)

Environment details

• OS type and version: MacOS
• Python version: 3.9
• pip version: 21.1.1
• google-cloud-storage version: 1.41.1

[cojenco]

Hi @evil-shrike, thanks for filing the issue. I'm investigating with the team on why this is causing an issue in the storage client.

The storage Client, same as the bigquery Client, both utilize cloud-core ClientWithProject and google.auth.default() when a project is not specified and can not be determined from the environment. @tseaver Is this WAI?

[tseaver]

@cojenco The fallback to google.auth.default is due to the need to determine the project for credentials which do not contain project_id.

• Credentials obtained from a service account file contain the default project, and so passing an explicit project is not needed when using them.
• OAuth2 flow credentials do not have a project: therefore, the user must either pass an explicit project to the client constructor, or set the project in the environment under GOOGLE_CLOUD_ID.

[cojenco]

Closing due to inactivity. Please reopen this issue if you have further questions.