Skip to content

BigQuery: Add debug logging statements to track when BQ Storage API is used.#8838

Merged
tswast merged 2 commits intogoogleapis:masterfrom
tswast:issue8826-debug-to_dataframe
Aug 1, 2019
Merged

BigQuery: Add debug logging statements to track when BQ Storage API is used.#8838
tswast merged 2 commits intogoogleapis:masterfrom
tswast:issue8826-debug-to_dataframe

Conversation

@tswast
Copy link
Contributor

@tswast tswast commented Jul 30, 2019

With BQ Storage API and logging.basicConfig(level=logging.DEBUG):

import google.cloud.bigquery_storage_v1beta1
import google.cloud.bigquery
client = google.cloud.bigquery.Client()
bqstorage = google.cloud.bigquery_storage_v1beta1.BigQueryStorageClient()
rows = client.list_rows("schema_examples.scalar_fields")

import logging
logging.basicConfig(level=logging.DEBUG)
df = rows.to_dataframe(bqstorage_client=bqstorage)

Outputs:

DEBUG:google.auth.transport.requests:Making request: POST https://oauth2.googleapis.com/token
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): oauth2.googleapis.com:443
DEBUG:urllib3.connectionpool:https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
DEBUG:google.cloud.bigquery._pandas_helpers:Started reading table 'swast-scratch.schema_examples.scalar_fields' with BQ Storage API session 'projects/swast-scratch/locations/us/sessions/CAISCEtpQ1dKVERqGgJpchoCaXcaAmpkGgJpbhoCamMaAmpx'.

With tabledata.list:

DEBUG:google.cloud.bigquery.table:Started reading table 'swast-scratch.schema_examples.scalar_fields' with tabledata.list.
DEBUG:urllib3.connectionpool:https://www.googleapis.com:443 "GET /bigquery/v2/projects/swast-scratch/datasets/schema_examples/tables/scalar_fields/data HTTP/1.1" 200 None

Closes #8826.

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jul 30, 2019
@tswast tswast marked this pull request as ready for review July 31, 2019 17:45
@tswast tswast requested review from a team and plamut July 31, 2019 17:45
@tswast tswast requested a review from steffnay July 31, 2019 21:32
Copy link
Contributor

@plamut plamut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

[
api_request=None,
path=None,
schema=[
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing the args as keyword arguments makes this more readable. 👍

@tswast tswast merged commit 1a4bd76 into googleapis:master Aug 1, 2019
@tswast tswast deleted the issue8826-debug-to_dataframe branch August 1, 2019 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BigQuery: Add debug logs in to_dataframe / to_arrow

4 participants