Skip to content

BigQuery insertAll always retries operation #3321

Description

@neuromantik33

Hi. First time reporter here.
I wanted to report a small but perhaps inconvenient bug in BigQuery, specifically regarding streaming inserts using an InsertAllRequest. It seems for some reason unknown to me that inserts are retried using the client's RetrySettings whenever row ids are supplied for deduplication.
When the current implementation always attempts a retry (which I discovered during these odd retry errors) due to the fact that Lists.transform() is strictly lazy.
Anyhow the fix is minor and I corrected it on my branch but I'm unsure how to write the necessary tests in order to validate my claim. My commit is here: neuromantik33@984b5bb, and after proper review I can submit a pull request.

Thanks in advance for any feedback.

Activity

  1. added
    type: questionRequest for information or clarification. Not an issue.
    api: bigqueryIssues related to the BigQuery API.
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on May 30, 2018
  2. andreamlin commented on Jun 4, 2018

    @andreamlin
    Contributor

    Hi @neuromantik33,

    Thanks for catching this issue! Would you mind turning your commit into a pull request? For testing, one option could be to add a unit test to BigQueryImplTest.java, set a spy on the BigQueryOptions options (see line 274), and assert that getRetrySettings() is called on that spy object when BigQueryImpl.insertAll is run or not run.

  3. self-assigned this
    on Jun 5, 2018
  4. removed
    🚨 criticalP0 critical issue. Requires immediate fix
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on Dec 3, 2018
  5. added a commit that references this issue on Dec 3, 2018
    79705d7
  6. added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    priority: p2Moderately-important priority. Fix may not be included in next release.
    and removed
    type: questionRequest for information or clarification. Not an issue.
    on Dec 4, 2018
  7. added a commit that references this issue on Dec 4, 2018
    174e62d
  8. added a commit that references this issue on Feb 3, 2019
    47c840e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

🚨 criticalP0 critical issue. Requires immediate fixapi: bigqueryIssues related to the BigQuery API.priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions