BigQuery NextPageToken incorrectly returns null

[cometfish]

At some point the NextPageToken field on BigQueryPage stopped working - it was working previously, but now it is always set to null.
I had not changed anything in my code, so I suspect the Google BigQuery API might have changed and broken something...?

Sample code:

BigQueryJob job = bqclient.CreateQueryJob("SELECT * FROM mytable", query_params, query_options);

BigQueryJob result;
do
{
    Thread.Sleep(500);
    result = bqclient.GetJob(job.Reference);
} while (result.Status.State.ToUpper() == "PENDING" || result.Status.State.ToUpper() == "RUNNING");

BigQueryResults res = result.GetQueryResults();
BigQueryPage page = res.ReadPage(100);

At the end of this code, res.TotalRows was 233, page.Rows.Count was 100, but page.NextPageToken was null

(I have since rewritten my code to use bqclient.ExecuteQuery() instead, but thought you might want to look into it.)

[jskeet]

Moved to google-cloud-dotnet as that's where the Google.Cloud.BigQuery.V2 code is.

When you say you've rewritten your code to use ExecuteQuery instead - does it then work? Or is it still failing in the same way? Could you tells us what query_params and query_options you're using? If I could reproduce this, that would really help.

Please also say what version of the library you're using - you say you haven't changed anything in the code, but have you changed library version, perhaps?

[cometfish]

Hi @jskeet , I posted in google-api-dotnet-client because that was the URL of the library in the NuGet package screen. My project is running .NET Framework 4.6.1 if that helps.

Yes, it works with ExecuteQuery (ExecuteQuery doesn't use pages, it just has all the results available).

The query_params only has parameters set for my query (ie. startDate and endDate).
The query_options is just a new QueryOptions(), no fields set/changed.
Were you unable to reproduce it then?

Good thought about updating library version: the library was Google.Apis.Bigquery.v2 version 1.43.0.1805, which I'd last updated in January 2020.
I had tried updating to the latest, version 1.44.1.1888, but the problem was still there.

I was running 1.40.3.1665 from August 2019 - January 2020. I'm not sure exactly when it broke (the function is not used everyday).
I'll revert back to 1.40.3.1665 tomorrow to test if that fixes it.

[jskeet]

I posted in google-api-dotnet-client because that was the URL of the library in the NuGet package screen

It shouldn't be for Google.Cloud.BigQuery.V2. Were you perhaps looking at Google.Apis.Bigquery.v2, which is the underlying library that Google.Cloud.BigQuery.V2 wraps? (That's the library you're referring to below, but it's not the library that contains the types you've been referring to.)

Good thought about updating library version: the library was Google.Apis.Bigquery.v2 version 1.43.0.1805, which I'd last updated in January 2020.
I had tried updating to the latest, version 1.44.1.1888, but the problem was still there.

The library containing BigQueryClient etc is Google.Cloud.BigQuery.V2. Which version of that are you using, and has it changed recently? (As in, have you changed which version you're using?)

Were you unable to reproduce it then?

I haven't tried yet - I wanted more information before I tried to do anything. I'd still like to know the versions of Google.Cloud.BigQuery.V2 before I delve into this further.

[cometfish]

Ok, sorry, I'm following now - the version of the Google.Cloud.BigQuery.V2 is 1.4.0. No new version available.
Was previously 1.3.0 from May 2019 to January 2020.

I will revert it back tomorrow and see if that fixes it.

[jskeet]

Okay, that's helpful, thanks. (There is a 2.x beta release, but no 2.x GA release yet.)

Of particular note, we reimplemented query handling in 1.4.0 to use GetQueryResults instead of ListRows, so that might explain things.

With that information, I'll try to reproduce the issue.

[jskeet]

Okay, I've now reproduced the issue. See the code below - you need to comment out the CreateDataset call after running it once.

Results:

• 1.3.0: valid token
• 1.4.0: null token
• 2.0.0-beta01: null token

Will look into what's causing that now.

using Google.Cloud.BigQuery.V2;
using System;
using System.Linq;

class Program
{
    const string DatasetId = "Issue4678";
    const string TableId = "SampleData";

    static void Main(string[] args)
    {
        if (args.Length != 1)
        {
            Console.WriteLine("Please specify a project ID as the only command line argument");
            return;
        }
        var client = BigQueryClient.Create(args[0]);
        CreateDataset(client);

        var table = client.GetTable(DatasetId, TableId);
        BigQueryJob queryJob = client.CreateQueryJob($"SELECT * FROM {table}", null);
        queryJob = queryJob.PollUntilCompleted();
        BigQueryResults res = queryJob.GetQueryResults();
        BigQueryPage page = res.ReadPage(100);
        Console.WriteLine($"Fetched {page.Rows.Count} rows");
        Console.WriteLine($"Next page token: '{page.NextPageToken}'");
    }

    static void CreateDataset(BigQueryClient client)
    {
        client.CreateDataset(DatasetId);
        var schema = new TableSchemaBuilder
        {
            { "TextField", BigQueryDbType.String }
        }.Build();
        var table = client.CreateTable(DatasetId, TableId, schema);
        var rows = Enumerable.Range(0, 300)
            .Select(index => new BigQueryInsertRow { ["TextField"] = $"Row {index}" });
        table.InsertRows(rows);
    }
}

[cometfish]

Great, thanks for looking into it :)

(also, I'm noting down .PollUntilCompleted() for future ref - much nicer than my example!)

[jskeet]

Okay, I've found and fixed the issue - it was a typo, basically, referring to an instance field instead of a local variable.
It fails the same way for me with ExecuteQuery.

Given the severity of this, I'm going to suggest creating a 1.4.1 patch version as well as fixing it for the next 2.x beta release.

Slightly alarmed that this wasn't caught by tests... that's the next thing to fix.