Progress counter on indexes is wrong (per-partition, not global)

[campoy]

I'm building an index on a really large dataset (the 96 repos with the most commits under google's github org).

Ran this:

CREATE INDEX file_by_lang ON files USING pilosa (LANGUAGE(file_path, blob_content));

And it started creating it right away, at 22:53:32.

time="2018-10-10T22:53:32Z" level=debug msg="executing query" query="\nCREATE INDEX file_by_lang ON files USING pilosa (LANGUAGE(file_path, blob_content));\n"
time="2018-10-10T22:53:32Z" level=info msg="starting to save the index" async=true driver=pilosa id=file_by_lang
time="2018-10-10T22:53:33Z" level=debug msg="still creating index" driver=pilosa duration=1.467891121s id=file_by_lang rows=10000

At 01:00:51 (that's over one hour later) I see this in the logs:

nohup.out:time="2018-10-11T01:00:51Z" level=debug msg="still creating index" driver=pilosa duration=1.394008128s id=file_by_lang rows=33070000
nohup.out:time="2018-10-11T01:00:59Z" level=debug msg="still creating index" driver=pilosa duration=5.915422321s id=file_by_lang rows=10000

WAT? After one hour of computing the index simply restarted and the rows went down to zero?

There's no other logs in between, so no other query was made that could have dropped the previous index or anything like that.

I've also considered whether this is some kind of overflow, but I don't see how. 33070000 is close to 33554432 (2^25) but since the increase of rows is 10000 I would have expected to see 33080000.

Any ideas on what might be going on in here?

You can find the logs attached: file_by_lang.log

Also, if you prefer to see all of the logs in my session, they're right here: nohup.log

[campoy]

update: it keeps on happening, i'll kill the server now

time="2018-10-10T22:53:33Z" level=debug msg="still creating index" driver=pilosa duration=1.467891121s id=file_by_lang rows=10000
time="2018-10-10T23:05:14Z" level=debug msg="still creating index" driver=pilosa duration=1.428551252s id=file_by_lang rows=10000
time="2018-10-11T01:00:59Z" level=debug msg="still creating index" driver=pilosa duration=5.915422321s id=file_by_lang rows=10000
time="2018-10-11T01:04:02Z" level=debug msg="still creating index" driver=pilosa duration=2.001288622s id=file_by_lang rows=10000
time="2018-10-11T01:04:29Z" level=debug msg="still creating index" driver=pilosa duration=1.817709661s id=file_by_lang rows=10000
time="2018-10-11T01:04:39Z" level=debug msg="still creating index" driver=pilosa duration=1.224002423s id=file_by_lang rows=10000
time="2018-10-11T01:05:42Z" level=debug msg="still creating index" driver=pilosa duration=7.263005495s id=file_by_lang rows=10000
time="2018-10-11T01:56:55Z" level=debug msg="still creating index" driver=pilosa duration=2.55158547s id=file_by_lang rows=10000
time="2018-10-11T01:59:55Z" level=debug msg="still creating index" driver=pilosa duration=3.607210652s id=file_by_lang rows=10000
time="2018-10-11T02:00:57Z" level=debug msg="still creating index" driver=pilosa duration=1.461399622s id=file_by_lang rows=10000
time="2018-10-11T02:05:09Z" level=debug msg="still creating index" driver=pilosa duration=4.931261785s id=file_by_lang rows=10000
time="2018-10-11T02:20:40Z" level=debug msg="still creating index" driver=pilosa duration=1.422601895s id=file_by_lang rows=10000
time="2018-10-11T02:21:13Z" level=debug msg="still creating index" driver=pilosa duration=1.509145105s id=file_by_lang rows=10000
time="2018-10-11T02:24:15Z" level=debug msg="still creating index" driver=pilosa duration=2.279904939s id=file_by_lang rows=10000
time="2018-10-11T03:13:31Z" level=debug msg="still creating index" driver=pilosa duration=1.6092169s id=file_by_lang rows=10000
time="2018-10-11T03:54:01Z" level=debug msg="still creating index" driver=pilosa duration=1.519160989s id=file_by_lang rows=10000
time="2018-10-11T03:58:27Z" level=debug msg="still creating index" driver=pilosa duration=1.207460815s id=file_by_lang rows=10000
time="2018-10-11T03:59:00Z" level=debug msg="still creating index" driver=pilosa duration=1.523911022s id=file_by_lang rows=10000
time="2018-10-11T04:00:07Z" level=debug msg="still creating index" driver=pilosa duration=1.444332686s id=file_by_lang rows=10000
time="2018-10-11T04:00:59Z" level=debug msg="still creating index" driver=pilosa duration=1.347002636s id=file_by_lang rows=10000
time="2018-10-11T04:04:30Z" level=debug msg="still creating index" driver=pilosa duration=1.540499027s id=file_by_lang rows=10000
time="2018-10-11T04:06:01Z" level=debug msg="still creating index" driver=pilosa duration=1.462244348s id=file_by_lang rows=10000
time="2018-10-11T04:11:38Z" level=debug msg="still creating index" driver=pilosa duration=5.355223753s id=file_by_lang rows=10000
time="2018-10-11T04:24:16Z" level=debug msg="still creating index" driver=pilosa duration=1.508795459s id=file_by_lang rows=10000
time="2018-10-11T04:25:06Z" level=debug msg="still creating index" driver=pilosa duration=2.431633323s id=file_by_lang rows=10000
time="2018-10-11T04:34:21Z" level=debug msg="still creating index" driver=pilosa duration=1.483318352s id=file_by_lang rows=10000
time="2018-10-11T04:35:15Z" level=debug msg="still creating index" driver=pilosa duration=2.44362682s id=file_by_lang rows=10000
time="2018-10-11T04:48:59Z" level=debug msg="still creating index" driver=pilosa duration=1.25975142s id=file_by_lang rows=10000
time="2018-10-11T04:49:39Z" level=debug msg="still creating index" driver=pilosa duration=1.464206805s id=file_by_lang rows=10000

[campoy]

This happens with the latest release and my older version too.
But it started happening once I started using the Google dataset containing the repos in this list:

Details

https://github.com/google/ggrc-core https://github.com/google/closure-compiler https://github.com/google/WebFundamentals https://github.com/google/skia-buildbot https://github.com/google/closure-library https://github.com/google/protobuf https://github.com/google/boringssl https://github.com/google/or-tools https://github.com/google/guava https://github.com/google/ExoPlayer https://github.com/google/j2objc https://github.com/google/paco https://github.com/google/conscrypt https://github.com/google/personfinder https://github.com/google/syzygy https://github.com/google/liquidfun https://github.com/google/material-design-lite https://github.com/google/caja https://github.com/google/error-prone https://github.com/google/trace-viewer https://github.com/google/honggfuzz https://github.com/google/certificate-transparency https://github.com/google/coursebuilder-core https://github.com/google/kythe https://github.com/google/cadvisor https://github.com/google/sanitizers https://github.com/google/oss-fuzz https://github.com/google/volley https://github.com/google/dagger https://github.com/google/shaka-player https://github.com/google/science-journal https://github.com/google/gapid https://github.com/google/google-api-dotnet-client https://github.com/google/traceur-compiler https://github.com/google/breakpad https://github.com/google/nomulus https://github.com/google/guice https://github.com/google/google-api-php-client https://github.com/google/closure-templates https://github.com/google/iosched https://github.com/google/googletest https://github.com/google/mtail https://github.com/google/oauth2client https://github.com/google/instrumentation-java https://github.com/google/gson https://github.com/google/codeworld https://github.com/google/rekall https://github.com/google/kati https://github.com/google/key-transparency https://github.com/google/keytransparency https://github.com/google/physical-web https://github.com/google/flatbuffers https://github.com/google/cpp-netlib https://github.com/google/tracing-framework https://github.com/google/fonts https://github.com/google/cayley https://github.com/google/blockly-android https://github.com/google/google-api-ruby-client https://github.com/google/gdata-python-client https://github.com/google/truth https://github.com/google/rekall-profiles https://github.com/google/trillian https://github.com/google/copybara https://github.com/google/lovefield https://github.com/google/gopacket https://github.com/google/namebench https://github.com/google/openhtf https://github.com/google/grr https://github.com/google/fruit https://github.com/google/xi-editor https://github.com/google/google-api-nodejs-client https://github.com/google/brotli https://github.com/google/google-api-python-client https://github.com/google/edash-packager https://github.com/google/shaka-packager https://github.com/google/auto https://github.com/google/seq2seq https://github.com/google/timesketch https://github.com/google/libcxx https://github.com/google/benchmark https://github.com/google/streamy-dart https://github.com/google/yapf https://github.com/google/jsonnet https://github.com/google/androguard https://github.com/google/go-github https://github.com/google/googleapis https://github.com/google/sagetv https://github.com/google/google-api-java-client https://github.com/google/upb https://github.com/google/puffs https://github.com/google/wuffs https://github.com/google/end-to-end https://github.com/google/vector_math.dart https://github.com/google/nsjail https://github.com/google/santa https://github.com/google/web-starter-kit

It keeps on occurring which impacts the performance of what we'll show on demos.

[jfontan]

The rows counter is now per repository since partitions were added. Each time it starts processing a new repo row count starts again in 0.

We may want to have a global counter as before.

[smola]

@jfontan So index building is not really restarting, it's just the counter that is wrong?

[jfontan]

@smola Yes, it's only the counter that counts rows of each repo independently. Indexing is not restarting.

[campoy]

Yes, we should have either a global counter, or maybe a repository countdown?
Basically showing something telling you how many more repositories are left to analyze.

[ajnavarro]

I tried to use SHOW PROCESSLIST to check the index process but is not working correctly. Issues to track this: src-d/go-mysql-server#454 and src-d/go-mysql-server#455