Wikidata:SPARQL query service/WDQS graph split

Tracked in Phabricator
Task T337013

The Search team is currently running an experiment to split the Wikidata Query Service graph and use federation for the queries that need access to all subgraphs. This is a breaking change, which will require a number of queries to be rewritten, either to access a new SPARQL endpoint, or to use federation. We want to have a good understanding of the trade-offs before we commit to any long-term solution.

The reasons behind the plan

The underlying issue that we are trying to address is the medium term scalability and stability of Wikidata Query Service, which can hinder access to and the possibility to query the data in Wikidata. The Wikidata Query Service runs on top of Blazegraph, and comprises 15 billion triples. The graph is currently growing at the rate of 1 billion triples per year.

With the current size and growth of the graph, we are experiencing a number of scalability issues:

• The reloading (rebuilding) of the graph from the Wikidata dumps takes between 1 and 2 months, sometimes more. In part this depends on how long the operation is, in part the reloading time is extended because it can unpredictably crash once the graph reaches a certain size, requiring the process to be restarted
• More frequent stability issues with WDQS
• The queries are taking a longer time to run, with more frequent timeouts

The ability to reload the graph is a critical function in order to ensure data consistency and be able to recover from potential critical data issues. It is an indication of the stability and scalability of the system. Furthermore, the instability of the data reload process is directly linked to the size of the graph, in a similar way that the runtime stability of WDQS is linked to the size of the graph.

The experiment: splitting the scholarly articles graph

For the reasons outlined before, we will run an experiment: we will split the graph, moving the scholarly articles to a separate graph. This will reduce the size of the constituent graphs, thereby increasing the stability of WDQS by reducing the size of the data queried across each of the graphs and size of data involved in a full graph refresh. It will support continued linear growth of the graph, providing time to assess other solutions to address the graph scalability issues.

It is important to remember that this is just an experiment, and that it will take time to implement. No decisions will be applied immediately, and all relevant decisions will be taken considering the input of the larger Wikidata community and the Scholia community, with the latter being the most impacted by such experiment. Our scope is to limit, as much as possible, the impact of such a decision on Wikidata’s tools and community actions.

The current proposal about the split is available at Wikidata:SPARQL query service/WDQS graph split/WDQS Split Refinement.

FAQs

What is the timeline of this project?

In February 2024, we set up the test servers and made them available to the public for testing and providing feedback. In April 2024, we conducted internal tests based on query logs. The refinement of the graph split was completed by the end of July.

In August 2024, the update process was modified to support a split graph, and production servers were made available with a split graph updated in real time. The “query-scholarly” endpoint serves all entities related to scholarly articles, while the “query-main” endpoint serves everything else. For more details, have a look at the WDQS Graph Split Rules.

Starting from September 2024, we plan a 6-months transition period where the three SPARQL endpoints (query.wikidata.org, query-main.wikidata.org, query-scholarly.wikidata.org) will remain in place until the end of the transition.

The transition period will end in March 2025, when query.wikidata.org will serve only the main Wikidata subgraph (without scholarly articles). The query-main and query-scholarly endpoints will continue to be available after the transition.

Are scholarly articles going to be removed from Wikidata?

No. The experiment will only split the current query graph, moving the scholarly articles to a separate graph in Blazegraph. This will not impact the management of current and future content via the Wikidata website and other APIs.

Is WMF asking to remove scholarly articles and/or other kinds of data from Wikidata?

No. Wikimedia Foundation does not want to and will never take editorial decisions on behalf of the community. Wikimedia Foundation also believes the Wikidata community should be the only actor in charge of deciding what is notable and what is not, throughout its established community processes.

Are you planning to test other splits after scholarly articles?

No. The experiment on scholarly articles is for the time being the only experiment that we are planning to do. At the moment, there is no need to plan on other experiments on other kinds of data in Wikidata.

What are you doing to limit the impact of such experiment on Wikidata tools?

We are in contact with the Scholia community, which will likely be the most impacted, to understand the extent of the impact and provide the necessary support to mitigate eventual negative effects. We are also investigating if other tools will be impacted and we plan on getting in contact with the tools’ maintainers to provide necessary support. If you anticipate this will have a significant effect on your tools and workflows, we would appreciate it if you could reach out to us in the talk page, in the Phabricator ticket for community feedback or by pinging directly Sannita (WMF).

My tool/gadget is or might be affected by the graph split. What can I do?

You can list your affected tools or gadgets on Wikidata:SPARQL query service/WDQS graph split/Affected tools, we'll try to mitigate the effect of the graph split on your tool or gadget during the transition period.

What if I need help to rewrite a query that doesn't work anymore following the split?

We updated the page Wikidata:Request a query to include the possibility for people to request a query rewrite in case they need it.

What are the plans for replacing Blazegraph right now?

For now we are focusing our work on this split, as we anticipate that it will buy us time to make better decisions about which alternative to move to. Reducing the size of the graph also makes the replacement easier in the future.

Where should feedback be sent?

You can reach out to us using the talk page, the Phabricator ticket for community feedback or by pinging directly Sannita (WMF).