Joining Wikidata IDs to OSM

[abhisheksaikia]

Objective: Joining matched wikidata id's to OSM City and Towns POIs

Worksheet https://docs.google.com/spreadsheets/d/1Gh2-T20DuXMoRhUwx_L6FYcaHufOwcwpHm-fe0gkRxI/edit#gid=0 - View

OSM Diary Scaling multilingual name tags with Wikidata

---

Why join wikidata and OSM?

An OpenStreetMap feature linked to a wikidata item is augmented with extra information that is stored in the wikidata database. The level of information that can be gathered for an OSM feature when it is linked to wikidata is immense. For example, you can get name translations from Wikidata for places in OSM, find population of countries - all using the API.

Workflow:

We have created spreadsheets for OSM city and town features which have corresponding wikidata entries with exact name matches.

1. Click on a link in the JOSM column, the OSM feature with it's corresponding wikidata tag is displayed in JOSM. Click on add selected tags to add the wikidata tag

1. Before uploading check:
2. Case 1: Match distance is close to 0
   • Wikidata type matches to a City> Town> Commune> human settlement > Municipality
   • Check if the address matches in both OSM and Wikidata
   • Check the description matches to a city or town
   • Case 1A: There is only one match candidate, add the wikidata tag to OSM
   • Case 1B: If there is still more than one match, Wikidata has duplicate entries for the place. eg. OSM Feature id 958710338; Wikidata entries: Q1993763, Q2001643

• Case 2: Match distance is high (>10km)
  • Wikidata type matches to a City> Town> Commune> human settlement > Municipality
  • Check if the address matches in both OSM and WIkidata
  • Case2A: If only one match, coordinate in Wikidata is wrong, add the wikidata id to OSM
• Case 3: Anything else has no match in Wikidata. Manual matching has to be done to find out if a Wikidata entry exists or not.

Changeset comment: Adding Wikidata tag to Cities and Towns https://github.com/mapbox/mapping/issues/242
Changeset source: Wikidata

In the added column in the spreadsheet, enter:

yes=added the match, manual=added a wikidata match after a manual search, no=did not add a wikidata tag

Resources:

• Adding wikidata entries through ID: https://www.mapbox.com/blog/id-wikidata/
• Linking wikidata to OSM through JOSM plugins: http://wiki.openstreetmap.org/wiki/Key:wikidata

cc @mapbox/team-data

[bhousel]

Nice work! This is a great step forward for OSM.
cc @pigsonthewing & @1ec5, thought you would find this interesting..

[pigsonthewing]

Thank you; yes. I've requested access to the worksheet.

[1ec5]

Should city boundary relations be tagged with wikidata too?

If there is still more than one match, Wikidata has duplicate entries for the place. eg. OSM Feature id 958710338; Wikidata entries: Q1993763, Q2001643

This turns out to be a village of the same name as a larger town nearby. Hopefully cases like this are rare.

Wikidata type matches to a City> Town> Commune> human settlement > Municipality

Note that Wikidata’s subclass hierarchy is a little unexpected. Normally I have to perform this search in the Wikidata Query Service:

SELECT ?settlement
WHERE {
  # Municipalities (and any subclasses thereof)
  { ?settlement wdt:P31/wdt:P279* wd:Q15284 . }
  # and cities (and any subclasses thereof)
  UNION { ?settlement wdt:P31/wdt:P279* wd:Q515 . }
  # but not metropolitan areas (or any subclasses thereof)
  FILTER NOT EXISTS { ?settlement wdt:P31/wdt:P279* wd:Q1907114 . }
  …
}

Note how city isn’t directly related to municipality, and how some metropolitan areas are also tagged as cities.

[planemad]

@1ec5 lesser than 20% of the WIkidata entries have a subclass (P31) , most of them have a description though which are really useful for context. But since its free-form text it requires a manual check. You can see here how some names match perfectly with Wikidata items electoral districts and railway station, but the subclass is not defined.

Except for these outliers, most entries seem to match perfectly even if it does not have a subclass.

[planemad]

Over 5,300 Cities and Towns have been joined to Wikidata in the last 4 days using this workflow.

http://overpass-turbo.eu/s/jGy

[1ec5]

lesser than 20% of the WIkidata entries have a subclass (P31) , most of them have a description though which are really useful for context.

You mean an entry has an “is a” (P31) statement, but it’s too generic to associate with an OSM place tag? Like how Q526757, depicted in the screenshot above as having no wd_type, is generically tagged as a human settlement? That’s true; my suggestion was only to make sure to look at municipalities and cities but not metropolitan areas. If you’re already looking at all human settlements, including Q526757, then that’s fine. 👍

[planemad]

Another 3,100 places got updated in the last two days.

http://overpass-turbo.eu/s/jWM

[maning]

OSM-Canada community started working on this for all place=town/city in the country!

• Dairy - http://www.openstreetmap.org/user/LogicalViolinist/diary/39872
• TM project - http://tasks.osmcanada.ca/project/40

[maning]

I also requested @mapconcierge -san to help us with the Japan places.

[bsrinivasa]

Here are the spreadsheet links for the following cities which doesn't have an english names and needs wikidata matching for neighbourhoods and suburbs.

• Tokyo
• Kyoto
• Osaka
• Chiang Mai

cc: @maning @mapconcierge -san

[DenisCarriere]

@maning Thanks for the post, we've also started doing all of Africa. We've only been mapping for about 1-2 hours at 3-5 people and we've gotten pretty far so far.

http://tasks.osmcanada.ca/project/42

Using Add #wikidata to #Africa places to track our changesets.

Here's the SPARQL query we are running:

http://tinyurl.com/zhs93zu

SELECT DISTINCT ?place ?location ?distance ?placeDescription ?name_en ?name_fr ?name_es ?name_de ?name_it ?name_ru WHERE { 
  # Search Instance of & Subclasses
  ?place wdt:P31/wdt:P279* ?subclass
  FILTER (?subclass in (wd:Q486972))

  # Search by Nearest
  SERVICE wikibase:around { 
    ?place wdt:P625 ?location . 
    bd:serviceParam wikibase:center "Point(13.397 52.514)"^^geo:wktLiteral .
    bd:serviceParam wikibase:radius "15" . 
    bd:serviceParam wikibase:distance ?distance .
  }

  # Filter by Exact Name
  OPTIONAL {?place rdfs:label ?name_en FILTER (lang(?name_en) = "en") . }
  OPTIONAL {?place rdfs:label ?name_fr FILTER (lang(?name_fr) = "fr") . }
  OPTIONAL {?place rdfs:label ?name_es FILTER (lang(?name_es) = "es") . }
  OPTIONAL {?place rdfs:label ?name_de FILTER (lang(?name_de) = "de") . }
  OPTIONAL {?place rdfs:label ?name_it FILTER (lang(?name_it) = "it") . }
  OPTIONAL {?place rdfs:label ?name_ru FILTER (lang(?name_ru) = "ru") . }

  FILTER (regex(?name_en, "^Berlin$") || regex(?name_fr, "^Berlin$") || regex(?name_es, "^Berlin$") || regex(?name_de, "^Berlin$") || regex(?name_it, "^Berlin$") || regex(?name_ru, "^Berlin$")) .

  # Get Descriptions
  SERVICE wikibase:label {
    bd:serviceParam wikibase:language "en,fr,es,de,it,ru"
  }

} ORDER BY ASC(?dist)

We decided to do 15km since a lot of cities weren't getting tagged, we could always drop it down to 10km if we wanted to, it's a parameter.

I've built a Javascript Geocoder that runs the SPARQL query mentioned above.

https://github.com/deniscarriere/geocoder-geojson

$ npm install -g geocoder-geojson
$ geocode --provider wikidata --nearest [-75.7,45.4] --radius 15 Ottawa | jq .features[0].id
"Q1930"

[woodpeck]

Everyone, please be aware that if you run a query against another database to retrieve tags that you then add to objects in OSM, you're doing a mechanical edit that requires prior consultation with the community.

If you edit an individual object in OSM that you have knowledge of and add the Wikidata tag, that's of course no problem at all. But editing objects that you don't know in areas you have never been to just because a SPARQL query suggests that you edit something is questionable. The addition of Wikidata tags by people without knowledge of the region regularly leads to data quality issues (e.g. different OSM entities for an area administered by a village and the village itself, but the same wikipedia entry for both, and the wikidata entry created from it claims to be just the village).