Getting the shops

shop=supermarket in Germany is about 34,000 objects. I fetch them from my selfhosted Overpass one federal state at a time, into one cache file per state. The state boundary comes from ISO3166-2, so there are no ids to look up:

[out:json][timeout:900];
relation["boundary"="administrative"]["admin_level"="4"]["ISO3166-2"="DE-BW"];
map_to_area->.state;
nwr["shop"="supermarket"](area.state);
out tags center;

That is 19 MB of JSON for all 16 states.

Assigning each shop to a district is the same trick as in the cycling post: fetch the 400 boundary relations by osm_id, stitch the member ways with linemerge and polygonize, subtract the inner ways so an enclaved kreisfreie Stadt does not count twice, then a shapely STRtree over the polygons. The boundaries are the biggest download at about 117 MB, cached as a 55 MB GeoJSON.

From brand to owner

My first version matched the brand tag against a list of chain names. That list grew to 40 entries, and every one of them is a decision I had to make myself: whether E-Center counts as Edeka, or whether a bare Netto is the unrelated Danish chain. And it says nothing about who owns what.

The better key was already in the data: 81.7% of the shops carry brand:wikidata. Wikidata answers the ownership question with owned by (P127) and parent organization (P749), in one query for all 64 QIDs that appear.

SELECT ?brand ?brandLabel ?ownerLabel WHERE {
  VALUES ?brand { wd:Q701755 wd:Q879858 ... }
  OPTIONAL { ?brand wdt:P127|wdt:P749 ?owner. }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en,de". }
}

Wikidata names the regional cooperative that formally owns a brand, so Edeka arrives as "Edeka Minden-Hannover" and "Edeka Südwest", and those get folded onto the group. Aldi Nord, Aldi Süd, Norma and Globus have no owner recorded at all, they are the group themselves.

For the 18.3% without a QID the chain-name matching is still useful as a fallback, and it recovers 941 shops: mostly Edeka, Norma, Penny and Lidl branches where the mapper typed the name and skipped the identifier. The remaining 5,265 stay not identifiable, and they are the long tail: 4,775 distinct names, 4,367 of which appear exactly once, i.e. Tante-M, Dorfladen, Ihr Kaufmann or Mein Markt.

Shops per group

group             shops  share %
----------------  -----  -------
Edeka              9982     34.9
Rewe               6370     22.3
Schwarz-Gruppe     4042     14.1
Aldi Nord          2203      7.7
Aldi Süd           2013      7.0
Norma              1347      4.7
Dennree             380      1.3
Salling Group       343      1.2
Migros              297      1.0

not identifiable   5265        -

Shares are over the 28,580 shops that can be attributed to a group, 84.4% of the total. The five big groups hold 24,610 of those, 86.1%.

Edeka's 34.9% is 5,181 shops under its own QID -- Edeka, E-Center, nah und gut -- plus 4,319 Netto Marken-Discount. Netto Marken-Discount is Edeka's discounter, so it looks like a competitor in the shop but belongs to the same company. Without that one ownership edge Edeka and Rewe would be a lot closer.

The leading group per district

group             districts led
----------------  -------------
Edeka                       315
Rewe                         73
Schwarz-Gruppe                4
Aldi Nord                     2
feneberg                      2
K+K Klaas & Kock              1
Migros                        1
Norma                         1
V-MARKT                       1

Four of the groups at the bottom are purely regional: feneberg in Kempten and Landkreis Oberallgäu, V-Markt in Kaufbeuren, K+K in Landkreis Grafschaft Bentheim, Norma in Fürth. Migros leads one district, Landkreis Fulda, where tegut has 20 of the 95 shops. Fulda is where tegut comes from.

Leading a district says nothing about how big the lead is. In 36 districts the top group holds more than half the shops, and the extreme is Landkreis Straubing-Bogen in Bavaria: 31 of 37 identifiable shops are Edeka group, 22 with an Edeka sign and 9 Netto Marken-Discount. That is a Herfindahl index of 7,093 on the 0--10,000 scale, where a competition authority calls anything above 2,500 highly concentrated. The median district sits at 2,421.

Store counts are a rough proxy for market share -- a Kaufland hypermarket and a Penny count as one shop each.

Supermarkets per inhabitant

Germany has 40.9 supermarkets per 100,000 inhabitants. Per federal state that runs from 53.2 in Mecklenburg-Vorpommern down to 34.6 in Hamburg, and per district from 73.0 in Landkreis Landsberg am Lech to 27.0 in Bottrop.

Both ends of both lists are the wrong way round from what I assumed. The correlation between population density and shops per 100,000 inhabitants is -0.33: the denser a district, the fewer supermarkets per person. Rural districts under 150 inhabitants per km² have a median of 46.6 per 100,000, urban districts over 1,500 have 37.7.

A rural district needs a shop in a lot of small towns, each of them serving a few thousand people, while a city can put one large store where 20,000 people walk past it. The per-capita number counts shops and says nothing about their size or how far away the next one is.

How good is the data

All of the above rests on OSM being evenly mapped, and it is not. brand:wikidata coverage runs from 52% in Delmenhorst to 98% in Landkreis Oberspreewald-Lausitz, and 58 of 400 districts are below 75%.

Per state the spread is smaller: Brandenburg 90%, Sachsen-Anhalt 89%, Mecklenburg-Vorpommern 88% at the top, Bremen 70%, Baden-Württemberg 77% and Hamburg 79% at the bottom. Coverage correlates -0.30 with population density -- the east German rural districts are the best-tagged part of the country, the western cities the worst.

Tagging quality does not explain the density result, though. Coverage against shops per 100,000 inhabitants correlates -0.04, so effectively not at all. Poorly tagged districts report the same number of shops, with less information attached.

For the concentration numbers the missing shops do matter. An unidentified shop is far more likely to be an independent than a chain, so leaving them out pushes every group's share up. The 86% for the big five is an upper bound on store count, not a measured market share.

Redirect and Pages

Redirects come from the wiki's own table dumps. wikidatawiki-latest-redirect.sql.gz is 36MB and holds roughly 5 million item redirects, about one for every 25 entities in the cache.

But it is only half of the answer. A row looks like this:

(3545,0,'Q92258250','','')

rd_title is the target and is already an entity id. rd_from is the source, and it is a page id. redirect is a per page attribute table, so it names its row the way every such table in MediaWiki does, by the page it belongs to, and the entity id appears nowhere in it.

The only place that mapping exists is the page dump (wikidatawiki-latest-page.sql.gz), which is 3.4GB. So the 36MB table costs a 3.4GB download for one join.

The import

The two rows that make up one redirect, one from each dump:

redirect  (3545,0,'Q92258250','','')
page      (3545,0,'Q2577',1,0,0.243262073359,'20250714122000','20240219185646',1511092905,62,'wikibase-item',NULL)

Page 3545 is Q2577, Messier 86, merged into Q92258250, a duplicate item for the same galaxy. The 1 in fourth position is page_is_redirect, and the page_len of 62 is the entire content of a redirect page. Since both rows open with a page id, a namespace and a title, one regular expression reads either dump.

The import needs two passes. The first reads redirect into a scratch table, the second streams page and keeps only the rows whose page id appears in it. That filter is a Python set rather than a lookup against the scratch table, because it runs once per row of page, about 120 million times, and discards all but five million on that first test. The set costs 300MB and saves 120 million index probes.

Answer with a 307

Special:EntityData answers a redirected id with the target entity, keyed under the target's id. The API could copy that, but then a caller that looks up the id it asked for finds nothing and stores an entity with no claims, and the response looks perfectly normal while it happens. A 307 to /Q106842805.json cannot be ignored by accident.

It does have to be opted into on the other side. httpx does not follow redirects unless told to:

client = httpx.Client(follow_redirects=True)

That alone was not enough. The cache keys its answer by the id in the path it ended up serving, so a request for Q116513636 comes back as:

{"entities": {"Q106842805": {...}}}

Every call site looked up entities[entity_id] with the id it started from, and that id is not in the response. The lookup has a default of {}, an empty entity has no claims, and storing an entity with no claims raises nothing and looks like an entity that simply has no data. They take the single entity out of the response now, whichever id it is keyed by.

Which table wins

Some of the merged ids still have an entity in the cache. They were canonical when an earlier pass imported them and were merged afterwards, and an upsert pass cannot see an absence, so nothing ever removed the blob. Every cycle adds the ids merged since the last one and none of them leave again.

Looking in entities first serves that stale copy for good and never reaches the redirect, so redirects is checked first instead.

The result

The two passes take 6.4 minutes and build just under 5 million redirects.

13 sources had no row in the page dump. That count is the check on whether the two dumps are from the same date, and 13 is low enough.

73 redirects point at another redirect and one of those chains runs three hops. Nothing in the client has to know that, since follow_redirects=True walks a chain the same as a single hop. No chain is a cycle and none of them ends on an id the cache does not have.

MediaWiki bots flatten double redirects, which is why it is 73 and not five figures. The first chain in the table is Q5485936 to Q27956032 to Q25506055, three items all called census of agriculture, and upstream it is already gone: both of the first two point straight at Q25506055 now. A dump only catches the chains created since the last bot pass.