Matrix status messages from a cron job

The Wikidata cache now updates itself from cron. If an update is happening I want to get a status message about the results. The script posts into a Matrix room when it loaded something, and stays quiet when no new Wikidata dump was found and no update happened.

The bot uses my own Matrix server instance and has its own account.

Creating the room

The bot account creates the room and invites me:

room = httpx.post(
    f"{base}/_matrix/client/v3/createRoom",
    headers={"Authorization": f"Bearer {token}"},
    json={
        "name": "wikidata-cache",
        "topic": "Status of the automatic Wikidata dump loads",
        "preset": "private_chat",
        "invite": ["@me:cress.space"],
    },
).json()

The response is one line:

{"room_id": "!AbCdEfGhIjKlMnOpQr-StUvWxYz0123456789abcdef"}

That id is stored and used by the script to send messages later. An alias like #wikidata-cache:cress.space has to be resolved into the id first, one request more per message.

Minting a token

The password, given by the create-user command in the conduit admin room, is only used once, to get an access token:

login = httpx.post(
    f"{base}/_matrix/client/v3/login",
    json={
        "type": "m.login.password",
        "identifier": {"type": "m.id.user", "user": "@bot:cress.space"},
        "password": password,
        "device_id": "wikidata-cache-cron",
        "initial_device_display_name": "wikidata-cache-cron",
    },
).json()

device_id is fixed, so a second login replaces that device instead of registering another one.

The token, not the password, is what goes on disk. A token can be revoked from any Matrix client without changing the account password. It does not expire on its own either: a login that does not ask for a refresh token gets one that is valid until the device is logged out, the password changes, or someone revokes it. Mine lives in a matrix.toml next to the script:

homeserver = "https://chat.cress.space"
room = "!AbCdEfGhIjKlMnOpQr-StUvWxYz0123456789abcdef"
token = "syt_..."

Sending the message

Sending is a PUT, with a transaction id in the URL that makes a repeated send idempotent:

httpx.put(
    f"{base}/_matrix/client/v3/rooms/{room}/send/m.room.message/{time.time_ns()}",
    headers={"Authorization": f"Bearer {token}"},
    json={"msgtype": "m.notice", "body": text},
).raise_for_status()

m.notice instead of m.text, so clients that mute notices can do so.

The room id goes percent-encoded into the URL: it starts with ! and holds a :, so urllib.parse.quote(room, safe="") makes %21AbCdEf...%3Acress.space of it.

Sending an image

Another cron of mine pushes a photo once a day, until now to ntfy. I moved it to Matrix so ntfy only gets alarms.

Matrix can handle images too, but it needs two calls instead of one. The image is uploaded to the media repository first, which answers with an mxc:// URI:

MXC=$(curl -sf -X POST \
    -H "Authorization: Bearer $MATRIX_TOKEN" \
    -H "Content-Type: image/jpeg" \
    --data-binary "@$IMAGE" \
    "$MATRIX_HOMESERVER/_matrix/media/v3/upload?filename=$NAME" | jq -r .content_uri)

The media endpoint takes a POST with --data-binary.

Then the same PUT as a text message, with m.image and the URI in it:

curl -sf -X PUT \
    -H "Authorization: Bearer $MATRIX_TOKEN" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg url "$MXC" --arg name "$NAME" \
        '{msgtype:"m.image", body:$name, url:$url, info:{mimetype:"image/jpeg"}}')" \
    "$MATRIX_HOMESERVER/_matrix/client/v3/rooms/$ROOM/send/m.room.message/$(date +%s%N)"

ntfy's prio:low has no equivalent here; m.image has no notice variant, so quiet is a per-room setting in the client.

Conduit Matrix homeserver behind Caddy

I added Conduit to the Caddy docker-compose setup on my small server. It is a Matrix homeserver in Rust, one container with an embedded RocksDB. The upstream docker-compose.yml works as is.

The homeserver runs on chat.cress.space, but the server_name is cress.space, so accounts are @user:cress.space.

conduit:
  image: matrixconduit/matrix-conduit:latest
  user: 1000:1000
  volumes:
    - ./conduit-db:/var/lib/matrix-conduit
  environment:
    CONDUIT_SERVER_NAME: cress.space
    CONDUIT_DATABASE_PATH: /var/lib/matrix-conduit/
    CONDUIT_DATABASE_BACKEND: rocksdb

CONDUIT_SERVER_NAME is written into the database on first start.

Delegation

Because the server_name and the host the server runs on are different, other servers have to be told where to look. Two static JSON responses in the cress.space Caddy block do that.

handle /.well-known/matrix/server {
  header Content-Type application/json
  respond `{"m.server": "chat.cress.space:443"}`
}

handle /.well-known/matrix/client {
  header Content-Type application/json
  respond `{"m.homeserver": {"base_url": "https://chat.cress.space"}}`
}

The Synapse delegation docs mention an optional port 8448, which I am not routing. Federation only falls back to 8448 when there is no delegation and no SRV record, and the .well-known sends everyone to chat.cress.space:443, where Caddy already has a certificate. I used federationtester and it reports FederationOK: true for IPv4 and for IPv6.

The first user

Registration is protected by CONDUIT_REGISTRATION_TOKEN, which needs a client supporting the m.login.registration_token flow. I used SchildiChat web for registration, but switched to Cinny afterwards. The homeserver to enter is cress.space, not chat.cress.space.

The first user that registers is invited into an admin room, where a @conduit:cress.space bot takes commands. Chatting for admin use cases is weird at first, but easier than opening a shell into the docker container.

Registration can not be closed from the start though. There is an admin command to toggle it, but it needs an admin, and there is no admin until somebody registered. So: open registration, register once, close it again.

Adding users later

After that registration can stay closed, because the admin room creates users directly. The command list is not in the docs, so I ran strings over the binary in the image.

cid=$(docker create matrixconduit/matrix-conduit:latest)
docker export $cid | tar -x -O --wildcards '*conduit' | strings -n 6 > conduit.txt

Besides the documented appservice commands there are create-user, reset-password, deactivate-user, show-config and allow-registration. create-user works no matter what allow_registration is set to, and without a password Conduit generates one and prints it.

@conduit:cress.space: create-user new-username

Next steps

Over the next weeks I want to move some of my crons from ntfy to Matrix. The plan is to use Matrix for status messages and ntfy for alerts.

Who owns the supermarkets in every German district

The YAML file with all German districts turned out to be useful for more than a cycling score. It has an osm_id, inhabitants and area per district, so anything counted per district can be normalized.

Counting supermarkets per brand is not very interesting, Aldi and Lidl are everywhere. I wanted to know who owns them, because a lot of the different signs in the German grocery trade belong to the same few groups.

Getting the shops

shop=supermarket in Germany is about 34,000 objects. I fetch them from my selfhosted Overpass one federal state at a time, into one cache file per state. The state boundary comes from ISO3166-2, so there are no ids to look up:

[out:json][timeout:900];
relation["boundary"="administrative"]["admin_level"="4"]["ISO3166-2"="DE-BW"];
map_to_area->.state;
nwr["shop"="supermarket"](area.state);
out tags center;

That is 19 MB of JSON for all 16 states.

Assigning each shop to a district is the same trick as in the cycling post: fetch the 400 boundary relations by osm_id, stitch the member ways with linemerge and polygonize, subtract the inner ways so an enclaved kreisfreie Stadt does not count twice, then a shapely STRtree over the polygons. The boundaries are the biggest download at about 117 MB, cached as a 55 MB GeoJSON.

From brand to owner

My first version matched the brand tag against a list of chain names. That list grew to 40 entries, and every one of them is a decision I had to make myself: whether E-Center counts as Edeka, or whether a bare Netto is the unrelated Danish chain. And it says nothing about who owns what.

The better key was already in the data: 81.7% of the shops carry brand:wikidata. Wikidata answers the ownership question with owned by (P127) and parent organization (P749), in one query for all 64 QIDs that appear.

SELECT ?brand ?brandLabel ?ownerLabel WHERE {
  VALUES ?brand { wd:Q701755 wd:Q879858 ... }
  OPTIONAL { ?brand wdt:P127|wdt:P749 ?owner. }
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en,de". }
}

Wikidata names the regional cooperative that formally owns a brand, so Edeka arrives as "Edeka Minden-Hannover" and "Edeka Südwest", and those get folded onto the group. Aldi Nord, Aldi Süd, Norma and Globus have no owner recorded at all, they are the group themselves.

For the 18.3% without a QID the chain-name matching is still useful as a fallback, and it recovers 941 shops: mostly Edeka, Norma, Penny and Lidl branches where the mapper typed the name and skipped the identifier. The remaining 5,265 stay not identifiable, and they are the long tail: 4,775 distinct names, 4,367 of which appear exactly once, i.e. Tante-M, Dorfladen, Ihr Kaufmann or Mein Markt.

Shops per group

group             shops  share %
----------------  -----  -------
Edeka              9982     34.9
Rewe               6370     22.3
Schwarz-Gruppe     4042     14.1
Aldi Nord          2203      7.7
Aldi Süd           2013      7.0
Norma              1347      4.7
Dennree             380      1.3
Salling Group       343      1.2
Migros              297      1.0

not identifiable   5265        -

Shares are over the 28,580 shops that can be attributed to a group, 84.4% of the total. The five big groups hold 24,610 of those, 86.1%.

Edeka's 34.9% is 5,181 shops under its own QID -- Edeka, E-Center, nah und gut -- plus 4,319 Netto Marken-Discount. Netto Marken-Discount is Edeka's discounter, so it looks like a competitor in the shop but belongs to the same company. Without that one ownership edge Edeka and Rewe would be a lot closer.

The leading group per district

group             districts led
----------------  -------------
Edeka                       315
Rewe                         73
Schwarz-Gruppe                4
Aldi Nord                     2
feneberg                      2
K+K Klaas & Kock              1
Migros                        1
Norma                         1
V-MARKT                       1

Four of the groups at the bottom are purely regional: feneberg in Kempten and Landkreis Oberallgäu, V-Markt in Kaufbeuren, K+K in Landkreis Grafschaft Bentheim, Norma in Fürth. Migros leads one district, Landkreis Fulda, where tegut has 20 of the 95 shops. Fulda is where tegut comes from.

Leading a district says nothing about how big the lead is. In 36 districts the top group holds more than half the shops, and the extreme is Landkreis Straubing-Bogen in Bavaria: 31 of 37 identifiable shops are Edeka group, 22 with an Edeka sign and 9 Netto Marken-Discount. That is a Herfindahl index of 7,093 on the 0--10,000 scale, where a competition authority calls anything above 2,500 highly concentrated. The median district sits at 2,421.

Store counts are a rough proxy for market share -- a Kaufland hypermarket and a Penny count as one shop each.

Supermarkets per inhabitant

Germany has 40.9 supermarkets per 100,000 inhabitants. Per federal state that runs from 53.2 in Mecklenburg-Vorpommern down to 34.6 in Hamburg, and per district from 73.0 in Landkreis Landsberg am Lech to 27.0 in Bottrop.

Both ends of both lists are the wrong way round from what I assumed. The correlation between population density and shops per 100,000 inhabitants is -0.33: the denser a district, the fewer supermarkets per person. Rural districts under 150 inhabitants per km² have a median of 46.6 per 100,000, urban districts over 1,500 have 37.7.

A rural district needs a shop in a lot of small towns, each of them serving a few thousand people, while a city can put one large store where 20,000 people walk past it. The per-capita number counts shops and says nothing about their size or how far away the next one is.

How good is the data

All of the above rests on OSM being evenly mapped, and it is not. brand:wikidata coverage runs from 52% in Delmenhorst to 98% in Landkreis Oberspreewald-Lausitz, and 58 of 400 districts are below 75%.

Per state the spread is smaller: Brandenburg 90%, Sachsen-Anhalt 89%, Mecklenburg-Vorpommern 88% at the top, Bremen 70%, Baden-Württemberg 77% and Hamburg 79% at the bottom. Coverage correlates -0.30 with population density -- the east German rural districts are the best-tagged part of the country, the western cities the worst.

Tagging quality does not explain the density result, though. Coverage against shops per 100,000 inhabitants correlates -0.04, so effectively not at all. Poorly tagged districts report the same number of shops, with less information attached.

For the concentration numbers the missing shops do matter. An unidentified shop is far more likely to be an independent than a chain, so leaving them out pushes every group's share up. The 86% for the big five is an upper bound on store count, not a measured market share.