In Wikidata cache I converted the Wikidata dump into a 396GB SQLite database with one bz2 blob per entity.
It has more than 120 million rows and a mean row of 2452 bytes.
Two things could be improved: the page size and the compression.
Page size
SQLite does not write rows to the file one after another.
It stores the database as a sequence of fixed size pages and packs rows into them, and the file is always a whole number of pages.
The default page size is 4096 bytes.
A row that does not fit in the space left on a page starts a new page.
A row too big for an entire page keeps part of itself there and puts the rest into a chain of extra pages, called overflow pages.
My mean row is 2452 bytes.
Two of them are 4904 bytes and do not fit in a 4096 byte page, one of them does.
So most pages hold a single row, and the remaining 1.6kB of each page is unusable.
The overflow pages are not where the waste is.
SQLite chooses how much of a row to keep in the main page so that whatever spills over fills its overflow pages exactly.
The leftover space in the ordinary pages is the whole problem, and a larger page makes it proportionally smaller, because more rows share the same leftover.
Sampling 20000 random rows and applying SQLite's layout rules to them gives the table for this database.
leaf slack is the wasted space and overflow/row is how many extra pages reading one row has to follow:
page_size table leaf slack overflow/row vs now
512 309.1G 10.6G 4.4
1,024 318.7G 20.9G 1.9
2,048 345.4G 47.6G 0.7
4,096 384.6G 87.7G 0.1 +0.0% <- now
8,192 362.9G 67.2G 0.0 -5.6%
16,384 328.5G 33.2G 0.0 -14.6%
32,768 311.9G 16.6G 0.0 -18.9%
65,536 303.8G 8.5G 0.0 -21.0%
88GB of the 396GB was dead space.
Small pages avoid it as well, because then almost everything lands in overflow pages, which are packed exactly.
But at 512 bytes every read follows 4.4 of them, so I went the other way, to 32768.
page_size can only be set on an empty database, so changing it means writing a new file.
VACUUM INTO does that without touching the source:
sqlite3 wikidata-cache.db "PRAGMA page_size=32768; VACUUM INTO 'wikidata-cache-new.db';"
90 minutes, 396GB to 324GB, 72GB recovered.
The old database served the API throughout and stayed the rollback until the new one was verified.
The output of VACUUM INTO is not in WAL mode whatever the source was, so PRAGMA journal_mode=WAL has to be set again after the swap.
Compression
The old database compressed each entity on its own with bz2.compress(), which defaults to level 9.
I picked that in the first version without comparing it to anything.
Measured on 15000 entities sampled from across the dump:
codec total ratio vs bz2-9 comp decomp
bz2-9 35.4M 6.34 +0.0% 9.4MB/s 64.2MB/s
zstd-3 33.8M 6.65 -4.5% 321.9MB/s 1079.9MB/s
zstd-12 31.5M 7.14 -11.1% 22.4MB/s 1139.0MB/s
zstd-19 30.8M 7.30 -13.1% 3.4MB/s 1066.1MB/s
zstd-12+dict 17.4M 12.91 -50.8% 36.9MB/s 1780.0MB/s
zstd-19+dict 16.2M 13.90 -54.3% 3.7MB/s 1936.0MB/s
xz-6 30.9M 7.27 -12.8% 5.7MB/s 151.5MB/s
bz2 is last on ratio and 34 times slower to compress than zstd-3.
zstd-19 costs ten times the CPU of zstd-12 for two more points.
xz-6 matches zstd-19 on ratio and decompresses eight times slower, which every API request would pay.
On ratio alone none of the plain codecs is worth much, 11 percent at best.
The speed is the real difference.
zstd-12 compresses 2.4 times faster than bz2 and decompresses 18 times faster, and zstd-3 compresses 34 times faster.
Decompression runs on every API request, so 64MB/s against 1139MB/s is what shows up day to day, and it is also why xz is out despite the good ratio.
The size only moves with a dictionary, the two rows at the bottom of the zstd ones.
The new database uses zstd level 12 with a dictionary, which is the next section.
The dictionary
A zstd dictionary is a fixed blob of typical data that the compressor is given up front.
An entity compresses to a couple of kB, and much of that is the first occurrence of boilerplate like mainsnak or snaktype, which the rest of the entity then back-references.
The dictionary pre-loads it, so it is never spelled out.
The size is a permanent decision, because every blob written with a dictionary is unreadable without it, so it is worth sweeping:
dict size total ratio vs no dict train
none 31.5M 7.14
112,640 17.4M 12.91 -44.7% 46s
262,144 15.8M 14.19 -49.7% 59s
524,288 15.4M 14.63 -51.2% 61s
1,048,576 15.3M 14.71 -51.5% 62s
2,097,152 15.4M 14.63 -51.2% 61s
The percentages are against zstd-12 without a dictionary, the 31.5M in the first row.
The ratio peaks at 1MB and falls back at 2MB, so it is a real optimum and not the end of the range.
The zstd-12+dict row in the codec table used zstd's default size of 112640 bytes and produced 17.4M.
At 1MB the same 15000 entities compress to 15.3M, so the default costs 12 percent.
Against the 35.4M of bz2-9 that is 56.8 percent smaller.
zstd hands back a smaller dictionary than asked for when the training data does not support it, roughly 100 times the dictionary size being the rule of thumb.
With 227MB of samples all five came back at the full requested size.
Sampling the dump
When benchmarking a dictionary on this dump the samples have to be spread out.
Consecutive Q-ids are batch imports, thousands of near-identical scholarly articles, genes or asteroids in a row.
15000 consecutive lines made the dictionary look 60.5 percent better than bz2 instead of 50.8, because it memorised one batch and then scored itself on more of the same batch.
The same window put the mean entity at 28.8kB instead of 14.6kB, since the head of the dump holds the old, heavily edited items.
Sampling every 2000th line across half the dump fixed both.
The new conversion
The old version ran eight workers writing eight separate SQLite files and merged them afterwards, because SQLite allows only one writer.
The new version keeps one writer but makes it the parent process: a pool of workers parses and compresses, the parent only inserts.
That removes the merge, which used to write the whole dataset a second time.
The database stays readable during a conversion.
Blobs identify themselves by their first bytes, zstd frames with 28 B5 2F FD and bz2 streams with BZh, so a half-converted table serves correctly and needs no format column:
def decompress(blob):
if blob[:4] == ZSTD_MAGIC:
dict_id = zstandard.get_frame_parameters(blob).dict_id
return _decompressor(dict_id).decompress(blob)
if blob[:3] == BZ2_MAGIC:
return bz2.decompress(blob)
A zstd frame records which dictionary it needs, so a missing one raises instead of returning garbage.
The dictionary itself exists twice.
It is 1MB, small enough to keep in the git repo, and that copy is what the conversion compresses with.
The conversion also writes it into a codec_dict table in the database it builds, and that is the copy the API loads at startup.
The database is self contained that way, so the container image does not need the file at all, and the blobs cannot end up somewhere without the dictionary that decodes them.
The number of lines read is committed in the same transaction as the rows it covers, so a crash loses at most one batch and a restart re-streams the prefix and skips it.
Serving while writing needed one change to the container.
It used to mount the database file itself, read only, which worked as long as nothing was writing to it.
In WAL mode the recent writes sit in wikidata-cache.db-wal next to the database, and a single file mount does not see that file.
SQLite also needs to write the -shm index to read a WAL database at all, so a read only mount cannot open one.
The volume is the directory now, and writable:
volumes:
- ./data:/app/data
The read only part is done by the application instead, with PRAGMA query_only = ON on its connection.
The next dump
Part of the work can be skipped next time.
modified is a column and not something inside the blob, so comparing it against the dump is one index lookup and no decompression.
Once a full pass has finished, the workers drop the entities whose modified did not change.
That saves the compression and the write, but not the bzip2 decompression of the dump and not the json parsing, and those are most of the CPU.
The first update done this way skipped 116.3 million of 121.2 million entities and took 5.9 hours, about twice as fast as a full pass.
The 121 million index lookups turned out to be cheap: the index stays in the page cache, so a lookup costs far less than the compression and the write it replaces.
The result
396GB to 141GB, so 2.8 times smaller.
72GB of that was the page size and the rest the codec with its dictionary.
Per row the compressed entity went from 2604 bytes to 1166.
Reads are 28 times faster to decompress, 64MB/s against 1780MB/s, so an API request now spends its time on SQLite and on json parsing rather than on bz2.
Compressing is 3.9 times faster too, which is what makes a full pass affordable at all.
The full conversion is 2.4 times faster than the old version that wrote eight databases and merged them, on the same machine.
Another part that matters is that the database was readable the whole way through, apart from a container restart when the new file replaced the old one.
I can update the database a lot faster now when a new wikidata dump is released.