I've just finished something I'd been circling for most of a year: a pipeline that takes Wikidata's JSON dumps and turns them into fully schematised, native Parquet datasets. I don't want to let the moment pass without writing it up.
The JSON schema inference library at its heart, polars-genson, appeared in my end of year writeup in a semi-finished state, and Wikidata was meant to be its first proper outing. When I wrote the first two parts of this series in January, I got as far as pointing it at real data and hit bugs that proved show-stopping, chiefly very high peak memory usage (RSS). The project stalled there.
In September I came back to it with coding agents. I'd describe the errors I was seeing, and the agents would write tests and benchmarks to reproduce them. Then they'd dig through the convoluted logic of how a schema inference program walks a large, deeply nested Wikidata file, and find things I had never managed to spot. They did incredibly well. I tweeted little glimpses as it went, and this post is my attempt to take stock of how far it actually came. The journals the agents kept as they worked make that much easier to reconstruct.
Speeding up polars-genson
First, a sense of where I'd left it. polars-genson 0.7.4, released back on 17th January, could handle the Map types. But pointed at a single real Wikidata source file (1,818 entities, 1.65 GB of claims JSON), it took almost 5 minutes just to infer the claims schema, and in the agent's 4 GB container that same file was killed for running out of memory on every version. Extrapolating from a handful of chunk files to the full 1.6 TB source gave roughly 148 hours single-threaded, and that was just for schema inference, before any of the actual processing. It wasn't a true pipeline so much as a proof that one could be written. Part of me felt like there were insurmountable problems owing to the allocator (a recurring theme in the Polars issue tracker that has at times been marked as a "won't fix"), but upon revisiting it I found there were more solvable causes of the high memory load.
The first headline numbers I tweeted on 24th September showed going from polars-genson 0.7.4 to 0.7.6 took a chunk from 35s to 3s, an 11× speedup. Soon after, another 25% came off the wall time and peak memory fell by a third. That one came from eliminating an unnecessary extra pass over the data, which I'd begrudgingly accepted as a workaround for a bug in the recursive schema inference that I couldn't fix (but Claude Opus 5.5 later could!).
That day I quickly iterated through another six releases from 0.7.5 to 0.7.10, in the course of about 16 hours. What made it possible to move that fast without breaking anything was a tiny benchmark harness (bench_claims) that ran inference on a 4,851-row slice of claims and printed the wall time alongside a hash of the output schema. Every change either kept the hash or it didn't, so the agent could chase speed without me having to re-check correctness by eye each time.
The profile it took of 0.7.6 was quite damning: building the schemas cost 4.6 CPU-seconds, while converting them to their final form and dropping the builders cost 18.4. to_schema was cloning every subtree on the way out, and moving the values instead of copying them took the slice from 3.1s to 2.2s (#189). Then the agent ran a neat ablation where it simply leaked each merged row schema rather than freeing it. That showed deallocation alone was ~30% of the serial merge, so the drops got pushed onto the thread pool. A few more allocation cuts brought it to 1.5s.
Once the serial merge was the bottleneck, the trick was to notice that the root object's properties are independent subtrees, so each one can merge on its own thread (#190). That was then bounded by the slowest property, so it recurses down through nested objects and arrays too, which brought it to 1.12s. Then came the ones that read like confessions (#191). At every object level, the extra-keywords handling deep-copied the properties it had already seen, which is quadratic in nesting depth. Fixing it cut peak memory from 3.24 to 2.18 GB. And rewrite_objects had a loop that recursed into a properties map as if it were a schema, rewriting every subtree again, 2^depth times over. Both fixes left the output byte-identical. Last came a one-liner-in-spirit: the Python plugin was copying the entire input column into owned Rust strings (~1.6 GB on a big chunk) when it could just borrow them (#193).
By the end of the day, on that same big source file that took 298s in January, 0.7.10 took 6.5s at 5 GB peak RSS: about 46× faster.
I also liked that the agent kept a "measured and not kept" list in its journal. It tried overlapping the merge of one chunk with the build of the next (the merge just doubled under contention), parallelising property serialisation (slower), and a fold/reduce merge (13.5s, and a different schema). It even hashed every property subschema to see if deduplicating them would pay off, and found 21 KB of duplication in 165 MB. That's the kind of negative result I'd never have bothered to write down myself, and it saves going back round the same loop later.
There was one side discovery that day that I'm quite pleased about. The inferred schema's key order changed between runs of the same input, and had done since 0.6.0. I'd noticed "unresolved non-determinism" back then and never tracked it down (I had two commits in that window more or less saying "still not found the source of randomness"). The agent bisected it by building the library at each commit, and landed on a simd-json 0.13 → 0.17 bump. The newer simd-json stores objects with more than 32 keys in a randomly seeded hash map. Wikidata claims are keyed by property ID, so any entity with more than 32 properties got a shuffled schema. Reading values from simd-json's tape instead, which walks keys in document order, fixed it at no cost in speed (#207).
The next bottleneck was outside inference entirely. Timing the whole claims step showed inference at 6.4s, while decoding the normalised output took 28.8s of about 50s. The pipeline wrote normalised JSON strings to a temporary parquet file, read it back, and decoded it with str.json_decode. PyArrow could decode the same data in under 3s, so the data was fine; the round trip through JSON strings was the waste. polars-genson 0.8.0 added a typed=True mode that writes typed Arrow structs directly (#195), and the pipeline switched over that same evening.
On 26th September, when I pulled out ‘invariants’ in the data processing approach the processing time fell 10x and peak memory load halved.
The invariants came out of a review session in which I had an agent measure a single real source file properly (10,000 entities, 625 MB of claims JSON). It found that 97.7% of the claims bytes were label maps. In the philippesaade parquet, every time a claim refers to another entity it carries that entity's entire multilingual label set along with it: the item's labels next to its ID, the property's labels next to every snak's property, the unit's labels next to every unit. That one file had 154,783 of these maps but only 14,257 distinct IDs, and no ID ever came with two different maps. With the maps stripped out, normalising the claims went from 26.3s and 5.75 GB peak RSS to 1.22s and 0.39 GB, with the same schema minus those three fields.
The naming took a bit of thought, which is all recorded in the journal. "Normalise" was already taken: in genson it means making each row conform to the inferred schema, so I couldn't use it in the database sense. The agent surveyed what others call this: normalizr calls it "normalize" into "entities", JSON:API calls it "sideloading", star schemas would call the result a dimension table, and dlt/Airbyte have "child tables" that split nested data out without deduplicating. The framing I liked best was the database one. Each label map is functionally dependent on its sibling key, which is its determinant, so the option became extract_invariants={"labels": "id", "property-labels": "property", "unit-labels": "unit"} (#206). Those fields get pulled out before inference, each distinct one is written once to a lookup table, and if a field turns out not to be invariant (one key, two different values) it's an error rather than silently picking one. On that big 1,818-entity file it took the claims step from 21.9s to 4.2s and peak RSS from 9.64 to 4.16 GB.
The less comfortable part of that review was that it turned up data I'd been losing without realising. The normalisation step wrote out only the normalised column, so the entity ID simply wasn't in any of the output tables. The checks that would have caught it were sitting there commented out. Null input rows were being skipped too, so an output could be shorter than its input and couldn't be lined back up by position (#196 fixed both, and added keep_columns to carry the ID through). My favourite was a quantity filter on unit-labels.list.len() > 0 and its negation, which was meant to split quantities with units from those without. Dimensionless quantities have null unit labels, so the comparison was null and both filters dropped them. That meant population (P1082) was being thrown away, which is not a small thing to lose from Wikidata!
The same day I had an agent write a proper docs site for polars-genson, and writing worked examples turned out to be a very effective way to find bugs. Several of them silently dropped data and went all the way back to 0.3.0:
- a field that's
1in one row and1.5in another inferred as a float, but the integer normalised to null (#203) - a field that's a string in some rows and an object in others lost the string under default options, unless you happened to pass
unify_maps=True, which luckily the Wikidata pipeline did (#201) - the JSON Schema route turned every nullable field into a
String, because a type array fell through to the default case (#202), so I made Avro the default route instead (#200) - always-empty arrays came out as
List(Boolean), because the first non-null branch of avrotize's catch-all union happens to be boolean (#204)
A field that's an integer in some rows and a string in others still keeps only the first type. The agent tried a fix, but it clashed with the string coercion option, so it went into the journal as a documented, parked problem instead.
And one that wasn't my bug at all: bumping arrow/parquet from 53 to 60 made the Rust parquet writer mark float columns with a new column order, which Polars before 1.43.2 rejects as "Invalid thrift". The agent found this by walking the raw Thrift footers of a file written by each version and diffing them. Every claims file has float columns (latitudes and longitudes), so the fix was simply to raise the minimum Polars version (#205).
Getting a full run through
With the library in decent shape, the bottleneck moved into my own pipeline code. That evening I set the full run going and it was managing a chunk every 313s on average. There are 7,449 chunks, which projected to about 27 days.
Profiling one chunk showed where it went: 366s, of which 74% was partitioning the claims by language. I had settled on a rule that a claim goes into language L if its property or its entity has a label in L, which seemed reasonable. The trouble is that core properties like P31 ("instance of") have labels in almost every language, so nearly every claim went into nearly every language. That one chunk of 10,000 entities turned into 48 million (claim, language) rows, and each language's "subset" of claims was practically the whole table again.
So I stopped splitting claims by language altogether. Each claim is now just one row, and the labels it refers to live in a separate claims_labels table (field, ref, language, label) which is split by language, alongside the entities' own labels. If you want claims in Welsh you join to the Welsh labels, and any language rule you like becomes a join you choose to do rather than one I bake in. The same night the pipeline also got grouped uploads with sha256 post-checks, a fresh subprocess per chunk (to contain memory), local files deleted as soon as they're processed, and every chunk's claims conformed to one stored schema.
The last snag before the run was snaks on properties that have since been deleted from Wikidata (P450, P4003). They show up malformed: either a main snak collapsed to a bare property ID string, or an error message sitting where the value should be. I first handled them in Polars after normalisation, by exploding down to individual snaks to find the bad ones and then rebuilding the claims column with nested list.eval/list.filter. That took about 9s of a 13s chunk to deal with 7 bad snaks across the first 190 chunks. The agent then wrote a design doc for doing it inside genson instead, as prune (#212). You name fields, they're removed from the schema, and any record holding one is dropped during normalisation. The drop cascades up to the nearest list element or map entry, and anything left empty goes too, with everything removed written to a side file so nothing is lost silently. Success criteria went in the design doc before any code (matching the old output exactly and staying within 5% of the time without the quarantine at all), and in the pipeline it came out about 3× faster.
The full run started at 20:03 on 27th September and finished its 7,449th chunk at 09:39 on the 29th: about 37.5 hours, or 18s a chunk against the 313s three days before. That's 73.8 million entities, uploaded to the Hub in 31 groups.
Making it nice to use
Finishing the run wasn't quite the same as finishing the datasets. Uploading one file per language per group meant labels had 17,075 files and links 25,057, averaging 58 KB each. When I downloaded the links dataset (1.45 GB in total) it crawled along at about 14 files/s, or 50 kB/s. The download speed followed the file count, not the bytes. So the next step was compaction: rewrite each language's files into files of about 500 MB, check each new file against the files it replaces with a row-hash fingerprint, and swap them over on the Hub in single commits so there's never a moment when the data is half there. In the end almost every language is one file. Along the way the agent found a genuine pyarrow bug: casting a struct with a null-typed field inside a sliced list returns a child array of the wrong length, which some cast methods catch and others don't. It wrote it up with a minimal repro and a matrix of which calls are affected (in the repo under docs/pyarrow_bug_report/) which I submitted and was soon patched.
The rows were still in the order of the source chunks, which wander back and forth through the ID space. So looking up one item by ID couldn't skip a single row group of the English labels. Sorting every table by ID fixed that. The sort is in string order, so that Parquet's min/max statistics prune exactly (numeric order would leave row groups straddling Q999999/Q1000000 useless), and it's stable, so alias and statement order survive. All 774 million claims rows, about 300 GB in Arrow memory, went through a bucketed sort. As a bonus the sort shrank everything (labels by 16%, descriptions by 25%) because sorted IDs compress better. Looking up Q42's English label now takes 45 ms, and all 337 of its claims rows come back from 34 files in 86 ms.
The dataset cards are now rendered from the data rather than hand-written: one subset per language with English as the default, sizes and row counts filled in, and a note on Wikidata's language fallback order (your language, then its MediaWiki fallbacks, then mul, then English) with a Polars example of how to coalesce along it. The renderer refuses to run if any of its figures are stale. I also learned the hard way that Norwegian's language code no becomes the boolean false in YAML unless you quote it.
Then, at last, I got to actually use it. With a local copy of all six datasets I wrote a bunch of little demo scripts: walking an item's ancestors, tabulating a country's divisions with their ranks and qualifiers, laying out an item's history from its dated statements, and finding which items carry BBC, Google Knowledge Graph or WordNet IDs. That snowballed into training a Matryoshka sparse autoencoder on the sets of external identifiers each item has (31.9 million items over 7,752 identifier properties, about an hour and a half on a 3090 for the first run). It turns out "which catalogues list you" is a surprisingly good fingerprint: the Kalman filter's nearest neighbours come out as random walk, Monte Carlo method and control theory. I also built a little browser app that reads the published parquet straight off the Hub with range requests, no server involved. It ran a browser tab out of memory on its first outing, until the postings went into typed arrays (508 MB → 76 MB of heap per search).
Closing the loop: from the official dump
Everything so far was built on the philippesaade parquet, which was a lightly ingested form of the Wikidata JSON dump with each field still a JSON string. To really close the loop I wanted to go from the official dump itself, the 103 GB wikidata-20260928-all.json.bz2 from dumps.wikimedia.org, all the way to the final datasets. That also meant keeping everything. philippesaade's copy had dropped a lot of fields (snak types, hashes, statement IDs, qualifier order, sitelink badges, datavalue types), and on a sample the statement IDs alone turned out to be a quarter of the processed claims bytes. The new split-dump streams the bz2 through lbzip2 into 10,000-entity chunks, keeping every field, and adds a seventh table for the entity-level fields. That came to 12,182 chunks and 121.8 million entities.
philippesaade's dataset card said it had filtered out "scholarly articles" but didn't give the rule, so I took a shot at working it out myself, comparing the IDs in the results I'd processed against the ones in the latest JSON dump (I also asked but didn't expect an immediate reply). An anti-join of the new dump's IDs against my existing claims, plus a survey of each missing entity's "instance of" values, found 46.4 million entities missing (the card's own figures imply 46.41 million). 45.4 million of those being scholarly articles. There are 38 classes that are 99% or more missing, and they're what you'd expect: editorials, case reports, meta-analyses, theses, errata, retraction notices… and for some reason MDMA (all 1,866 entities that are instances of it). So now each release is 'routed' into two sets: the scholarly works (46.4 million entities) go to new wikidata-scholar-* datasets, and everything else (75.4 million) goes to the existing ones. Each set is built on a branch and only 'promoted' (i.e. merged) to main once it's finished, with the old philippesaade-based data kept under a 20260507 tag.
The new source also found three more genson bugs, each one stopping the run in turn, so I fixed and released each one and set it going again:
- sitelinks where one item's badges were
[]and another's["Q…"]didn't unify, because an empty array's schema has no item type (#213) - a chunk where every entity's aliases were
{}inferred an empty struct, which Parquet refuses to write; it's now a map of nulls (#214) - and the best one (#215): from 10 rows up, inference switches to a parallel path that ran each row's schema through the forced-field-types pass before merging, and that pass reset every forced map's values to
string. So any forced map over 10 or more rows came out as a map of strings.
That last one means the sitelinks in the whole first run had come out as JSON strings all along, quietly papered over by a json_decode further down the pipeline. It only showed up now because the first few dozen scholarly chunks had just a handful of entities each, and it finally bit on chunk 57 (49 entities, two of them with a sitelink).
The release run itself is a single command, just release 20260928 20260507, which processes the scholarly set and then the main set, finalises both (the claims labels have to wait until both sets' labels exist), and then promotes them. Every step can be resumed just by rerunning it. Watching it go, each chunk cost about 4.4s fixed plus 1.5s per MB of source, and the machine sat at about half its 20 cores. Part of that fixed cost turned out to be every chunk reading the state files of all 13,000 chunks, about six times over at 0.4s a time. So on 4th October chunks started reading only their own state, running several at a time (each still in its own process), and uploading finished groups in the background while the next chunks carry on. A quick benchmark gave 1.48× going from 2 to 8 workers on a small sample, with peak memory flat at 16–20 GB however many workers there were, so it now defaults to 6.
There was one self-inflicted wound. Each chunk's process imports the pipeline code fresh from disk, so editing the code mid-run changes what the next chunk runs. For a few minutes one function took keyword-only arguments, the still-running parent called it positionally, and a chunk fell over. Putting the signature back fixed it and the chunk reran cleanly, but it's a good reminder that a long-running job is effectively running whatever is on disk at that moment. And on 5th October the main set's chunk numbers passed 9,999, which broke the 4-digit file names (they stopped sorting in order, and compaction stopped dead), so those now pad to the width of the last chunk number. The sort also now reuses the files compaction just uploaded instead of downloading them all again, which would have been 44.7 GB for the claims alone.
I am fairly sure this is still only the beginning of what can be done in terms of speeding up the processing, and throwing more agent compute at the problem could get the total pipeline duration down further, but I'm mainly very happy to have gotten over the finish line in the first place, and then the next iteration going from the raw JSON with the latest data dump and all scholarly articles is just icing on the cake.
Lest it go unsaid, this new pipeline and the resulting datasets are a step change in the accessibility of Wikidata, which can now be consumed without the rigmarole nor the compute toll of over a terabyte of hard drive space and many hours of processing in which it is very possible for data to systematically be corrupted or go missing. I am very keen to make the dataset obtained from this both visible and comprehensible, and my next steps will involve improving the documentation around this from a user story perspective (which again coding agents are helping me simulate).
I've already made a start on that (though it won't come to fruition until the latest run from raw JSON completes).
This weekend I had an agent read only the README and the source code, and write down every question a confused user might go hunting in the docs for: "which table has the English name of Q42?", "what is mul?", "how do I get the best-ranked value per property, like the Wikidata UI does?", "do I need a token just to read?", and so on, 96 in all.
A second pass then graded each question by how many pages you'd have to click through from where you'd naturally start before reaching the sentence that answers it (which I termed the 'hops', indicating whether the docs are intuitive to navigate), or whether it was answered anywhere at all (which I termed 'reachability', again appropriating graph theory terminology).
The biggest oversight was structural rather than any single gap, which was that I had overlooked that the docs site was written entirely for someone running the pipeline. Everything a person using the data needs (the claims schema, the joins, language fallbacks, the sort order, subsets) lived on the Hugging Face dataset cards, which the docs site doesn't include. The README had also fallen behind and still described six tables built from the philippesaade copy, so you couldn't tell which build main actually holds. Besides that, several things the pipeline simply doesn't support, like building only some languages or skipping the scholarly set, were never stated, so their absence read as "undocumented" rather than "no". These are in turn potential hints at ways to improve on the pipeline.
The first round of fixes, staged on a branch for now, adds a "Using the data" page organised by task: which table to use for what, choosing statements by rank, dates and their precision (BCE included), quantity units, exploding qualifiers and references, pinning a release, and a list of terms. It also says plainly what isn't supported, and which settings are environment variables versus edits to config.py. A few questions are still open: reading the tables from pandas, DuckDB or Spark, a list of every datatype value, redirects and deleted entities, a minimum amount of RAM, and how often releases will be updated. The dataset cards themselves (defining mul where they use it, linking the terms list) have to wait until this run finishes, since they get rendered and pushed as part of finalising it.
Besides the UX side of things, there is still an unsatisfying number of passes over the data which some more careful thought might be able to evade. I wrote the "compacting" and sorting passes over the data with constraints of hard drive space in mind, but an empirical approach might be able to find a way to use different data structures or what have you to finalise the files sooner.
I think the core pass is around 12 hours and then the dataset reprocessing adds another couple hours or so on top: any improvement here could be a big deal. This is where I see coding agents adding a big boost to development: the parts that would be nice to have but require a total rewriting in pursuit of some refreshed end goal become trivial mechanical operations that can be derisked thoroughly. The risk to one's mental model falling apart when not having these tools for though to lean on can be a barrier to this kind of experimentation.
More specifically, it can also allow rapid optimisations of processes with guarantees on them being safe to resume (without losing progress) to allow for experimentation with paralellisation in the moment, which is how I just stopped the parquet packing operation and restarted it rewritten with paralellism (3x lower wall time). While of course such feats were possible prior to coding agents, they implied a risk (at most, of wasting time, that diluted the potential upside) that is now close to instantaneous.
I wrote a script for ETA estimation during one of the steps, and then later had it revised in moments to give a "you are here" view for the entire pipeline. I'd like to intensify this with some profiling of where time was spent overall, and of course this is trivial to add now. With this approach I don't doubt that the bottlenecks can be driven down even further.