WMDE/Wikidata/ORES
ORES is a machine learning platform maintained by WMF. We use ORES extensively in Wikidata and have developed parts of it related to Wikidata. Most notably item quality model.
How to build item quality model
For models, you need to find the repo that holds the model and its features. For item quality it's articlequality. Clone it.
After installing python requirements (requirements.txt file), you can simply run "make wikidatawiki_models". The Makefile has more details on what commands will be run and what can cause issues. The makefile will also create the model info file which exists in model_info/wikidatawiki.item_quality.md. It also creates the model binary file in models/wikidatawiki.item_quality.gradient_boosting.model.
Model info file has critical information on ratio of false positives, thresholds, ROC-AUC, etc.
To test the model file you can simply run such python code:
import mwapi
from revscoring import Model
from revscoring.extractors.api.extractor import Extractor
with open("models/wikidatawiki.item_quality.gradient_boosting.model") as f:
scorer_model = Model.load(f)
extractor = Extractor(mwapi.Session(host="https://www.wikidata.org",
user_agent="test"))
feature_values = list(extractor.extract(123456789, scorer_model.features))
print(scorer_model.score(feature_values))
Preparing the model for deployment
If you added features and you're sure it'll improve the system, after getting the features reviewed and merged, you need to retrain the model. The binary files merged to the articlequality repo must be the exact same as production (yeah, I know...). You need to find out what OS ores100x is on (currently stretch) and retrain the model in on a VM that has the same OS. Retraining the model as said is simply running "make wikidata_models". If this command doesn't do anything, simply delete the model file (and datasets/wikidatawiki.labeling_revisions.w_cache*) first.
If you're adding more labeled data (and not adding features), you need to change the makefile, look at dependencies of the wikidata model in the make file, look where the datasets are coming from and how they are built, simply add another dataset, cat it in the final step and then run "make wikidata_models" to trigger a retrain.
For deployment, follow the general guideline: ORES/Deployment
Run the model on a dump
If you re-trained the model and now want to have scores on revisions of wikidata based on a dump, you can run extract_scores utility in articlequality (there is no guarantee it would work, it's ancient, you might have to tweak it). Here's the bash file that produces the dump monthly in stat1005:
month=$(date +"%Y%m")
day="${month}01"
source /home/ladsgroup/p3/bin/activate
cd /home/ladsgroup/articlequality
./utility extract_scores /mnt/data/xmldatadumps/public/wikidatawiki/${day}/wikidatawiki-${day}-pages-articles[1234567890]?*.xml-*.bz2 --model models/wikidatawiki.item_quality.gradient_boosting.model --sunset ${day}000000 --processes=10 --score-at monthly --verbose > run_${month}.out 2> run_${month}.err
grep ${month} run_${month}.out > run_${month}_1.out
mv run_${month}_1.out run_${month}.out
mv run_${month}.out wikidata_quality_snapshot_${month}.tsv
gzip -k wikidata_quality_snapshot_${month}.tsv
The resulting output is something like:
page_id item_id rev_id timestamp class weighted_sum 15791782 Q14126127 1006231840 20191001000000 C 2.9696165555741985 21934496 Q20219489 1022984964 20191001000000 D 2.029816923665906 20434497 Q18881996 900205935 20191001000000 C 3.3918374898232493 25711961 Q23708018 982636318 20191001000000 C 2.91177394658644 23914320 Q21877494 1013198591 20191001000000 C 2.9855789615867576 14840084 Q13223924 881834471 20191001000000 C 3.0241936745417326 25414320 Q23405795 914178204 20191001000000 C 2.918603674849337 21934498 Q20219490 1015121079 20191001000000 D 2.029353280459076 23914321 Q21877495 1011135621 20191001000000 B 3.555012588220033