Page History

...

The 1.5.0 SourceForge models must be fully compatible with the 1.5.3
release. In this test all the English models are tested for compatibility
on the English 300k sentences Leipzig Corpus (Which file to download??). It is tested that
the output produced with the same model by both versions has the same md5 hash.

Component	Model	Perf 1.5.2	Perf 1.5.3	Tester	Passed	Comment
Sentence Detector	en-sent.bin	44870.8 sent/s	42733.8 sent/s	William	yes
Tokenizer	en-token.bin	2824.2 sent/s	2833.3 sent/s	William	yes
Name Finder	en-ner-person.bin	781.3 sent/s	761.6 sent/s	William	yes
POS Tagger	en-pos-maxent.bin	773.3 sent/s	816.2 sent/s	William	yes
POS Tagger	en-pos-perceptron.bin		1138.6 sent/s	1117.1 sent/s	William	yes
Chunker	en-chunker.bin		183.7 sent/s	181.1 sent/s	William	yes
Parser	en-parser-chunking.bin		16.0 sent/s	16.3 sent/s	William	yes

Note: Test was done on Hardware running Operational System
and Java Java VersionMacBook Pro 15", 2 GHz Core i7, 16GB Ram, 500GB HD running OS X 10.7.5
and Java 1.5.0_30. The performance varies because light weight tasks have been performed in the background while testing.

...

To pass the test the event hash and the model output must be identical.

Component	Model	Training Time 1.5.3	Tester	Passed	Comment
Sentence Detector	en-sent.bin		Jörn	yes
Tokenizer	en-token.bin	Jörn	yes
POS Tagger	en-pos-maxent.bin		Jörn	yes
POS Tagger	en-pos-perceptron.bin	Jörn	yes
Parser	en-parser-chunking.bin		Jörn		yes	Tested on 10k sentences

Note: Time was measured with the time command, the value is the "real" time value.

...

Component	Data	Tester	Tagging Perf 1.5.2	Tagging Perf 1.5.3	Comment
Sentence Detector
Tokenizer
Name Finder	CONLL 2002 Dutch Person ned.testa	jkosin	Precision: 0.7552941176470588 Recall: 0.4566145092460882 F-Measure: 0.5691489361702128			Precision: 0.7552941176470588 Recall: 0.4566145092460882 F-Measure: 0.5691489361702128
Name Finder	CONLL 2002 Dutch Person ned.testb	jkosin	Precision: 0.8505025125628141 Recall: 0.6165755919854281 F-Measure: 0.7148891235480465	Precision: 0.8505025125628141 Recall: 0.6165755919854281 F-Measure: 0.7148891235480465
Name Finder	CONLL 2002 Dutch Organization ned.testa	jkosin	Precision: 0.8561872909698997 Recall: 0.37317784256559766 F-Measure: 0.5197969543147207	Precision: 0.8561872909698997 Recall: 0.37317784256559766 F-Measure: 0.5197969543147207
Name Finder	CONLL 2002 Dutch Organization ned.testb	jkosin	Precision: 0.7830374753451677 Recall: 0.4501133786848073 F-Measure: 0.5716342692584593	Precision: 0.7830374753451677 Recall: 0.4501133786848073 F-Measure: 0.5716342692584593
Name Finder	CONLL 2002 Dutch Location ned.testa	jkosin	Precision: 0.8458333333333333 Recall: 0.42379958246346555 F-Measure: 0.564673157162726			Name Finder	CONLL 2002 Dutch Location ned.testb		Precision: 0.8816326530612245 8458333333333333 Recall: 0.5581395348837209 42379958246346555 F-Measure: 0.6835443037974683 564673157162726
Name Finder	CONLL 2002 Dutch Misc Location ned.testa testb	jkosin	Precision: 0.8354114713216958 8816326530612245 Recall: 0.44786096256684493 5581395348837209 F-Measure: 0.5831157528285466 6835443037974683	Precision: 0.8816326530612245 Recall: 0.5581395348837209 F-Measure: 0.6835443037974683
Name Finder	CONLL 2002 Dutch Misc ned.testb testa	jkosin	Precision: 0.8264984227129337 8354114713216958 Recall: 0.44144903117101936 44786096256684493 F-Measure: 0.5755079626578803			Name Finder	CONLL 2002 Combined ned.testa	5831157528285466	Precision: 0.6509695290858726 8354114713216958 Recall: 0.628822629969419 44786096256684493 F-Measure: 0.6397044526540929 5831157528285466		1000 iterations
Name Finder	CONLL 2002 Dutch Combined Misc ned.testb	jkosin	Precision: 0.6869929337869668 8264984227129337 Recall: 0.6660746003552398 44144903117101936 F-Measure: 0.6763720690543674		1000 iterations	Name Finder	CONLL 2002 Spanish Person esp.testa	5755079626578803	Precision: 0.9010695187165776 8264984227129337 Recall: 0.5515548281505729 44144903117101936 F-Measure: 0.684263959390863 5755079626578803
Name Finder	CONLL 2002 Spanish Person esp.testb Combined ned.testa	jkosin	Precision: 0.9195205479452054 6509695290858726 Recall: 0.7306122448979592 628822629969419 F-Measure: 0.8142532221379833			6397044526540929	Precision: 0.664424218440839 Recall: 0.6418195718654435 F-Measure: 0.6529263076025666	1000 iterations OPENNLP-417
Name Name Finder	CONLL 2002 Spanish Organization esp.testa Dutch Combined ned.testb	jkosin	Precision: 0.8288942695722357 6869929337869668 Recall: 0.6041176470588235 6660746003552398 F-Measure: 0.6988771691051379			Name Finder	CONLL 2002 Spanish Organization esp.testb	6763720690543674	Precision: 0.8036277602523659 7006019366657943 Recall: 0.7278571428571429 679269221009896 F-Measure: 0.7638680659670164 6897706776603968		1000 iterations OPENNLP-417
Name Finder	CONLL 2002 Spanish Location Person esp.testa	jkosin	Precision: 0.7743016759776536 9010695187165776 Recall: 0.7042682926829268 5515548281505729 F-Measure: 0.7376263970196913			Name Finder	CONLL 2002 Spanish Location esp.testb		684263959390863	Precision: 0.8301886792452831 9010695187165776 Recall: 0.5682656826568265 5515548281505729 F-Measure: 0.6746987951807228 684263959390863
Name Finder	CONLL 2002 Spanish Misc Person esp.testa testb	jkosin	Precision: 0.6492890995260664 9195205479452054 Recall: 0.30786516853932583 7306122448979592 F-Measure: 0.4176829268292683			8142532221379833	Precision: 0.9195205479452054 Recall: 0.7306122448979592 F-Measure: 0.8142532221379833
Name Finder	CONLL 2002 Spanish Misc Organization esp.testb testa	jkosin	Precision: 0.686046511627907 8288942695722357 Recall: 0.3480825958702065 6041176470588235 F-Measure: 0.461839530332681			Name Finder	CONLL 2002 Spanish Combined esp.testa	6988771691051379	Precision: 0.7005423249233671 8288942695722357 Recall: 0.6828315329809239 6041176470588235 F-Measure: 0.6915735567970205 6988771691051379	1000 iterations
Name Finder	CONLL 2002 Spanish Combined Organization esp.testb	jkosin	Precision: 0.756635931824532 8036277602523659 Recall: 0.7611017425519955 7278571428571429 F-Measure: 0.7588622670589884		7638680659670164	Precision: 0.8036277602523659 Recall: 0.7278571428571429 F-Measure: 0.7638680659670164	1000 iterations
Name Finder	CONLL 2003 English Person eng2002 Spanish Location esp.testa	jkosin	Precision: 0.9523195876288659 7743016759776536 Recall: 0.8023887079261672 7042682926829268 F-Measure: 0.8709487330583382 7376263970196913	Precision: 0.95231958762886597743016759776536 Recall: 0.80238870792616727042682926829268 F-Measure: 0.8709487330583382 7376263970196913
Name Finder	CONLL 2003 English Person eng2002 Spanish Location esp.testb	jkosin	Precision: 0.9391727493917275 8301886792452831 Recall: 0.7161410018552876 5682656826568265 F-Measure: 0.8126315789473685 6746987951807228	Precision: 0.93917274939172758301886792452831 Recall: 0.71614100185528765682656826568265 F-Measure: 0.8126315789473685 6746987951807228
Name Finder	CONLL 2003 English Organization eng2002 Spanish Misc esp.testa	jkosin	Precision: 0.8768046198267565 6492890995260664 Recall: 0.6793437733035048 30786516853932583 F-Measure: 0.7655462184873949 4176829268292683	Precision: 0.87680461982675656492890995260664 Recall: 0.679343773303504830786516853932583 F-Measure: 0.7655462184873949 4176829268292683
Name Finder	CONLL 2003 English Organization eng2002 Spanish Misc esp.testb	jkosin	Precision: 0.8435980551053485 686046511627907 Recall: 0.6267308850090307 3480825958702065 F-Measure: 0.7191709844559586 461839530332681	Precision: 0.8435980551053485686046511627907 Recall: 0.62673088500903073480825958702065 F-Measure: 0.7191709844559586 461839530332681
Name Finder	CONLL 2003 English Location eng2002 Spanish Combined esp.testa	jkosin	Precision: 0.9361421988150099 7005423249233671 Recall: 0.7740881872618399 6828315329809239 F-Measure: 0.8474374255065554 6915735567970205	Precision: 0.93614219881500997047866069323273 Recall: 0.77408818726183996869685129855205 F-Measure: 0.8474374255065554 6957635009310986	1000 iterations OPENNLP-417
Name Finder	CONLL 2003 English Location eng2002 Spanish Combined esp.testb	jkosin	Precision: 0.9206349206349206 756635931824532 Recall: 0.7302158273381295 7611017425519955 F-Measure: 0.8144433299899699 7588622670589884	Precision: 0.92063492063492067588711930706902 Recall: 0.73021582733812957633501967397415 F-Measure: 0.8144433299899699 7611041053664006	1000 iterations OPENNLP-417
Name Finder	CONLL 2003 English Misc Person eng.testa	jkosin	Precision: 0.9027982326951399 9523195876288659 Recall: 0.6648590021691974 8023887079261672 F-Measure: 0.7657713928794503 8709487330583382	Precision: 0.90279823269513999523195876288659 Recall: 0.66485900216919748023887079261672 F-Measure: 0.7657713928794503 8709487330583382
Name Finder	CONLL 2003 English Misc Person eng.testb	jkosin	Precision: 0.8592436974789915 9391727493917275 Recall: 0.5826210826210826 7161410018552876 F-Measure: 0.6943972835314092 8126315789473685	Precision: 0.85924369747899159391727493917275 Recall: 0.58262108262108267161410018552876 F-Measure: 0.6943972835314092 8126315789473685
Name Finder	CONLL 2003 English Combined Organization eng.testa	jkosin	Precision: 0.861812521618817 8768046198267565 Recall: 0.8386065297879501 6793437733035048 F-Measure: 0.8500511770726714 7655462184873949	Precision: 0.86406087858872368768046198267565 Recall: 0.84079434533826996793437733035048 F-Measure: 0.8522688502217672 7655462184873949	1000 iterations
Name Finder	CONLL 2003 English Combined Organization eng.testb	jkosin	Precision: 0.8041311831853597 8435980551053485 Recall: 0.7857648725212465 6267308850090307 F-Measure: 0.7948419450165667 7191709844559586	Precision: 0.80648668236999458435980551053485 Recall: 0.78806657223796046267308850090307 F-Measure: 0.7971702337243664 7191709844559586	1000 iterations
Name Finder	CONLL 2003 German Person deuEnglish Location eng.testa	jkosin	Precision: 0.9132653061224489 9361421988150099 Recall: 0.25553176302640973 7740881872618399 F-Measure: 0.3993307306190742		Name Finder	CONLL 2003 German Person deu.testb	8474374255065554	Precision: 0.9361421988150099 Recall: 0.7740881872618399 F-Measure: 0.8474374255065554
Name Finder	CONLL 2003 English Location eng.testb	jkosin	Precision: 0.8732106339468303 9206349206349206 Recall: 0.3573221757322176 7302158273381295 F-Measure: 0.507125890736342			Name Finder	CONLL 2003 German Organization deu.testa	8144433299899699	Precision: 0.8407224958949097 9206349206349206 Recall: 0.4125705076551168 7302158273381295 F-Measure: 0.5535135135135135 8144433299899699
Name Finder	CONLL 2003 German Organization deu.testb English Misc eng.testa	jkosin	Precision: 0.8014705882352942 9027982326951399 Recall: 0.4230271668822768 6648590021691974 F-Measure: 0.5537679932260795			Name Finder	CONLL 2003 German Location deu.testa		7657713928794503	Precision: 0.7816326530612245 9027982326951399 Recall: 0.32430143945808637 6648590021691974 F-Measure: 0.45840813883901854 7657713928794503
Name Finder	CONLL 2003 German Location deuEnglish Misc eng.testb	jkosin	Precision: 0.8033826638477801 8592436974789915 Recall: 0.3671497584541063 5826210826210826 F-Measure: 0.5039787798408487			6943972835314092	Precision: 0.8592436974789915 Recall: 0.5826210826210826 F-Measure: 0.6943972835314092
Name Finder	CONLL 2003 German Misc deuEnglish Combined eng.testa	jkosin	Precision: 0.7055555555555556 861812521618817 Recall: 0.12574257425742574 8386065297879501 F-Measure: 0.21344537815126052			Name Finder	CONLL 2003 German Misc deu.testb	8500511770726714	Precision: 0.6601307189542484 8640608785887236 Recall: 0.15074626865671642 8407943453382699 F-Measure: 0.2454434993924666 8522688502217672		1000 iterations OPENNLP-417
Name Finder	CONLL 2003 German Combined deu.testa English Combined eng.testb	jkosin	Precision: 0.8041311831853597 Recall: 0.7857648725212465 F-Measure: 0.7948419450165667	Precision: 0.7718859429714857 8064866823699945 Recall: 0.319263397475688 7880665722379604 F-Measure: 0.4516978922716628 7971702337243664		1000 iterations OPENNLP-417
Name Finder	CONLL 2003 German Combined Person deu.testb testa	jkosin	Precision: 0.7467566165023353 9132653061224489 Recall: 0.3917778382793357 25553176302640973 F-Measure: 0.5139285714285715			POS Tagger	CONLL 2006 Danish		3993307306190742	Precision: 0.9132653061224489 Recall: 0.25553176302640973 F-Measure Accuracy: 0.9511278195488722 3993307306190742
Name Finder	POS Tagger	CONLL 2006 Dutch		CONLL 2003 German Person deu.testb	jkosin	Precision Accuracy: 0.9324977618621307			POS Tagger	CONLL 2006 Portuguese		Accuracy8732106339468303 Recall: 0.9659110277825124
POS Tagger	CONLL 2006 Swedish		Accuracy: 0.9275106082036775
3573221757322176 F-Measure: 0.507125890736342	Precision: 0.8732106339468303 Recall: 0.3573221757322176 F-Measure: 0.507125890736342
Name Finder	CONLL 2003 German Organization deu.testa	jkosin	Chunker	CONLL 2000	William	Precision: 0.9257575757575758 8407224958949097 Recall: 0.9221868187154117 4125705076551168 F-Measure: 0.9239687473746113			5535135135135135	Precision: 0.8407224958949097 Recall: 0.4125705076551168 F-Measure: 0.5535135135135135
Name Finder	CONLL 2003 German Organization deu.testb	jkosin	Precision: 0.8014705882352942 Recall: 0.4230271668822768	Chunker	Arvores Deitadas (10-fold cross-validation)	William	Precision: 0.9403445830378374 Recall: 0.9373141775994345 F-Measure: 00.9388269348910339

Test UIMA Integration

The test ensures that the Analysis Engine can run and not not
crash trough simple runtime time code errors. We need to add
more sophisticated testing with the next releases.

Analysis Engine	Tester	Passed	Comment
Sentence Detector
Sentence Detector Trainer
Tokenizer ME
Tokenizer Trainer
Name Finder
Name Finder Trainer
Chunker
Chunker Trainer
POS Tagger
POS Tagger Trainer
Parser
createPear.sh
Sample PEAR

Distribution Review

Please ensure that the listed files below are included in the distributions
and are in a good state.

...

Package

...

File or Test

...

Tester

...

Passed

...

Comment

...

Binary

...

LICENSE

...

AL 2.0 and BSD for JWNL

...

Binary

...

NOTICE

...

standard notice, dates are correct. JWNL is mentioned

...

Binary

...

README

...

File was reviewed on the dev list.

...

Binary

...

RELEASE_NOTES.html

...

issue list is generated correctly

...

Binary

...

Test signatures: .md5, .sha1, .asc

...

rc4

...

Binary

...

JIRA issue list created

...

Binary

...

Contains maxent, tools, uima and jwnl jars

...

Source

...

LICENSE

...

standard AL 2.0 file

...

Source

...

NOTICE

...

standard notice, dates are correct

...

Source

...

Test signatures: .md5, .sha1, .asc

...

rc4

...

Source

...

Can build from source?

...

5537679932260795	Precision: 0.8014705882352942 Recall: 0.4230271668822768 F-Measure: 0.5537679932260795
Name Finder	CONLL 2003 German Location deu.testa	jkosin	Precision: 0.7816326530612245 Recall: 0.32430143945808637 F-Measure: 0.45840813883901854	Precision: 0.7816326530612245 Recall: 0.32430143945808637 F-Measure: 0.45840813883901854
Name Finder	CONLL 2003 German Location deu.testb	jkosin	Precision: 0.8033826638477801 Recall: 0.3671497584541063 F-Measure: 0.5039787798408487	Precision: 0.8033826638477801 Recall: 0.3671497584541063 F-Measure: 0.5039787798408487
Name Finder	CONLL 2003 German Misc deu.testa	jkosin	Precision: 0.7055555555555556 Recall: 0.12574257425742574 F-Measure: 0.21344537815126052	Precision: 0.7055555555555556 Recall: 0.12574257425742574 F-Measure: 0.21344537815126052
Name Finder	CONLL 2003 German Misc deu.testb	jkosin	Precision: 0.6601307189542484 Recall: 0.15074626865671642 F-Measure: 0.2454434993924666	Precision: 0.6601307189542484 Recall: 0.15074626865671642 F-Measure: 0.2454434993924666
Name Finder	CONLL 2003 German Combined deu.testa	jkosin	Precision: 0.7718859429714857 Recall: 0.319263397475688 F-Measure: 0.4516978922716628	Precision: 0.7783891945972986 Recall: 0.32195323815435545 F-Measure: 0.45550351288056207	OPENNLP-417
Name Finder	CONLL 2003 German Combined deu.testb	jkosin	Precision: 0.7467566165023353 Recall: 0.3917778382793357 F-Measure: 0.5139285714285715	Precision: 0.749351323300467 Recall: 0.3931391233324258 F-Measure: 0.5157142857142857	OPENNLP-417
POS Tagger	CONLL 2006 Danish	Jörn / ?	Accuracy: 0.9511278195488722	Accuracy: 0.9512987012987013	Jörn: Same result as other tester
POS Tagger	CONLL 2006 Dutch	Jörn	Accuracy: 0.9324977618621307	Accuracy: 0.9324977618621307
POS Tagger	CONLL 2006 Portuguese	Jörn / ?	Accuracy: 0.9659110277825124	Accuracy: 0.9659110277825124	Jörn: Same result as other tester
POS Tagger	CONLL 2006 Swedish	Jörn	Accuracy: 0.9275106082036775	Accuracy: 0.9275106082036775
Chunker	CONLL 2000	William	Precision: 0.9257575757575758 Recall: 0.9221868187154117 F-Measure: 0.9239687473746113	Precision: 0.9257575757575758 Recall: 0.9221868187154117 F-Measure: 0.9239687473746113
Sentence Detector	Arvores Deitadas (Floresta Virgem) (10-fold cross-validation)	William		Precision: 0.9891491491491492 Recall: 0.9894066523820013 F-Measure: 0.9892778840089301	PERCEPTRON Cutoff 0 1.5.2 works poorly because we didn't have configurable EOS
Tokenizer	Arvores Deitadas (Floresta Virgem) (10-fold cross-validation)	William		Precision: 0.9995231988260895 Recall: 0.9994542652270997 F-Measure: 0.9994887308380267	PERCEPTRON Cutoff 0 alphaNumOpt
Chunker	Arvores Deitadas (10-fold cross-validation)	William	Precision: 0.9404684925220583 Recall: 0.9374181341871635 F-Measure: 0.9389408359191154	Precision: 0.9562405864042575 Recall: 0.9582419351592844 F-Measure: 0.9572402147035765	OPENNLP-541, OPENNLP-423

Test UIMA Integration

The test ensures that the Analysis Engine can run and not not
crash trough simple runtime time code errors. We need to add
more sophisticated testing with the next releases.

Analysis Engine	Tester	Passed
Sentence Detector
Sentence Detector Trainer
Tokenizer ME
Tokenizer Trainer
Name Finder
Name Finder Trainer
Chunker
Chunker Trainer
POS Tagger
POS Tagger Trainer
Parser
createPear.sh	Jörn	yes
Sample PEAR	Jörn	yes

Distribution Review

Please ensure that the listed files below are included in the distributions
and are in a good state.

Package	File or Test	Tester	Passed	Comment
Binary	LICENSE	Jörn	Yes	AL 2.0 and BSD for JWNL
Binary	NOTICE	Jörn	Yes	standard notice, dates are correct. JWNL is mentioned
Binary	README	Jörn	Yes
Binary	RELEASE_NOTES.html	Jörn	Yes
Binary	Test signatures: .md5, .sha1, .asc	Jörn	Yes	tested for rc3
Binary	JIRA issue list created	William	Yes	Minor issue: the project.version was not filled.
Binary	Contains maxent, tools, uima and jwnl jars	Jörn	Yes
Source	LICENSE	Jörn	Yes	standard AL 2.0 file
Source	NOTICE	Jörn	Yes	standard notice, dates are correct
Source	Test signatures: .md5, .sha1, .asc	Jörn		tested for rc3
Source	Can build from source?	Jörn	Yes	Test should be done without jwnl and opennlp in local m2 repo.

Notes about testing

Compatibility tests

The following commands can be used to reproduce the compatibility tests with Leipzig corpus.

Code Block


# Corpus preparation: the following command will create documents from the corpus. Sed is used to remove the language prefix

sh bin/opennlp DoccatConverter leipzig -data ../eng_news_2010_300K-text/eng_news_2010_300K-sentences.txt -encoding UTF-8 -lang en | sed -E 's/^en[[:space:]]//g' > ../out-tokenized-documents.test

# Corpus preparation: this forces the detokenization of the documents

sh bin/opennlp SentenceDetectorConverter namefinder -data ../out-tokenized-documents.test -encoding UTF-8 -detokenizer trunk/opennlp-tools/lang/en/tokenizer/en-detokenizer.xml > ../out-documents.test

# Now the actually tests. Execute it for the previous release and for the current RC. Compare the output using diff:

time sh bin/opennlp SentenceDetector ../models/en-sent.bin < ../out-documents.test > ../out-sentences_1.5.2.test

time sh bin/opennlp TokenizerME ../models/en-token.bin < ../out-sentences_1.5.2.test > ../out-toks_1.5.2.test

time sh bin/opennlp TokenNameFinder ../models/en-ner-person.bin < ../out-toks_1.5.2.test > ../out-ner_1.5.2.test

time sh bin/opennlp POSTagger ../models/en-pos-maxent.bin < ../out-toks_1.5.2.test > ../out-pos_maxent_1.5.2.test

time sh bin/opennlp POSTagger ../models/en-pos-perceptron.bin < ../out-toks_1.5.2.test > ../out-pos_pers_1.5.2.test

time sh bin/opennlp ChunkerME ../models/en-chunker.bin < ../out-pos_pers_1.5.2.test > ../out-chk_1.5.2.test

time sh bin/opennlp Parser ../models/en-parser-chunking.bin < ../out-toks_1.5.2.test > ../out-parse_1.5.2.test

Child pages

Versions Compared

Old Version 9

New Version Current

Key

Test UIMA Integration

Distribution Review

Test UIMA Integration

Distribution Review

Notes about testing

Compatibility tests