You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 17 Next »

Test Plan for Apache OpenNLP 1.5.3

This page contains the test plan for the 1.5.3 release.

The 1.5.3 release does not introduce any changes to the feature
generation.

Compatibility Test with OpenNLP 1.5.0 SourceForge Models

The 1.5.0 SourceForge models must be fully compatible with the 1.5.3
release. In this test all the English models are tested for compatibility
on the English 300k sentences Leipzig Corpus. It is tested that
the output produced with the same model by both versions has the same md5 hash.

Component

Model

Perf 1.5.2

Perf 1.5.3

Tester

Passed

Comment

Sentence Detector

en-sent.bin

44870.8 sent/s

42733.8 sent/s

William

yes

 

Tokenizer

en-token.bin

2824.2 sent/s

2833.3 sent/s

William

yes

 

Name Finder

en-ner-person.bin

781.3 sent/s

761.6 sent/s

William

yes

 

POS Tagger

en-pos-maxent.bin

773.3 sent/s

816.2 sent/s

William

yes

 

POS Tagger

en-pos-perceptron.bin

1138.6 sent/s

1117.1 sent/s

William

yes

 

Chunker

en-chunker.bin

183.7 sent/s

181.1 sent/s

William

yes

 

Parser

en-parser-chunking.bin

16.0 sent/s

16.3 sent/s

William

yes

 

Note: Test was done on MacBook Pro 15", 2 GHz Core i7, 16GB Ram, 500GB HD running OS X 10.7.5
and Java 1.5.0_30. The performance varies because light weight tasks have been performed in the background while testing.

Note: "Concurrent" in the comment means that both tests where started at the same time.

Regression Test Training with (private) English data

The training of both versions with the same data must produce
a model with identical output. The model output is tested with
the procedure from the previous test.

To pass the test the event hash and the model output must be identical.

Component

Model

Training Time 1.5.2

Training Time 1.5.3

Tester

Passed

Comment

Sentence Detector

en-sent.bin

 

 

Jörn

 

 

Tokenizer

en-token.bin

 

 

Jörn

 

 

POS Tagger

en-pos-maxent.bin

 

 

Jörn

 

 

POS Tagger

en-pos-perceptron.bin

 

 

Jörn

 

 

Parser

en-parser-chunking.bin

 

 

Jörn

 

 

Note: Time was measured with the time command, the value is the "real" time value.

Performance test with public data

Test the tagging performance with all the publicly available training
and test data for various languages.

It is assumed that the training will be done with a cutoff of 5 and 100 iterations,
if different values are used please write them into the comment.

Component

Data

Tester

Tagging Perf 1.5.2

Tagging Perf 1.5.3

Comment

Sentence Detector

 

 

 

 

 

Tokenizer

 

 

 

 

 

Name Finder

CONLL 2002 Dutch Person ned.testa

 

Precision: 0.7552941176470588
Recall: 0.4566145092460882
F-Measure: 0.5691489361702128

 

 

Name Finder

CONLL 2002 Dutch Person ned.testb

 

Precision: 0.8505025125628141
Recall: 0.6165755919854281
F-Measure: 0.7148891235480465

 

 

Name Finder

CONLL 2002 Dutch Organization ned.testa

 

Precision: 0.8561872909698997
Recall: 0.37317784256559766
F-Measure: 0.5197969543147207

 

 

Name Finder

CONLL 2002 Dutch Organization ned.testb

 

Precision: 0.7830374753451677
Recall: 0.4501133786848073
F-Measure: 0.5716342692584593

 

 

Name Finder

CONLL 2002 Dutch Location ned.testa

 

Precision: 0.8458333333333333
Recall: 0.42379958246346555
F-Measure: 0.564673157162726

 

 

Name Finder

CONLL 2002 Dutch Location ned.testb

 

Precision: 0.8816326530612245
Recall: 0.5581395348837209
F-Measure: 0.6835443037974683

 

 

Name Finder

CONLL 2002 Dutch Misc ned.testa

 

Precision: 0.8354114713216958
Recall: 0.44786096256684493
F-Measure: 0.5831157528285466

 

 

Name Finder

CONLL 2002 Dutch Misc ned.testb

 

Precision: 0.8264984227129337
Recall: 0.44144903117101936
F-Measure: 0.5755079626578803

 

 

Name Finder

CONLL 2002 Combined ned.testa

 

Precision: 0.6509695290858726
Recall: 0.628822629969419
F-Measure: 0.6397044526540929

 

1000 iterations

Name Finder

CONLL 2002 Dutch Combined ned.testb

 

Precision: 0.6869929337869668
Recall: 0.6660746003552398
F-Measure: 0.6763720690543674

 

1000 iterations

Name Finder

CONLL 2002 Spanish Person esp.testa

 

Precision: 0.9010695187165776
Recall: 0.5515548281505729
F-Measure: 0.684263959390863

 

 

Name Finder

CONLL 2002 Spanish Person esp.testb

 

Precision: 0.9195205479452054
Recall: 0.7306122448979592
F-Measure: 0.8142532221379833

 

 

Name Finder

CONLL 2002 Spanish Organization esp.testa

 

Precision: 0.8288942695722357
Recall: 0.6041176470588235
F-Measure: 0.6988771691051379

 

 

Name Finder

CONLL 2002 Spanish Organization esp.testb

 

Precision: 0.8036277602523659
Recall: 0.7278571428571429
F-Measure: 0.7638680659670164

 

 

Name Finder

CONLL 2002 Spanish Location esp.testa

 

Precision: 0.7743016759776536
Recall: 0.7042682926829268
F-Measure: 0.7376263970196913

 

 

Name Finder

CONLL 2002 Spanish Location esp.testb

 

Precision: 0.8301886792452831
Recall: 0.5682656826568265
F-Measure: 0.6746987951807228

 

 

Name Finder

CONLL 2002 Spanish Misc esp.testa

 

Precision: 0.6492890995260664
Recall: 0.30786516853932583
F-Measure: 0.4176829268292683

 

 

Name Finder

CONLL 2002 Spanish Misc esp.testb

 

Precision: 0.686046511627907
Recall: 0.3480825958702065
F-Measure: 0.461839530332681

 

 

Name Finder

CONLL 2002 Spanish Combined esp.testa

 

Precision: 0.7005423249233671
Recall: 0.6828315329809239
F-Measure: 0.6915735567970205

 

1000 iterations

Name Finder

CONLL 2002 Spanish Combined esp.testb

 

Precision: 0.756635931824532
Recall: 0.7611017425519955
F-Measure: 0.7588622670589884

 

1000 iterations

Name Finder

CONLL 2003 English Person eng.testa

jkosin

Precision: 0.9523195876288659
Recall: 0.8023887079261672
F-Measure: 0.8709487330583382

Precision: 0.9523195876288659
Recall: 0.8023887079261672
F-Measure: 0.8709487330583382

 

Name Finder

CONLL 2003 English Person eng.testb

jkosin

Precision: 0.9391727493917275
Recall: 0.7161410018552876
F-Measure: 0.8126315789473685

Precision: 0.9391727493917275
Recall: 0.7161410018552876
F-Measure: 0.8126315789473685

 

Name Finder

CONLL 2003 English Organization eng.testa

jkosin

Precision: 0.8768046198267565
Recall: 0.6793437733035048
F-Measure: 0.7655462184873949

Precision: 0.8768046198267565
Recall: 0.6793437733035048
F-Measure: 0.7655462184873949

 

Name Finder

CONLL 2003 English Organization eng.testb

jkosin

Precision: 0.8435980551053485
Recall: 0.6267308850090307
F-Measure: 0.7191709844559586

Precision: 0.8435980551053485
Recall: 0.6267308850090307
F-Measure: 0.7191709844559586

 

Name Finder

CONLL 2003 English Location eng.testa

jkosin

Precision: 0.9361421988150099
Recall: 0.7740881872618399
F-Measure: 0.8474374255065554

Precision: 0.9361421988150099
Recall: 0.7740881872618399
F-Measure: 0.8474374255065554

 

Name Finder

CONLL 2003 English Location eng.testb

jkosin

Precision: 0.9206349206349206
Recall: 0.7302158273381295
F-Measure: 0.8144433299899699

Precision: 0.9206349206349206
Recall: 0.7302158273381295
F-Measure: 0.8144433299899699

 

Name Finder

CONLL 2003 English Misc eng.testa

jkosin

Precision: 0.9027982326951399
Recall: 0.6648590021691974
F-Measure: 0.7657713928794503

Precision: 0.9027982326951399
Recall: 0.6648590021691974
F-Measure: 0.7657713928794503

 

Name Finder

CONLL 2003 English Misc eng.testb

jkosin

Precision: 0.8592436974789915
Recall: 0.5826210826210826
F-Measure: 0.6943972835314092

Precision: 0.8592436974789915
Recall: 0.5826210826210826
F-Measure: 0.6943972835314092

 

Name Finder

CONLL 2003 English Combined eng.testa

jkosin

Precision: 0.861812521618817
Recall: 0.8386065297879501
F-Measure: 0.8500511770726714

Precision: 0.8640608785887236
Recall: 0.8407943453382699
F-Measure: 0.8522688502217672

1000 iterations
OPENNLP-417

Name Finder

CONLL 2003 English Combined eng.testb

jkosin

Precision: 0.8041311831853597
Recall: 0.7857648725212465
F-Measure: 0.7948419450165667

Precision: 0.8064866823699945
Recall: 0.7880665722379604
F-Measure: 0.7971702337243664

1000 iterations
OPENNLP-417

Name Finder

CONLL 2003 German Person deu.testa

jkosin

Precision: 0.9132653061224489
Recall: 0.25553176302640973
F-Measure: 0.3993307306190742

Precision: 0.9132653061224489
Recall: 0.25553176302640973
F-Measure: 0.3993307306190742

 

Name Finder

CONLL 2003 German Person deu.testb

jkosin

Precision: 0.8732106339468303
Recall: 0.3573221757322176
F-Measure: 0.507125890736342

Precision: 0.8732106339468303
Recall: 0.3573221757322176
F-Measure: 0.507125890736342

 

Name Finder

CONLL 2003 German Organization deu.testa

jkosin

Precision: 0.8407224958949097
Recall: 0.4125705076551168
F-Measure: 0.5535135135135135

Precision: 0.8407224958949097
Recall: 0.4125705076551168
F-Measure: 0.5535135135135135

 

Name Finder

CONLL 2003 German Organization deu.testb

jkosin

Precision: 0.8014705882352942
Recall: 0.4230271668822768
F-Measure: 0.5537679932260795

Precision: 0.8014705882352942
Recall: 0.4230271668822768
F-Measure: 0.5537679932260795

 

Name Finder

CONLL 2003 German Location deu.testa

jkosin

Precision: 0.7816326530612245
Recall: 0.32430143945808637
F-Measure: 0.45840813883901854

Precision: 0.7816326530612245
Recall: 0.32430143945808637
F-Measure: 0.45840813883901854

 

Name Finder

CONLL 2003 German Location deu.testb

jkosin

Precision: 0.8033826638477801
Recall: 0.3671497584541063
F-Measure: 0.5039787798408487

Precision: 0.8033826638477801
Recall: 0.3671497584541063
F-Measure: 0.5039787798408487

 

Name Finder

CONLL 2003 German Misc deu.testa

jkosin

Precision: 0.7055555555555556
Recall: 0.12574257425742574
F-Measure: 0.21344537815126052

Precision: 0.7055555555555556
Recall: 0.12574257425742574
F-Measure: 0.21344537815126052

 

Name Finder

CONLL 2003 German Misc deu.testb

jkosin

Precision: 0.6601307189542484
Recall: 0.15074626865671642
F-Measure: 0.2454434993924666

Precision: 0.6601307189542484
Recall: 0.15074626865671642
F-Measure: 0.2454434993924666

 

Name Finder

CONLL 2003 German Combined deu.testa

 

Precision: 0.7718859429714857
Recall: 0.319263397475688
F-Measure: 0.4516978922716628

 

 

Name Finder

CONLL 2003 German Combined deu.testb

 

Precision: 0.7467566165023353
Recall: 0.3917778382793357
F-Measure: 0.5139285714285715

 

 

POS Tagger

CONLL 2006 Danish

 

Accuracy: 0.9511278195488722

 

 

POS Tagger

CONLL 2006 Dutch

 

Accuracy: 0.9324977618621307

 

 

POS Tagger

CONLL 2006 Portuguese

 

Accuracy: 0.9659110277825124

 

 

POS Tagger

CONLL 2006 Swedish

 

Accuracy: 0.9275106082036775

 

 

Chunker

CONLL 2000

William

Precision: 0.9257575757575758
Recall: 0.9221868187154117
F-Measure: 0.9239687473746113

Precision: 0.9257575757575758
Recall: 0.9221868187154117
F-Measure: 0.9239687473746113

 

Chunker

Arvores Deitadas
(10-fold cross-validation)

William

Precision: 0.9404684925220583
Recall: 0.9374181341871635
F-Measure: 0.9389408359191154

Precision: 0.9562405864042575
Recall: 0.9582419351592844
F-Measure: 0.9572402147035765

OPENNLP-541, OPENNLP-423

Test UIMA Integration

The test ensures that the Analysis Engine can run and not not
crash trough simple runtime time code errors. We need to add
more sophisticated testing with the next releases.

Analysis Engine

Tester

Passed

Comment

Sentence Detector

 

 

 

Sentence Detector Trainer

 

 

 

Tokenizer ME

 

 

 

Tokenizer Trainer

 

 

 

Name Finder

 

 

 

Name Finder Trainer

 

 

 

Chunker

 

 

 

Chunker Trainer

 

 

 

POS Tagger

 

 

 

POS Tagger Trainer

 

 

 

Parser

 

 

 

createPear.sh

 

 

 

Sample PEAR

 

 

 

Distribution Review

Please ensure that the listed files below are included in the distributions
and are in a good state.

Package

File or Test

Tester

Passed

Comment

Binary

LICENSE

 

 

AL 2.0 and BSD for JWNL

Binary

NOTICE

 

 

standard notice, dates are correct. JWNL is mentioned

Binary

README

 

 

File was reviewed on the dev list.

Binary

RELEASE_NOTES.html

 

 

issue list is generated correctly

Binary

Test signatures: .md5, .sha1, .asc

 

 

rc4

Binary

JIRA issue list created

William

No

The list is empty

Binary

Contains maxent, tools, uima and jwnl jars

 

 

 

Source

LICENSE

 

 

standard AL 2.0 file

Source

NOTICE

 

 

standard notice, dates are correct

Source

Test signatures: .md5, .sha1, .asc

 

 

rc1

Source

Can build from source?

 

 

Test should be done without jwnl and opennlp in local m2 repo.
Test was done on Ubuntu 10.10.

Notes about testing

Compatibility tests

The following commands can be used to reproduce the compatibility tests with Leipzig corpus.

 
# Corpus preparation: the following command will create documents from the corpus. Sed is used to remove the language prefix

sh bin/opennlp DoccatConverter leipzig -data ../eng_news_2010_300K-text/eng_news_2010_300K-sentences.txt -encoding UTF-8 -lang en | sed -E 's/^en[[:space:]]//g' > ../out-tokenized-documents.test

# Corpus preparation: this forces the detokenization of the documents

sh bin/opennlp SentenceDetectorConverter namefinder -data ../out-tokenized-documents.test -encoding UTF-8 -detokenizer trunk/opennlp-tools/lang/en/tokenizer/en-detokenizer.xml > ../out-documents.test

# Now the actually tests. Execute it for the previous release and for the current RC. Compare the output using diff:

time sh bin/opennlp SentenceDetector ../models/en-sent.bin < ../out-documents.test > ../out-sentences_1.5.2.test

time sh bin/opennlp TokenizerME ../models/en-token.bin < ../out-sentences_1.5.2.test > ../out-toks_1.5.2.test

time sh bin/opennlp TokenNameFinder ../models/en-ner-person.bin < ../out-toks_1.5.2.test > ../out-ner_1.5.2.test

time sh bin/opennlp POSTagger ../models/en-pos-maxent.bin < ../out-toks_1.5.2.test > ../out-pos_maxent_1.5.2.test

time sh bin/opennlp POSTagger ../models/en-pos-perceptron.bin < ../out-toks_1.5.2.test > ../out-pos_pers_1.5.2.test

time sh bin/opennlp ChunkerME ../models/en-chunker.bin < ../out-pos_pers_1.5.2.test > ../out-chk_1.5.2.test

time sh bin/opennlp Parser ../models/en-parser-chunking.bin < ../out-toks_1.5.2.test > ../out-parse_1.5.2.test
  • No labels