Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 4.0

Test Plan for Apache OpenNLP 1.5.

...

2

This page contains the test plan for the 1.5.2 release.

The 1.5.1 2 release does not introduce any changes to the feature
generation (expect OPENNLP-138) and should produce exactly the
same output as the 1.5.0 releaseexpect for the name finder which might generate different
token class features for words with special letters.

Compatibility Test with OpenNLP 1.5.0 SourceForge Models

The 1.5.0 SourceForge models must be fully compatible with the 1.5.12
release. In this test all the English models are tested for compatibiltycompatibility
on the English 300k sentences Leipzig Corpus. It is tested that
the output produced with the same model by both versions has the same md5 hash.

Component

Model

Perf 1.5.01

Perf 1.5.12

Tester

Passed

Comment

Sentence Detector

en-sent.bin 42565.4 sent/s

42186.7 sent/s

 

joern

no

It did not pass because of OPENNLP-202.
The diff showed that in the first 20 compared cases didn't made a mistake compared to 1.5.1.

Tokenizer

yes

 

Tokenizer

en-token.bin

3059 3091.5 8 sent/s
3091

2300.8 sent4 sent/s

joern

yes

 

Name Finder

en-ner-person.bin

290 614.7 4 sent/s

487 650.1 6 sent/s

joern

no

yes

output identical, measurement was done on a idle system,
the new name finder is roughly 10% faster OPENNLP-138, feature-gen fix

POS Tagger

en-pos-maxent.bin

721 732.3 1 sent/s
732

816.1 9 sent/s

joern

yes

 

POS Tagger

en-pos-perceptron.bin

1097.7 sent/s

1110.6 sent/s

 

joern

 

no

Perceptron normalization was changed. OPENNLP-155 might improve accuracy a little

Chunker

en-chunker.bin

169,5 sent/s

167,3 sent/s

166.4 sent/s

joern colen

yes

computerB, tested with CONLL2000 (2012 sentences)  

Parser

en-parser-chunking.bin 4.3 sent/s

11.6 sent/s

 

joern

yes

Macbook was sleeping a little while doing 1.5.0

no

A very few sentences are parsed differently due to OPENNLP-233.
The parser code itself it not affected by this only the code in the cmd line package.

Note: Test was done Note: Test was done on MacBook Pro 13" 7.1, 2.66 GHz Core 2 Duo, 8GB Ram, 256GB SSD running OS X 10.6.6
and Java 1.6.0_22 26 64-Bit Server.The performance varies because light weight tasks have been performed in the background while testing.

Note: computerB is a DualCore T8100, 4GB Ram, 250GB HD running Ubuntu 10.10 64-Bit and Java 1.6.0_20Note: "Concurrent" in the comment means that both tests where started at the same time.

...

Component

Model

Training Time 1.5.01

Training Time 1.5.12

Tester

Passed

Comment

Sentence Detector

en-sent.bin

0m12.847s

0m11.255s

 

joern

yes

no

The new version is more accurate due to OPENLP-202.  

Tokenizer

en-token.bin

2m16 2m30.694s 115s
2m30

1m35.115s 414s

joern

yes

Re-test tagging was very slow, only 250 sent/s

 

POS Tagger Name Finder

en-nerpos-datemaxent.bin

 

 

joern

yes

Test is still done, because tagdict is not tested with public data

POS Tagger

no

OPENNLP-138

Name Finder

en-nerpos-locationperceptron.bin

 

 

joern

no

OPENNLP-138

Perceptron code was changed

Parser Name Finder

en-nerparser-moneychunking.bin

138m9.045s  

 

joern

no

There are small differences due to OPENNLP-138

Name Finder

en-ner-organization.bin

 

 

joern

no

OPENNLP-138

Name Finder

en-ner-percentage.bin

 

 

joern

no

OPENNLP-138

Name Finder

en-ner-person.bin

 

 

joern

no

OPENNLP-138

POS Tagger

en-pos-maxent.bin

 

 

joern

 

 

POS Tagger

en-pos-perceptron.bin

 

 

joern

 

 

Chunker

en-chunker.bin

 

 

joern

 

 

Parser

en-parser-chunking.bin

110m8.712s

138m9.045s

joern

yes

 

Note: Time was measured with the time command, the value is the "real" time value.

Performance test with public data

Test the tagging performance with all the publicly available training
and test data for various languages.

It is assumed that the training will be done with a cutoff of 5 and 100 iterations,
if different values are used please write them into the comment.

233.
Changes to the training code seems not to cause regressions.

Note: Time was measured with the time command, the value is the "real" time value.

Performance test with public data

Test the tagging performance with all the publicly available training
and test data for various languages.

It is assumed that the training will be done with a cutoff of 5 and 100 iterations,
if different values are used please write them into the comment.

Component

Data

Tester

Tagging Perf 1.5.1

Tagging Perf 1.5.2

Comment

Sentence Detector

 

 

 

 

 

Tokenizer

 

 

 

 

 

Name Finder

CONLL 2002 Dutch Person ned.testa

jkosin

Precision: 0.7906976744186046
Recall: 0.48364153627311524 
F-Measure: 0.6001765225066196

Precision: 0.7552941176470588
Recall: 0.4566145092460882
F-Measure: 0.5691489361702128

Performance Change due to OPENNLP-294 and more...

Name Finder

CONLL 2002 Dutch Person ned.testb

jkosin

Precision: 0.8527980535279805
Recall: 0.6384335154826958 
F-Measure: 0.7302083333333333

Precision: 0.8505025125628141
Recall: 0.6165755919854281
F-Measure: 0.7148891235480465

 

Name Finder

CONLL 2002 Dutch Organization ned.testa

jkosin

Precision: 0.8386075949367089
Recall: 0.38629737609329445 
F-Measure: 0.5289421157684631

Precision: 0.8561872909698997
Recall: 0.37317784256559766
F-Measure: 0.5197969543147207

 

Name Finder

CONLL 2002 Dutch Organization ned.testb

jkosin

Precision: 0.7784200385356455
Recall: 0.4580498866213152 
F-Measure: 0.5767309064953604

Precision: 0.7830374753451677
Recall: 0.4501133786848073
F-Measure: 0.5716342692584593

 

Name Finder

CONLL 2002 Dutch Location ned.testa

jkosin

Precision: 0.8362831858407079
Recall: 0.3945720250521921 
F-Measure: 0.5361702127659574

Precision: 0.8458333333333333
Recall: 0.42379958246346555
F-Measure: 0.564673157162726

 

Name Finder

CONLL 2002 Dutch Location ned.testb

jkosin

Precision: 0.854251012145749 
Recall: 0.5452196382428941 
F-Measure: 0.665615141955836

Precision: 0.8816326530612245
Recall: 0.5581395348837209
F-Measure: 0.6835443037974683

 

Name Finder

CONLL 2002 Dutch Misc ned.testa

jkosin

Precision: 0.8300492610837439
Recall: 0.4505347593582888 
F-Measure: 0.5840554592720971

Precision: 0.8354114713216958
Recall: 0.44786096256684493
F-Measure: 0.5831157528285466

 

Name Finder

CONLL 2002 Dutch Misc ned.testb

jkosin

Precision: 0.8373205741626795
Recall: 0.44229149115417016 
F-Measure: 0.5788313120176405

Precision: 0.8264984227129337
Recall: 0.44144903117101936
F-Measure: 0.5755079626578803

 

Name Finder

CONLL 2002 Combined ned.testa

jkosin

Precision: 0.7906976744186046
Recall: 0.48364153627311524 
F-Measure: 0.6001765225066196

Precision: 0.6509695290858726
Recall: 0.628822629969419
F-Measure: 0.6397044526540929

1000 iterations
OPENNLP-335 Exporting of all tags...

Name Finder

CONLL 2002 Dutch Combined ned.testb

jkosin

Component

Data

Tester

Tagging Perf 1.5.0

Tagging Perf 1.5.1

Comment

Sentence Detector

 

 

 

 

Will not be done in this release.

Tokenizer

 

 

 

 

We need a de-tokenizer dictionary first, will be done in next release.

Name Finder

CONLL 2002 Dutch Person ned.testa

joern

 

Precision: 0.7906976744186046
Recall: 0.48364153627311524
F-Measure: 0.6001765225066196

 

Name Finder

CONLL 2002 Dutch Person ned.testb

joern

 

Precision: 0.8527980535279805
Recall: 0.6384335154826958 6384335154826958 
F-Measure: 0.7302083333333333

 

Name Finder

CONLL 2002 Dutch Organization ned.testa

joern

 .7302083333333333

Precision: 0.8386075949367089 6869929337869668
Recall: 0.38629737609329445 6660746003552398
F-Measure: 0.5289421157684631 6763720690543674

  1000 iterations

Name Finder

CONLL 2002 Dutch Organization ned.testb

joern

Spanish Person esp.testa

jkosin  

Precision: 0.7784200385356455 8982630272952854
Recall: 0.4580498866213152 5924713584288053 
F-Measure: 0.5767309064953604

 

Name Finder

CONLL 2002 Dutch Location ned.testa

joern

 

7140039447731755

Precision: 0.8362831858407079 9010695187165776
Recall: 0.3945720250521921 5515548281505729
F-Measure: 0.5361702127659574 684263959390863

 

Name Finder

CONLL 2002 Dutch Location nedSpanish Person esp.testb

joern

  jkosin

Precision: 0.854251012145749 9008 
Recall: 0.5452196382428941 7659863945578231 
F-Measure: 0.665615141955836

 

Name Finder

CONLL 2002 Dutch Misc ned.testa

joern

 

8279411764705882

Precision: 0.8300492610837439 9195205479452054
Recall: 0.4505347593582888 7306122448979592
F-Measure: 0.5840554592720971 8142532221379833

 

Name Finder

CONLL 2002 Dutch Misc ned.testb

joern

Spanish Organization esp.testa

jkosin  

Precision: 0.8373205741626795 8216258879242304
Recall: 0.44229149115417016 6123529411764705 
F-Measure: 0.5788313120176405

 

Name Finder

CONLL 2002 Combined ned.testa

joern

.7017189079878665  

Precision: 0.7906976744186046 8288942695722357
Recall: 0.48364153627311524 6041176470588235
F-Measure: 0.6001765225066196 6988771691051379

 

Name Finder

CONLL 2002 Dutch Combined nedSpanish Organization esp.testb

joern

  jkosin

Precision: 0.8527980535279805 8009331259720062
Recall: 0.6384335154826958 7357142857142858  
F-Measure: 0.7302083333333333

 

Name Finder

CONLL 2002 Spanish Person esp.testa

joern

 

7669396872673119

Precision: 0.8982630272952854 8036277602523659
Recall: 0.5924713584288053 7278571428571429
F-Measure: 0.7140039447731755 7638680659670164

 

Name Finder

CONLL 2002 Spanish Person esp.testb

joern

Location esp.testa

jkosin

Precision: 0.7481789802289281
Recall: 0.7306910569105691 
F-Measure: 0.739331619537275  

Precision: 0.9008 7743016759776536
Recall: 0.7659863945578231 7042682926829268
F-Measure: 0.8279411764705882 7376263970196913

 

Name Finder

CONLL 2002 Spanish Organization Location esp.testa testb

joern jkosin

 

Precision: 0.8216258879242304 8226221079691517
Recall: 0.6123529411764705 5904059040590406 
F-Measure: 0.7017189079878665

 

Name Finder

CONLL 2002 Spanish Organization esp.testb

joern

6874328678839956  

Precision: 0.8009331259720062 8301886792452831
Recall: 0.7357142857142858  5682656826568265
F-Measure: 0.7669396872673119 6746987951807228

 

Name Finder

CONLL 2002 Spanish Location Misc esp.testa

joern

  jkosin

Precision: 0.7481789802289281 6446886446886447
Recall: 0.7306910569105691 3955056179775281 
F-Measure: 0.739331619537275

 

Name Finder

CONLL 2002 Spanish Location esp.testb

joern

 

49025069637883006

Precision: 0.8226221079691517 6492890995260664
Recall: 0.5904059040590406 30786516853932583
F-Measure: 0.6874328678839956 4176829268292683

 

Name Finder

CONLL 2002 Spanish Misc esp.testa testb

joern jkosin

 

Precision: 0.6446886446886447 6595744680851063
Recall: 0.3955056179775281 36578171091445427 
F-Measure: 0.49025069637883006

 

Name Finder

CONLL 2002 Spanish Misc esp.testb

joern

4705882352941176  

Precision: 0.6595744680851063 686046511627907
Recall: 0.36578171091445427 3480825958702065
F-Measure: 0.4705882352941176 461839530332681

 

Name Finder

CONLL 2002 Spanish Combined esp.testa

joern

jkosin

Precision: 0.8982630272952854  
Recall: 0.5924713584288053 
F-Measure: 0.7140039447731755  

Precision: 0.8982630272952854  7005423249233671
Recall: 0.5924713584288053 6828315329809239
F-Measure: 0.7140039447731755 6915735567970205

  1000 iterations

Name Finder

CONLL 2002 Spanish Combined esp.testb

joern

jkosin

Precision: 0.9008 
Recall: 0.7659863945578231 
F-Measure: 0.8279411764705882  

Precision: 0.9008 756635931824532
Recall: 0.7659863945578231 7611017425519955
F-Measure: 0.8279411764705882 7588622670589884

  1000 iterations

Name Finder

CONLL 2003 English Person eng.testa

jkosin

Precision:  00.901992661721591 9352201257861635
Recall:  00.7263843648208469 8072747014115093 
F-Measure:  00.8047194918352375 8665501165501166

Precision: 0.9352201257861635 9523195876288659
Recall: 0.8072747014115093 8023887079261672
F-Measure: 0.8665501165501166 8709487330583382

 

Name Finder

CONLL 2003 English Person eng.testb

jkosin

Precision:  00.8977988745723299 8873546511627907
Recall:  00.6821273964131107 7551020408163265 
F-Measure:  00.7752427693131103 8159037754761109

Precision: 0.8873546511627907 9391727493917275
Recall: 0.7551020408163265 7161410018552876
F-Measure: 0.8159037754761109 8126315789473685

 

Name Finder

CONLL 2003 English Organization eng.testa

jkosin

Precision:  00.8290322580645161 8528584817244611
Recall:  00.6226696495152871 6785980611483967 
F-Measure:  00.711183505195638 7558139534883722

Precision: 0.8528584817244611 8768046198267565
Recall: 0.6785980611483967 6793437733035048
F-Measure: 0.7558139534883722 7655462184873949

 

Name Finder

CONLL 2003 English Organization eng.testb

jkosin

Precision:  00.818058934847256 8263422818791947
Recall:  00.5394340758579169 5930162552679109 
F-Measure:  00.6501526888707977 6905012267788293

Precision: 0.8263422818791947 8435980551053485
Recall: 0.5930162552679109 6267308850090307
F-Measure: 0.6905012267788293 7191709844559586

 

Name Finder

CONLL 2003 English Location eng.testa

jkosin

Precision:  00.9584186939820742 9283837056504599
Recall:  00.7408818726183996 769188894937398 
F-Measure:  00.8357262402029991 8413218219708247

Precision: 0.9283837056504599 9361421988150099
Recall: 0.769188894937398 7740881872618399
F-Measure: 0.8413218219708247 8474374255065554

 

Name Finder

CONLL 2003 English Location eng.testb

jkosin

Precision:  00.9485177151120753 9156180606957809
Recall:  00.7182254196642686 7416067146282974 
F-Measure:  00.8174619349330977 8194766478966545

Precision: 0.9156180606957809 9206349206349206
Recall: 0.7416067146282974 7302158273381295
F-Measure: 0.8194766478966545 8144433299899699

 

Name Finder

CONLL 2003 English Misc eng.testa

jkosin

Precision:  00.8492613111726685 8539007092198582
Recall:  00.6052060737527115 6529284164859002 
F-Measure:  00.706757826338278 7400122925629993

Precision: 0.8539007092198582 9027982326951399
Recall: 0.6529284164859002 6648590021691974
F-Measure: 0.7400122925629993 7657713928794503

 

Name Finder

CONLL 2003 English Misc eng.testb

jkosin

Precision:  00.8979300499643112 8599137931034483
Recall:  00.5299145299145299 5683760683760684 
F-Measure:  00.6664957615531857 6843910806174958

Precision: 0.8599137931034483 8592436974789915
Recall: 0.5683760683760684 5826210826210826
F-Measure: 0.6843910806174958 6943972835314092

 

Name Finder

CONLL 2003 English Combined eng.testa

jkosin

Precision: 0.8230655223984119 8601818493738206
Recall: 0.8039380679905755 8438236284079434 
F-Measure: 0.8133893616650641 8519242205420101

Precision: 0.8601818493738206 861812521618817
Recall: 0.8438236284079434 8386065297879501
F-Measure: 0.8519242205420101 8500511770726714

1000 iterations

Name Finder

CONLL 2003 English Combined eng.testb

jkosin

Precision: 0.7849405582672956 8036415565869333
Recall: 0.7563739376770539 7970963172804533 
F-Measure: 0.7703925220469681 8003555555555556

Precision: 0.8036415565869333 8041311831853597
Recall: 0.7970963172804533 7857648725212465
F-Measure: 0.8003555555555556 7948419450165667

1000 iterations

Name Finder

CONLL 2003 German Person deu.testa

joern

Precision: 0.8272041489863272  8602620087336245
Recall: 0.22626695217701642  28122769450392576 
F-Measure:  00.35533762893472637 4238838084991931

Precision: 0.8602620087336245 9132653061224489
Recall: 0.28122769450392576 25553176302640973
F-Measure: 0.4238838084991931 3993307306190742

 

Name Finder

CONLL 2003 German Person deu.testb

joern

Precision: 0.7535042735042735  878 
Recall: 0.2602510460251046  3673640167364017 
F-Measure:  00.38687890773270717 5179941002949853

Precision: 0.878 8732106339468303
Recall: 0.3673640167364017 3573221757322176
F-Measure: 0.5179941002949853 507125890736342

 

Name Finder

CONLL 2003 German Organization deu.testa

joern

Precision: 0.6615148726058698  8365695792880259
Recall: 0.29814665592264306  41659951651893634 
F-Measure:  00.4110375194740828 5562130177514794

Precision: 0.8365695792880259 8407224958949097
Recall: 0.41659951651893634 4125705076551168
F-Measure: 0.5562130177514794 5535135135135135

 

Name Finder

CONLL 2003 German Organization deu.testb

joern

Precision: 0.690884820747521  7942583732057417
Recall: 0.3311772315653299  4294954721862872 
F-Measure:  00.4477327413690855 5575146935348446

Precision: 0.7942583732057417 8014705882352942
Recall: 0.4294954721862872 4230271668822768
F-Measure: 0.5575146935348446 5537679932260795

 

Name Finder

CONLL 2003 German Location deu.testa

joern

Precision: 0.8779137529137528  7362637362637363
Recall: 0.32006773920406434  34038950042337 
F-Measure:  00.46910886680647634 4655471916618414

Precision: 0.7362637362637363 7816326530612245
Recall: 0.34038950042337 32430143945808637
F-Measure: 0.4655471916618414 45840813883901854

 

Name Finder

CONLL 2003 German Location deu.testb

joern

Precision: 0.741636798088411  75 
Recall: 0.3169082125603865  3652173913043478 
F-Measure:  00.44406386065180703 4912280701754385

Precision: 0.75 8033826638477801
Recall: 0.3652173913043478 3671497584541063
F-Measure: 0.4912280701754385 5039787798408487

 

Name Finder

CONLL 2003 German Misc deu.testa

joern

Precision: 0.8151658767772512  7213930348258707
Recall: 0.12178217821782178  14356435643564355 
F-Measure:  00.21190646707366545 2394715111478117

Precision: 0.7213930348258707 7055555555555556
Recall: 0.14356435643564355 12574257425742574
F-Measure: 0.2394715111478117 21344537815126052

 

Name Finder

CONLL 2003 German Misc deu.testb

joern

Precision: 0.8125  6198830409356725
Recall: 0.15074626865671642  1582089552238806 
F-Measure:  00.2543095099748208 2520808561236623

Precision: 0.6198830409356725 6601307189542484
Recall: 0.1582089552238806 15074626865671642
F-Measure: 0.2520808561236623 2454434993924666

 

Name Finder

CONLL 2003 German Combined deu.testa

joern

Precision: 0.6622805891862553  7675205413243112
Recall: 0.28698530933167804  32857438444030623 
F-Measure:  00.400445860424834 46015647638365687

Precision: 0.7675205413243112 7718859429714857
Recall: 0.32857438444030623 319263397475688
F-Measure: 0.46015647638365687 4516978922716628

 

Name Finder

CONLL 2003 German Combined deu.testb

joern

Precision: 0.6632526799570968  7553418803418803
Recall: 0.33324258099646065  3849714130138851 
F-Measure:  00.44360278183404916 5100090171325519

Precision: 0.7553418803418803 7467566165023353
Recall: 0.3849714130138851 3917778382793357
F-Measure: 0.5100090171325519 5139285714285715

 

POS Tagger

CONLL 2006 Danish

joern

Accuracy: 0.9511278195488722

Accuracy: 0.9511278195488722

 

POS Tagger

CONLL 2006 Dutch

joern

Accuracy: 0.9324977618621307

Accuracy: 0.9324977618621307

 

POS Tagger

CONLL 2006 Portuguese

joern

Accuracy: 0.9659110277825124

Accuracy: 0.9659110277825124

 

POS Tagger

CONLL 2006 Swedish

joern

Accuracy: 0.9275106082036775

Accuracy: 0.9275106082036775

 

Chunker

CONLL 2000

colen  

Precision: 0.9255923572240226
Recall: 0.9220610430991112 9220610430991112 
F-Measure: 0.9238233255623465 Evaluator was not available in 1.5.0. To evaluate if something changed I compared the output of 1.5.0 and 1.5.1. The output changed a little because of a bug fixed in 1.5.1 (missing trailing closing bracket)

Precision: 0.9257575757575758
Recall: 0.9221868187154117
F-Measure: 0.9239687473746113

Perf change due to OPENNLP-242

Chunker

Arvores Deitadas
(10-fold cross-validation)

colen

 

Precision: 0.9406086044071353 9413606010016694
Recall: 0.9364814040952779 9379938451301671
F-Measure: 0.9396742073907428

Precision: 0.9385404669668097 AD format for Chunker was not available for 1.5.0 9403445830378374
Recall: 0.9373141775994345
F-Measure: 0.9388269348910339

Perf change due to OPENNLP-242 and OPENNLP-186

The results of the tagging performance might differ compared to the
1.5.0 release since a precision bug in the calculation of the score has been fixed:
https://issues.apache.org/jira/browse/OPENNLP-59
A problem was corrected for the CoNLL 02 data being improperly converted to the wrong encoding.

Test UIMA Integration

The test ensures that the Analysis Engine can run and not not
crash trough simple runtime time code errors. We need to add
more sophisticated testing with the next releases.

Analysis Engine

Tester

Passed

Comment

Sentence Detector

joern

yes

Used to process millions of news articles  

Sentence Detector Trainer

Tommaso joern

yes

Trained and tested with cmd line tool with a UIMA pipeline

Tokenizer ME

joern

yes

Used to process millions of news articles  

Tokenizer Trainer

Tommaso joern

  yes

Trained and tested with cmd line tool with a UIMA pipeline

Name Finder

joern

yes

Used to process millions of news articles  

Name Finder Trainer

Tommaso joern

yes

Trained and tested with cmd line tool with a UIMA pipeline

Chunker

joern

yes

as part of sample pear

Chunker Trainer

 

 

 

POS Tagger

joern

yes

as part of sample pear

POS Tagger Trainer

Tommaso  

yes  

Trained and tested with cmd line tool

Parser

 

 

 

createPear.sh

joern

no, retest with RC5

yes

  Test that pear is build and works. Now fixed after OPENNLP-143.

Sample PEAR

joern

yes

installed and run over sample text

...

Package

File or Test

Tester

Passed

Comment

Binary

LICENSE

joern

yes

AL 2.0 and BSD for JWNL

Binary

NOTICE

joern

yes

standard notice, dates are correct. JWNL is mentioned

Binary

README

colen, jason, james, joern

yes  

File was reviewed on the dev list.

Binary

RELEASE_NOTES.html

joern, james

yes

issue list is generated correctly

Binary

Test signatures: .md5, .sha1, .asc

joern

yes

rc4

Binary

JIRA issue list created

joern

yes

 

Binary

Contains maxent, tools, uima and jwnl jars

joern

yes

  rc7

Source

LICENSE

joern

yes

standard AL 2.0 file

Source

NOTICE

joern

yes

standard notice, dates are correct

Source

Test signatures: .md5, .sha1, .asc

joern

yes

rc7 rc4

Source

Can build from source?

joern

yes

Test should be done without jwnl and opennlp in local m2 repo.
Test was done on Ubuntu 10.10.