View Source

h1. Make Luke a Lucene/Solr Module

----
*Student Name* : Ajay Bhat

*Student e-mail* : a.ajay.bhat@gmail.com

*Apache Project Name* : Apache Lucene

*Mentor name* : Mark Miller

*Link to project idea JIRA* : [Make Luke a Lucene/Solr Module|https://issues.apache.org/jira/browse/LUCENE-2562]
----
h2. *1. Abstract*

Apache Lucene(TM) is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.

Lucene offers powerful features through a simple API:

* Scalable, High-Performance Indexing
* Powerful, Accurate and Efficient Search Algorithms
* Cross-Platform Solution

Luke accesses already existing Lucene indexes and allows you to display and modify their content in several ways:
* Browse by document number, or by term
* View documents / copy to clipboard
* Retrieve a ranked list of most frequent terms
* Execute a search. browse and analyze search results
* Selectively delete documents from the index
* reconstruct the original document fields, edit them and re-insert to the index
* optimize indexes

 !http://i1175.photobucket.com/albums/r628/otto_netz/Luke-ALE-1_zpsc0744c92.png|border=1!


Currently the Luke application uses Thinlet (LGPL Licensed API) for many of the GUI features. The UI needs to be ported to an Apache Licensed API for adding it to the trunk. The porting has been done to support Lucene 4.0.0. However lot of work still needs to be done. In this project I am mainly focusing on porting some of the plugins present in the Luke version, so it is more beneficial to the community and users who want to make use of these plugins.

Current (sandbox) version of Luke can be accessed at :&nbsp;[https://svn.apache.org/repos/asf/lucene/sandbox/luke|https://svn.apache.org/repos/asf/lucene/sandbox/luke]&nbsp;
The original repo created by Andrzej Bialecki can be accessed at :&nbsp;[http://luke.googlecode.com/svn/trunk/|http://luke.googlecode.com/svn/trunk/]

h2. 2. Detailed Description

2. Detailed Description

2.1 BXML used in Pivot for GUI

{color:#252525}BXML is an XML-based markup language for simplifying the construction of Java object hierarchies. While it is most often used to define the user interface of an Apache Pivot application, it is not limited to user interface construction, and can actually be used to create hierarchies of any object type.{color}

{color:#252525}In BXML, an XML namespace represents a Java package. Declaring a namespace associates the namespace prefix with the package, similar to how the import keyword is used in Java.{color}

{color:#252525}For example, the following simple BXML associates the package "com.foo" with the "foo" namespace prefix:{color}{color:#252525}&nbsp;{color}

{color:#252525}<foo:Bar xmlns:foo="com.foo"/>{color}&nbsp;
{color:#252525}&nbsp;{color}
{color:#252525}The above window is created when BXML file is written thusly:{color}

{color:#252525}<Window title="Hello BXML\!" maximized="true"{color}

{color:#252525}&nbsp;&nbsp;&nbsp;xmlns:bxml="{color}[http://pivot.apache.org/bxml|http://pivot.apache.org/bxml]{color:#252525}"{color}

{color:#252525}&nbsp;&nbsp;&nbsp;xmlns="org.apache.pivot.wtk">{color}

{color:#252525}&nbsp;&nbsp;&nbsp;<Label text="Hello BXML\!"{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;styles="{font:'Arial bold 24', color:'#ff0000',{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;horizontalAlignment:'center', verticalAlignment:'center'}"/>{color}

{color:#252525}</Window>{color}
{color:#252525}&nbsp;{color}
{color:#252525}and then we use the BXML file in our Java source code as:{color}
{color:#252525}&nbsp;{color}
{color:#252525}import org.apache.pivot.beans.BXMLSerializer;{color}

{color:#252525}import org.apache.pivot.collections.Map;{color}

{color:#252525}import org.apache.pivot.wtk.Application;{color}

{color:#252525}import org.apache.pivot.wtk.Display;{color}

{color:#252525}import org.apache.pivot.wtk.Window;{color}

{color:#252525}public class HelloBXML implements Application {{color}

{color:#252525}&nbsp;&nbsp;&nbsp;private Window window = null;{color}

{color:#252525}&nbsp;&nbsp;&nbsp;@Override{color}

{color:#252525}&nbsp;&nbsp;&nbsp;public void startup(Display display, Map<String, String> properties){color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;throws Exception {{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;BXMLSerializer bxmlSerializer = new BXMLSerializer();{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;window = (Window)bxmlSerializer.readObject(HelloBXML.class, "hello.bxml");{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;window.open(display);{color}

{color:#252525}&nbsp;&nbsp;&nbsp;}{color}

{color:#252525}&nbsp;&nbsp;&nbsp;@Override{color}

{color:#252525}&nbsp;&nbsp;&nbsp;public boolean shutdown(boolean optional) {{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (window \!= null) {{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;window.close();{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return false;{color}

{color:#252525}&nbsp;&nbsp;&nbsp;}{color}

{color:#252525}&nbsp;&nbsp;&nbsp;@Override{color}

{color:#252525}&nbsp;&nbsp;&nbsp;public void suspend() {{color}

{color:#252525}&nbsp;&nbsp;&nbsp;}{color}

{color:#252525}&nbsp;&nbsp;&nbsp;@Override{color}

{color:#252525}&nbsp;&nbsp;&nbsp;public void resume() {{color}

{color:#252525}&nbsp;&nbsp;&nbsp;}{color}
{color:#252525}&nbsp;{color}
{color:#252525}&nbsp;&nbsp;&nbsp;public static void main(String\[\] args) {{color}

{color:#252525}&nbsp;&nbsp;&nbsp;{color} {color:#252525}&nbsp;DesktopApplicationContext.main(HelloBXML.class, args);{color}

{color:#252525}&nbsp;&nbsp;&nbsp;}{color}
{color:#252525}&nbsp;{color}
{color:#252525}}{color}
&nbsp;
&nbsp;
2.2 Requirements &nbsp;

*Apache Pivot API* is used to implement the GUI features of the Luke application as it is Apache Licensed and can be used in the Lucene trunk.
&nbsp;
*Tabs for Plugins*

Tabbed pane is already done using Pivot API for the Luke application. There is a pane for plugins that is empty in the sandboxed version.

A Tab Pane is added as a BXML file as :
<TabPane bxml:id="tabPane">

<tabs>

<Border styles="{backgroundColor:11,thickness:0}">

<TabPane.tabData>

<content:ButtonData icon="/img/info.gif" text="%lukeWindow_overviewTabText" />

</TabPane.tabData>

<content>

<bxml:include bxml:id="abc" src="MyTab.bxml" />

</content>

</Border>

</tabs>

</TabPane>



*Hadoop Plugin*

\--TO BE ADDED-\-
*Vocabulary Analysis Tool*

The tool basically works using a SlowThread class which extends Thread and the class having run() defined as:

 public final void run() {

&nbsp;&nbsp;&nbsp;app.add(ui);

&nbsp;&nbsp;&nbsp;try {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;execute();

&nbsp;&nbsp;&nbsp;} catch (Throwable t) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;t.printStackTrace();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;app.showStatus(t.getMessage());

&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;app.remove(ui);

&nbsp;&nbsp;&nbsp;app.repaint();

&nbsp;}
Where app is the pane showing results.
The analysis is done as:

SlowThread st = new SlowThread(app) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;public void execute() {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;int numAgeGroups = 100;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;float numDocs = ir.maxDoc();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (numDocs < numAgeGroups) numAgeGroups = ir.maxDoc();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;float ageTotals\[\] = new float\[numAgeGroups\];

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;TermsEnum te = MultiFields.getTerms(ir, field).iterator(null);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;while (te.next() \!= null) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DocsEnum td = te.docs(null, null, 0);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;td.nextDoc();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;float firstDocId = td.docID();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;int ageBracket = (int) ((firstDocId / numDocs) * numAgeGroups);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ageTotals\[ageBracket\]++;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;float total = 0.0f;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;float max = 0.0f;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for (int i = 0; i < ageTotals.length; i++) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (ageTotals\[i\] > max) max = ageTotals\[i\];

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;total \+= ageTotals\[i\];

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (i > 0 && cumul) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ageTotals\[i\] \+= ageTotals\[i - 1\]; // make totals cumulative

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Object maxpct = app.find(myUi, "maxpct");

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (cumul) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;app.setString(maxpct, "text", "100 %");

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} else {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;app.setString(maxpct, "text", (float)Math.round(max * 10000.0f / total) / 100.0f + " %");

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;chart.setScores(ageTotals);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;chart.invalidate();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;app.repaint();

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} catch (Exception e) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;app.showStatus("ERROR: " + e.getMessage());

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;};

This only involves some basic string manipulation, arithmetic and some classes imported from Lucene index, specifically org.apache.lucene.index.MultiFields, org.apache.lucene.index.DocsEnum and org.apache.lucene.index.TermsEnum.
So there’s no need of specific GUI needed for analysis. However for displaying we need Pivot.
*Analyzer Tool*

The Analyzer tool is a lexical analyzer. It analyzes first for tokens, leaving out any special characters and spaces including @ symbol. An example is:


The Analyzer also provides functionality for highlighting selected text.

Right now the highlighting is done by requestFocus() method which works as:

Object inputText = app.find(myUi, "inputText");

&nbsp;&nbsp;&nbsp;if (as.hasAttribute(OffsetAttribute.class)) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OffsetAttribute off = (OffsetAttribute)as.getAttribute(OffsetAttribute.class);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;app.setInteger(inputText, "start", 0);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;app.setInteger(inputText, "end", off.endOffset());

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;app.setInteger(inputText, "start", off.startOffset());

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;app.requestFocus(inputText);

&nbsp;&nbsp;&nbsp;}
where setInteger() is inherited from superclass:

public void setInteger(Object component, String key, int value) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Object\[\] definition = getDefinition(getClass(component), key, "integer");

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (setInteger(component, (String) definition\[1\], value, ((Integer) definition\[3\]).intValue())) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;update(component, definition\[2\]);

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;}
and getDefinition() is :
private static Object\[\] getDefinition(Object classname, String key, String type) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Object currentname = classname;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;while (classname \!= null) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for (int i = 0; i < dtd.length; i \+= 3) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (dtd\[i\] == classname) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Object\[\]\[\] attributes = (Object\[\]\[\]) dtd\[i + 2\];

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (attributes \!= null) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for (int j = 0; j < attributes.length; j++) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (attributes\[j\]\[1\].equals(key)) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if ((type \!= null) && (type \!= attributes\[j\]\[0\])) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;throw new IllegalArgumentException(attributes\[j\]\[0\].toString());

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return attributes\[j\];

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;classname = dtd\[i + 1\];

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;break;

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;}

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;throw new IllegalArgumentException("unknown " + key + " " + type + " for " + currentname);

&nbsp;&nbsp;&nbsp;}



Highlighting can be implemented in Pivot as one of two methods:
* {color:#222222}setSelection(int selectionStart, int selectionLength){color}
* {color:#222222}setSelection(Span selection){color}

{color:#222222}If we already know the offset and length of where we want to select, then we can use the first one:{color}

{color:#222222}void doSomething(){{color}

{color:#222222}int start = <starting offset of text to select>;{color}

{color:#222222}int length = <length of text>;{color}

{color:#222222}textArea.clearSelection();{color}

{color:#222222}textArea.setSelection(start, length);{color}

{color:#222222}}{color}
{color:#222222}Alternatively, we can use the method that involve a Span object:{color}

{color:#222222}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{color}{color:#222222}Span span = new Span(start, end);{color}

{color:#222222}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;textArea.setSelection(span);{color}
The token analysis displayed as a Table Pane. In Pivot we implement Table pane using BXML:

{color:#252525}<layout:SimpleTablePanes bxml:id="window" title="Table Panes" maximized="true"{color}

{color:#252525}&nbsp;&nbsp;&nbsp;xmlns:bxml="{color}[http://pivot.apache.org/bxml|http://pivot.apache.org/bxml]{color:#252525}"{color}

{color:#252525}&nbsp;&nbsp;&nbsp;xmlns:layout="org.apache.pivot.tutorials.layout"{color}

{color:#252525}&nbsp;&nbsp;&nbsp;xmlns="org.apache.pivot.wtk">{color}

{color:#252525}&nbsp;&nbsp;&nbsp;<Border>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<TablePane bxml:id="tablePane" styles="{verticalSpacing:1, showHorizontalGridLines:true,{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;horizontalSpacing:1, showVerticalGridLines:true}">{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<columns>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<TablePane.Column width="100"/>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<TablePane.Column width="2*"/>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<TablePane.Column width="300"/>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</columns>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<TablePane.Row height="-1">{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<TablePane.Filler/>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<Label text="Type" styles="{horizontalAlignment:'center'}"/>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<Label text="Implementation" styles="{horizontalAlignment:'center'}"/>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<Label text="Value" styles="{horizontalAlignment:'center'}"/>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TablePane.Row>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<TablePane.Row height="-1">{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<Label styles="{verticalAlignment:'center'}"/>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TablePane.Row>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<TablePane.Row height="1*">{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<Label styles="{verticalAlignment:'center'}"/>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TablePane.Row>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<TablePane.Row height="1*">{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<Label styles="{verticalAlignment:'center'}"/>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TablePane.Row>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</TablePane>{color}

{color:#252525}&nbsp;&nbsp;&nbsp;</Border>{color}

{color:#252525}</layout:SimpleTablePanes>{color}
and use the BXML file as:

{color:#252525}public class SimpleTablePanes extends Window implements Bindable {{color}

{color:#252525}&nbsp;&nbsp;&nbsp;private TablePane tablePane = null;{color}

{color:#252525}&nbsp;&nbsp;&nbsp;@Override{color}

{color:#252525}&nbsp;&nbsp;&nbsp;public void initialize(Map<String, Object> namespace, URL location, Resources resources) {{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;tablePane = (TablePane)namespace.get("tablePane");{color}

 {color:#252525}}{color}

{color:#252525}}{color}

{color:#252525}&nbsp;&nbsp;&nbsp;{color}

giving a pane similar in look to :
*Status Bar*

A minor feature that is in the Luke application but not in the sandboxed version is a status bar.

It can be implemented using Box Pane.

{color:#252525}<BoxPane styles="{padding:4, verticalAlignment:'center'}">{color}

{color:#252525}&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<Label text="Index : Not added"/>{color}

{color:#252525}</BoxPane>{color}

*Themes*

Another nice, if minor, touch to the application would be different color schemes.
Hex values of the current themes available are as follows.
| {0xece9d0, 0x000000, 0xf5f4f0, 0x919b9a, 0xb0b0b0, 0xeeeeee, 0xb9b9b9, 0xff8080, 0xc5c5dd} | Default |
| {0xe6e6e6, 0x000000, 0xffffff, 0x909090, 0xb0b0b0, 0xededed, 0xb9b9b9, 0x89899a, 0xc5c5dd} | Gray |
| {0xeeeecc, 0x000000, 0xffffff, 0x999966, 0xb0b096, 0xededcb, 0xcccc99, 0xcc6600, 0xffcc66} | Sandstone |
| {0xf0f0ff, 0x0000a0, 0xffffff, 0x8080ff, 0xb0b0b0, 0xededed, 0xb0b0ff, 0xff0000, 0xfde0e0} | Sky |
| {0x6375d6, 0xffffff, 0x7f8fdd, 0xd6dff5, 0x9caae5, 0x666666, 0x003399, 0xff3333, 0x666666} | Navy |





h2. 3. Timeline

\--to be added-\-