\YTEX annotators
All the Pipelines in the YTEX_HOME/config/desc directory use the following
This pipeline is located under CTAKES_HOME/desc/ytex-uima/desc/analysis_engine/AggregatePlaintextUMLSProcessor.xml. It is a copy of the 'typical' cTAKES pipeline with the following modifications:
The YTEXPipeline, located under CTAKES_HOME/desc/ytex-uima/desc/analysis_engine/YTEXPipeline.xml, is a minimal pipeline used in previous cTAKES versions. It is based on the cTAKES clinical document pipeline. You are of course welcome to modify/add/remove annotators to the analysis engine and modify configuration parameters.
Clinical Documents from an EMR are typically stored in a database. We developed an UIMA CollectionReader that retrieves these documents for annotation (thereby avoiding exporting the documents to the file system). The DBCollection reader works as follows:
Documents in a database typically are given a Unique ID, and are cross-referenced with other information (e.g. patient id, document type, document date, etc.). This unique id can be stored in the database along with other document annotations. This allows you to cross-reference annotation data with other document data.
By default, YTEX supports an integer document identifier (INSTANCE_ID) and/or a string document identifier (INSTANCE_KEY). When using the FileSystemCollection reader, the file name is stored in the INSTANCE_KEY.
The YTEX DBCollection reader is parameterized by 2 queries: a key query and a document query. The key query loads document unique ids from a database, and the document query loads a document for a given unique id. For more information, refer to the example below.
To illustrate this, we can simply retrieve sample documents stored in the fracture_demo table. To configure and execute the CPE, do the following:
Document Key Query (replace schema to match your configuration)
select note_id INSTANCE_ID from <schema>.fracture_demo
Document Query
select note_text from <schema>.fracture_demo where note_id = :instance_id
Document Key Query
select note_id INSTANCE_ID from fracture_demo
Document Query
select note_text from fracture_demo where note_id = :instance_id
Document Key Query
select note_id INSTANCE_ID from fracture_demo
Document Query
select note_text from fracture_demo where note_id = :instance_id
Click on the 'Add' button in the 'Analysis Engine' section, and select CTAKES_HOME\desc\ytex-uima\desc\analysis_engine\AggregatePlaintextUMLSProcessor.xml
select * from
v_document where analysis_batch = 'test2'
Notice that the instance_id column is set, and refers to the fracture_demo.note_id.
This example demonstrates how to use custom document keys. Let's assume that clinical documents have a unique identifier that comprises 2 fields - note_id (integer) and site_id (character) - and that these documents are stored in the fracture_demo table. We would like to link our annotations to the original document, so we need to store both the note_id and site_id in the ytex document table. In this example, we map these columns to the instance_id and site_id columns in the document table. (We could map the site_id to INSTANCE_KEY, but that would be no fun.)
To run this example, do the following:
alter table document add site_id varchar(20)
Windows: Run YTEX_HOME/ytexCPE.cmd Unix: from a shell run the following commands
. ${HOME}/ytex.profile
cd ${YTEX_HOME}
java ${JAVA_OPTS} org.apache.uima.tools.cpm.CpmFrame
Go to File->Open, and select YTEX_HOME/examples/cpe-fracture/fracture-demo.cpe.xml. The query to get the keys looks like this (notice how note_id was renamed to instance_id):
select note_id instance_id, site_id from fracture_demo
The query to get the document looks like this:
select note_text from fracture_demo where note_id = :instance_id and site_id = :site_id
The CPE config "Store Doc Text" checkbox is unchecked: we will note store the document text in the document.doc_text column, because it is already in the database - we can join the document and fracture_demo tables on the uid/site_id columns to get the corresponding text.
select * from document where analysis_batch = 'cpe-fracture'
The UIMA CPE Configurator (yes it is ugly) allows you to override parameters defined in the UIMA Pipeline. Some of these properties are:
Typically, you will want to annotate different document collections, or you may want to annotate the same document collection with different pipelines/configurations. The analysis_batch is a way to identify document annotation runs or document collections. It is stored in thedocument.analysis_batch column.
Some simple queries (replace schema to match your configuration):
select top 1000 document_id, doc_text
from <schema>.v_document
select *
from <schema>.v_document_cui_sent
where code = 'C0024228'
and certainty = -1
Refer to the YTEX Data Model page for more information.
For a graphical representation of document annotations, use the DBAnnotationViewer. This modified viewer retrieves the document CAS from the database (as opposed to the plain-vanilla AnnotationViewer which retrieves the CAS from the file system).. TODO Update this
By default, YTEX configures the cTAKES DictionaryLookup algorithm to annotate documents with concepts from the SNOMED and RXNORM vocabularies. These vocabularies are quite expansive; nevertheless, they may lack certain concepts relevant to your project. Refer to Dictionary Lookup Configuration for information on how to configure the dictionary lookup algorithm.
The Dictionary Lookup Algorithm may not be flexible enough to identify all variants of a concept; YTEX can also identify concepts using regular expressions. Simply add a row to the ref_named_entity_regex table:
insert into <schema>.ref_named_entity_regex (regex, coding_scheme, code)
values ('(?i)\bREFER\s+TO\s+.*#{0,1}+\s*\d+','ESLD','DOCREF')
;
| column | description |
| regex | regular expression (see java pattern) |
| coding_scheme | similar to sourcetype in umls lookup table (see above) |
| code | concept id/code |
| context | the document section to which the regular expression search should be limited (see sections below) |
YTEX can identify sections within a document using regular expressions. Update the ref_segment_regex table, e.g.:
insert into <schema>.ref_segment_regex (segment_id, regex) values('FINDINGS', '\nFINDINGS:|\nTECHNIQUE AND FINDINGS|\nPROCEDURE AND FINDINGS:|\nFindings:');
| column | description |
| segment_id | the segment identifier |
| regex | the regular expression that finds the section heading, or finds the entire section |
| limit_to_regex | 0 - false - the regular expression only identifies the section heading. In this case, the section will span the text from the heading to the next section. 1 - true - the regular expression identifies the entire section. The section spans from the beginning to the end of the text covered by the regular expression |
For data mining purposes, text can be represented as a 'Bag-of-Words': a matrix with 'words' as columns and documents as rows. The value of the column represents the (weighted) word frequency, or is an indicator representing the presence of the word in the document. The words can be the raw natural language word, the stemmed word, or concept identifiers. This is typically a very high-dimensional feature space, i.e. the number of columns (distinct words) can be very large. This space is typically sparse: most of the words assume the value 0.
To deal with sparse, high-dimensional spaces efficiently, data mining packages support a sparse file format. YTEX currently supports exporting sparse matrices in the following formats:
YTEX provides great flexibility in choosing which features or class of features to export. Before we get into details, you should run either the WEKA or R example.
TODO update this
In this example, we develop a classifier that identifies documents that assert the presence of a fracture. The documents and their class labels are stored in the fracture_demo table. We decided to use 2/3 of the notes as a training set, and 1/3 as a test set, and store this assignment in thefracture_demo table as well.
Before going through the following steps, please annotate the fracture documents using the YTEX collection processing engine as documented above (Example - Custom Key Mapping).
In this example, we annotate documents, export two different representations of these documents, and train classifiers using either R or Weka. We export a bag-of-cuis representation where each variable represents the frequency of an affirmed UMLS concept within a document; we also export a bag-of-words representation where each variable represents the frequency of a stemmed word.
The SparseDataExporter takes a Java property file as input; the property file specifies the SQL queries used to obtain instances (i.e. documents), their class labels, and their 'attributes'.
Start a command prompt/shell, change to the YTEX_HOME\examples\fracture\cui or YTEX_HOME\examples\fracture\word directory (for the bag-of-cuis vs. bag-of-words example respectively), and run the following commands.
Windows:
..\..\..\setenv.cmd
java %JAVA_OPTS% ytex.kernel.SparseDataExporterImpl -prop export.xml -type weka
Linux:
. ${HOME}/ytex.profile
java ${JAVA_OPTS} ytex.kernel.SparseDataExporterImpl -prop export.xml -type weka
After executing this you should see 2 arff files in this directory corresponding to the training and test sets.
java -Xmx1500m weka.gui.GUIChooser
The rest is standard WEKA usage.
Execute the sparse data exporter with the -type sparsematrix option. You should see 3 files:
What all this is should become clear if you look at YTEX_HOME\examples\fracture\classify.R
To train and test a classifier, do the following
source("../classify.R")
This will train a decision tree on the training data, print out the decision tree, run the decision tree on the test data, and print the results.
The SparseDataExporter is parameterized by java properties that contain queries that determine how data will be exported. Refer to export.xmlfor sample queries.
The queries return data in instance-attribute-value triples.
Retrieves instance ids (i.e. document ids) and their class labels. This query must return the following columns:
Retrieves numeric instance attributes for all attribute-instance combinations. Must return 3 columns:
Either a numericWordQuery, a nominalWordQuery, or both must be specified. Retrieves nominal instance attributes for all attribute-instance combinations. Must return 3 columns:
For weka, nominal attributes are created in the ARFF file. For the sparsematrix format, each nominal attribute level is turned into a numeric attribute with a binary indicator.
Applicable only to Weka. This is the internal 'name' of the dataset