Tika-normaliser/1.9.0

From WebLab Wiki

(Redirected from Normaliser using Tika)
Jump to: navigation, search
Normaliser using Tika
Details
Service Interfaces Analyser
Exchange model: WebLab 1.2.5
Versions:

1.8 1.8.2 1.9.0

Licence LGPL 2.1
Supported OS Windows/Linux/MacOs
Integrated COTS Tika
Binary tika-normaliser-1.9.0.war
Sources tika-normaliser-1.9.0-sources.jar
Javadoc tika-normaliser-1.9.0-javadoc.jar
SVN tika-normaliser
Maven Artifact

<groupId>org.ow2.weblab.webservices</groupId>

<artifactId>tika-normaliser</artifactId>

<version>1.9.0</version>
Release Note


This service aims to integrate Tika into the Weblab platform. Tika is use to create Weblab resource by extracting the textual part from various kind of files such as HTML page, PDF file, Microsoft Office files, text file amongst others.

Contents

Configuration

Configuration is achieved through 3 files located under the web application folder in WEB-INF/classes:

  • log4j.properties : allowing to change the logging facility
  • contentManager.properties : configuring the way Tika Normalisation service accesses the file pointed out by hasNativeContent annotation
  • fileContentManager.properties : configuring the default implementation for contentManager which try to retrieve the native content by file, FTP, HTTP and HTTPS URLs

UsageContext effects

As this service is stateless, the UsageContext has no effect.

Examples of SOAP Input/Output

<soapenv:Envelope xmlns:ns8="http://weblab.ow2.org/core/1.2/model#" 
                  xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                  xmlns:anal="http://weblab.ow2.org/core/1.2/services/analyser">
   <soapenv:Header/>
   <soapenv:Body>
      <anal:processArgs xmlns:ns2="http://weblab.ow2.org/core/1.2/services/resourcecontainer"
                        xmlns:ns4="http://weblab.ow2.org/core/1.2/services/configurable"
                        xmlns:ns3="http://weblab.ow2.org/core/1.2/services/trainable" 
                        xmlns:ns9="http://weblab.ow2.org/core/1.2/services/exception" 
                        xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
                        xmlns:ns12="http://weblab.ow2.org/core/1.2/services/queuemanager"
                        xmlns:ns5="http://weblab.ow2.org/core/1.2/services/reportprovider"
                        xmlns:ns6="http://weblab.ow2.org/core/1.2/model#"
                        xmlns:ns7="http://weblab.ow2.org/core/1.2/services/analyser"
                        xmlns:ns10="http://weblab.ow2.org/core/1.2/services/searcher"
                        xmlns:ns8="http://weblab.ow2.org/core/1.2/services/sourcereader"
                        xmlns:ns11="http://weblab.ow2.org/core/1.2/services/indexer">
         <resource xsi:type="ns6:Document" uri="weblab://fileRepository/1326441871065/res_940" 
                   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <annotation uri="weblab://fileRepository/1326441871065/res_940#a0">
               <data xmlns:ns2="http://weblab.ow2.org/core/1.2/services/sourcereader" 
                     xmlns:ns4="http://weblab.ow2.org/core/1.2/services/queuemanager"
                     xmlns:ns3="http://weblab.ow2.org/core/1.2/services/configurable">
                  <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
                           xmlns:wp="http://weblab.ow2.org/core/1.2/ontology/processing#">
                     <rdf:Description rdf:about="weblab://fileRepository/1326441871065/res_940" 
                          xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dcterms="http://purl.org/dc/terms/">
                        <wp:hasNativeContent rdf:resource="http://perdu.com"/>
                        <wp:hasOriginalFileName>index.html</wp:hasOriginalFileName>
                        <wp:hasOriginalFileSize>163</wp:hasOriginalFileSize>
                        <dcterms:extent>163 bytes</dcterms:extent>
                        <dcterms:modified>2010-03-02T19:52:21+0100</dcterms:modified>
                        <wp:hasGatheringDate>2011-10-15T00:11:00+0200</wp:hasGatheringDate>
                        <dc:source>http://perdu.com</dc:source>
                        <dc:title>Vous Etes Perdu</dc:title>
                     </rdf:Description>
                  </rdf:RDF>
               </data>
            </annotation>
         </resource>
      </anal:processArgs>
   </soapenv:Body>
</soapenv:Envelope>

According to configuration, the Tika Normaliser service will download the file specified by the hasNativeContent annotation and normalize it as follows : 

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Body>
      <ns4:processReturn xmlns:ns2="http://weblab.ow2.org/core/1.2/services/exception"
                         xmlns:ns3="http://weblab.ow2.org/core/1.2/services/trainable" 
                         xmlns:ns4="http://weblab.ow2.org/core/1.2/services/analyser" 
                         xmlns:ns5="http://weblab.ow2.org/core/1.2/services/resourcecontainer"
                         xmlns:ns6="http://weblab.ow2.org/core/1.2/services/queuemanager" 
                         xmlns:ns7="http://weblab.ow2.org/core/1.2/services/searcher" 
                         xmlns:ns8="http://weblab.ow2.org/core/1.2/services/configurable"
                         xmlns:wl="http://weblab.ow2.org/core/1.2/model#"
                         xmlns:ns10="http://weblab.ow2.org/core/1.2/services/reportprovider"
                         xmlns:ns11="http://weblab.ow2.org/core/1.2/services/sourcereader"
                         xmlns:ns12="http://weblab.ow2.org/core/1.2/services/indexer">
         <resource xsi:type="wl:Document" uri="weblab://fileRepository/1326441871065/res_940" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <annotation uri="weblab://fileRepository/1326441871065/res_940#a0">
               <data xmlns:ns10="http://weblab.ow2.org/core/1.2/services/searcher"
                     xmlns:ns11="http://weblab.ow2.org/core/1.2/services/indexer"
                     xmlns:ns12="http://weblab.ow2.org/core/1.2/services/queuemanager"
                     xmlns:ns2="http://weblab.ow2.org/core/1.2/services/sourcereader"
                     xmlns:ns3="http://weblab.ow2.org/core/1.2/services/configurable"
                     xmlns:ns4="http://weblab.ow2.org/core/1.2/services/queuemanager"
                     xmlns:ns5="http://weblab.ow2.org/core/1.2/services/reportprovider"
                     xmlns:ns6="http://weblab.ow2.org/core/1.2/model#"
                     xmlns:ns7="http://weblab.ow2.org/core/1.2/services/analyser"
                     xmlns:ns8="http://weblab.ow2.org/core/1.2/services/sourcereader"
                     xmlns:ns9="http://weblab.ow2.org/core/1.2/services/exception">
                  <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:wp="http://weblab.ow2.org/core/1.2/ontology/processing#">
                     <rdf:Description rdf:about="weblab://fileRepository/1326441871065/res_940" xmlns:dc="http://purl.org/dc/elements/1.1/" 
                                      xmlns:dcterms="http://purl.org/dc/terms/">
                        <wp:hasNativeContent rdf:resource="http://perdu.com"/>
                        <wp:hasOriginalFileName>index.html</wp:hasOriginalFileName>
                        <wp:hasOriginalFileSize>163</wp:hasOriginalFileSize>
                        <dcterms:extent>163 bytes</dcterms:extent>
                        <dcterms:modified>2010-03-02T19:52:21+0100</dcterms:modified>
                        <wp:hasGatheringDate>2011-10-15T00:11:00+0200</wp:hasGatheringDate>
                        <dc:source>http://perdu.com</dc:source>
                        <dc:title>Vous Etes Perdu</dc:title>
                        <wp:hasNormalisedContent rdf:resource="file:/home/fred/Dev/apache-tomcat-6.0.20/bin/data/content/weblab.7127726849319234860.content"/>
                     </rdf:Description>
                  </rdf:RDF>
               </data>
            </annotation>
            <annotation uri="weblab://fileRepository/1326441871065/res_940#a4">
               <data>
                  <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
                           xmlns:dc="http://purl.org/dc/elements/1.1/"
                           xmlns:dct="http://purl.org/dc/terms/" 
                           xmlns:wlp="http://weblab.ow2.org/core/1.2/ontology/processing#"
                           xmlns:wlr="http://weblab.ow2.org/core/1.2/ontology/retrieval#">
                     <rdf:Description rdf:about="weblab://fileRepository/1326441871065/res_940#a4">
                        <dct:created>2012-03-29T15:20:59.625+02:00</dct:created>
                        <wlp:isProducedBy rdf:resource="http://weblab.ow2.org/service/normaliser/tika"/>
                     </rdf:Description>
                     <rdf:Description rdf:about="weblab://fileRepository/1326441871065/res_940">
                        <dc:title>Vous Etes Perdu ?</dc:title>
                     </rdf:Description>
                  </rdf:RDF>
               </data>
            </annotation>
            <mediaUnit xsi:type="wl:Text" uri="weblab://fileRepository/1326441871065/res_940#3">
               <content>Perdu sur l'Internet ? Pas de panique, on va vous aider *</content>
            </mediaUnit>
         </resource>
      </ns4:processReturn>
   </soap:Body>
</soap:Envelope>

Known Limitations

N/A

Dependencies

List off all dependencies of this service:

  • org.ow2.weblab.core model 1.2.5-SNAPSHOT
  • org.ow2.weblab.core extended 1.2.5-SNAPSHOT
  • org.ow2.weblab.components content-manager 2.0.0-SNAPSHOT
  • org.ow2.weblab.core.helpers rdf-helper-jena 1.4.0-SNAPSHOT
  • org.apache.tika tika-core 1.1
  • org.apache.tika tika-parsers 1.1
  • com.ibm.icu icu4j 3.8
  • javax.mail mail 1.4.1
  • org.ow2.weblab.core annotator 1.2.5-SNAPSHOT
  • org.apache.cxf cxf-rt-frontend-jaxws 2.6.1
  • org.apache.cxf cxf-rt-transports-http 2.6.1
  • org.springframework spring-web 3.0.7.RELEASE
  • xalan xalan 2.7.1
  • xml-apis xml-apis 1.3.04
  • commons-logging commons-logging 1.1.1
  • junit junit 4.10
  • javax.servlet servlet-api 2.5
Retrieved from "http://weblab-project.org/index.php?title=Tika-normaliser/1.9.0"
Personal tools