Question

我正在利用solr ExtractingRequestHandler来提取和索引HTML内容。我的问题来自它产生的提取链接部分。返回的提取内容插入了“rect”，它们在HTML源代码中不存在。

我的solrconfig单元配置如下：

  <requestHandler name="/upate/extract" 
              startup="lazy"
              class="solr.extraction.ExtractingRequestHandler" >
<lst name="defaults">
  <str name="lowernames">true</str>
  <!-- capture link hrefs but ignore div attributes -->
  <str name="captureAttr">true</str>
  <str name="fmap.div">ignored_</str>
</lst>

我的solr schema.xml包含以下语言：

   <field name="content_type" type="string" indexed="true" stored="true" multiValued="true"/>
   <field name="links" type="string" indexed="true" stored="true" multiValued="true"/>
   <field name="meta" type="string" indexed="true" stored="true" multiValued="true"/>
   <field name="content_encoding" type="string" indexed="false" stored="true" multiValued="false"/>
   <field name="content" type="text_general" indexed="false" stored="true" multiValued="true"/>

我将以下HTML发布到sorl cell：

<!DOCTYPE html>
<html>
<body>
  <h1>Heading1</h1><a href="http://www.google.com">Link to Google</a><a href=
  "http://www.google.com">Link to Google2</a><a href="http://www.google.com">Link to
  Google3</a><a href="http://www.google.com">Link to Google</a>

  <p>Paragraph1</p>
</body>
</html>

Solr具有以下索引：

      {
    "meta": [
      "Content-Encoding",
      "ISO-8859-1",
      "ignored_hbaseindexer_mime_type",
      "text/html",
      "Content-Type",
      "text/html; charset=ISO-8859-1"
    ],
    "links": [
      "rect",
      "http://www.google.com",
      "rect",
      "http://www.google.com",
      "rect",
      "http://www.google.com",
      "rect",
      "http://www.google.com"
    ],
    "content_encoding": "ISO-8859-1",
    "content_type": [
      "text/html; charset=ISO-8859-1"
    ],
    "content": [
      "             Heading1  Link to Google  Link to Google2  Link to Google3  Link to Google  Paragraph1   "
    ],
    "id": "row69",
    "_version_": 1461665607851180000
  }

注意每个链接之间的“rect”。为什么solr cell或tika插入这些？我没有定义要使用的tika配置文件。我需要配置tika吗？

Answer 1

尽管是一个老问题，但在通过Solr 8.7.0为HTML文档建立索引时，我也遇到了这个问题。

<requestHandler name="/update/extract" 
    class="org.apache.solr.handler.extraction.ExtractingRequestHandler">
    ....

HTML：

<p>My website is <a href="https://buriedtruth.com/">BuriedTruth.com</a>.</p>

结果：

My website is rect https://buriedtruth.com/ BuriedTruth.com .

[我正在Linux命令行上发布/编制索引：solr restart; sleep 1; post -c gettingstarted /mnt/Vancouver/programming/datasci/solr/test/solr_test9.html; ]

我用grep（ripgrep：rg --color=always -w -e 'rect' . |less）这个单词的Solr代码，但是什么也没找到，因此索引网址中rect http...的来源使我难以理解。

我的解决方案是向我的solrconfig.xml添加一个正则表达式处理器：

  <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}"
           processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
    <processor class="solr.LogUpdateProcessorFactory"/>
    <processor class="solr.DistributedUpdateProcessorFactory"/>
    <!-- ======================================== -->
    <!-- https://lucene.apache.org/solr/7_4_0/solr-core/org/apache/solr/update/processor/RegexReplaceProcessorFactory.html -->
    <!-- Solr bug? URLs parse as "rect https..."  Managed-schema (Admin UI): defined p as text_general -->
    <!-- but did not parse. Looking at content | title: text_general copied to string, so added  -->
    <!-- copyfield of p (text_general) as p_str ... regex below now works! -->
    <processor class="solr.RegexReplaceProcessorFactory">
      <str name="fieldName">content</str>
      <str name="fieldName">title</str>
      <str name="fieldName">p</str>
      <!-- Case-sensitive (and only one pattern:replacement allowed, so use as many copies): -->
      <!-- of this processor as needed: -->
      <str name="pattern">rect http</str>
      <str name="replacement">http</str>
      <bool name="literalReplacement">true</bool>
    </processor>
    <!-- ======================================== -->
    <!-- This needs to be last (may need to clear documents and reindex to see changes, e.g. Solr Admin UI): -->
    <processor class="solr.RunUpdateProcessorFactory"/>
  </updateRequestProcessorChain>

正如在该处理器中的评论所暗示的那样，我正在将<p />格式的HTML内容提取到p字段（field: p | type: text_general）中。

该内容未使用RegexReplaceProcessorFactory处理器进行解析。

在Solr Admin UI中，我注意到title和content被复制为字符串（例如：field: content | type: text_general | copied to: content_str），所以我复制字段（p >> p_str）解决了正则表达式问题。

为完整起见，这是我的solrconfig.xml中与HTML文档索引相关的部分，

  <lib dir="${solr.install.dir:../../..}/contrib/extraction/lib" regex=".*\.jar" />
  <lib dir="${solr.install.dir:../../..}/dist/" regex="solr-cell-\d.*\.jar" />

  <!-- https://lucene.472066.n3.nabble.com/Prons-an-Cons-of-Startup-Lazy-a-Handler-td4059111.html -->
                  <!-- startup="lazy" -->

  <requestHandler name="/update/extract"
                  class="org.apache.solr.handler.extraction.ExtractingRequestHandler">
    <lst name="defaults">
      <str name="lowernames">true</str>
      <str name="uprefix">ignored_</str>
      <str name="capture">div</str>
      <str name="fmap.div">div</str>
      <str name="capture">p</str>
      <str name="fmap.p">p</str>
    </lst>
  </requestHandler>

...再次指出，我是通过Solr管理员界面将字段添加到managed-schema的。

结果：

My website is https://buriedtruth.com/ BuriedTruth.com .

  <field name="p" type="text_general" uninvertible="true" indexed="true" stored="true"/>
  <copyField source="p" dest="p_str"/>

另请参阅：

re：<requestHandler name="/update/extract"...：
- Solr 8.6.3 could not index html file
- https://lucene.apache.org/solr/guide/8_6/uploading-data-with-solr-cell-using-apache-tika.html#configuring-the-extractingrequesthandler-in-solrconfig-xml
当我从Solr的updateRequestProcessorChain />切换到经典的managed-schema
时，我的回答（与上面的schema.xml相关的问题）
- How to extract metatags from HTML files and index them in SOLR and TIKA

Solr ExtractingRequestHandler在链接中提取“rect”

1 个答案: