Question

尝试使用NUTCH 1.17进行爬网，但该URL被拒绝，有＃！在网址中例如：xxmydomain.com/xxx /＃！/ xxx / abc.html

我也试图加入

+ ^ /

+ ^＃！在我的regex-urlfilter中

Answer 1

如果您特别签入regex-normalize.xml文件这个特定的规则文件将作为urlnormalizer-regex插件的一部分应用。默认情况下，此插件包含在nutch-site.xml中的plugin-includes中。

作为URL规范化g的一部分，如果在URLFragment之后出现任何内容，此特定行将截断URL

<!-- removes interpage href anchors such as site.com#location -->
<regex>
  <pattern>#.*?(\?|&amp;|$)</pattern>
  <substitution>$1</substitution>
</regex>

您可以通过注释禁用此规则。（推荐方式） （或），您可以从nutch-site.xml的plugin-include conf中删除urlnormalizer-regex。

还有另外一个地方，URL标准化部分urlnormalizer-basic中的URL片段部分被忽略

BasicURLNormalizer用于对URL进行一般规范化（即删除多个立即斜杠并使用百分比编码正确编码）

    public String normalize(String urlString, String scope)
      throws MalformedURLException {
    
    if ("".equals(urlString)) // permit empty
      return urlString;

    urlString = urlString.trim(); // remove extra spaces

    URL url = new URL(urlString);

    String protocol = url.getProtocol();
    String host = url.getHost();
    int port = url.getPort();
    String file = url.getFile();

    boolean changed = false;
    boolean normalizePath = false;

    if (!urlString.startsWith(protocol)) // protocol was lowercased
      changed = true;

    if ("http".equals(protocol) || "https".equals(protocol)
        || "ftp".equals(protocol)) {

      if (host != null && url.getAuthority() != null) {
        String newHost = normalizeHostName(host);
        if (!host.equals(newHost)) {
          host = newHost;
          changed = true;
        } else if (!url.getAuthority().equals(newHost)) {
          // authority (http://<...>/) contains other elements (port, user,
          // etc.) which will likely cause a change if left away
          changed = true;
        }
      } else {
        // no host or authority: recompose the URL from components
        changed = true;
      }

      if (port == url.getDefaultPort()) { // uses default port
        port = -1; // so don't specify it
        changed = true;
      }

      normalizePath = true;
      if (file == null || "".equals(file)) {
        file = "/";
        changed = true;
        normalizePath = false; // no further path normalization required
      } else if (!file.startsWith("/")) {
        file = "/" + file;
        changed = true;
        normalizePath = false; // no further path normalization required
      }

      if (url.getRef() != null) { // remove the ref
        changed = true;
      }

    } else if (protocol.equals("file")) {
      normalizePath = true;
    }

    // properly encode characters in path/file using percent-encoding
    String file2 = unescapePath(file);
    file2 = escapePath(file2);
    if (!file.equals(file2)) {
      changed = true;
      file = file2;
    }

    if (normalizePath) {
      // check for unnecessary use of "/../", "/./", and "//"
      if (changed) {
        url = new URL(protocol, host, port, file);
      }
      file2 = getFileWithNormalizedPath(url);
      if (!file.equals(file2)) {
        changed = true;
        file = file2;
      }
    }

    if (changed) {
      url = new URL(protocol, host, port, file);
      urlString = url.toString();
    }

    return urlString;
  }

您可以从代码中看到。.它完全忽略了包含URLFragment的**url.getRef**信息。

因此，我们所能做的就是简单地替换url = new URL(protocol, host, port, file);

规格化方法（line number）

的结尾

与url = new URL(protocol, host, port, file+"#"+url.getRef());

我如何验证的？。

scala> val url = new URL("https://www.codepublishing.com/CA/AlisoViejo/#!/AlisoViejo01/AlisoViejo01.html");
url: java.net.URL = https://www.codepublishing.com/CA/AlisoViejo/#!/AlisoViejo01/AlisoViejo01.html

scala> val protocol = url.getProtocol();
protocol: String = https

scala>     val host = url.getHost();
host: String = www.codepublishing.com

scala>     val port = url.getPort();
port: Int = -1

scala>     val file = url.getFile();
file: String = /CA/AlisoViejo/

scala> //when we construct back new url using the above information we end up loosing fragment information like shown in below

scala> new URL(protocol, host, port, file).toString
res69: String = https://www.codepublishing.com/CA/AlisoViejo/

scala> //if we use url.getRef Information in constructing url we can retain back URL fragment information

scala> //like shown below

scala> new URL(protocol, host, port, file+"#"+url.getRef).toString
res70: String = https://www.codepublishing.com/CA/AlisoViejo/#!/AlisoViejo01/AlisoViejo01.html

scala> // so we can replace the url construction object as explained above to retain url fragment information

注意： UrlFragment将在页面内提供本地对象引用。在大多数情况下，对这些URL进行爬网是没有意义的（这就是为什么要使用上述规则对URL进行归一化），因为HTML将保持不变。

由于存在特殊字符，因此无法抓取网址

1 个答案: