Question

我们的产品利用ascii折叠令牌过滤器，我们的客户要求提供有关它的具体信息。具体来说，他们希望将unicode字符映射到ASCII等效字符。虽然我相信大多数转换是显而易见的（例如ü= u），但有一些像ß这样的“棘手”的转换，我相信这些转换为“ss”。

我用谷歌搜索但未能找到明确的映射。我有什么地方可以获得这些信息吗？

感谢您的帮助，埃里克

Answer 1

You can just read the source code for ASCIIFoldingFilter.

来自该来源的样本：

      case '\u00C0': // À  [LATIN CAPITAL LETTER A WITH GRAVE]
      case '\u00C1': // Á  [LATIN CAPITAL LETTER A WITH ACUTE]
      case '\u00C2': // Â  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX]
      case '\u00C3': // Ã  [LATIN CAPITAL LETTER A WITH TILDE]
      case '\u00C4': // Ä  [LATIN CAPITAL LETTER A WITH DIAERESIS]
      case '\u00C5': // Å  [LATIN CAPITAL LETTER A WITH RING ABOVE]
      case '\u0100': // Ā  [LATIN CAPITAL LETTER A WITH MACRON]
      case '\u0102': // Ă  [LATIN CAPITAL LETTER A WITH BREVE]
      case '\u0104': // Ą  [LATIN CAPITAL LETTER A WITH OGONEK]
      case '\u018F': // Ə  http://en.wikipedia.org/wiki/Schwa  [LATIN CAPITAL LETTER SCHWA]
      case '\u01CD': // Ǎ  [LATIN CAPITAL LETTER A WITH CARON]
      case '\u01DE': // Ǟ  [LATIN CAPITAL LETTER A WITH DIAERESIS AND MACRON]
      case '\u01E0': // Ǡ  [LATIN CAPITAL LETTER A WITH DOT ABOVE AND MACRON]
      case '\u01FA': // Ǻ  [LATIN CAPITAL LETTER A WITH RING ABOVE AND ACUTE]
      case '\u0200': // Ȁ  [LATIN CAPITAL LETTER A WITH DOUBLE GRAVE]
      case '\u0202': // Ȃ  [LATIN CAPITAL LETTER A WITH INVERTED BREVE]
      case '\u0226': // Ȧ  [LATIN CAPITAL LETTER A WITH DOT ABOVE]
      case '\u023A': // Ⱥ  [LATIN CAPITAL LETTER A WITH STROKE]
      case '\u1D00': // ᴀ  [LATIN LETTER SMALL CAPITAL A]
      case '\u1E00': // Ḁ  [LATIN CAPITAL LETTER A WITH RING BELOW]
      case '\u1EA0': // Ạ  [LATIN CAPITAL LETTER A WITH DOT BELOW]
      case '\u1EA2': // Ả  [LATIN CAPITAL LETTER A WITH HOOK ABOVE]
      case '\u1EA4': // Ấ  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND ACUTE]
      case '\u1EA6': // Ầ  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND GRAVE]
      case '\u1EA8': // Ẩ  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND HOOK ABOVE]
      case '\u1EAA': // Ẫ  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND TILDE]
      case '\u1EAC': // Ậ  [LATIN CAPITAL LETTER A WITH CIRCUMFLEX AND DOT BELOW]
      case '\u1EAE': // Ắ  [LATIN CAPITAL LETTER A WITH BREVE AND ACUTE]
      case '\u1EB0': // Ằ  [LATIN CAPITAL LETTER A WITH BREVE AND GRAVE]
      case '\u1EB2': // Ẳ  [LATIN CAPITAL LETTER A WITH BREVE AND HOOK ABOVE]
      case '\u1EB4': // Ẵ  [LATIN CAPITAL LETTER A WITH BREVE AND TILDE]
      case '\u1EB6': // Ặ  [LATIN CAPITAL LETTER A WITH BREVE AND DOT BELOW]
      case '\u24B6': // Ⓐ  [CIRCLED LATIN CAPITAL LETTER A]
      case '\uFF21': // Ａ  [FULLWIDTH LATIN CAPITAL LETTER A]
        output[outputPos++] = 'A';
        break;

正如你所看到的，它对希腊语和西里尔字母都没有做任何事情，更不用说其他字母了。

另外。正如您猜对，ß转换为ss：

      case '\u00DF': // ß  [LATIN SMALL LETTER SHARP S]
        output[outputPos++] = 's';
        output[outputPos++] = 's';
        break;

对于ElasticSearch的ascii折叠令牌过滤器，unicode字符到前127个ASCII字符的映射是什么？

1 个答案: