Question

如何轻松创建从UTF-8字节流到Unicode代码点数组的映射？为了澄清，例如我是否有字节序列：

c3 a5 76 aa e2 82 ac

映射应生成两个长度相同的数组;一个具有UTF-8字节序列，另一个具有相应的Unicode代码点。然后，阵列可以并排打印，如：

UTF8                UNICODE             
----------------------------------------
C3 A5               000000E5            
76                  00000076            
AA                  0000FFFD            
E2 82 AC            000020AC

Answer 1

适用于流的解决方案：

use READ_SIZE => 64*1024;

my $buf = '';
while (1) {
   my $rv = sysread($fh, $buf, READ_SIZE, length($buf));
   die("Read error: $!\n") if !defined($rv);
   last if !$rv;

   while (length($buf)) {
      if ($buf =~ s/
         ^
         ( [\x00-\x7F]
         | [\xC2-\xDF] [\x80-\xBF]
         | \xE0        [\xA0-\xBF] [\x80-\xBF]
         | [\xE1-\xEF] [\x80-\xBF] [\x80-\xBF]
         | \xF0        [\x90-\xBF] [\x80-\xBF] [\x80-\xBF]
         | [\xF1-\xF7] [\x80-\xBF] [\x80-\xBF] [\x80-\xBF]
         )
      //x) {
         # Something valid
         my $utf8 = $1;
         utf8::decode( my $ucp = $utf8 );
         handle($utf8, $ucp);
      }

      elsif ($buf =~ s/
         ^
         (?: [\xC2-\xDF]
         |   \xE0            [\xA0-\xBF]?
         |   [\xE1-\xEF]     [\x80-\xBF]?
         |   \xF0        (?: [\x90-\xBF] [\x80-\xBF]? )?
         |   [\xF1-\xF7] (?: [\x80-\xBF] [\x80-\xBF]? )?
         )
         \z
      //x) {
         # Something possibly valid
         last;
      }

      else {
         # Something invalid
         handle(substr($buf, 0, 1, ''), "\x{FFFD}");
      }
}

while (length($buf)) {
   handle(substr($buf, 0, 1, ''), "\x{FFFD}");
}

以上只返回U + FFFD，因为Encode::decode('UTF-8', $bytes)认为格式不正确。换句话说，它仅在遇到以下内容时返回U + FFFD：

意外的延续字节。
一个起始字节后面没有足够的连续字节。
＆＃34; overlong＆＃34;的第一个字节编码。

仍然需要进行解码后检查才能返回U + FFFD以查找Encode::decode('UTF-8', $bytes)认为非法的内容。

Answer 2

Encode有一个增量解码API，但它没有记录，您的里程可能会有所不同！它由Encode::Encoding和PerlIO::encoding的子类使用。与任何未记录的API一样，它可以随时更改。已经努力document the API。

#!/usr/bin/perl
use strict;
use warnings;

use Encode qw[STOP_AT_PARTIAL];

my $encoding = Encode::find_encoding('UTF-8');

my @octets = map { pack 'C', hex } qw<C3 A5 76 AA E2 82 AC F0 9F 90 A2>;
my $buffer = '';
while (@octets) {
    my $octets = $buffer . shift @octets;

    printf "--> processing: <%s>\n", 
      join ' ', map { sprintf '%.2X', ord } split //, $octets;

    my $string = $encoding->decode($octets, STOP_AT_PARTIAL);

    $buffer = $octets;

    if (length $buffer) {
        printf "buffered code units: <%s>\n", 
          join ' ', map { sprintf '%.2X', ord } split //, $buffer;
    }

    if (length $string) {
        printf "received code points: <%s>\n",
          join ' ', map { sprintf 'U+%.4X', ord } split //, $string;
    }
}

输出：

--> processing: <C3>
buffered code units: <C3>
--> processing: <C3 A5>
received code points: <U+00E5>
--> processing: <76>
received code points: <U+0076>
--> processing: <AA>
received code points: <U+FFFD>
--> processing: <E2>
buffered code units: <E2>
--> processing: <E2 82>
buffered code units: <E2 82>
--> processing: <E2 82 AC>
received code points: <U+20AC>
--> processing: <F0>
buffered code units: <F0>
--> processing: <F0 9F>
buffered code units: <F0 9F>
--> processing: <F0 9F 90>
buffered code units: <F0 9F 90>
--> processing: <F0 9F 90 A2>
received code points: <U+1F422>

Answer 3

这是一种方法（脚本将字节序列作为第一个命令行参数）：

use feature qw(say);
use strict;
use warnings;

use Encode;

my @hex = split " ", shift;
my $bytes = join '', map { chr hex } @hex;
my @abytes;
my @achr;
while (1) {
    my $str = decode( 'UTF-8', $bytes, Encode::FB_QUIET );
    if ( length $str > 0 ) {
        for my $char ( split //, $str ) {
            my $bytes = encode( "UTF-8", $char, Encode::FB_CROAK | Encode::LEAVE_SRC);
            push @abytes, $bytes;
            push @achr, $char;
        }
    }
    last if length $bytes == 0;
    push @abytes, substr $bytes, 0, 1;
    push @achr, chr 0xfffd;
    $bytes = substr $bytes, 1;
}

my $fmt = '%-20s%-20s';
say sprintf $fmt, qw(UTF8 UNICODE);
say "-" x 40;
for my $char ( @achr ) {
    my $bytes = shift @abytes;
    my $str1 = join ' ', map { sprintf '%X', ord $_} split //, $bytes;
    my $str2 = sprintf '%08X', ord $char;
    say sprintf $fmt, $str1, $str2;
}

将UTF-8字节流转换为Unicode

3 个答案: