ShiftJIS::Collate - collation of Shift-JIS strings
use ShiftJIS::Collate;
@sorted = ShiftJIS::Collate->new(%tailoring)->sort(@source);
This module provides some functions to compare and sort strings in Shift-JIS based on JIS X 4061:1996, collation of Japanese character strings, ``Nihongo mojiretsu shogo junban''.
This module is an implementation of JIS X 4061:1996 and the collation rules are based on that standard. See Conformance to the Standard.
The new method returns a collator object.
$Collator = ShiftJIS::Collate->new(
ignoreChar => $regexIgnoredChar,
kanji => $kanji_class,
katakana_before_hiragana => $bool,
level => $collationLevel,
position_in_bytes => $bool,
tounicode => \&sjis_to_unicode,
preprocess => \&preprocess,
upper_before_lower => $bool,
);
# if %tailoring is false (empty),
# $Collator should do the default collation.
e.g. If you want to ignore KATAKANA-HIRAGANA PROLONGED SOUND MARK
and its halfwidth form, say
ignoreChar => '^(?:\x81\x5B|\xB0)',
If the parameter is true, this is reversed.
Class 1: 'saisho' (minimal) Class 2: 'kihon' (basic) Class 3: 'kakucho' (extended)
The kanji class is specified as 1, 2, or 3. If omitted, class 2 is applied.
This module does not support 'kakucho' kanji class, since the repertoire of Shift-JIS does not define all the Unicode CJK unified ideographs.
But if the kanji class 3 is specified, you can collate kanji in the unicode order. In this case you must provide tounicode coderef which gives a unicode codepoint from a Shift-JIS character.
Level 1: alphabetic ordering Level 2: diacritic ordering Level 3: case ordering Level 4: script ordering Level 5: width ordering
The collation level is specified as a number between 1 and 5. If omitted, level 4 is applied.
Such a subroutine should map a string comprising of a kanji of level 1 and 2 in Shift-JIS to a codepoint in the range between 0x4E00 and 0x9FFF.
index method returns its results in characters.
If this parameter is true, it returns the results in bytes.
If the parameter is true, this is reversed.
$result = $Collator->cmp($a, $b)1 (when $a is greater than $b)
or 0 (when $a is equal to $b)
or -1 (when $a is lesser than $b).
$result = $Collator->eq($a, $b)$result = $Collator->ne($a, $b)$result = $Collator->gt($a, $b)$result = $Collator->ge($a, $b)$result = $Collator->lt($a, $b)$result = $Collator->le($a, $b)
$sortKey = $Collator->getSortKey($string)You compare the sort keys using a binary comparison and get the result of the comparison of the strings.
$Collator->getSortKey($a) cmp $Collator->getSortKey($b)
is equivalent to
$Collator->cmp($a, $b)
@sorted = $Collator->sort(@source)@sorted = $Collator->sortYomi(@source)Yomi-hyoki shogo is carried out through two comparison stages.
First, order by reading; next, order by spelling among strings having the same reading.
See also sample/yomi.txt.
@sorted = $Collator->sortDaihyo(@source)Note: This module does not support kihon-daihyo-yomi shogo: 'the basic representative reading collation', in which daihyo-yomi jisho, a dictionary of representative readings, is required.
kan'i-daihyo-yomi shogo is carried out through five comparison stages.
This ordered list is an example of the result of "kan'i-daihyo-yomi shogo".
(1) Compare the character class of the first character of the spell.
(2) Compare the first character of the reading.
(3) Compare the first character of the spell.
(4) Compare the whole string of the reading.
(5) Compare the whole string of the spell.
See also sample/daihyo.txt.
$position = $Collator->index($string, $substring)($position, $length) = $Collator->index($string, $substring)$substring matches a part of $string, returns
the position of the first occurrence of the matching part in scalar context;
in list context, returns a two-element list of
the position and the length of the matching part.
Notice that the length of the matching part may differ from
the length of $substring.
If $substring does not match any part of $string,
returns -1 in scalar context and an empty list in list context.
If your substr function is aware of Shift-JIS,
specify true as position_in_bytes. See Constructor and Tailoring.
The following criteria are considered in order until the collation order is determined. By default, Levels 1 to 4 are applied and Level 5 is ignored, for JIS X 4061 does not specify level 5.
1 space
2 'kijutsu kigou' (punctuation marks)
3 'kakko kigou' (quotes, parentheses, and braces)
4 'gakujutsu kigou' (mathematical and technical signs)
5 'ippan kigou' (general symbols)
6 'tan-i kigou' (unit signs)
7 digits (European)
8 'ouji kigou' (Greek and Cyrillic letters, as "European signs")
9 Latin alphabets
10 kana (hiragana and katakana)
11 kanji
12 'geta' mark
In the class, alphabets are collated alphabetically; kana letters are AIUEO-betically (in the Gozyuon order); kanji are in the JIS X 0208 order.
According to JIS X 4061, GETA MARK (0x81AC, U+3013)
must be the greatest character (ordered at the last).
Any character for which the order is no defined, like control characters, box drawings, unassigned characters, etc. is regarded as a completely ignorable character.
A voiceless kana, the voiced, then the semi-voiced (if exists).
In kana, the order is as shown the following list. see Replacement of PROLONGED SOUND MARK and ITERATION MARKs.
Replaced PROLONGED SOUND MARKs (U+30FC and U+FF70);
Small Kana;
Replaced ITERATION MARKs (U+309D, U+309E, U+30FD, and U+30FE);
then, Normal Kana.
Halfwidth and Fullwidth Forms
is greater than the corresponding normal character.
BN: JIS X 4061 does not mention this level. Level 5 is an extention by this module.
JIS X 4061 specifies three Kanji Classes. This modules supports 'Saisho' and 'Kihon' Kanji Classes.
0x8156 (U+3003)
0x8157 (U+4EDD)
0x8158 (U+3005)
0x8159 (U+3006)
0x815A (U+3007)
Any kanji except U+4EDD are ignored on collation.
SJIS UCS Name
0x815B U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK 0xB0 U+FF70 HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK 0x8154 U+309D HIRAGANA ITERATION MARK 0x8155 U+309E HIRAGANA VOICED ITERATION MARK 0x8152 U+30FD KATAKANA ITERATION MARK 0x8153 U+30FE KATAKANA VOICED ITERATION MARK
These characters, if replaced, are secondary equal to the replacing kana, while ternary not equal to.
The unreplaced characters are primary greater than any kana.
e.g. CJK Ideograph followed by PROLONGED SOUND MARK
Digit followed by ITERATION MARK
Kana that has no voiced variant followed by VOICED ITERATION MARK
(1) charset: Shift-JIS.
(2) No limit of the number of characters in the string considered
to collate.
(3) No character class is added.
(4) The following characters are added as collation elements.
IDEOGRAPHIC SPACE is added in the space class.
ACUTE ACCENT, GRAVE ACCENT, DIAERESIS, CIRCUMFLEX ACCENT
are added in 'kijutsu kigou'.
APOSTROPHE, QUOTATION MARK are added in 'kakko kigou'.
HYPHEN-MINUS is added in 'gakujutsu kigou'.
(5) Collation of Latin alphabets with macron and with circumflex
is not supported.
(6) Kanji Classes:
i) the minimal kanji class (Five kanji-like chars).
ii) the basic kanji class (Levels 1 and 2 kanji of JIS)..
SADAHIRO Tomoyuki <SADAHIRO@cpan.org>
Copyright(C) 2001-2007, SADAHIRO Tomoyuki. Japan. All rights reserved.
This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.