Uni codes and Japanese characters

During the preparation process, the data is extracted (E), transformed (L) and loaded (L). Illustration: © TOLERANT Software

Japanese duplicates

However, duplicates are not only caused by different encodings; they can also be culturally determined. In Japanese, for example, there are three alphabets and all three are used in combination in Japanese texts. Words of Chinese origin are usually written in the Kanji symbol alphabet, grammatical words such as prepositions or verb endings are written in the Hiragana alphabet and the Japanese use the Katakana alphabet for foreign words from English. As the pronunciation of kanji characters is not always clear even for native speakers, names written in kanji are often also transcribed into the hiragana or katakana syllable alphabets. Similar to the Uni-Code, duplicates occur in Japanese data records when transcribed names are not assigned to their Kanji equivalents. Using three examples, we present the challenges that the Japanese language poses for data migration and how duplicates can arise.

Five spellings for one and the same name

One example is the name 髙橋 眞國 (Takahashi Makuni). In this example, the first kanji character 髙 is written in a traditional kanji variant, such as that used in the family register. Depending on the machine, the kanji character can also be written in a slightly different way 高. There is also a modern equivalent for the traditional kanji characters 眞國 in the second part of the name, which looks like this 真国. In the Katakana syllable alphabet, the complete name would be transcribed as タカハシ マサクニ, and in the Hiragana syllable alphabet it would look like soたかはし まくに. During data migration, it is now important to link the 5 different data sets of the same name (i.e. traditional kanji, modern kanji, katakana, hiragana and finally the variant written in Latin letters) in such a way that they can always be found and correctly assigned, regardless of the data input. The consequence of a missing link would be five duplicates.

Error-tolerant duplicate recognition

Another challenge posed by Japanese script is fuzzy or error-tolerant duplicate recognition. A search is fuzzy or error-tolerant if duplicates are found even in the case of incorrect data entry (e.g. misspelled letters, typing errors or minimal spelling differences). In addition to three different alphabets, the Japanese script also distinguishes between upper and lower case letters. When the hiragana sounds よ(yo)、や (ya) andゆ (yu) as well as the katakana soundsヨ (yo)、ヤ (ya) and ユ(yu) merge with a consonant, they are written in a slightly smaller font size. An example is the name 中島 京子 (Nakashima Kyoko), which is transcribed in Hiragana as なかしま きょこ and in Katakana as ナカシマ キョウコ. The characters ょ and ョ follow きand キ respectively and are therefore written slightly smaller. If these characters are written in capital letters, the name from the example above would be transcribed incorrectly as “Nakashima Kiyoko”. The challenge when migrating Japanese data records is therefore to recognise duplicates even with slight spelling differences (such as upper and lower case) and still assign them to the correct kanji characters.

Different kanji readings

Finally, the different readings of kanji characters are particularly challenging for Japanese duplicate recognition. For example, the name 東海林太郎 can be read and pronounced either as Shouji Tarō or as Taorin Tarō. In order to successfully master the task of data migration, data quality tools must be able to identify the katakana transcription ショウジ タロウ (Shouji Tarō) and the variant トウカイリン タロウ(Taorin Tarō) as different readings of the same kanji character and recognise them as duplicates.

Conclusion

Problems with data migration can be avoided by checking data records from different databases before merging them. Incompatible character codes must be cleaned up using ETL processes before data migration and recoded if necessary. Data quality tools that can handle and read all three Japanese writing systems help to avoid errors during data migration. The data quality specialist TOLERANT Software specialises in different writing systems and encodings in many countries and cultures. The data quality tools from TOLERANT Software eliminate the language-related, specific problems of incompatible encodings in many different countries and cultures.