Extend character support to the full range of Unicode Code Points from $000000 to $10FFFF.

Up to now, DIUnicode stored code points as WideChars. This limited Unicode support to the Basic Multilingual Plane (BMP) from $0000 to $FFFF. Code points from the Supplementary Planes were converted to the $FFFD replacement character. This went well with a great number of languages. But less common scripts did not work, just like the increasingly popular emojis from the Symbols and Pictographs Unicode blocks.

DIUnicode 7.0.0 overcomes these limitations and now covers the complete Unicode range. Changes are almost entirely internal and maintain backwards compatibility as much as possible. Existing applications should compile with no or minor changes only. WideChar routines are marked as deprecated and hint at their new complementary UCP routines.

TDIUnicode.Data is still a WideChar buffer. However, its contents is now fully UTF-16 encoded. This means that it may contain code points > $FFFF which take up two WideChars (surrogate pairs). As a result, indexed access to the buffer is no longer guaranteed. TDIUnicode.Data related methods, like TDIUnicode.DataAsStrTrimW are adjusted accordingly.

UnicodeString utility routines are rewritten to handle full UTF-16, including surrogate pairs. Most of them are in DIUtils.pas. YuUtf.pas also contains new utility routines for UTF-16 testing, encoding, and decoding. If possible, string handling routines now take NativeInt type parameters for the buffer length.

Other noteworthy changes:

• TDIUnicode.UCP complements TDIUnicode.Char.
• Changed the type of TDIUnicode.StartCol, EndCol, StartLine, EndLine, StartPos, and EndPos from unsigned Cardinal to signed NativeInt.
• Removed conditional compilation directives DI_No_Classes and DI_No_Unicode_Component. TDIUnicode always descends from TComponent and the Classes unit is always used. Source code only.
• Improve DIUtils.pas Unicode processing to support Unicode Code Points from $000000 to $10FFFF. Adjust remaining source code accordingly.
• Update DIUtils Unicode functions to Unicode 12.1.0.
• Delphi 4 and Delphi 5 crash when compiling DIUtils. There is no error message, so it is not possible to work around the problem. Support for these compilers is therefore removed. At least Delphi 6 is required to compile DIUnicode.
• Remove DI.inc include file. Directly link in DICompilers.inc instead. Source code only.

• Fix potential TDIUnicodeWriter memory leak if TDIUnicodeWriteMethods.Init allocates its own memory.
• TDIUnicodeWriter.Clear calls TDIUnicodeWriteMethods.Flush to reset encoder state.
• KOI8-U converter now maps 0xB4 to U+0404 instead of U+0403.
• Update DIUtils Unicode functions to Unicode 12.
• Compatibility update with DIConverters 1.18.0. These changes only affect projects using DIConverters:
  • Add ISO-2022-CP-MS encoding: Read_iso_2022_jp_ms read methods and Write_iso_2022_jp_ms write methods.
  • DIConverters converter functions now use the native unsigned integer type for the length of a string and support stings longer than 2 GB.
  • UTF-8 converter functions reject surrogates and out-of-range code points, namely the in the ranges 0xD800..0xDFFF and >= 0x110000.
  • Fix error handling in UCS-2, UCS-4, and UTF-32 decoder functions.
  • Tweak the GB18030 converter functions to map 0x8135F437 to U+E7C7.
  • Update the CP1255 converter functions to map 0xCA to U+05BA.

• Support Delphi 10.3 Rio Win32 and Win64.

• Support Delphi 10.2 Tokyo Win32 and Win64.

• Support Delphi 10.1 Berlin Win32 and Win64.

• Support Delphi 10 Seattle Win32 and Win64.

• Add support for Delphi XE8 Win32 and Win64.

• Support Delphi XE7 Win32 and Win64.
• Improved documentation shows inherited class members.

• Support Delphi XE6 Win32 and Win64.

• Compatibility update with other Yunqa products.

• Support Delphi XE5 Win32 and Win64.

• Support Delphi XE4 Win32 and Win64.

• Support Delphi XE3 Win32 and Win64.

• Fix: When reading from TDIUnicodeReader.SourceStream, the size of the internal source buffer was not correctly calculated. Depending on the decoding, this slowed down reading or even stoped it before the end of the stream was reached.
• Fix: TDIUnicodeReader.SkipEmptyLines consumed additional chars after the line break.
• Work around a compiler warning in TDIUnicodeReader.FillSourceBuffer (source code edition only).
• Add unit scope to demo projects.

• Support Delphi XE2 Win64.

• Support Delphi XE2 Win32 (binary editions) and Win64 (source code edition only right now).

• Library source code compiles with FreePascal (Win32).

• Delphi XE support.
• New TDIUnicodeWriter.WriteStr8 and WriteBuf8 methods.
• New TDIUnicodeReader.DataAsStrTrim8 method.
• Improved help layout for better navigation and readability. Send your comments!

• Additions and bug fixes to DIUtils.

• Delphi 2010 support.
• Update character properties, case folding, and case mapping to Unicode 5.10.

• Work around an unexpected Delphi 2009 automatic numeric AnsiChar Unicode conversion in DIUtils.pas which caused a compiler error on Eastern European and Asian language settings.

• Delphi 2009 support.

• Improve compatibility for parallel installation with other DI packages.
• Some code cleanup.

• Delphi 2007 Support.
• Compatibility with DIContainers 1.11.
• Add XP Themes to Demo projects.

• Added compatibility with Delphi 2006 Win32.

• Fixed an error which could prematurely stop TDIUnicodeReader when a pushed source was popped at the end of a nested document.
• Added Delphi 3 compatibility to the utility units.
• Included the missing DIUnicodeCodePages.pas unit which is required to compile the FontCharSet example project.
• Resolved dependency issues when DIUnicode is used in parallel with other DI products.

• Added support for Delphi 2005 Win32.
• Added TDIUnicodeReader.ReadBOM function which returns the Byte Order Mark (BOM) found at the current position and advances the position accordingly.
• Added TDIUnicodeReader.SourceFile property as a simple means to read from a file.
• Added optional WriteByteOrderMark parameter to TDIUnicodeReader.SaveDataToFile and TDIUnicodeReader.SaveDataToStream which controls if a UTF-16/UCS-2 little endian byte order mark is being written in front of the data.
• Other, smaller improvements and bug fixes.

• Added the possibility to link DIUnicode against DIConverters, which gives access to 130+ character encodings.
• Added Pascal implementation for reading / decoding and writing / encoding the following character sets:
  • Mac Arabic, Mac Dingbats, Mac Central Europe, Mac Croatian, Mac Cyrillic, Mac Farsi, Mac Greek, Mac Hebrew, Mac Iceland, Mac Roman, Mac Romanian, Mac Thai, Mac Turkish
  • UCS-2 LE, CS-2 BE
  • UCS-4 LE, UCS-4 BE
  • UTF-32 LE, UTF-32 BE
  • UTF-7 (Write_UTF_7 / Read_UTF_7)
  • UTF-7 Optional Direct Characters (Write_UTF_7_ODC / reads as Read_UTF_7)
  • JIS X0201, NextStep, TIS 620
• Bug fixes.

• Added support for reading and writing UTF-7 according to RFC 2152. Writing UTF-7 comes in two flavors, with (Write_UTF_7) or without (Write_UTF_7_ODC) encoding optional direct characters. UTF-7 reading (Read_UTF_7) works equaly well for both writing methods.
• Implementation of UTF-7 made it necessary to change the reading and writing implementation of TDIUnicodeReader and TDIUnicodeWriter to allow data buffering between consecutive reads and writes.
• TDIUnicodeReader.PushSource and TDIUnicodeReader.PopSource methods added to which allow to insert one source into another, like for Pascal {$INCLUDE …} directive.
• TDIUnicodeReader can optionally free its source stream if the reading reached the end of the stream. This is especially usefull when reading nested files using the TDIUnicodeReader.PushSource and TDIUnicodeReader.PopSource methods. The protected property TDIUnicodeReader.AutoFreeSourceStreams may be used by descendent classes which implement specialized reading / parsing.
• Added Methods for reading digits and hexadecimal characters to TDIUnicodeReader, as well as for retrieving data as trimmed strings.
• Various other improvements, code clean-ups and minor bug fixes.

• Initial release.