utf 8 - Is there an easy way to work around a Delphi utf8-file flaw? -

i have discovered (the hard way) if file has valid utf-8 bom contains invalid utf8 encodings, , read of delphi (2009+) encoding-enabled methods such loadfromfile, result empty file no error indication. in several of applications, prefer lose few bad encodings, if no error report in case either.

debugging reveals multibytetowidechar called twice, first output buffer size, conversion. tencoding.utf8 contains private fmbtowcharflags value these calls, , initialized mb_err_invalid_chars value. call charcount returns 0 , loaded file empty. calling api without flag 'silently drop illegal code points'.

my question how best weave through nest of classes in encoding area work around fact private value (and needs be, because class var threads). think add custom utf8 encoding, using guidance in marco cantu's delphi 2009 book. , optionally raise exception if multibytetowidechar has returned encoding error, after calling again without flag. not solve problem of how custom encoding used instead of tencoding.utf8.

if set default application @ initialization, perhaps modifying class var tencoding.uft8, sufficient.

of course, need solution without waiting lodge qc report asking more robust design, getting accepted, , seeing changed.

any ideas welcome. , can confirm still issue xe4, have not yet installed?

i ran mb_err_invalid_chars issue when first updated indy support tencoding, , ended implementing custom tencoding-derived class utf-8 handling avoid specifying mb_err_invalid_chars. didn't think use class helper.

however, issue not limited utf-8. decoding failure of of tencoding classes result in blank result, not exception being raised. why embarcadero chose route, when of rtl/vcl uses exceptions instead, beyond me. not raising exception on error caused fair amount of issues in indy had worked around.