Subj : not all is lost but far too much for far too long
To : Ozz Nixon
From : mark lewis
Date : Sat Jun 29 2019 01:25 am
On 2019 Jun 28 21:23:36, you wrote to Maurice Kinal:
ON> FTN Header versus actual message body conveying Unicode.
ON> When I telnet to a SQL server that speaks Unicode only, it always
ON> returns the following characters (pascal): #239#187#191
that's a UTF8 BOM (Byte Order Mark)...
ON> When I telnet to a web page that speaks Unicode, it too returns
ON> #239#187#191 plus the <!doctype html> etc.
i'm sure you know what it is but if not, it is a magic number that may appear
at the start of a text stream... it signals at least one of several things to
program processing the stream...
1. the byte order, or endianness, of the stream
2. that the text stream's encoding is unicode
3. which nnicode encoding the stream is using
ON> So... would it not stand true that systems that are posting UTF8 do
ON> the same introduction on the message body?
they could but it is not required... it actually interfers with software using
UTF8 that do not expect non-ascii bytes at the beginning of a stream...
ON> Then authors *know* it potentially has Unicode
see above... it does indicate that the stream is unicode... not potentially...
ON> and leave it damn well alone, and also parse it based upon UTF8
ON> instead of 8bit char...
it is an idea except that everyone else that uses plain ascii will be saying,
what's that garbage at the beginning of these messages?
ON> This is how I am coding things here, just based upon NexusSQL,
ON> PremierSQL, MS SQL, Apache and Nexus Web Service. I do not have access
ON> to my Oracle box nor the MySQL 5 server to see if they do the same
ON> during the initial connection negotiation(s).
it is probably a configuration option... apache shouldn't care as it just sends
whatever is in the file... i don't know about nexus...
ON> A quick google: It's the utf8 byte order mark. Some editors save the
ON> BOM inside the file (in order to be used as a header) which regularly
ON> causes confusion because it is optional.
ahh, you found it :)
ON> So, if we wanted to help enforce at a reader (or even tosser level)
ON> how to handle, I would offer this up as a required BOM to the message
ON> body that is UTF8.
tossers shouldn't be modifying message bodies anyway... that's in the specs...
the problem is how some coders interpreted "ignore"... the funny thing is if
they chose to ignore the problematic character by stripping it, they actually
added code to remove it... if they had selected the other form of ignore and
left it in the stream, their code would be (slightly) smaller and faster... it
is kinda funny on the one hand...
)\/(ark
Always Mount a Scratch Monkey
Do you manage your own servers? If you are not running an IDS/IPS yer doin' it
wrong...
... You know you're in YK when: you have to break your dog loose from the tree.
---
* Origin: (1:3634/12.73)
