Re: Is it me or does NNTP convert all bytes in the Extended ASCII range to '?'

Giganews Newsgroups
Subject: Re: Is it me or does NNTP convert all bytes in the Extended ASCII range to '?'
Posted by:  Remy Lebeau \(TeamB\) (no.spam@no.spam.com)
Date: Mon, 19 Jun 2006

"NEws" <…@a.com> wrote in message news:AA4BDAACFAFCE24…@a.com...

> The IOHandler is modifying the raw bytes to ASCII 127 and lower.

The NNTP component does not support Extended ASCII or Unicode characters.

> I have tried modifying the input buffer encoding to utf-8 and ansii but
> both create the same output.

NNTP is an ASCII-based protocol, and UTF-8 is an ASCII-based encoding
scheme.  The NNTP protocol does not support Unicode at all, so it doesn't
matter what you set Indy to read.  The server is always going to send ASCII
data.

A large part of Indy is ASCII-based.  Most of the Internet protocols that
Indy implements (NNTP included) only support 7-bit ASCII characters.  When
Indy receives bytes from a protocol server, the bytes are treated as ASCII
characters, even if they were not transmitted as ASCII (except in cases like
FTP where explicit Unicode support is implemented in the protocol).

In the case of .NET, all .NET strings are Unicode, so they have to be
converted to/from ASCII in order for Indy to use them in most protocols.
Indy uses .NET's own Encoding classes for the actual conversions.  In this
case, the "System.Text.Encoding.ASCII" class is used.  When the lines are
read from the socket, they are converted to Unicode strings for .NET, and
then converted back to ASCII when written to your stream.  The
Encoding.ASCII class only supports 7-bit ASCII characters, so Extended ASCII
characters cannot be handled at all.

The NNTP protocol does not allow 8-bit characters anyway, so your message
data should never be including Extended ASCII characters to begin with.  If
it is, then the sender is not complying with established standards.  Besides
that, sending 8-bit data over a 7-bit protocol should be using MIME instead
of yEnc to encode the data.

Gambit

Replies

In response to

Is it me or does NNTP convert all bytes in the Extended ASCII range to '?' posted by NEws on Mon, 19 Jun 2006