Proposal: `showLitChar` (and `show @Char`) shouldn't escape readable Unicode characters

[ksqsf]

Motivation

Currently, ghci cannot print readable Unicode directly,

ghci> 'λ'
'\955'
ghci> "世界"
"\19990\30028"
ghci> '😂'
'\128514'

According to the documentation,

• Show: "Conversion of values to readable Strings."

  Escaped strings are often not very readable.

• showLitChar: "Convert a character to a string using only printable characters, using Haskell source-language escape conventions. "

  It does not consider Unicode-printable characters at all.

Conclusion 1: There's a bug, either in the code, or in the doc.

You might argue that people are using Show as a lightweight serialization mechanism, and this proposal will cause unexpected breakage for them. However, this breakage is expected only when they are sending serialized data over network, assuming ASCII-compatible encodings on both sides. And there is actually an inconsistency, and your code is probably buggy if you rely on escaping:

ghci> data Γ a = Γ a deriving Show
ghci> Γ 'Γ'
Γ '\915'   -- Uh-oh.

Conclusion 2: If you rely on Show's escaping for encoding-independent serialization today, your code is probably buggy.

Therefore, we have pretty strong reasons to correct the behavior of showLitChar (and in turn show @Char and show @String):

1. It's much friendlier to learners around the globe: you can now use your native language freely in ghci.
2. Fix a longstanding inconsistency issue.
3. Align better with the documentation.
4. String is always a list of Unicode code points. Why escape it in the first place? Encoding and decoding is already handled elsewhere.
5. showLitChar remains unchanged since 1999, a time when GHC had no good Unicode support. They had reasons to escape then, we don't today.

See also previous discussions at gitlab, libraries@, r/haskell, and another proposal.

Proposed Changes

• Relax showLitChar so that readable Unicode characters in Char and String are not escaped.
• Add a newtype wrapper Ascii, and supporting functions to really escape all non-ASCII characters. And escapeAscii guarantees that oldshow x = asciiToString . escapeAscii . newshow $ x where x :: String.

newtype Ascii = MkAscii String

escapeAscii :: String -> Ascii
asciiToString :: Ascii -> String

• GHCi supports only Unicode-compatible terminals after this proposal is implemented.

Breakage and Migration

This is a breaking change with a large impact. All libraries with custom Show instances will probably need a major version bump, as their show now can output different results. However,

1. The new behavior is usually what you and the users really want. Even if the results are different. Maybe you don't need to do anything at all:
   • Existing serialized data will still be supported by Read.
   • New serialized data will still be supported by an existing program compiled with an older version of base.
2. If you rely on the old behavior, consider using asciiToString . escapeAscii to get the old behavior back:

-- Previous
instance Show T where
  show x = _tostring x

-- Now
instance Show T where
  show x = asciiToString . escapeAscii $ _tostring x

Implementation

I will implement this proposal.

update: a PoC has been implemented here.

[Bodigrim]

You might argue that people are using Show as a lightweight serialization mechanism

I don't quite follow: how does the proposal impact this use case? Seems users can continue to use Show for lightweight all-bets-off serialization?

[ksqsf]

I don't quite follow: how does the proposal impact this use case? Seems users can continue to use Show for lightweight all-bets-off serialization?

Yes, I'm not arguing they can't. Here, I'm specifically referring to the case where the program is relying on the escaping, so that transmitting a serialized file over network is "reliable".

• A serializes a value in a file using A's local encoding.
• A sends this file to B, and B deserializes this file using B's local encoding.

This is an expected breakage, and indeed there are concerns around it. However, as the example has shown, even today this behavior is not reliable, since data constructors are not at all escaped. Any program that relies on escaping is already buggy, and should move to a better solution. To smooth the migration, they can use the proposed escapeAscii, which reliably escapes all non-ASCII characters.

I've edited the proposal to clarify it.

[Bodigrim]

Here, I'm specifically referring to the case where the program is relying on the escaping, so that transmitting a serialized file over network is "reliable".

* A serializes a value in a file using A's local encoding.

* A sends this file to B, and B deserializes this file using B's local encoding.

But this is not reliable even now: imagine that A's local encoding is UTF8, but B's local encoding is UTF16. Any usage of Show for serialization requires the same local encoding on both sides; the proposal does not impact it in the slightest.

What about show for String? Is it intentional to leave it escaped, even while changing instance Show Char?

[ksqsf]

But this is not reliable even now: imagine that A's local encoding is UTF8, but B's local encoding is UTF16. Any usage of Show for serialization requires the same local encoding on both sides; the proposal does not impact it in the slightest.

Indeed! I forgot the existence of multibyte encodings...

What about show for String? Is it intentional to leave it escaped, even while changing instance Show Char?

Strings will be unescaped, since it's implemented as Char's showList. I've edited the proposal.

[gwils]

I'm in favour of fixing this. I think a lot of newer Haskellers will expect this behaviour, and this will give a better experience for non-English users (of which there are surely many!).
I think the breakage is likely to be acceptable. Relying on Show/Read for (de)serialisation is something that feels to me like it wouldn't be seriously used by anyone but beginners, and as you've described, it should keep working in most cases except for a rather contrived one.
I'd like to hear counter-arguments here, if anybody has them. Can anyone demonstrate that this breaks anything real?

[chreekat]

Relying on Show/Read for (de)serialisation is something that feels to me like it wouldn't be seriously used by anyone but beginners

Every code base I've ever interacted with has fought a (losing) battle with using Show and Read for this purpose. :) Maybe I'm surrounded by beginners, but even so, this comment feels to me like it's trivializing a real phenomenon.

I'm in favor of this change, so I'm commenting in the spirit of avoiding another Eq-of-no-Neq situation.

[gwils]

this comment feels to me like it's trivializing a real phenomenon

I apologise; I could have worded that more respectfully.

[NorfairKing]

+1 as long as the proposal can specify the desired behaviour in these cases:

1. ANSII codes in String
2. The show instance of Text
3. The show instance of ByteString

EDIT: I think the HPC coverage reports rely on Show/Read. I hope we can check that those are not broken by this change.

[jberryman]

Is there any benefit to this outside of ghci? Is there a way to add another hack to ghci so that we Show in the friendlier way that you propose here?

The downside to a hack is that Show will work differently in "production"... but then you justify the proposal by pointing out (rightly I think) that Show isn't really "suitable for production". And either way the output could be pasted into a file or passed to read.

If that's not possible then my opinion is the potential for breakage is not worth the benefit (though I'm biased as an English speaker).

A good survey of show calls in popular libraries and projects would improve the proposal imo.

[stephen-smith]

ANSII codes in String

Neither 027 nor 0177 characters are "readable Unicode characters" so I don't think ANSI would be meaningfully changed by this proposal.

[ekmett]

So we're proposing to break use of Show/Read and ghci for non-utf8 terminal / codepages? (along with all serialization tests that wrote out data beforehand and are checking golden datasets, and serialization of data with Show from code running on new GHC versions to old?)

If we wanted to make this configurable, there's not really a good way to smuggle knowledge of the surrounding host environment into Show without changing a lot of code, e.g. by adding an argument to showsPrec, diverging further from the standard, or doing something awful like plumbing around a big implicit parameter indicating if we expect to be serializing with unicode, or passing flags to the executable and hoping you don't need to use things with different semantics in different parts of your code. But then if you are going to go that far with settings, adding all sorts of options for indentation and spacing and whether it should use record syntax or not seems to be in order, whereupon it has become a full fledged pretty printing library.

[ekmett]

The main difference with the Show/Read for a unicode-named type mentioned above is it is often have a lot more control over the data types I show than the contents of the strings users put in them.

[ksqsf]

Thanks for the feedback. I will try to respond to them based on my current understanding of situations.

2. The show instance of Text

text maintainers will have to make their own decisions on whether to keep the old behavior. My expectation is that text will eventually be unescaped.

3. The show instance of ByteString

Likewise, it's up to bytestring maintainers. My expectation is that ByteString will still be 8bit-truncated, so its output will not be affected by this proposal.

Is there any benefit to this outside of ghci?

Show is used by a lot of Haskell programs. E.g., IHP uses Show in its admin interface. I agree that perhaps a better solution is to drop Show in favor of text-display.

So we're proposing to break use of Show/Read and ghci for non-utf8 terminal / codepages?

I'm not sure if this is the entire story, but I'd say yes. My understanding is that GHCi and other String I/O functions already handle host encodings correctly, as they always consider the external encoding. It's, however, probable that some valid Unicode characters cannot be encoded using the external encoding. (I'm not super familiar with the GHC encoding mechanism so please correct me if I'm wrong here.)

(along with all serialization tests that wrote out data beforehand and are checking golden datasets,

Unfortunately, character-by-character comparison will be broken, yes.

and serialization of data with Show from code running on new GHC versions to old?)

As long as the data is decoded correctly, this use case will not be affected.

[tomjaguarpaw]

Is there a way to add another hack to ghci so that we Show in the friendlier way that you propose here?

Yes

% cabal repl --build-depends pretty-simple
...
Prelude> :set -interactive-print=Text.Pretty.Simple.pPrint
Prelude> "λ"
"λ"
Prelude> "世界"
"世界"
Prelude> "😂"
"😂"

(It doesn't work with Char yet, only String, but that is an easily-addressed issue.)

I've mentioned this in OP's previous issue reports but he hasn't chosen to list it as an alternative.

• https://gitlab.haskell.org/ghc/ghc/-/issues/20027#note_362926

• Proposal: Add non-escaping methods to Show #25 (comment)