u8 literals should support literal concatenation

[stephentoub]

Version Used:
32a81dd

Steps to Reproduce:

using System;
public class C {
    public ReadOnlySpan<byte> M() => "abc"u8 + "def"u8;
}

Expected Behavior:
Produces a result equivalent to:

using System;
public class C {
    public ReadOnlySpan<byte> M() => "abcdef"u8;
}

This is valuable for very long UTF-8 literals such that you want to split them across lines.

Actual Behavior:
Fails to compile.

Relates to test plan #58848

[danmoseley]

Here's a 500 line motivating example: https://github.com/dotnet/runtime/blob/1313a689fd5e1d282f8910a4c77fbe15315d6d98/src/libraries/System.Private.CoreLib/src/System/Globalization/IcuLocaleData.cs#L27-L606

[jcouv]

Request seems sensible and useful, but I don't remember that we discussed it in LDM. Assigned to @AlekseyTs to drive discussion/resolution.

[AlekseyTs]

@stephentoub, in order to expedite this and make chances of this happening feasible, I suggest you to put a concrete language proposal together. I.e. what specific language rules you propose to add/change around + operator, etc. taking into account the latest LDM decisions around the feature.

[am11]

At first glance, inability to concatenate u8 literals appeared as a bug in implementation rather than missing language design; given how it is already possible to write it without the u8 suffix:

public ReadOnlySpan<byte> M() => "abc" + "def";
// without u8 suffix compiler still implicitly converts to UTF-8 bytes,
// instead of of Unicode / UTF-16 bytes (which I personally found unintuitive).

[AlekseyTs]

At first glance, inability to concatenate u8 literals appeared as a bug in implementation rather than missing language design; given how it is already possible to write it without the u8 suffix

That was not a bug, UTF-8 conversion of string constants and u8 suffix were two independent features, each with their own specified behavior. BTW, based on latest LDM decisions (https://github.com/dotnet/csharplang/blob/main/meetings/2022/LDM-2022-04-18.md#issues-with-utf8-string-literals), we are going to remove UTF-8 conversion of string constants.

[stephentoub]

I suggest you to put a concrete language proposal together. I.e. what specific language rules you propose to add/change around + operator, etc. taking into account the latest LDM decisions around the feature.

I'm proposing the language behave as if there's a:

u8literal operator +(u8literal x, u8literal y);

operator that produces a new literal that in turn yields the concatenation of all the bytes from x with all the bytes from y. So you could write:

ReadOnlySpan<byte> s = "abc"u8 + "def"u8;

and that would be compiled identically to:

ReadOnlySpan<byte> s = "abcdef"u8;

Similarly,

ReadOnlySpan<byte> s = "abc"u8 + "def"u8 + "ghi"u8;

would be compiled identically to:

ReadOnlySpan<byte> s = "abcdefghi"u8;

as would:

ReadOnlySpan<byte> s = ("abc"u8 + "def"u8) + "ghi"u8;

and

ReadOnlySpan<byte> s = "abc"u8 + ("def"u8 + "ghi"u8);

This would only apply to literals and not to the natural type ReadOnlySpan<byte> form, so:

ReadOnlySpan<byte> s = "abc"u8 + (ReadOnlySpan<byte>)"def"u8; // error

would be an error, as would:

ReadOnlySpan<byte> s1 = ...;
ReadOnlySpan<byte> s2 = "abc"u8 + s1; // error

[AlekseyTs]

@stephentoub

Just to confirm, this proposal requires each literal to be successfully convertible to UTF-8 byte representation individually. Correct?
So the code below, won't compile:

ReadOnlySpan<byte> span = "\uD83D"u8 +  // high surrogate
                                                 "\uDE00"u8;     // low surrogate;

Even though this code will:

ReadOnlySpan<byte> span = "\uD83D\uDE00"u8; 

[stephentoub]

this proposal requires each literal to be successfully convertible to UTF-8 byte representation individually. Correct?

Yes. Otherwise a standalone "\uD83D"u8 is an invalid literal but concatenating it with an entirely separate literal somehow makes it valid, which feels weird to me. The downside is that you can't split a u8 literal at arbitrary points, only at valid boundaries, but I think that's the lesser evil. (There's an argument to be made that "\uD83D" is a valid string literal, so why not a u8 literal, but the compiler also isn't trying to interpret the contents of the string literal, whereas it is with u8 literals. Given that, I think every u8 literal needs to be able to stand on its own.)

@GrabYourPitchforks, thoughts?

[GrabYourPitchforks]

I'm totally fine with the compiler not allowing splits like this. If you're writing a u8 string literal, it's really weird (IMO) for people to use "\uXXXX\uYYYY" syntax anyway to represent supplementary code points. They should either use "\U00ZZZZZZ" syntax or just have the actual raw characters directly in source, which would prevent splits like this in the first place.

[AlekseyTs]

Implemented in #62044.