[Proposal] Allow explicit field endianness

[ShadedBlink]

We can make C# much better for interop with binary formats by specifying exact endianness of the field like such:

public lend int Field1;
public bend int Field2;
public void Method(ref lend int arg1, ref bend int arg2) {
    lend int var1 = 123;
    bend int var2 = 123;
}

Binary formats always require us to use strict endianess to operate with fields, thus it would be very nice to have such attribute to specify that field is supposed to be strictly in such endianness. This should work exactly same as volatile keyword, i.e. all read/write operations with such field should emit a new opcode like OpCodes.LEnd or OpCodes.BEnd. For the backwards compatibility we can just compile old languages with explicit batches of opcodes to perform such operations.

Example

lend short x = 1;
x += 1;

New .Net

There is just IL code. JIT will just emit corresponding machine code inline without any need for custom methods.

lend. // Modifier for the following opcode, works exactly like volatile.
ldloc.0 // Reading will be performed as little-endian.
ldc.i4.1
add
lend. // Modifier for the following opcode.
stloc.0 // Writing will be performed as little-endian

netstandard2.1

short x = 1;
BinaryPrimitives.WriteInt32LittleEndian(new Span<int>(&x, 1), BinaryPrimitives.ReadInt32LittleEndian(new Span<int>(&x, 1)) + 1);

.Net prior netstandard2.1.

Of course, it looks bulk and heavy, but I think there is definitely a way to implement it much better. Not as good as for new .Net, but still much better than the way I wrote it. For example, compiler will look for a public/internal class with fixed name like BinaryPrimitives that may be either internally implemented or attached as nuget package. Then compiler will just take corresponding public static Read/Write() method from this class and use it. It already works same way for init, required and other attributes on older frameworks.

short x = 1;
if (BitConverter.IsLittleEndian) {
    x += 1;
} else {
    var y = (short)(((x & 255) << 8) & (x >> 8)); // No real variable created, just a value in the stack for clean text.
    y += 1;
    x = (short)(((y & 255) << 8) & (y >> 8));
}

Conversions

For the questions about conversions between ref lend, ref bend and ref - we should allow such conversions, they just have to be explicit. For example, ref lend T Unsafe.AsRefLittleEndian(ref T x). Meanwhile value conversion should be implicit since this keyword doesn't affect value in any way, it affects only read/write operations.

Operations

This endianness proposal involves only READ/WRITE operations, i.e. only process of reading or storing value from memory. It SHOULD NOT affect any other operation, like <<. >>, ^, & and etc. All of them are performed on READ values, i.e. AFTER conversion from our specified endianness to system native conversion. Thus all those operations work as is, lend/bend modifiers are not applicable to them.

lend int x = 0x00000010; // Value 16 is stored in memory as `0x10000000`
Console.WriteLine(x + 1); // 17
bend int x = 0x00000010; // Value 16 is stored in memory as `0x00000010`
Console.WriteLine(x + 1); // 17

[ShadedBlink]

Also we can expand custom types with such interface:

interface ISupportsEndianness<T> {
    T AsLittleEndian { get; set; }
    T AsBigEndian { get; set; }
}

It should be especially applicable to System.Guid due to it's conflicts with global Uuid since Guid is little-endian and Uuid is big-endian. In such case we can just mark Uuid cases as bend Guid.
So this case

bend Guid uuid = Guid.NewGuid();
return uuid.ToString();

Is translated to

Guid uuid;
uuid.ValueAsBigEndian = Guid.NewGuid();
return uuid.ValueAsBigEndian.ToString();

[colejohnson66]

This, IMO, is a really interesting idea. But it’s so niche I can’t see the LDM entertaining it. The vast majority of the time you’re doing interop, everything is in native endianness. Only when serializing to the wire does endianness tend to matter, and that can already be accomplished through a custom marshaller.

Obligatory, “source generators can help here” sentence.

[ShadedBlink]

I am working with SCSI, VHD(X) and WinAPI, thus I have to store unmanaged or format-specific data structures in memory and work with them. Serializer case requires me to add another memory block that will represent result of format->native conversion, which I think is excessive, since I have to read data block, then parse it to native, work on it, then parse it back to format and finally write back. Also it is makes another problem that I have to either store both views in memory(format and native, which I also have to always update simultaneously, i.e. x2 code) or always create a new formatted instance from native when I want to save data back. With this proposal I would be able to just read, perform work and write it back as is. I know today is time of JSON and other fine-looking formats, but binary formats are still pretty popular, especially in high-performance area.
Source generators can provide us with properties and methods, but still they are unapplicable for raw field access.

[HaloFour]

Conversion as close to the point of interop feels more appropriate. In terms of overhead, having the compiler or runtime convert back and forth anytime you want to access and work with the underlying values is going to add overhead on every access anyway.

[ShadedBlink]

I believe that access overhead is more applicable than code complexity or memory overhead. x86 has a single bswap instruction to perform endianness swap, so performance overhead is pretty close to unnoticeable. And we still can just store such values in variables in case of multiple consecutive operations that will have native endianness. I believe such cases are corner cases and very unlikely to happen.

[ShadedBlink]

Also JIT already knows how to optimize code, so impossible branches depending on BitConverter.IsLittleEndian or etc in BinaryPrimitives are gonna be optimized away anyway. So reading little endian fields on little endian machines guaranteed to not have any overhead.

[HaloFour]

This reads much more like a runtime proposal than a C# proposal since the suggestion is to modify the IL opcodes necessary to read/write these fields.

IMO, baking this into the runtime+language isn't necessary, you could accomplish this just as easily by defining properties where the accessor methods handle the translation.

[FredyFerrari]

IMO this does not make sense, the data in the own code should be normalized and only input/output should honor the correct endianness. Otherwise you open a rabbit hole for Debugging and all other kind of issues.

[ShadedBlink]

Actually language part is necessary there, while runtime part is optional. netstandard2.1 approach for compiler is already very simple, versatile and applicable, we can even leave new opcode part and live with just this compiler enhancement. I just believe that opcode variant would be more performant since JIT would just insert necessary instructions directly in method. The main point in this proposal is actually to leave endianness conversion to compiler instead of making tons of BinaryPrimitives.ReverseEndianness() and etc.
At the moment I already use properties as the point of conversion, but this just makes code more complicated and forbids me from getting fields addresses, leaving me with the only way of specifying constant offsets for structure.

[ShadedBlink]

@FredyFerrari , I believe that endianness of the field won't affect debugging experience, since debugger will read values according to the field endianness. Thus you are gonna read values the way the are meant to be read.

[FredyFerrari]

But this sounds like a major issue for a very niche problem. There is probably already a small percentage of C# programmers who know what the endianness is and even a smaller amount who need different endianess support by the compiler itself.
Just my opinion. For hardware access or endian dependent code I still use C++ most of the time.

[ShadedBlink]

Yes, it is pretty niche, but still binary formats are very popular in high-performance areas. Json just can't reach heights of bson or protobuf. Of course, random developer most of the time can just use those tools and avoid diving deeper in internals, but still we can make life much better for those binary developers. All of them are pretty niche, but together they represent a pretty big community that have to provide .net community with implementations to support different formats. It will make huge impact to make C# more binary-friendly.

[Korporal]

I suspect there's no use case for accessing a managed datum that needs to worry about endianess.

The only use case I see is where code needs to access unmanaged memory using a different endianness to the current environment, and it might be useful to then be able to declare a pointer to have a specific endianess.

[FredyFerrari]

There are indeed use cases and I had need to write more than one program where I had to do this in my career. Like reading/writing binary data from other systems, writing emulators, drivers, etc.

But then I always normalized the data during read/write.

[Korporal]

There are indeed use cases and I had need to write more than one program where I had to do this in my career. Like reading/writing binary data from other systems, writing emulators, drivers, etc.

But then I always normalized the data during read/write.

Right, that's why I referred to managed data. We'd never have a use case where we'd need to declare managed variables along with their endianess. I'm alluding to the fact that the scenarios would all be with interop or external data or mapped memory (which is unmanaged).

I suppose one could consider an attribute or a new constructor for the MarshalAs attribute on structs. One could also consider adding more overloads to the BitConverter static class too like:

public static long ToInt64 (byte[] value, int startIndex, Endian endianess);

Where Endian is a new enum or something...there are several possible types too like Big, Little, Bit and Middle and maybe others.

[colejohnson66]

A program should never have to deal with bitwise endianness. That should be a property of the hardware.

[ShadedBlink]

tldr: strict endianness is necessary for IO, not for calculations.

@colejohnson66 , if it will be only a property of hardware, you would never be able to pass binary files between devices - one device writes file in big-endian, while other attempts to read it as little-endian. For example, VHD defines all content in big-endian, while VHDX uses little-endian. Your PC will be able to use either VHD or VHDX, but not both of them. In such case system endianness would be a mandatory parameter of the system along with architecture and supported instructions, but you never see it noted in device specs, just because your apps always have to solve it for you.

[huoyaoyuan]

See dotnet/runtime#95207