Version 2.3.0

This version wraps the new functionality in version 1.0.27 of the native libdmf library. The following members have been added to the MarketDataFeed class and the IMarketDataFeed interface:

• 5327a64 An IntPtr Extract(out ushort mref, out ulong insref, out uint len) method that can be used to extract an unmanaged pointer to the current message from the feed handle
• 5b75d1e An int Inject(IntPtr ptr, uint len) method that injects a previously extracted message
• 5b75d1e Another int Inject(ReadOnlySpan<byte> data) overload to inject the byte data of a previously extracted message into the handle
• 14a22c8 A read-only ReadBufferSize property that gets the number of bytes waiting to be processed in the internal read buffer after a call to the Consume method
• 94f6522 A read-write ReadBufferMaxSize property that gets and sets the current size of the internal read buffer

Other changes in this release include this:

• 5ee24f5 The following ReadOnlySpan<byte> method overloads of the Message class have been added to the IMessage interface:

  • bool AddNumeric(uint tag, ReadOnlySpan<byte> value);
  • bool AddNumeric(Field tag, ReadOnlySpan<byte> value);
  • bool AddString(uint tag, ReadOnlySpan<byte> value);
  • bool AddString(uint tag, ReadOnlySpan<byte> value, int length);
  • bool AddString(Field tag, ReadOnlySpan<byte> value);
  • bool AddString(Field tag, ReadOnlySpan<byte> value, int length);
  • bool AddDate(uint tag, ReadOnlySpan<byte> value);
  • bool AddDate(Field tag, ReadOnlySpan<byte> value);
  • bool AddTime(uint tag, ReadOnlySpan<byte> value);
  • bool AddTime(Field tag, ReadOnlySpan<byte> value);
  • bool AddList(uint tag, ReadOnlySpan<byte> value);
  • bool AddList(Field tag, ReadOnlySpan<byte> value);

• a959bd2 The following enumerations have been marked with the [ObsoleteAttribute] and will probably be remove in a future version:

  • CompressionLevel
  • CorporateAction
  • Field
  • MessageClasses
  • MessageReference
  • RequestClass
  • RequestType
  • TradeCodes

  Field has been replaced by a static Fields class that contains constants for all fields.
  Similarly MessageReference has been replaced by a static MessageReferences class that contains constants for all message references, RequestClass has been replaced by a RequestClasses class with constants for all request classes and RequestType has been replaced by a RequestTypes class that contains the constants for all request types.

• b758a9c MDF_F_EVENTSTATUS, MDF_F_SMA20, MDF_F_SMA50, MDF_F_SMA200, MDF_F_RSI14,
  MDF_F_MACD, MDF_F_TRADESTATEACTIONS, MDF_F_SHARECLASS have been added to the deprecated Field enumeration and as constants in the new Fields class

• ffec795 The finalizer of the Message class no longer throws a NullReferenceException when you pass an invalid native library path to the constructor overload that accepts a string

Version 2.2.0

• 3d3a82b2 Methods that previously threw an ObjectDisposedException now simply return false if the instance has been disposed. Property getters return the default value and property setters silently fail to set the backing field.

• d7371efc An InvalidOperationException is no longer thrown when a function is missing from the installed version of the wrapped libmdf library. Methods that previously threw in this case now simply return false while property getters return the default value and property setters silently fail (just as with 3d3a82b2).

• 56215e44 The Deserialize and Move methods of the Message class and the Connect and Send methods of the MarketDataFeed class no longer throw an ArgumentNullException. Instead, these methods return false when an invalid (null) argument is supplied.

• f414dfda The AddInt64 and AddUInt64 methods of the Message class no longer throw an ArgumentException when the decimals parameter is not between 0 and 19. Instead, they bypass the check to the wrapped libmdf implementation and return false if the argument is invalid.

• 8bf8a133 The System.Runtime.InteropServices.NativeLibrary API is now internally used to load the wrapped libmdf library on supported (.NET Core 3.0 and later) platforms. This removes the dependency on libdl on Linux platforms.

• d4ae641b MDF_F_CLOSEPRICEDATE, MDF_F_PRIIP00001, MDF_F_PRIIP00002, MDF_F_PRIIP00004, MDF_F_PRIIP00005, MDF_F_PRIIP00006, MDF_F_PRIIP00007, MDF_F_PRIIP00008, MDF_F_PRIIP00009, MDF_F_PRIIP00015, MDF_F_PRIIP00016, MDF_F_PRIIP00017, MDF_F_PRIIP00075, MDF_F_PRIIP00110, MDF_F_PRIIP01095, MDF_F_PRIIP01125, MDF_F_PRIIP02032, MDF_F_PRIIP02035, MDF_F_PRIIP02062, MDF_F_PRIIP02065, MDF_F_PRIIP02092, MDF_F_PRIIP02095, MDF_F_PRIIP02122, MDF_F_PRIIP02125, MDF_F_PRIIP02185, MDF_F_PRIIP02190, MDF_F_PRIIP02200, MDF_F_PRIIP02210, MDF_F_PRIIP02220, MDF_F_PRIIP04086, MDF_F_PRIIP04087, MDF_F_PRIIP04088, MDF_F_PRIIP04089, MDF_F_PRIIP04120, MDF_F_PRIIP04130, MDF_F_PRIIP04140, MDF_F_PRIIP04150, MDF_F_PRIIP04160, MDF_F_PRIIP06005, MDF_F_PRIIP07005, MDF_F_CLOSEPRICE4Y, MDF_F_CLOSEPRICE6, MDF_F_CLOSEPRICE7Y, MDF_F_CLOSEPRICE8Y, MDF_F_CLOSEPRICE9Y, MDF_F_CLOSEYIELD4Y, MDF_F_CLOSEYIELD6, MDF_F_CLOSEYIELD7Y, MDF_F_CLOSEYIELD8Y, MDF_F_CLOSEYIELD9Y, MDF_F_SWINGFACTORSUB, MDF_F_CLEARING, MDF_F_LASTMOD, MDF_F_APPLICABLENAV, MDF_F_FUNDCUTOFFTIME, MDF_F_AVGTURNOVER1W, MDF_F_AVGTURNOVER1M, MDF_F_AVGTURNOVER3M, MDF_F_AVGTURNOVER1Y and MDF_F_AVGTURNOVERYTD have been added to the Field enumeration

• 95e9bf56 d1b22553 The default marshalling for strings is now internally used for the overloads of the AddDate, AddList, AddNumeric, AddTime and Deserialize methods of the Message class that accept string arguments, as well as for the Connect method of the MarketDataFeed class

• b873ab4a The internal implementation of the MarketFeedData and Message classes now use the shared System.Buffers.ArrayPool<T> class instead of stack allocating data to be passed to the wrapped libmdf library

• d50747c0 The System.Memory NuGet package dependency has been updated to version 4.5.5

Version 2.1.0

This version wraps all new functionality in version 1.0.26 of the native data feed API (MDF) while still staying backwards compatible with version 1.0.25.

• 3e0be47 A HandleDelay property that lets you enable delay-mode on the connection has been added to the generic MarketDataFeed class

• 0b6de2d A Delay property that returns the intended delay of the current received message if delay-mode is enabled on the connection has been added to the generic MarketDataFeed class

• 3371e67 A MessageClass property that returns the message class of the current received message has been added to the generic MarketDataFeed class

• 0f0f4aa A Timeout property that returns the number of seconds you have to wait in maximum before having to call the Consume method to let it handle internal idle timers has been added to the generic MarketDataFeed class

• ddd3e67 A Delay property that gets or sets the intended delay of the message has been added to the Message class

• 6894e25 A FieldCount property that returns the number of added fields to the current message has been added to the Message class

• 04a2736 A new message type MDF_M_QUOTEEX has been added to the MessageReference enumeration,
  a new MDF_MC_QUOTEEX message class has been added to the MessageClasses enumeration and a new MDF_RC_QUOTEEX request class has been added to the RequestClass enumeration

• 19a0e1e New overloads of the GetNextMessage method without the mclass parameter - this one lets you retrieve the message class of the fetched message - have been added to the generic MarketDataFeed class. With the addition of the new MDF_MC_QUOTEEX message class, the value may no longer fit in an int and can instead be fetched using the new MessageClass property that returns a ulong value. The old overloads of the GetNextMessage method are still available for backward compatibility reasons.

• 9d0569b MDF_F_COMBOLEGS, MDF_F_SWINGFACTOR, MDF_F_SWINGMETHOD, MDF_F_INCOMEPROPMAN, MDF_F_CLOSEPRICE1DDATE and MDF_F_CLOSEPRICEDATE have been added to the Field enumeration

• 13fffc3 An InvalidOperationException is now explicitly thrown from the CompressionLevel, Delay, FieldCount and Utf8Validation properties of the Messageclass and the Delay and MessageClass properties of the generic MarketDataFeed class when a function that is required to retrieve or set a value is missing from the current installed version of the native library.

  An exception is also thrown from the AddDate, AddInt64, AddString, AddTime2, AddTime3, AddUInt64, Deserialize, Move and Serialize methods of the Message class if the corresponding native function is missing from the installed version of the native library.

• 2c178b2 The generic IMarketDataFeed interface has been extended to include all public members of the generic MarketDataFeed class

Version 2.0.1

• d24978c A fix to avoid trying to load the native library from the default path when a custom path is supplied
• 83fdc34 Added MDF_F_S12, MDF_F_S13, MDF_F_S14, MDF_F_S15, MDF_F_N6,
  MDF_F_N7, MDF_F_N8, MDF_F_N9, MDF_F_N10, MDF_F_I6, MDF_F_I7,
  MDF_F_I8, MDF_F_I9, MDF_F_I10, MDF_F_INSREF1, MDF_F_INSREF2,
  MDF_F_INSREF3, MDF_F_INSREF4 and MDF_F_INSREF5 to the Field enumeration

Version 2.0.0

This version is backwards incompatible (breaking) but adds support for:

• Requesting data by creating and sending messages just like you do using the wrapped libmdf C/C++ API
• Sending data to the system (assuming you have sufficient permissions)
• Creating messages using a new Message class which is a managed implementation of the mdf_message_t handle that can contain several messages
• Adding any numeric, integer, string, date, time and list fields that are supported by the native API to a message
• Consuming data and fetching messages and fields "manually" on the same thread using a new MarketDataFeed class which is a managed implementation of the native mdf_t API handle and a replacement for the removed DataFeed class
• Registering data and status callbacks with strongly-typed custom and settable user data
• Sending of multiple messages
• Resetting and reusing of messages
• Removal of a message from a message handle
• Moving of messages between message handles
• Serialization and deserialization of messages
• Setting the zlib compression level and disabling UTF-8 validation
• Setting the current API error code and the number of sent and received bytes between the client and the server
• Getting the file descriptor used by the connection and the time difference between the client and the server in nanoseconds
• Setting a numerical bind address to which the API will bind before attempting to connect
• Enabling or disabling encryption
• Limiting the supported message digests used to authenticate messages between the server and the client
• Limiting the supported ciphers used to encrypt messages between the server and the client
• Two-factor authentication using an extra credential
• Specifying a custom path to the native dependency
• A bit field (flags) MessageClasses enumeration (enum) that defines all message classes

The latest connections statuses, fields/tags, message references and request classes in libmdf 1.0.25 have been added to the existing enums. For each method that accepts an enum value, an overload that accepts the underlying value (int or uint) has been added to make the API forward compatible and being able to handle new fields/tags, messages, and request classes without updating.

string constants for all message classes and request types have also been defined to make them easier to use in Message.AddNumeric (which is the managed counterpart of mdf_message_add_numeric), e.g.:

message.AddNumeric(Field.MDF_F_REQUESTTYPE, StringConstants.RequestTypes.MDF_RT_FULL);

• Besides the DataFeed class, the RequestMessage, ResponseMessage, SubscribeMessage, UnsubscribeMessage, ConnnectionStatusChangedEventHandler and ConnectionStatusChangedEventArgs types have also been removed (breaking)

DataTypes Version 1.1.0

• Number now implements IFormattable and supports the common standard numeric format strings
• Leading and trailing zeros have been removed from the output of a parsed Number
• Improved ReadOnlySpan<byte> parsing performance for most types

Version 1.1.2

• Changed the type of the InstrumentReference property of the ResponseMessage class from UInt32 (uint) to UInt64 (ulong)
• Added support for issuing a single request for all request classes, aka "wildcard" requests
• Added new fields
• Upgraded to the latest stable versions of the System.Buffers and System.Memory dependencies
• Installed the System.Collections.Immutable package to be able to use the ImmutableHashSet<T> type internally
• Decreased the soft limit of instrument references to the correct value of 1,000,000.

DataTypes Version 1.0

v1.0.0-datatypes

Added implementations of the Millistream data types.

Version 1.1.1

Added macOS support and some new fields and message references.

Version 1.1.0

DataFeed.cs

• The DataReceived event has been removed and replaced by an IObservable<ResponseMessage> property called Data. (breaking)
• A Recycle(ResponseMessage) method has been added. It enables you to reuse instances of ResponseMessage using an internal object pool instead of allocating new ones for each message received from the feed.

ResponseMessage.cs

• The Fields property now returns an IReadOnlyDictionary<Field, ReadOnlyMemory<byte>> instead of an IReadOnlyDictionary<Field, string>. No strings are allocated by default. (breaking)

In version 1.0.* you subscribed to the DataReceived event and received response messages containing fields with UTF-8 string values like this:

dataFeed.DataReceived += (s, e) =>
{
    Console.WriteLine($"{DateTime.Now.ToShortTimeString()} - " +
        $"Received a {e.Message.MessageReference} message with the following fields:");
    foreach (var field in e.Message.Fields)
        Console.WriteLine($"{field.Key}: {field.Value}");
};

The problem with this approach is that it creates a lot of garbage. For each message received from the data feed, there was an instance of the ResponseMessage class created as well as an instance of the DataReceivedEventArgs class.

In version 1.1.0, the DataReceived event has been replaced by an IObservable<ResponseMessage> property. This removes the need for the DataReceivedEventArgs class completely. You now subscribe to data by calling the Subscribe method of the Data property and passing in an instance of a class that implements the IObserver<ResponseMessage> interface, e.g.:

dataFeed.Data.Subscribe(new Observer(dataFeed));
...
class Observer : IObserver<ResponseMessage>
{
    private readonly IDataFeed _dataFeed;

    public Observer(IDataFeed dataFeed) => _dataFeed = dataFeed;

    public void OnNext(ResponseMessage message)
    {
        Console.WriteLine($"{DateTime.Now.ToShortTimeString()} - " +
            $"Received a {message.MessageReference} message with the following fields:");
        foreach (var field in message.Fields)
            Console.WriteLine($"{field.Key}: {Encoding.UTF8.GetString(field.Value.Span)}");
        _dataFeed?.Recycle(message);
    }

    public void OnCompleted() { }

    public void OnError(Exception exception) => Console.WriteLine(exception.Message);
}

If you are using the Reactive Extensions (Rx.NET) library, you don't have to explicitly implement IObserver<T> as there is an extension Subscribe method that accepts an Action<T>:

dataFeed.Data.Subscribe(responseMessage => 
{
    Console.WriteLine($"{DateTime.Now.ToShortTimeString()} - " + 
        $"Received a {message.MessageReference} message with the following fields:");
    foreach (var field in message.Fields)
        Console.WriteLine($"{field.Key}: {Encoding.UTF8.GetString(field.Value.Span)}");
    dataFeed.Recycle(message);
});

Millistream.Streaming has no dependency on Rx.NET though. The IObservable<T> and IObserver<T> interfaces are included in both .NET Standard 1.2 and .NET Framework 4.5.

Besides this breaking change, a Recycle(ResponseMessage) method has been added to the DataFeed class. It resets and recycles an instance of the ResponseMessage class. This means that you now only need to allocate a single instance of ResponseMessage assuming that you subscribe to and handle all response messages in a single thread. You simply call the Recycle method once you have processed the response message as shown above.

The DataFeed class internally uses a generic ObjectPool<T> class to keep track of the recycled objects. The implementation is based on the implementation from the .NET compiler platform Roslyn and uses a simple array to store the recycled object(s).

The other breaking change in this version is that the type of the Fields property of the ResponseMessage class has changed from IReadOnlyDictionary<Field, string> to IReadOnlyDictionary<Field, ReadOnlyMemory<byte>>. In the previous version a string was allocated for each field of each message regardless of whether the consumer was actually interested in all values of all fields. In the new version, no strings are allocated by default. Instead the DataFeed class adds all bytes that it reads from the feed for a message to an internal ExtendableArray<byte> property of the ResponseMessage class. It then uses the ReadOnlyMemory<byte> class that was introduced in .NET Core 2.1 (and is available as a separate System.Memory NuGet package in previous versions) to assign each field a portion of the underlying array without having to allocate a new byte array (byte[]) for each field.

ExtendableArray<byte> is an internal type that wraps a byte array (byte[]) and enables you to dynamically add items to it. It resizes the underlying array just like the built-in List<T> class does, but unlike List<T> it also has an Items property that returns the underlying array of items without creating a copy of it. This array is passed to the constructor of the ReadOnlyMemory<byte> that represents the value of a field along with the offset and length.

To convert a field value to a string in the version, you can use the Encoding.UTF8.GetString method in the consuming application. In .NET Core 2.1 there is an optimized overload that accepts a ReadOnlySpan<byte>. You can use the Span property of the ReadOnlyMemory<byte> class to get one, e.g.:

foreach (KeyValuePair<Field, ReadOnlySpan<byte>> field in message.Fields)
    Console.WriteLine($"{field.Key}: {Encoding.UTF8.GetString(field.Value.Span)}");

If you are interested in numeric values, the System.Buffers.Text.Utf8Parser class has several overloads of the TryParse method that you can use to convert a ReadOnlySpan<byte> to any primitve data type without allocating a string first.

Apart from the public API changes, there has also been some other performance improvements introduced in the DataFeed class under the hood. For example, it now uses the ReadOnlySpan<byte> class to read bytes directly from native memory without allocating managed arrays and copying bytes. Also, the slow calls to Enum.IsDefined have been replaced by faster calls to the Contains method of two cached hash sets that determine whether a field tag received from the feed is a valid MessageReference or Field enumeration value.