Open API

Introduction

This document provides an introduction to OpenAPI Streaming.

OpenAPI Streaming provides real time updates on quotes, positions, orders, balances etc. to client application without having to poll the OpenAPI at high frequencies. Streaming is feature where server push the content to connected clients instantly as it becomes available, so results in less network traffic and lower latency.  

Below steps are required for a client application to use OpenAPI Streaming:

  1. Get a valid OpenAPI access token.
  2. Establish connection to the streaming server.
  3. Setup one or more subscriptions, to specify what you want to subscribe to.
  4. OpenAPI Streaming server sends streaming updates to the client application.
  5. Server send heartbeats to client in case no data is being transmitted for a certain period of time.

Examples and Code Snippets in this article are from attached JavaScript sample.

 

Get OpenAPI Access Token

All OpenAPI end points require client application to pass a valid OpenAPI access token. Client application can get access token by interacting with our security system. Depending on client application type, there are different authentication flows details of which can be found here OpenAPI Security.

Streaming sample applications do not use any authentication flow, as described in OpenAPI Security, instead they require access token to be placed in configuration files. So please refer to developer's portal to get valid access token for sample applications.

Establish Connection with Streaming Server

The OpenAPI Streaming feature depends on Microsoft's SignalR libraries, so a client application should have a reference to the latest stable version of Microsoft.AspNet.SignalR libraries depending on the client application type i.e. windows or web client.

Once we have dependencies in place , next steps are 

  • Create connection proxy.
  • Register event handlers with connection proxy. This refers to handling of wide range of connection lifetime events in OpenAPIs .
  • Start Connection to connect with streaming server.

In Web based JavaScript application, a client can create the connection proxy by using $.connection method of SignalR.JS library as shown in code snippet below:

Make Connection
contextId = encodeURIComponent("C" + Date.now());
        
var queryString = "authorization=" + encodeURIComponent(accessToken) + "&context=" + contextId;
 
console.info("Initializing  Streaming Connection, Url: " + streamingConnectionUrl);
        
//Create connection proxy.
_connection = $.connection(streamingConnectionUrl, queryString, true);
        
//Register eventHandlers with connection proxy.
_connection.stateChanged(onStateChanged.bind(this));
_connection.received(onMessagesReceived.bind(this));
        
//Connect to streaming server.
_connection.start();

 

Read the section OpenAPI Stickiness and Cookies if you are using a non JavaScript client to setup the connection.

 Parameters explanation

  • streamingConnectionUrl - The URL of the streaming server. For more information , please refer OpenAPI Environments
  • QueryString -  Client have to send data to the server while making connection. Client must send followings 2 information's in query string parameter: 
    • contextId : The streaming context id that this request is associated with.
      Within a user session the context id binds subscriptions to streaming connections. That makes it possible for a single user session to have several streaming connections open each with a distinct set of subscription, e.g. when opening multiple tabs in a web browser. The context id is client generated, making it possible to set up subscriptions and a streaming connection in parallel, and can be up to 50 characters long (a-z, A-Z, 0-9, dash, and underscore).
       
    • authorization : Valid OpenAPI access token, you got in Step 1. Finally query string becomes like this -


  • Logging - it is an optional parameter, with defaults to false. SignalR has a built-in logging function that you can enable on the client for troubleshooting. For example if you enable logging, logs tell you which of the following transport methods SignalR is using like this :

Setup Subscriptions to specify what you want to subscribe to

After making a connection, client application can set up subscription(s) with resources which support streaming and later can unsubscribe with resources to stop streaming or real time updates.

Each streaming resource supports following endpoints. You should call these endpoints as per your requirement.

  • Subscribe              --> /openapi/{Service}/{Version}/{Some-Resource}/subscriptions/
  • Unsubscribe (Single)   --> /openapi/{Service}/{Version}/{Some-Resource}/subscriptions/{contextId}/{referenceId}
  • Unsubscribe (Multiple) --> /openapi/{Service}/{Version}/{Some-Resource}/subscriptions/{contextId}

Subscribe:  /openapi/{Service}/{Version}/{Some-Resource}/subscriptions/

Create endpoint is used to set up a subscription with a resource. Client must send POST request on this endpoint with required parameters.

Error rendering macro 'code': Invalid value specified for parameter 'lang'
POST /openapi/trade/v1/infoprices/subscriptions/

Request parameters of create endpoint

In request parameter, a serialized object whose parameters are as shown below is sent in the body of the request. 

NameTypeOriginDescription
ArgumentsConcrete TypeBody

Arguments for the subscription request. This is a concrete type and varies for each resource. For example for Balances, it is BalanceRequest.

For more information about concrete type of a resource please look at the Reference Documentation

ContextId
required
StringBody

The streaming context id this request is associated with. It must be same that was send in query string while making streaming connection with the server. 

This context id is used to route messages from a subscription to a specific streaming connection.

FormatStringBodyOptional Media type (RFC 2046) of the serialized data updates that are streamed to the client. Streaming currently only supports application/json.
If an unrecognized format is specified, the subscription end point will return HTTP status code 400 - Bad format.
ReferenceId
required
StringBody

Mandatory client specified reference id for the subscription. It should be unique for each subscription. it identifies the subscription (within the context of a specific service/subscription type.

This parameter must only contain alphanumeric characters as well as - (dash) and _ (underscore). It is case insensitive. Max length is 50 characters.

RefreshRateIntBodyOptional custom refresh rate, measured in milliseconds, between each data update. Note that it is not possible to get a refresh rate lower than the rate specified in the customer service level agreement (SLA).
TagStringBodyOptional client specified tag used for grouping. you can group your subscriptions by tagging them with a common tag. This helps you in deleting multiple subscriptions at once.


Below is sample request with bare minimal required parameters only. For more information about concrete type of a resource please look at the Reference Documentation

Subscription Request
Post /openapi/trade/v1/infoprices/subscriptions/
{
  "Arguments": {
    "AssetType": "FxSpot",
    "Uics": "21",
  },
  "ContextId": "29931122",
  "Format": "application/json",
  "ReferenceId": "0f8fad5b-d9cb-469f-a165-70867728950e",
  "RefreshRate": 5,
  "Tag": "2345223"
} 

Response

As soon as you call this endpoint on the resource, it makes a subscription with the streaming server and returns a data snapshot. The data snapshot is returned as the response of the endpoint. Immediately there after the streaming server starts sending  data to the client on connection made earlier and client receives it in the connection.received event handler.

Please note that the data snapshot is returned as the response of the Create endpoint while streaming data is pushed on the client connection by the steaming server. Refer point 3 and 4 in  "OpenAPI Streaming flow" image , at the beginning to this documentation. 


Structure of data snapshot

NameTypeDescription
ContextIdStringThe streaming context id that this response is associated with.
FormatStringThe media type (RFC 2046), of the serialized data updates that are streamed to the client.
InactivityTimeoutIntThe time (in seconds) that the client should accept the subscription to be inactive before considering it invalid.
ReferenceIdStringThe reference id that (along with streaming context id and session id) identifies the subscription (within the context of a specific service/subscription type)
RefreshRateIntActual refresh rate assigned to the subscription according to the customers SLA.
SnapshotData[]

Snapshot of the current data on hand, when subscription was created. This is an array of  concrete type and varies for each resource. For example for Balances, it is BalanceResponse

For more information about concrete type of a specific resource please look at Reference Documentation

StateSubscriptionStateThe current state of the subscription.
TagStringClient specified tag assigned to the subscription, if specified in the request

Unsubscribe (Single): /openapi/{Service}/{Version}/{Some-Resource}/subscriptions/{contextId}/{referenceId}

The Unsubscribe(Single ) endpoint is used to delete a subscription from current session identified by the reference id.

Example structure of DELETE Request to Unsubscribe (Single):

Error rendering macro 'code': Invalid value specified for parameter 'lang'
Delete /openapi/trade/v1/infoprices/subscriptions/{contextId}/{referenceId}

Request parameters of remove (single) endpoint

NameTypeTypeOriginOriginDescriptionDescrDescription
contextId                                                           requiredStringRouteThe context id part of the streaming session (used to identify the subscription within a streaming session).
referenceId                                                        requiredStringRouteUnique ID that identifies the subscription

Unsubscribe (Multiple): /openapi/{Service}/{Version}/{Some-Resource}/subscriptions/{contextId}

Unsubscribe(Multiple) endpoint is used to delete all subscriptions and free all resources on the server.

Example structure of DELETE Request to Unsubscribe (Multiple):

Error rendering macro 'code': Invalid value specified for parameter 'lang'
Delete /openapi/trade/v1/infoprices/subscriptions/{contextId}

Ensure stickiness

Read the section OpenAPI Stickiness and Cookies to ensure that your client is sticky against the API (important when dealing with subscriptions).

Streaming Updates

All streaming messages are sent as incremental updates to the initial snapshot. The client must therefore maintain a representation of the snapshot and when an update is received apply that update to the snapshot.

Structure of streaming updates 

NameTypeDescription
ReferenceIdStringClient specified reference ID assigned to the subscription, if specified in the request.
__pnint The partition number of the current update (only present if the update is partitioned)
__pc
intThe total number of partitions in the message, which the current update is a part of (only present if the update is partitioned)
TimestampUtcDateTimeTime when this message was generated
DataData []

The resource information being sent. Remember, most of the time it is only the delta which will be sent as mentioned above.


When an update is large, it may be split into a number of partitions each sent as an incremental update instruction. For example, if an update contains updates to 100 elements in a list, it may be sent as two separate updates each containing 50 list elements. Such updates will typically be sent immediately after each other but there may be other messages in between. In order for a client to determine whether an incremental update is a partition, it should inspect the __pn and __pc properties. __pc indicates the total number of partitions in the message and __pn indicates the number of the partition, starting from 0. When __pn=__pc-1 the last partition has been received.

Client may receive streaming messages before the subscription response. In that case, streaming messages must be queued up and handled when the snapshot arrives. 

Streaming samples has given idea of it but we totally leave it to client application developers how to handle it.

For example in JavaScript sample:

/***************************************************************************
* Concept note: Delta’s can come before snapshot, Please handle accordingly.
* This is just an example of handling.
**************************************************************************/
snapshot = data;
snapshot.LastUpdatedTimestamp = Math.floor((new Date()).getTime() / 1000);
snapShotReceived = true;
onMessagesReceived(data, false);


Once again just for the reminder, here is where you will be receiving the streaming updates:

connection.received
connection.received(function (data) {
       //This method receives streaming updates from server.
        onMessagesReceived(data,true);
   });

It is important to understand that messages over the streaming channel includes both data and control messages, and the client application must be able to handle both types of messages correctly.

Handling streaming data

How interpretation of the streaming data should be done and what is the structure of the data can be determined by looking at the ReferenceId property of the data. Following table informs us about the various kind of instructions we can get from the server:

ReferenceId
Instruction type
"_heartbeat"Subscription heartbeat indicating that a subscription is still alive event though no data has been sent on it for a while
"_resetsubscriptions" Reset some or all subscriptions due to an error
"_disconnect" The server has disconnected the subscription and no more messages will be sent. The client should close the subscripition after this, and ask the user to authenticate again before sending new requests. 
Refers to a known subscription

Apply incremental update to the snapshot of the subscription with the given reference id


Is none of the above Ignore the instruction 


The following subsections describe how the Data property should be interpreted depending on the data type.

Object Updates

Object updates contain only the properties whose values have changed. If a property is of a structured type (object or list), the property value is an also update and must be applied as such. Otherwise, the value in the snapshot can be replaced directly. In the following example, the object has a simple Age value and a structured Address property where the Street field is changed.

The initial snapshot:

{
   "Name": "Mister Green",
   "Age": 42,
   "Address": {
     "Street": "Green Boulevard",
     "City": "Green Town"
   }
}


The incremental update that is sent to the client to reflect a new Age and Street value:

{
   "Age": 43,                   // Simple: Value in snapshot must be replaced
   "Address": {                 // Structured: Update must be applied recursively
     "Street": "Red Boulevard"  // Simple: Value in snapshot must be replaced
   }
}


The snapshot after updating the Age and the Street fields:

{
   "Name": "Mister Green",
   "Age": 43,
   "Address": {
     "Street": "Red Boulevard",
     "City": "Green Town"
   }
}

 Array Updates Without Key Properties

An update for an array where the array elements have no key always contains the fully specified array value. Clients should replace the snapshot directly with the update value. This is true for both simple and structured types.

Array Updates With Key Properties

Updates to arrays of objects with key properties are sent as arrays that can contain three different types of elements: add elements, update elements, and delete elements. All the array elements in the update contains the key properties so that the corresponding element in the client's snapshot can be found. 

For each of the elements in the update array the client must apply the update to the snapshot in the following way:

  • If the snapshot does not contain an element with the same key, the update is a fully specified object that should be added to the snapshot.
  • If the snapshot contains an element with the same key, the update can either be a deletion or an update to that element:
    • If the update has a property called "__meta_deleted", the element must be deleted from the client's snapshot
    • Otherwise the element in the client's snapshot must be updated (not replaced) with the new value


In the example below, objects in the list are identified by the Name property. 

The snapshot contains two element:

[
   {
     "Name": "Mister Red",
     "Age": 42,
     "Address": {
       "Street": "Red Boulevard",
       "City": "Red Town"
    },
    {
     "Name": "Mister Green",
     "Age": 42,
     "Address": {
       "Street": "Green Boulevard",
       "City": "Green Town"
    }
]


The update contains one element update (Mister Red), one delete (Mister Green), and one add (Mister Blue).

[
   // Updated element contains only updated properties
   {
     "Name": "Mister Red",
     "Age": 43
   },
   // Deleted element contains a "__meta_deleted" property and the key properties
   {
     "Name": "Mister Green",
     "__meta_deleted": true
   },
   // New element is fully specified
   {
     "Name": "Mister Blue",
     "Age": 42,
     "Address": {
       "Street": "Blue Boulevard",
       "City": "Blue Town"
    }
]

The result of applying the update is:

[
   {
     "Name": "Mister Red",
     "Age": 43,
     "Address": {
       "Street": "Red Boulevard",
       "City": "Red Town"
    },
    {
     "Name": "Mister Blue",
     "Age": 42,
     "Address": {
       "Street": "Blue Boulevard",
       "City": "Blue Town"
    }
]
 

Heartbeat

When no data is sent for a while on a subscription, streaming server starts sending out heartbeats to let the client know that it is still active.

Heartbeat instructions have following format:

Property
Type
Description
ReferenceId stringThe value "_heartbeat"
Timestamp  stringThe time at which the instruction was generated
HeartbeatsHeartbeat[] (see below)The heartbeats


Like other streaming data, client receives it too in connection.received event handler. A Heartbeat is an object with the following properties: 

PropertyTypeDescription
OriginatingReferenceIdstringThe reference id identifying the subscription
Reasonstring 

One of the following values, indicating the reason that the heartbeat was sent:

  • Property not specified or it has the value "NoNewData": No new data is available. This is the normal case.
  • "SubscriptionTemporarilyDisabled": No data is currently being sent due to a recoverable error on the server, e.g. failure to connect to a database or internal service. The client should accept that data is not available at the expected rate and disable functions accordingly until data is available again. The subscription will continue to deliver heartbeats until data is available again.
  • "SubscriptionPermanentlyDisabled": No data will be available on the subscription. Most likely this is due to a misconfiguration. The client should not attempt to reset the subscription and should not expect any data on it. The subscription will continue to deliver heartbeats until the subscription is removed.


When no data or heartbeat has been received for the amount of time specified in the subscription response (InactivityTimeout), the client should reset the subscription using a new reference id.

[
   {
     {
       "ReferenceId": "_heartbeat",
       "Timestamp": "...",
       "Heartbeats": [
         {
           "OriginatingReferenceId": "MyReferenceId1",
           "Reason": "SubscriptionTemporarilyDisabled"
         },
         {
            "OriginatingReferenceId": "MyReferenceId2",
            "Reason": "NoNewData"
          },
       ]
   }
]

Subscription reset instructions

In some error scenarios, the streaming server may send subscription reset instructions to instruct the client to reset a given subset of its subscriptions. In this case, value of ReferenceID in the stream data will be "_resetsubscriptions" and reference ids of the subscriptions which should be reset is sent in parameter TargetReferenceIds.

Resetting a subscription means that the subscription should be deleted and another one created in its place using a new reference id. Any update received for a subscription after a reset instruction must be ignored.

Reset instruction have the following format :

Property
Type
Description
ReferenceIdstringThe value "_resetsubscriptions"
Timestamp stringThe time at which the instruction was generated
TargetReferenceIdsstring[] The list of reference ids for the subscriptions that must be reset. If this property is an empty array or not specified, the client should reset all its subscriptions.

Refers to a known subscription

In this case Reference id of the streaming instruction will be equal to one of the reference Id's of the made subscriptions. In this case apply incremental update to the snapshot of the subscription with the given reference id. Structure of instruction is what is defined above in streaming updates section.

Subscription disconnected instructions

It is possible that a subscription will be closed by the server. This can happen if a user changes their account password. When this happens a "_disconnect" instruction will be sent out on the stream and after that, no more messages will be sent. The client cannot create a new subscription after receiving a disconnect message. All subsequent requests will be rejected with a 401 - Unauthorized response from the server until the user is logged out and logs back in. The message will usually be sent to all open subscriptions at the same time.

Disconnecting a subscription Means that the entire user session should be closed and any messages received after this message should be ignored. The user should be logged out and asked to log in again.

Disconnect instructions have the following format:

Property
Type
Description
ReferenceIdstringThe value "_disconnect"
Timestamp stringThe time at which the instruction was generated


References

How to set up a developer machine

OpenAPI Security

OpenAPI Environments

Reference Documentation