Skip to content

Datasource

Jump to bottom
Denis Hilt edited this page Feb 17, 2022 · 8 revisions

Overview

Datasource is the main part of the integration of the end App and the Virtual scroll engine. Can be seen as the data-fetching mechanism that the end App developer should provide when using vscroll module (directly or via consumer). Through the dependency injection the Datasource gets into the heart of the Virtual scroll engine, and the Scroller requests the Datasource each time it needs the data from the outside to render new data rows. For example, when the user scrolls, it triggers the appropriate processes inside the Virtual scroll engine, which can cause the Scroller to decide that it needs a new pack of data items. The request is achieved by calling the Datasource.get method, which works on the App side and delivers the required piece of data to the Scroller. The Scroller then processes it and updates the UI.

Definition

Assume the Datasource as an object of the following type

interface IDatasource<Data = unknown> {
 get: DatasourceGet<Data>;
 settings?: Settings<Data>;
 devSettings?: DevSettings;
}

The technical details of this interface can be taken from the source code, but the practical meaning is that we need method get and can provide some settings. There are two ways to define the Datasource on the App side: as an object literal or as an instance of the Datasource class that takes the same object literal as a constructor argument.

const datasource1 = { get, settings }; // object literal
const datasource2 = new Datasource({ get, settings }); // instance

The Datasource class should be provided by the VScroll consumer, but in the case of using the vscroll core module, this class can be obtained through the makeDatasource factory:

import { makeDatasource } form 'vscroll';
...
const Datasource = makeDatasource();
const datasource2 = new Datasource({ get, settings });

All of the examples below will use the instance class notation.

Get method

The implementation of the Datasource is the App developer responsibility. It should provides access to the data we want to virtualize, and below is a simplest sample of an unlimited synchronous stream of data generated in runtime by index-count parameters:

const datasource = new Datasource({
  get: (index, count, success) => {
    const data = [];
    for (let i = index; i < index + count; i++) {
      data.push({ id: i, text: 'item #' + i });
    }
    success(data);
  }
});

The get method is the only mandatory property of the Datasource definition. It has some signatures, the simplest one is callback-based:

type DatasourceGet<T> = (index: number, count: number, success: (data: T[]) => void) => void;

The requirement for this method is that it should pass via success callback an array of count data-items starting at index. In other words, the App thus should always be ready to provide the required number of data items to the Scroller. Let's consider a bit more complex sample, where the dataset is pre-defined and limited:

const MIN = -99;
const MAX = 100;
const DATA = [];

for (let i = MIN; i <= MAX; ++i) {
  DATA.push({ text: 'item #' + i });
}

const datasource = new Datasource({
  get: (index, count, success) => {
    const shift = -Math.min(MIN, 0);
    const start = Math.max(index + shift, 0);
    const end = Math.min(index + count - 1, MAX) + shift;
    if (start <= end) {
      success(DATA.slice(start, end + 1));
    } else {
      success([]);
    }
  }
});

It is important that the get method can return empty array or array of length that is less than count. This cases are treated as EOF/BOF by the Scroller and prevent it from making new requests to the Datasource via the get method in appropriate direction. Another signature of the get method is promise-based. The following is equivalent to the first sample:

const datasource = new Datasource({
  get: (index, count) => new Promise(resolve => {
    const data = [];
    for (let i = index; i < index + count; i++) {
      data.push({ id: i, text: 'item #' + i });
    }
    resolve(data);
  })
});

Both signatures provide asynchronisity: both success and resolve callbacks can be invoked at an uncertain moment. The implementation of the Datasource.get method can be very complicated and multifunctional. More samples of the get method implementations can be found at ngx-ui-scroll demo page, including inverted and pages cases. And there is one case of particular interest, which is considered separately from the others in this doc: the caching datasource.

Cache

The Scroller can request the same data multiple times becase it destroys the rows when they are out of view and re-creates them as they become visible again. There is currently no caching option on the Scroller's end, and if the source of data is remote, it might be very important to implement an intermediate Cache layer before the Scroller. It could be done as an HTTP Service Interceptor which is global for the all Application, but here we'll give a try the Datasource to provide necessary functionality.

Since we are dealing with natural indexes, let Cache be a JavaScript Map which key is the Datasource.get index argument. We will ask the Cache for index-count items, and if some (or all) of them are not in the Cache, we will request them via our remote-data service. When the remote-data service responds, we'll store the result in the Cache, merge it with what we got from the Cache before the request and pass it to the Scroller via success callback.

import { remoteDataService } form "./myAPI";

const cache = new Map();

const datasource = new Datasource({
  get: (index, count, success) => {
    const result = [], _index = null;
    for (let i = index; i < index + count; i++) {
      const item = this.cache.get(i);
      if (item) {
        result.push(item);
      } else {
        _index = i;
        break;
      }
    }
    if (_index === null) {
      // all requested data was taken from cached
      success(result);
      return;
    }
    // request missing data from remote
    const _count = index + count - _index;
    remoteDataService.request(_index, _count).then(_result => {
      // cache received data
      _result.forEach((item, index) => this.cache.set(_index + index, item));
      // merge restored and requested data
      success([...result, ..._result]);
    });
  }
});

Settings

Along with the get method, Datasource implementation may include a settings object. Settings are being applied during the Scroller initialization and have an impact on how the Scroller behaves. Below is the list of available settings with descriptions, defaults, types and demos taken from the ngx-ui-scroll repository (which is a vscroll consumer built for Angular apps).

Name Type Default Description
bufferSize number,
integer		 5 Fixes minimal size of the pack of the datasource items to be requested per single Datasource.get call. Can't be less than 1.
padding number,
float		 0.5 Determines the viewport outlets containing real but not visible items. The value is relative to the viewport's size. For example, 0.25 means that there will be as many items at a moment as needed to fill out 100% of the visible part of the viewport, + 25% of the viewport size in the backward direction and + 25% in the forward direction. The value can't be less than 0.01.
startIndex number,
integer		 1 Specifies item index to be requested/rendered first. Can be any, but the real datasource boundaries should be taken into account.
minIndex number,
integer		 -Infinity Fixes absolute minimal index of the dataset. The datasource left boundary.
maxIndex number,
integer		 +Infinity Fixes absolute maximal index of the dataset. The datasource right boundary.
infinite boolean false Enables "infinite" mode, when items rendered once are never removed.
horizontal boolean false Enables "horizontal" mode, when the viewport's orientation is horizontal.
sizeStrategy string enum, 'average' | 'frequent' | 'constant' 'average' Defines how the default item size is calculated. If item has never been rendered, its size is assumed to be the default size: an average or most frequent among all items that have been rendered before, or constant. This has an impact on the process of virtualization.
windowViewport boolean false Enables "entire window scrollable" mode, when the entire window becomes the scrollable viewport.

There is also devSettings property. The development settings are not documented. Information about it can be taken directly from the source code. The Scroller has "debug" mode with powerful logging which can be enabled via devSettings.debug = true, see Dev Log doc.

Clone this wiki locally