FetchListV3.make

Overview

When you need to process large volumes of data from list methods of REST API version 3 in parts (chunks), use FetchListV3.make().

This method implements a fast algorithm for iterating over large data sets without loading all data into memory at once. Each iteration returns the next page/batch of results until all data is received.

// Basic usage
const generator = $b24.actions.v3.fetchList.make({
  method: 'main.eventlog.list',
  params: {
    filter: [
      ['userId', '=', 1]
    ],
    select: ['id', 'userId']
  },
  idKey: 'id',
  customKeyForResult: 'items',
  requestId: 'unique-request-id',
  limit: 600
})

for await (const chunk of generator) {
  // Process chunk (e.g., save to database, analyze, etc.)
  console.log(`Processing ${chunk.length} items`)
}

When to Use FetchListV3.make()

1. Very large data volumes: When the number of records is in the thousands or tens of thousands.
2. Stream processing: When data needs to be processed as it arrives.
3. Long operations: When processing each record requires significant time.

Method Signature

make<T = unknown>(
  options: ActionFetchListV3
): AsyncGenerator<T[]>

Parameters

The options object contains the following properties:

Return Value

AsyncGenerator<T[]> — an asynchronous generator that returns data chunks as arrays of type T.

Each iteration of the generator returns:

• An array of elements (chunk) with up to options.limit records.
• The generator completes when all data is received.

Key Concepts

AsyncGenerator vs Promise

Unlike CallListV3.make(), which returns a Promise with all data at once, FetchListV3.make() returns an asynchronous generator:

Performance Optimization

The method implements the Bitrix24 recommended algorithm for efficient work with large data volumes:

1. Filtering by increasing id: Each subsequent query uses a [cursorIdKey, '>', id] filter with the id of the last retrieved element (read via idKey).
2. Stream processing: Data is processed as it is received, saving memory.
3. Automatic data end detection: Requests stop when an empty array is received, or when a page is shorter than the largest page seen so far. This keys the stop on the page size the server actually returns rather than the requested options.limit, so a method that silently caps the page below limit (e.g. tasks.task.list, fixed at 50) still pages through every record instead of stopping after the first capped page. A short page ends the walk only once the cursor read from it has been checked — a page that is short and repeats the cursor is a server ignoring the page condition, not the end of the data, and is reported rather than returned.

REST API v3 Response Structure

In REST API version 3, various methods return data in the same structures:

• Grouped array: { result: { items: [...] } }

The customKeyForResult parameter allows you to specify the key where the data is located in the response.

Limitations

• Page size is per method, not a global 1000: limit states what you want; the method decides what it gives. Measured on an on-premise build, tasks.task.list returns 50 however much is asked, and nothing in the response says it was capped. The walker keys end-of-data on the largest page the server actually returned, so it pages through everything regardless — a hand-rolled loop on call.make does not.
• Sorting is fixed: The method always sorts by cursorIdKey (which defaults to idKey) ascending, because cursor pagination relies on [cursorIdKey, '>', nextId] filters to walk the dataset. A user-supplied order value would break that invariant, so the declared order property is Omitted from the type — though the [key: string]: unknown index signature it inherits means the compiler still accepts one — and any value passed at runtime is stripped with a warning log entry. To narrow the result set, use filter instead.
• filter must be the v3 array form: [['id', '>', 100]], or the output of FilterV3.build(...). The restApi:v2 object dialect ({ '>id': 100 }) is accepted by TypeCallParamsV3 for backward compatibility and works with a plain call, but not here: cursor pagination appends [cursorIdKey, '>', nextId] to the same filter on every page, so an array is the only shape it can extend. Passing an object throws SdkError with code JSSDK_ACTION_V3_LIST_FILTER_NOT_ARRAY. It used to be accepted and then failed mid-walk with filter is not iterable.
• A cursor that stops advancing is fatal: if the last idKey value on a page equals the one already filtered on, the >cursorIdKey condition was dropped and the same page will keep arriving. A full page cannot end the walk on its own — the largest-page-seen stop never fires — and a short one is checked before it is taken for the end of the data, so it throws SdkError with code JSSDK_ACTION_CURSOR_STALLED instead of collecting duplicates for ever. The usual cause is idKey naming the response field while cursorIdKey is left to default to it: tasks.task.list returns a lowercase id but filters on an uppercase ID, so it needs both (idKey: 'id', cursorIdKey: 'ID'). A cursor that moves the wrong way is caught too, and separately: a server alternating between two pages never repeats the value just sent, so the equality test above cannot see it — the walk is stopped with JSSDK_ACTION_CURSOR_WENT_BACKWARDS on the first backwards step. And neither check waits for a full page any more: a short page whose cursor did not advance used to end the walk as end-of-data, returning a truncated and overlapping result with no error at all.
• Only for list methods: Intended only for methods that return data arrays.

Error Handling

Since the method returns an asynchronous generator, errors are handled differently than in CallListV3.make():

// @check-ignore: some.method is a placeholder, not a portal method

import { SdkError } from '@bitrix24/b24jssdk'

try {
  const generator = $b24.actions.v3.fetchList.make({
    method: 'some.method',
    params: { /* some_params */ },
    idKey: 'id',
    customKeyForResult: 'items',
    requestId: 'unique-request-id',
    limit: 600
  })

  for await (const chunk of generator) {
    // Process chunk (e.g., save to database, analyze, etc.)
    console.log(`Processing ${chunk.length} items`)
  }
} catch (error) {
  // Handling error
  if (
    error instanceof SdkError
    && error.code === 'JSSDK_CORE_B24_FETCH_LIST_METHOD_API_V3'
  ) {
    console.error(`${error.message}`, { code: error.code })
  } else {
    console.error('Some error', error)
  }
}

Examples

Step-by-step processing of a large number of Event Log Items

import { B24Hook, LoggerFactory, Text, SdkError, AjaxError } from '@bitrix24/b24jssdk'

type MainEventLogItem = {
  id: number
  userId: number
}

const devMode = !!(typeof import.meta !== 'undefined' && (import.meta.dev || import.meta.env?.DEV))
const $logger = LoggerFactory.createForBrowser('Example:ProcessMainEventLogItems', devMode)
const $b24 = B24Hook.fromWebhookUrl('https://your_domain.bitrix24.com/rest/1/webhook_code/')

async function processMainEventLogItem(): Promise<void> {
  let batchNumber = 0
  let totalItems = 0

  const sixMonthAgo = new Date()
  sixMonthAgo.setMonth((new Date()).getMonth() - 6)
  sixMonthAgo.setHours(0, 0, 0)
  
  const requestId = 'some-main-event-log-item-list'
  
  try {
    const generator = $b24.actions.v3.fetchList.make<MainEventLogItem>({
      method: 'main.eventlog.list',
      params: {
        filter: [
          ['timestampX', '>=', Text.toB24Format(sixMonthAgo)] // created at least 6 months ago
        ],
        select: ['id', 'userId']
      },
      idKey: 'id',
      customKeyForResult: 'items',
      requestId,
      limit: 60
    })
    
    for await (const chunk of generator) {
      batchNumber++
      totalItems += chunk.length

      $logger.info(`Processing batch #${batchNumber}`, {
        batchSize: chunk.length,
        totalSoFar: totalItems
      })
      
      // Example: saving to database
      await saveToDatabase(chunk)
      
      // Example: sending to message queue
      await sendToMessageQueue(chunk)
    }
    
    $logger.notice(`Processed ${totalItems} elements in ${batchNumber} batches`)
  } catch (error) {
    if (error instanceof SdkError) {
      $logger.error(`Processing error: ${error.message}`, {
        code: error.code,
        batchNumber,
        totalItems
      })
    } else {
      $logger.error('Unknown error', { error, batchNumber, totalItems })
    }
    throw error
  }
}

// Helper functions
// Database save implementation
async function saveToDatabase(items: MainEventLogItem[]): Promise<void> {
  await new Promise(resolve => setTimeout(resolve, 100)) // Simulation
}

// Message queue send implementation
async function sendToMessageQueue(items: MainEventLogItem[]): Promise<void> {
  await new Promise(resolve => setTimeout(resolve, 50)) // Simulation
}

// Usage
try {
  await processMainEventLogItem()
} catch (error) {
  $logger.critical('A problem occurred', { error })
}

When the request and response id fields differ

Some methods sort / filter by one field name but return another (for example an uppercase ID in the request vs a lowercase id in the payload). Set idKey to the response field and cursorIdKey to the request field:

// @check-ignore: some.list is a placeholder, not a portal method

const generator = $b24.actions.v3.fetchList.make({
  method: 'some.list',
  params: { select: ['ID', 'TITLE'] },
  idKey: 'id', // read the id from each returned item
  cursorIdKey: 'ID', // order + ["ID", ">", n] page filter in the request
  customKeyForResult: 'items'
})

for await (const chunk of generator) {
  console.log(`Got ${chunk.length} records`)
}

The two ways of getting this wrong fail differently, and the difference matters:

• idKey cannot be read from a full page — you named a field the response does not carry. The SDK logs a warning and stops paginating rather than silently truncating; the Result holds what arrived so far, and it is real data.
• cursorIdKey names a field the server does not filter on — idKey reads fine, so the walk keeps going, but the page condition is dropped and the same page comes back for ever. That one throws SdkError with code JSSDK_ACTION_CURSOR_STALLED, because everything collected at that point is one page repeated.

Alternatives and Recommendations

• Chunk size: Each chunk holds up to options.limit records (default 50; the method applies its own maximum, so a chunk can be smaller than asked). Larger chunks mean fewer HTTP requests but more memory pressure per iteration.
• Optimal concurrency: When processing data from the generator, limited concurrency (3-5 simultaneous operations) is recommended.
• Error handling: Always handle errors inside the for await...of loop to prevent the entire process from stopping.
• Progress monitoring: Implement progress logging for long-running operations.
• For sequential requests: Use Call for single calls.
• For batch operations: Use Batch to execute up to 50 commands in a single request.
• On the client-side (browser): Use the built-in B24Frame object.
• See also: Filtering — building the filter (and why order is stripped).

Bounding the walk

The walk pages until the data runs out. Three options bound it.

maxPages stops after that many pages and throws JSSDK_ACTION_MAX_PAGES_EXCEEDED, naming the method, after yielding every page it read. It defaults to 10 000 — a backstop rather than a policy: on restApi:v3 that is 500 000 rows at the default page size of 50, and more only where the method's own cap allows a larger limit, and at the default drain rate of 2 requests/second roughly 83 minutes, so a walk that never ends is bounded without capping a read anyone performs. Raise it when a read genuinely needs more.

It errors rather than truncating. A short list that looks complete is the failure mode this whole class of bug keeps producing, so nothing is returned — though the pages already yielded are yours to keep.

signal takes an AbortSignal and throws JSSDK_ACTION_ABORTED. It is checked at the top of each iteration, so an already-aborted signal costs no request at all.

The streaming walkers take no progress: they hand you each page as it arrives, so counting them is your own for await body.