FetchListV3.make

Overview

When you need to process large volumes of data from list methods of REST API version 3 in parts (chunks), use FetchListV3.make().

This method implements a fast algorithm for iterating over large data sets without loading all data into memory at once. Each iteration returns the next page/batch of results until all data is received.

// Basic usage
const generator = $b24.actions.v3.fetchList.make({
  method: 'main.eventlog.list',
  params: {
    filter: [
      ['userId', '=', 1]
    ],
    select: ['id', 'userId']
  },
  idKey: 'id',
  customKeyForResult: 'items',
  requestId: 'unique-request-id',
  limit: 600
})

for await (const chunk of generator) {
  // Process chunk (e.g., save to database, analyze, etc.)
  console.log(`Processing ${chunk.length} items`)
}

When to Use FetchListV3.make()

1. Very large data volumes: When the number of records is in the thousands or tens of thousands.
2. Stream processing: When data needs to be processed as it arrives.
3. Long operations: When processing each record requires significant time.

Method Signature

make<T = unknown>(
  options: ActionFetchListV3
): AsyncGenerator<T[]>

Parameters

The options object contains the following properties:

Return Value

AsyncGenerator<T[]> — an asynchronous generator that returns data chunks as arrays of type T.

Each iteration of the generator returns:

• An array of elements (chunk) with up to options.limit records.
• The generator completes when all data is received.

Key Concepts

AsyncGenerator vs Promise

Unlike CallListV3.make(), which returns a Promise with all data at once, FetchListV3.make() returns an asynchronous generator:

Performance Optimization

The method implements the Bitrix24 recommended algorithm for efficient work with large data volumes:

1. Filtering by increasing id: Each subsequent query uses a [cursorIdKey, '>', id] filter with the id of the last retrieved element (read via idKey).
2. Stream processing: Data is processed as it is received, saving memory.
3. Automatic data end detection: Requests stop when an empty array is received, or when a page is shorter than the largest page seen so far. This keys the stop on the page size the server actually returns rather than the requested options.limit, so a method that silently caps the page below limit (e.g. tasks.task.list, fixed at 50) still pages through every record instead of stopping after the first capped page.

REST API v3 Response Structure

In REST API version 3, various methods return data in the same structures:

• Grouped array: { result: { items: [...] } }

The customKeyForResult parameter allows you to specify the key where the data is located in the response.

Limitations

• Page size: Bitrix24 REST API version 3 limitation — maximum 1000 records per request.
• Sorting is fixed: The method always sorts by cursorIdKey (which defaults to idKey) ascending, because cursor pagination relies on [cursorIdKey, '>', nextId] filters to walk the dataset. A user-supplied order value would break that invariant, so the type signature excludes order and any value passed at runtime is stripped with a warning log entry. To narrow the result set, use filter instead.
• Only for list methods: Intended only for methods that return data arrays.

Error Handling

Since the method returns an asynchronous generator, errors are handled differently than in CallListV3.make():

import { SdkError } from '@bitrix24/b24jssdk'

try {
  const generator = $b24.actions.v3.fetchList.make({
    method: 'some.method',
    params: { /* some_params */ },
    idKey: 'id',
    customKeyForResult: 'items',
    requestId: 'unique-request-id',
    limit: 600
  })

  for await (const chunk of generator) {
    // Process chunk (e.g., save to database, analyze, etc.)
    console.log(`Processing ${chunk.length} items`)
  }
} catch (error) {
  // Handling error
  if (
    error instanceof SdkError
    && error.code === 'JSSDK_CORE_B24_FETCH_LIST_METHOD_API_V3'
  ) {
    console.error(`${error.message}`, { code: error.code })
  } else {
    console.error('Some error', error)
  }
}

Examples

Step-by-step processing of a large number of Event Log Items

import { B24Hook, LoggerFactory, Text, SdkError, AjaxError } from '@bitrix24/b24jssdk'

type MainEventLogItem = {
  id: number
  userId: number
}

const devMode = !!(typeof import.meta !== 'undefined' && (import.meta.dev || import.meta.env?.DEV))
const $logger = LoggerFactory.createForBrowser('Example:ProcessMainEventLogItems', devMode)
const $b24 = B24Hook.fromWebhookUrl('https://your_domain.bitrix24.com/rest/1/webhook_code/')

async function processMainEventLogItem(): Promise<void> {
  let batchNumber = 0
  let totalItems = 0

  const sixMonthAgo = new Date()
  sixMonthAgo.setMonth((new Date()).getMonth() - 6)
  sixMonthAgo.setHours(0, 0, 0)
  
  const requestId = 'some-main-event-log-item-list'
  
  try {
    const generator = $b24.actions.v3.fetchList.make<MainEventLogItem>({
      method: 'main.eventlog.list',
      params: {
        filter: [
          ['timestampX', '>=', Text.toB24Format(sixMonthAgo)] // created at least 6 months ago
        ],
        select: ['id', 'userId']
      },
      idKey: 'id',
      customKeyForResult: 'items',
      requestId,
      limit: 60
    })
    
    for await (const chunk of generator) {
      batchNumber++
      totalItems += chunk.length

      $logger.info(`Processing batch #${batchNumber}`, {
        batchSize: chunk.length,
        totalSoFar: totalItems
      })
      
      // Example: saving to database
      await saveToDatabase(chunk)
      
      // Example: sending to message queue
      await sendToMessageQueue(chunk)
    }
    
    $logger.notice(`Processed ${totalItems} elements in ${batchNumber} batches`)
  } catch (error) {
    if (error instanceof SdkError) {
      $logger.error(`Processing error: ${error.message}`, {
        code: error.code,
        batchNumber,
        totalItems
      })
    } else {
      $logger.error('Unknown error', { error, batchNumber, totalItems })
    }
    throw error
  }
}

// Helper functions
// Database save implementation
async function saveToDatabase(items: MainEventLogItem[]): Promise<void> {
  await new Promise(resolve => setTimeout(resolve, 100)) // Simulation
}

// Message queue send implementation
async function sendToMessageQueue(items: MainEventLogItem[]): Promise<void> {
  await new Promise(resolve => setTimeout(resolve, 50)) // Simulation
}

// Usage
try {
  await processMainEventLogItem()
} catch (error) {
  $logger.critical('A problem occurred', { error })
}

When the request and response id fields differ

Some methods sort / filter by one field name but return another (for example an uppercase ID in the request vs a lowercase id in the payload). Set idKey to the response field and cursorIdKey to the request field:

const generator = $b24.actions.v3.fetchList.make({
  method: 'some.list',
  params: { select: ['ID', 'TITLE'] },
  idKey: 'id', // read the id from each returned item
  cursorIdKey: 'ID', // order + ["ID", ">", n] page filter in the request
  customKeyForResult: 'items'
})

for await (const chunk of generator) {
  console.log(`Got ${chunk.length} records`)
}

If idKey can't be read from a full page, the SDK logs a warning and stops paginating instead of silently truncating.

Alternatives and Recommendations

• Chunk size: Each chunk holds up to options.limit records (default 50, max 1000). Larger chunks mean fewer HTTP requests but more memory pressure per iteration.
• Optimal concurrency: When processing data from the generator, limited concurrency (3-5 simultaneous operations) is recommended.
• Error handling: Always handle errors inside the for await...of loop to prevent the entire process from stopping.
• Progress monitoring: Implement progress logging for long-running operations.
• For sequential requests: Use Call for single calls.
• For batch operations: Use Batch to execute up to 50 commands in a single request.
• On the client-side (browser): Use the built-in B24Frame object.