FetchListV3.make

Overview

When you need to process large volumes of data from list methods of REST API version 3 in parts (chunks), use FetchListV3.make().

This method implements a fast algorithm for iterating over large data sets without loading all data into memory at once. Each iteration returns the next page/batch of results until all data is received.

// Basic usage
const generator = $b24.actions.v3.fetchList.make({
  method: 'main.eventlog.list',
  params: {
    filter: [
      ['userId', '=', 1]
    ],
    select: ['id', 'userId']
  },
  idKey: 'id',
  customKeyForResult: 'items',
  requestId: 'unique-request-id',
  limit: 600
})

for await (const chunk of generator) {
  // Process chunk (e.g., save to database, analyze, etc.)
  console.log(`Processing ${chunk.length} items`)
}

When to Use FetchListV3.make()

1. Very large data volumes: When the number of records is in the thousands or tens of thousands.
2. Stream processing: When data needs to be processed as it arrives.
3. Long operations: When processing each record requires significant time.

Method Signature

make<T = unknown>(
  options: ActionFetchListV3
): AsyncGenerator<T[]>

Parameters

The options object contains the following properties:

Return Value

AsyncGenerator<T[]> — an asynchronous generator that returns data chunks as arrays of type T.

Each iteration of the generator returns:

• An array of elements (chunk) with up to options.limit records.
• The generator completes when all data is received.

Key Concepts

AsyncGenerator vs Promise

Unlike CallListV3.make(), which returns a Promise with all data at once, FetchListV3.make() returns an asynchronous generator:

Performance Optimization

The method implements the Bitrix24 recommended algorithm for efficient work with large data volumes:

1. Filtering by increasing id: Each subsequent query uses a >id filter with the id of the last retrieved element.
2. Stream processing: Data is processed as it is received, saving memory.
3. Automatic data end detection: Requests stop when an empty array is received or the number of elements is less than the page size (options.limit).

REST API v3 Response Structure

In REST API version 3, various methods return data in the same structures:

• Grouped array: { result: { items: [...] } }

The customKeyForResult parameter allows you to specify the key where the data is located in the response.

Limitations

• Page size: Bitrix24 REST API version 3 limitation — maximum 1000 records per request.
• Sorting: The method always adds sorting by idKey in ascending order for correct pagination.
• Only for list methods: Intended only for methods that return data arrays.

Error Handling

Since the method returns an asynchronous generator, errors are handled differently than in CallListV3.make():

try {
  const generator = $b24.actions.v3.fetchList.make({
    method: 'some.method',
    params: { /* some_params */ },
    idKey: 'id',
    customKeyForResult: 'items',
    requestId: 'unique-request-id',
    limit: 600
  })

  for await (const chunk of generator) {
    // Process chunk (e.g., save to database, analyze, etc.)
    console.log(`Processing ${chunk.length} items`)
  }
} catch (error) {
  // Handling error
  if (
    error instanceof SdkError
    && error.code === 'JSSDK_CORE_B24_FETCH_LIST_METHOD_API_V3'
  ) {
    console.error(`${error.message}`, { code: error.code })
  } else {
    console.error('Some error', error)
  }
}

Examples

Step-by-step processing of a large number of Event Log Items

import { B24Hook, LoggerFactory, Text, SdkError, AjaxError } from '@bitrix24/b24jssdk'

type MainEventLogItem = {
  id: number
  userId: number
}

const devMode = typeof import.meta !== 'undefined' && (import.meta.dev || import.meta.env?.DEV)
const $logger = LoggerFactory.createForBrowser('Example:ProcessMainEventLogItems', devMode)
const $b24 = B24Hook.fromWebhookUrl('https://your_domain.bitrix24.com/rest/1/webhook_code/')

async function processMainEventLogItem(): Promise<void> {
  let batchNumber = 0
  let totalItems = 0

  const sixMonthAgo = new Date()
  sixMonthAgo.setMonth((new Date()).getMonth() - 6)
  sixMonthAgo.setHours(0, 0, 0)
  
  const requestId = 'some-main-event-log-item-list'
  
  try {
    const generator = $b24.actions.v3.fetchList.make<MainEventLogItem>({
      method: 'main.eventlog.list',
      params: {
        filter: [
          ['timestampX', '>=', Text.toB24Format(sixMonthAgo)] // created at least 6 months ago
        ],
        select: ['id', 'userId']
      },
      idKey: 'id',
      customKeyForResult: 'items',
      requestId,
      limit: 60
    })
    
    for await (const chunk of generator) {
      batchNumber++
      totalItems += chunk.length

      $logger.info(`Processing batch #${batchNumber}`, {
        batchSize: chunk.length,
        totalSoFar: totalItems
      })
      
      // Example: saving to database
      await saveToDatabase(chunk)
      
      // Example: sending to message queue
      await sendToMessageQueue(chunk)
    }
    
    $logger.notice(`Processed ${totalItems} elements in ${batchNumber} batches`)
  } catch (error) {
    if (error instanceof SdkError) {
      $logger.error(`Processing error: ${error.message}`, {
        code: error.code,
        batchNumber,
        totalItems
      })
    } else {
      $logger.error('Unknown error', { error, batchNumber, totalItems })
    }
    throw error
  }
}

// Helper functions
// Database save implementation
async function saveToDatabase(items: MainEventLogItem[]): Promise<void> {
  await new Promise(resolve => setTimeout(resolve, 100)) // Simulation
}

// Message queue send implementation
async function sendToMessageQueue(items: MainEventLogItem[]): Promise<void> {
  await new Promise(resolve => setTimeout(resolve, 50)) // Simulation
}

// Usage
try {
  await processMainEventLogItem()
} catch (error) {
  $logger.critical('A problem occurred', { error })
}

Alternatives and Recommendations

• Chunk size: Bitrix24 REST API version 3 always returns up to 50 records per request.
• Optimal concurrency: When processing data from the generator, limited concurrency (3-5 simultaneous operations) is recommended.
• Error handling: Always handle errors inside the for await...of loop to prevent the entire process from stopping.
• Progress monitoring: Implement progress logging for long-running operations.
• For sequential requests: Use Call for single calls.
• For batch operations: Use Batch to execute up to 50 commands in a single request.
• On the client-side (browser): Use the built-in B24Frame object.