I am encountering an issue with the paginate() function designed to handle pagination for querying DynamoDB with filters. As DynamoDB imposes certain limitations on filtered queries, I developed this function to manage pagination and ensure that the results are properly limited. However, the function is returning more results than I actually have in the database.

const docClient = new aws.DynamoDB.DocumentClient();

async paginate(params,TableName){    
  if (!params.Limit) {
    params.Limit = 10
  }

  let result = []
  let items
  let scannedItemCount = 0
  do {
    items = await docClient.scan({ TableName, ...params })
    result = result.concat(items.Items)
    if (items.Count !== null && items.Count !== undefined) {
      scannedItemCount += items.Count
    }
    params.ExclusiveStartKey = items.LastEvaluatedKey
  } while (scannedItemCount < params.Limit && items.LastEvaluatedKey)

  result.splice(params.Limit)

  return {
    items: result,
    LastEvaluatedKey: result.length > 0 ? { ...result[result.length - 1] } : undefined
  }
}

Issue: The paginate() function is not effectively limiting the results for filtered queries in DynamoDB. When executing the function with a specified params.Limit, it appears to return more results than the given limit, potentially exceeding the limitations imposed by DynamoDB on filtered queries.

Expected behavior: The paginate() function should be able to handle DynamoDB s limitations on filtered queries and correctly return results up to the specified params.Limit. If there are more results beyond the limit, they should be truncated, and the LastEvaluatedKey should be provided for subsequent pagination.

Possible fix: Given the limitations imposed by DynamoDB on filtered queries, it is essential to consider alternative approaches to handle pagination effectively. I have already attempted to address the issue by checking if items.Count is not null or undefined before incrementing the scannedItemCount, but the issue still persists.

I am seeking assistance from the community to help identify the root cause of the bug and suggest potential fixes or workarounds to ensure the paginate() function correctly limits the results for filtered queries in DynamoDB.