Crossbar ledger missing records bug

[Barnaby Puttick]

Hi,

I seem to have discovered a bit of an issue using crossbar to retrieve ledger entries.

There is a situation where crossbar does not return all expected records due to a bug when chunking is used (the default)

Crossbar expects the "key" to be unique so it can pass the next_key to get the following chunked result and to determine EOF when last_key==next_key (chunking exhausted).

cb_ledger use the "list_by_source" view to iterate through the records which has the "key" defined as [{service}, {pvt_created}] - however if there are 2 or more service entries that were created at the same time (multiple legs) this key is no longer unique and when this falls on a chunked boundary crossbar misinterprets this as EOF returning an incomplete result.

This can be easily demonstrated by setting the chuck_size to 1 so that every entry is a chunk boundary and any duplicate key triggers a result exhaustion. 

For most part, this chunk exhaustion will not occur because the default chunk_size and page_size are both 50 resulting in a page exhaustion first. 

The question is, should this be fixed in crossbar's chunking algorithm or should the ledger view be changed to force a unique key?

A temporary fix is to use is_chunked=false query parameter to disable chunking if you're experiencing missing records.

Barnaby 

 

Edited April 29, 2019 by Barnaby Puttick (see edit history)

[Barnaby Puttick]

This was tested on 4.3.29 and seems to be still prevalent in master and 4.3 (untested)