Spaces:
Sleeping
Sleeping
File size: 2,622 Bytes
287a0bc |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
# CIP-5: Large Batch Handling Improvements Proposal
## Status
Current Status: `Under Discussion`
## **Motivation**
As users start putting Chroma in its paces and storing ever-increasing datasets, we must ensure that errors
related to significant and potentially expensive batches are handled gracefully. This CIP proposes to add a new
setting, `max_batch_size` API, on the local segment API and use it to split large batches into smaller ones.
## **Public Interfaces**
The following interfaces are impacted:
- New Server API endpoint - `/pre-flight-checks`
- New `max_batch_size` property on the `API` interface
- Updated `_add`, `_update` and `_upsert` methods on `chromadb.api.segment.SegmentAPI`
- Updated `_add`, `_update` and `_upsert` methods on `chromadb.api.fastapi.FastAPI`
- New utility library `batch_utils.py`
- New exception raised when batch size exceeds `max_batch_size`
## **Proposed Changes**
We propose the following changes:
- The new `max_batch_size` property is now available in the `API` interface. The property relies on the
underlying `Producer` class
to fetch the actual value. The property will be implemented by both `chromadb.api.segment.SegmentAPI`
and `chromadb.api.fastapi.FastAPI`
- `chromadb.api.segment.SegmentAPI` will implement the `max_batch_size` property by fetching the value from the
`Producer` class.
- `chromadb.api.fastapi.FastAPI` will implement the `max_batch_size` by fetching it from a new `/pre-flight-checks`
endpoint on the Server.
- New `/pre-flight-checks` endpoint on the Server will return a dictionary with pre-flight checks the client must
fulfil to integrate with the server side. For now, we propose using this only for `max_batch_size`, but we can
add more checks in the future. The pre-flight checks will be only fetched once per client and cached for the duration
of the client's lifetime.
- Updated `_add`, `_update` and `_upsert` method on `chromadb.api.segment.SegmentAPI` to validate batch size.
- Updated `_add`, `_update` and `_upsert` method on `chromadb.api.fastapi.FastAPI` to validate batch size (client-side
validation)
- New utility library `batch_utils.py` will contain the logic for splitting batches into smaller ones.
## **Compatibility, Deprecation, and Migration Plan**
The change will be fully compatible with existing implementations. The changes will be transparent to the user.
## **Test Plan**
New tests:
- Batch splitting tests for `chromadb.api.segment.SegmentAPI`
- Batch splitting tests for `chromadb.api.fastapi.FastAPI`
- Tests for `/pre-flight-checks` endpoint
## **Rejected Alternatives**
N/A
|