Skip to content

feat: support redis #1550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 36 commits into from
Jul 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
df47474
feat: support redis
jupyterjazz May 17, 2023
0920adc
chore: merge main
jupyterjazz Jun 12, 2023
51558b7
fix: index creation
jupyterjazz Jun 15, 2023
12da714
feat: 1st draft, needs polishing
jupyterjazz Jun 21, 2023
be7bed7
feat: query builder, tests
jupyterjazz Jun 28, 2023
9365327
Merge branch 'main' into feat-add-redis
jupyterjazz Jun 28, 2023
cb25869
chore: update poetry lock
jupyterjazz Jun 28, 2023
341fa9a
chore: run tests
jupyterjazz Jun 28, 2023
d37b28f
Merge branch 'main' into feat-add-redis
jupyterjazz Jun 28, 2023
abca1dd
fix: defaultdict for column config
jupyterjazz Jun 28, 2023
edc8a34
chore: update branch
jupyterjazz Jun 28, 2023
c5abd80
style: ignore mypy errors
jupyterjazz Jun 28, 2023
22434f1
refactor: put vectorfield args in column info
jupyterjazz Jun 28, 2023
4a96194
chore: remove unused code
jupyterjazz Jun 28, 2023
a52cdfd
docs: add docstrings
jupyterjazz Jun 28, 2023
9ac683f
test: add tensorflow test
jupyterjazz Jun 28, 2023
a7f54c3
fix: tensorflow test
jupyterjazz Jun 28, 2023
be9d771
fix: tf tst
jupyterjazz Jun 28, 2023
4dc99e6
refactor: reduce ignore types
jupyterjazz Jun 28, 2023
d2718b4
style: remove other type ignores
jupyterjazz Jun 28, 2023
9f58474
style: try removing import ignores
jupyterjazz Jun 28, 2023
7e791da
style: i think mypy hates me
jupyterjazz Jun 28, 2023
8ed3dbb
feat: batch indexing
jupyterjazz Jun 29, 2023
83afb5f
chore: bump redis version
jupyterjazz Jun 29, 2023
4b0bc73
feat: subindex not fully finished
jupyterjazz Jul 2, 2023
0b15adc
feat: finalize subindex
jupyterjazz Jul 5, 2023
e8fbc47
docs: update readme
jupyterjazz Jul 5, 2023
848e95d
chore: commits not showing
jupyterjazz Jul 5, 2023
c00abe2
Merge branch 'main' into feat-add-redis
jupyterjazz Jul 5, 2023
27dd29a
feat: del and get batched
jupyterjazz Jul 6, 2023
162c6a8
docs: update batchsize docstring
jupyterjazz Jul 6, 2023
16c4323
Merge branch 'main' into feat-add-redis
jupyterjazz Jul 6, 2023
56829d6
refactor: index name
jupyterjazz Jul 9, 2023
c414836
Merge branch 'main' into feat-add-redis
jupyterjazz Jul 9, 2023
7a5ed5e
refactor: default index name following schema
jupyterjazz Jul 9, 2023
0fecb48
chore: update branch
jupyterjazz Jul 9, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ jobs:
fail-fast: false
matrix:
python-version: [3.8]
db_test_folder: [base_classes, elastic, hnswlib, qdrant, weaviate]
db_test_folder: [base_classes, elastic, hnswlib, qdrant, weaviate, redis]
steps:
- uses: actions/checkout@v2.5.0
- name: Set up Python ${{ matrix.python-version }}
Expand Down
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ DocArray is a Python library expertly crafted for the [representation](#represen

- :fire: Offers native support for **[NumPy](https://github.com/numpy/numpy)**, **[PyTorch](https://github.com/pytorch/pytorch)**, and **[TensorFlow](https://github.com/tensorflow/tensorflow)**, catering specifically to **model training scenarios**.
- :zap: Based on **[Pydantic](https://github.com/pydantic/pydantic)**, and instantly compatible with web and microservice frameworks like **[FastAPI](https://github.com/tiangolo/fastapi/)** and **[Jina](https://github.com/jina-ai/jina/)**.
- :package: Provides support for vector databases such as **[Weaviate](https://weaviate.io/), [Qdrant](https://qdrant.tech/), [ElasticSearch](https://www.elastic.co/de/elasticsearch/)**, and **[HNSWLib](https://github.com/nmslib/hnswlib)**.
- :package: Provides support for vector databases such as **[Weaviate](https://weaviate.io/), [Qdrant](https://qdrant.tech/), [ElasticSearch](https://www.elastic.co/de/elasticsearch/), [Redis](https://redis.io/)**, and **[HNSWLib](https://github.com/nmslib/hnswlib)**.
- :chains: Allows data transmission as JSON over **HTTP** or as **[Protobuf](https://protobuf.dev/)** over **[gRPC](https://grpc.io/)**.

## Installation
Expand Down Expand Up @@ -349,7 +349,7 @@ This is useful for:
- :mag: **Neural search** applications
- :bulb: **Recommender systems**

Currently, Document Indexes support **[Weaviate](https://weaviate.io/)**, **[Qdrant](https://qdrant.tech/)**, **[ElasticSearch](https://www.elastic.co/)**, and **[HNSWLib](https://github.com/nmslib/hnswlib)**, with more to come!
Currently, Document Indexes support **[Weaviate](https://weaviate.io/)**, **[Qdrant](https://qdrant.tech/)**, **[ElasticSearch](https://www.elastic.co/)**, **[Redis](https://redis.io/)**, and **[HNSWLib](https://github.com/nmslib/hnswlib)**, with more to come!

The Document Index interface lets you index and retrieve Documents from multiple vector databases, all with the same user interface.

Expand Down Expand Up @@ -421,7 +421,7 @@ They are now called **Document Indexes** and offer the following improvements (s
- **Production-ready:** The new Document Indexes are a much thinner wrapper around the various vector DB libraries, making them more robust and easier to maintain
- **Increased flexibility:** We strive to support any configuration or setting that you could perform through the DB's first-party client

For now, Document Indexes support **[Weaviate](https://weaviate.io/)**, **[Qdrant](https://qdrant.tech/)**, **[ElasticSearch](https://www.elastic.co/)**, and **[HNSWLib](https://github.com/nmslib/hnswlib)**, with more to come.
For now, Document Indexes support **[Weaviate](https://weaviate.io/)**, **[Qdrant](https://qdrant.tech/)**, **[ElasticSearch](https://www.elastic.co/)**, **[Redis](https://redis.io/)**, and **[HNSWLib](https://github.com/nmslib/hnswlib)**, with more to come.

</details>

Expand Down Expand Up @@ -775,6 +775,7 @@ Currently, DocArray supports the following vector databases:
- [Weaviate](https://www.weaviate.io/)
- [Qdrant](https://qdrant.tech/)
- [Elasticsearch](https://www.elastic.co/elasticsearch/) v8 and v7
- [Redis](https://redis.io/)
- [HNSWlib](https://github.com/nmslib/hnswlib) as a local-first alternative

An integration of [OpenSearch](https://opensearch.org/) is currently in progress.
Expand Down Expand Up @@ -836,6 +837,7 @@ from docarray.index import (
WeaviateDocumentIndex,
QdrantDocumentIndex,
ElasticDocIndex,
RedisDocumentIndex,
)

# Select a suitable backend and initialize it with data
Expand Down
4 changes: 4 additions & 0 deletions docarray/index/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
from docarray.index.backends.hnswlib import HnswDocumentIndex # noqa: F401
from docarray.index.backends.qdrant import QdrantDocumentIndex # noqa: F401
from docarray.index.backends.weaviate import WeaviateDocumentIndex # noqa: F401
from docarray.index.backends.redis import RedisDocumentIndex # noqa: F401

__all__ = ['InMemoryExactNNIndex']

Expand All @@ -34,6 +35,9 @@ def __getattr__(name: str):
elif name == 'WeaviateDocumentIndex':
import_library('weaviate', raise_error=True)
import docarray.index.backends.weaviate as lib
elif name == 'RedisDocumentIndex':
import_library('redis', raise_error=True)
import docarray.index.backends.redis as lib
else:
raise ImportError(
f'cannot import name \'{name}\' from \'{_get_path_from_docarray_root_level(__file__)}\''
Expand Down
Loading
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy