Skip to content

docs: make document indices self-contained #1678

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 35 commits into from
Aug 1, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
caa00fd
chore: first pr
jupyterjazz Jun 28, 2023
b45e3a6
docs: modify hnsw
jupyterjazz Jul 6, 2023
cad4e60
Merge branch 'main' into docs-self-contained-indices
jupyterjazz Jul 6, 2023
11bda62
docs: rough versions of inmemory and hnsw
jupyterjazz Jul 6, 2023
96319ca
chore: update branch
jupyterjazz Jul 6, 2023
f5825f8
docs: weaviate v1
jupyterjazz Jul 6, 2023
8aaedbe
docs: elastic v1
jupyterjazz Jul 17, 2023
4a3e25c
docs: introduction page
jupyterjazz Jul 17, 2023
db77beb
docs: redis v1
jupyterjazz Jul 17, 2023
82afb99
docs: qdrant v1
jupyterjazz Jul 17, 2023
befc786
docs: validate intro inmemory and hnsw examples
jupyterjazz Jul 17, 2023
9bdb0dc
docs: validate elastic and qdrant examples
jupyterjazz Jul 17, 2023
64f83bf
docs: validate code examples for redis and weaviate
jupyterjazz Jul 18, 2023
759900c
Merge branch 'main' into docs-self-contained-indices
jupyterjazz Jul 19, 2023
60cd4d4
chore: merge recent updates
jupyterjazz Jul 19, 2023
ca25feb
docs: milvus v1
jupyterjazz Jul 19, 2023
7fef5d8
Merge branch 'main' into docs-self-contained-indices
jupyterjazz Jul 24, 2023
fe572da
docs: validate milvus code
jupyterjazz Jul 24, 2023
10bc14b
docs: make redis and milvus visible
jupyterjazz Jul 24, 2023
6199a2a
docs: refine vol1
jupyterjazz Jul 26, 2023
fa8f919
Merge branch 'main' into docs-self-contained-indices
jupyterjazz Jul 26, 2023
c257a4e
docs: refine vol2
jupyterjazz Jul 26, 2023
ccf17e1
chore: pull recent updates
jupyterjazz Jul 26, 2023
f3ca77c
docs: update api reference
jupyterjazz Jul 27, 2023
21e3ad2
Merge branch 'main' into docs-self-contained-indices
jupyterjazz Jul 27, 2023
e6ef9c4
docs: apply suggestions
jupyterjazz Jul 31, 2023
19045ec
docs: separate nested data section
jupyterjazz Jul 31, 2023
5736334
Merge branch 'main' into docs-self-contained-indices
jupyterjazz Jul 31, 2023
41c7307
docs: apply suggestions vol2
jupyterjazz Jul 31, 2023
a32a1e5
fix: nested data imports
jupyterjazz Jul 31, 2023
8a8aa33
Merge branch 'main' into docs-self-contained-indices
jupyterjazz Aug 1, 2023
ef0b7ef
docs: apply johannes suggestions
jupyterjazz Aug 1, 2023
6818688
chore: merge conflicts
jupyterjazz Aug 1, 2023
9268161
docs: apply suggestions
jupyterjazz Aug 1, 2023
b402802
docs: app sgg
jupyterjazz Aug 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
docs: app sgg
Signed-off-by: jupyterjazz <saba.sturua@jina.ai>
  • Loading branch information
jupyterjazz committed Aug 1, 2023
commit b4028024d386704c14df0bc5d2676125ea69a67f
2 changes: 1 addition & 1 deletion docs/user_guide/storing/docindex.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ doc_index.index(docs)

### Perform a vector similarity search
Now, let's perform a similarity search on the document embeddings.
As a result, we'll retrieve ten most similar documents and their corresponding similarity scores.
As a result, we'll retrieve the ten most similar documents and their corresponding similarity scores.
```python
query = np.ones(128)
retrieved_docs, scores = doc_index.find(query, search_field='embedding', limit=10)
Expand Down
2 changes: 1 addition & 1 deletion docs/user_guide/storing/index_elastic.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ but will also work for [ElasticV7DocIndex][docarray.index.backends.elasticv7.Ela
## Basic usage
This snippet demonstrates the basic usage of [ElasticDocIndex][docarray.index.backends.elastic.ElasticDocIndex]. It defines a document schema with a title and an embedding,
creates ten dummy documents with random embeddings, initializes an instance of [ElasticDocIndex][docarray.index.backends.elastic.ElasticDocIndex] to index these documents,
and performs a vector similarity search to retrieve ten most similar documents to a given query vector.
and performs a vector similarity search to retrieve the ten most similar documents to a given query vector.

```python
from docarray import BaseDoc, DocList
Expand Down
2 changes: 1 addition & 1 deletion docs/user_guide/storing/index_hnswlib.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ It stores vectors on disk in [hnswlib](https://github.com/nmslib/hnswlib), and s
## Basic usage
This snippet demonstrates the basic usage of [HnswDocumentIndex][docarray.index.backends.hnswlib.HnswDocumentIndex]. It defines a document schema with a title and an embedding,
creates ten dummy documents with random embeddings, initializes an instance of [HnswDocumentIndex][docarray.index.backends.hnswlib.HnswDocumentIndex] to index these documents,
and performs a vector similarity search to retrieve ten most similar documents to a given query vector.
and performs a vector similarity search to retrieve the ten most similar documents to a given query vector.

```python
from docarray import BaseDoc, DocList
Expand Down
4 changes: 2 additions & 2 deletions docs/user_guide/storing/index_in_memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
[InMemoryExactNNIndex][docarray.index.backends.in_memory.InMemoryExactNNIndex] stores all documents in memory using DocLists.
It is a great starting point for small datasets, where you may not want to launch a database server.

For vector search and filtering the [InMemoryExactNNIndex][docarray.index.backends.in_memory.InMemoryExactNNIndex]
For vector search and filtering [InMemoryExactNNIndex][docarray.index.backends.in_memory.InMemoryExactNNIndex]
utilizes DocArray's [`find()`][docarray.utils.find.find] and [`filter_docs()`][docarray.utils.filter.filter_docs] functions.

!!! note "Production readiness"
Expand All @@ -23,7 +23,7 @@ utilizes DocArray's [`find()`][docarray.utils.find.find] and [`filter_docs()`][d
## Basic usage
This snippet demonstrates the basic usage of [InMemoryExactNNIndex][docarray.index.backends.in_memory.InMemoryExactNNIndex]. It defines a document schema with a title and an embedding,
creates ten dummy documents with random embeddings, initializes an instance of [InMemoryExactNNIndex][docarray.index.backends.in_memory.InMemoryExactNNIndex] to index these documents,
and performs a vector similarity search to retrieve ten most similar documents to a given query vector.
and performs a vector similarity search to retrieve the ten most similar documents to a given query vector.

```python
from docarray import BaseDoc, DocList
Expand Down
2 changes: 1 addition & 1 deletion docs/user_guide/storing/index_milvus.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ focusing on special features and configurations of Milvus.
## Basic usage
This snippet demonstrates the basic usage of [MilvusDocumentIndex][docarray.index.backends.milvus.MilvusDocumentIndex]. It defines a document schema with a title and an embedding,
creates ten dummy documents with random embeddings, initializes an instance of [MilvusDocumentIndex][docarray.index.backends.milvus.MilvusDocumentIndex] to index these documents,
and performs a vector similarity search to retrieve ten most similar documents to a given query vector.
and performs a vector similarity search to retrieve the ten most similar documents to a given query vector.

!!! note "Single Search Field Requirement"
In order to utilize vector search, it's necessary to define 'is_embedding' for one field only.
Expand Down
2 changes: 1 addition & 1 deletion docs/user_guide/storing/index_qdrant.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ based on the [Qdrant](https://qdrant.tech/) vector search engine.
## Basic usage
This snippet demonstrates the basic usage of [QdrantDocumentIndex][docarray.index.backends.qdrant.QdrantDocumentIndex]. It defines a document schema with a title and an embedding,
creates ten dummy documents with random embeddings, initializes an instance of [QdrantDocumentIndex][docarray.index.backends.qdrant.QdrantDocumentIndex] to index these documents,
and performs a vector similarity search to retrieve ten most similar documents to a given query vector.
and performs a vector similarity search to retrieve the ten most similar documents to a given query vector.

```python
from docarray import BaseDoc, DocList
Expand Down
2 changes: 1 addition & 1 deletion docs/user_guide/storing/index_redis.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ focusing on special features and configurations of Redis.
## Basic usage
This snippet demonstrates the basic usage of [RedisDocumentIndex][docarray.index.backends.redis.RedisDocumentIndex]. It defines a document schema with a title and an embedding,
creates ten dummy documents with random embeddings, initializes an instance of [RedisDocumentIndex][docarray.index.backends.redis.RedisDocumentIndex] to index these documents,
and performs a vector similarity search to retrieve ten most similar documents to a given query vector.
and performs a vector similarity search to retrieve the ten most similar documents to a given query vector.

```python
from docarray import BaseDoc, DocList
Expand Down
2 changes: 1 addition & 1 deletion docs/user_guide/storing/index_weaviate.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ focusing on special features and configurations of Weaviate.
## Basic usage
This snippet demonstrates the basic usage of [WeaviateDocumentIndex][docarray.index.backends.weaviate.WeaviateDocumentIndex]. It defines a document schema with a title and an embedding,
creates ten dummy documents with random embeddings, initializes an instance of [WeaviateDocumentIndex][docarray.index.backends.weaviate.WeaviateDocumentIndex] to index these documents,
and performs a vector similarity search to retrieve ten most similar documents to a given query vector.
and performs a vector similarity search to retrieve the ten most similar documents to a given query vector.

!!! note "Single Search Field Requirement"
In order to utilize vector search, it's necessary to define 'is_embedding' for one field only.
Expand Down
3 changes: 2 additions & 1 deletion docs/user_guide/storing/nested_data.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,8 @@ doc_index.index(index_docs)

### Search

You can perform search on any level by using `find_subindex()` method and the dunder operator `'root__subindex'` to specify the index to search on.
You can perform search on any level by using [`find_subindex()`][docarray.index.abstract.BaseDocIndex.find_subindex] method
and the dunder operator `'root__subindex'` to specify the index to search on:

```python
# find by the `VideoDoc` tensor
Expand Down
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy