Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ pip3 install -r deeplake/requirements/tests.txt
- `pytest .`: Run all tests with memory only.
- `pytest . --local`: Run all tests with memory and local.
- `pytest . --s3`: Run all tests with memory and s3.
- `pytest . --gcs`: Run all tests with memory and GCS
- `pytest . --gcs`: Run all tests with memory and GCS
- `pytest . --kaggle`: Run all tests that use the kaggle API.
- `pytest . --memory-skip --hub-cloud`: Run all tests with hub cloud only.
#### Backwards Compatibility Tests
Expand Down Expand Up @@ -107,4 +107,4 @@ API docs are hosted at: https://api-docs.activeloop.ai/
Run the below command to generate API documentation:
```
pdoc3 --html --output-dir api_docs --template-dir pdoc/templates hub
```
```
24 changes: 12 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@
<h3 align="center">
<a href="https://docs.deeplake.ai/?utm_source=github&utm_medium=github&utm_campaign=github_readme&utm_id=readme"><b>Docs</b></a> &bull;
<a href="https://docs.deeplake.ai/latest/getting-started/quickstart/?utm_source=github&utm_medium=github&utm_campaign=github_readme&utm_id=readme"><b>Get Started</b></a> &bull;
<a href="https://docs.deeplake.ai/latest/api/dataset/?utm_source=github&utm_medium=github&utm_campaign=github_readme&utm_id=readme"><b>API Reference</b></a> &bull;
<a href="https://docs.deeplake.ai/latest/api/dataset/?utm_source=github&utm_medium=github&utm_campaign=github_readme&utm_id=readme"><b>API Reference</b></a> &bull;
<a href="http://learn.activeloop.ai"><b>LangChain & VectorDBs Course</b></a> &bull;
<a href="https://www.activeloop.ai/resources/?utm_source=github&utm_medium=github&utm_campaign=github_readme&utm_id=readme"><b>Blog</b></a> &bull;
<a href="https://www.deeplake.ai/?utm_source=github&utm_medium=github&utm_campaign=github_readme&utm_id=readme"><b>Whitepaper</b></a> &bull;
<a href="https://www.deeplake.ai/?utm_source=github&utm_medium=github&utm_campaign=github_readme&utm_id=readme"><b>Whitepaper</b></a> &bull;
<a href="http://slack.activeloop.ai"><b>Slack</b></a> &bull;
<a href="https://twitter.com/intent/tweet?url=https%3A%2F%2Factiveloop.ai%2F&via=activeloopai&text=Deep%20Lake%20is%20the%20Database%20for%20all%20AI%20data.%20Check%20it%20out%21&hashtags=DeepLake%2Cactiveloop%2Copensource"><b>Twitter</b></a>
</h3>
Expand All @@ -26,14 +26,14 @@ Deep Lake is a Database for AI powered by a storage format optimized for deep-le

1. Storing and searching data plus vectors while building LLM applications
2. Managing datasets while training deep learning models
Deep Lake simplifies the deployment of enterprise-grade LLM-based products by offering storage for all data types (embeddings, audio, text, videos, images, dicom, pdfs, annotations, [and more](https://docs.deeplake.ai/latest/api/types/)), querying and vector search, data streaming while training models at scale, data versioning and lineage, and integrations with popular tools such as LangChain, LlamaIndex, Weights & Biases, and many more. Deep Lake works with data of any size, it is serverless, and it enables you to store all of your data in your own cloud and in one place. Deep Lake is used by Intel, Bayer Radiology, Matterport, ZERO Systems, Red Cross, Yale, & Oxford.

Deep Lake simplifies the deployment of enterprise-grade LLM-based products by offering storage for all data types (embeddings, audio, text, videos, images, dicom, pdfs, annotations, [and more](https://docs.deeplake.ai/latest/api/types/)), querying and vector search, data streaming while training models at scale, data versioning and lineage, and integrations with popular tools such as LangChain, LlamaIndex, Weights & Biases, and many more. Deep Lake works with data of any size, it is serverless, and it enables you to store all of your data in your own cloud and in one place. Deep Lake is used by Intel, Bayer Radiology, Matterport, ZERO Systems, Red Cross, Yale, & Oxford.

### Deep Lake includes the following features:

<details>
<summary><b>Multi-Cloud Support (S3, GCP, Azure)</b></summary>
Use one API to upload, download, and stream datasets to/from S3, Azure, GCP, Activeloop cloud, local storage, or in-memory storage. Compatible with any S3-compatible storage such as MinIO.
Use one API to upload, download, and stream datasets to/from S3, Azure, GCP, Activeloop cloud, local storage, or in-memory storage. Compatible with any S3-compatible storage such as MinIO.
</details>
<details>
<summary><b>Native Compression with Lazy NumPy-like Indexing</b></summary>
Expand Down Expand Up @@ -101,35 +101,35 @@ Deep Lake users can access and visualize a variety of popular datasets through a

<details>
<summary><b>Deep Lake vs Chroma </b></summary>

Both Deep Lake & ChromaDB enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. ChromaDB is a Vector Database that can be deployed locally or on a server using Docker and will offer a hosted solution shortly. Deep Lake is a serverless Vector Store deployed on the user’s own cloud, locally, or in-memory. All computations run client-side, which enables users to support lightweight production apps in seconds. Unlike ChromaDB, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. ChromaDB is limited to light metadata on top of the embeddings and has no visualization. Deep Lake datasets can be visualized and version controlled. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

</details>

<details>
<summary><b>Deep Lake vs Pinecone</b></summary>

Both Deep Lake and Pinecone enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. Pinecone is a fully-managed Vector Database that is optimized for highly demanding applications requiring a search for billions of vectors. Deep Lake is serverless. All computations run client-side, which enables users to get started in seconds. Unlike Pinecone, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. Deep Lake datasets can be visualized and version controlled. Pinecone is limited to light metadata on top of the embeddings and has no visualization. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

</details>

<details>
<summary><b>Deep Lake vs Weaviate</b></summary>

Both Deep Lake and Weaviate enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. Weaviate is a Vector Database that can be deployed in a managed service or by the user via Kubernetes or Docker. Deep Lake is serverless. All computations run client-side, which enables users to support lightweight production apps in seconds. Unlike Weaviate, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. Deep Lake datasets can be visualized and version controlled. Weaviate is limited to light metadata on top of the embeddings and has no visualization. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

</details>

<details>
<summary><b>Deep Lake vs DVC</b></summary>

Deep Lake and DVC offer dataset version control similar to git for data, but their methods for storing data differ significantly. Deep Lake converts and stores data as chunked compressed arrays, which enables rapid streaming to ML models, whereas DVC operates on top of data stored in less efficient traditional file structures. The Deep Lake format makes dataset versioning significantly easier compared to traditional file structures by DVC when datasets are composed of many files (i.e., many images). An additional distinction is that DVC primarily uses a command-line interface, whereas Deep Lake is a Python package. Lastly, Deep Lake offers an API to easily connect datasets to ML frameworks and other common ML tools and enables instant dataset visualization through [Activeloop's visualization tool](http://app.activeloop.ai/?utm_source=github&utm_medium=repo&utm_campaign=readme).

</details>

<details>
<summary><b>Deep Lake vs MosaicML MDS format </b></summary>

* **Data Storage Format:** Deep Lake operates on a columnar storage format, whereas MDS utilizes a row-wise storage approach. This fundamentally impacts how data is read, written, and organized in each system.
* **Compression:** Deep Lake offers a more flexible compression scheme, allowing control over both chunk-level and sample-level compression for each column or tensor. This feature eliminates the need for additional compressions like zstd, which would otherwise demand more CPU cycles for decompressing on top of formats like jpeg.
* **Shuffling:** MDS currently offers more advanced shuffling strategies.
Expand All @@ -139,7 +139,7 @@ Deep Lake and DVC offer dataset version control similar to git for data, but the

<details>
<summary><b>Deep Lake vs TensorFlow Datasets (TFDS)</b></summary>

Deep Lake and TFDS seamlessly connect popular datasets to ML frameworks. Deep Lake datasets are compatible with both PyTorch and TensorFlow, whereas TFDS are only compatible with TensorFlow. A key difference between Deep Lake and TFDS is that Deep Lake datasets are designed for streaming from the cloud, whereas TFDS must be downloaded locally prior to use. As a result, with Deep Lake, one can import datasets directly from TensorFlow Datasets and stream them either to PyTorch or TensorFlow. In addition to providing access to popular publicly available datasets, Deep Lake also offers powerful tools for creating custom datasets, storing them on a variety of cloud storage providers, and collaborating with others via simple API. TFDS is primarily focused on giving the public easy access to commonly available datasets, and management of custom datasets is not the primary focus. A full comparison article can be found [here](https://www.activeloop.ai/resources/tensor-flow-tf-data-activeloop-hub-how-to-implement-your-tensor-flow-data-pipelines-with-hub/).

</details>
Expand Down Expand Up @@ -192,7 +192,7 @@ Using Deep Lake? Add a README badge to let everyone know:

<details>
<summary><b> Dataset Licenses</b></summary>

Deep Lake users may have access to a variety of publicly available datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have a license to use the datasets. It is your responsibility to determine whether you have permission to use the datasets under their license.

If you're a dataset owner and do not want your dataset to be included in this library, please get in touch through a [GitHub issue](https://github.com/activeloopai/deeplake/issues/new). Thank you for your contribution to the ML community!
Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Please include the requested information listed below (as much as you can provid
* Proof-of-concept or exploit code (if possible)
* Impact of the issue, including how an attacker might exploit the issue

You should receive a response within 72 hours. If for some reason you do not, please follow up via email to ensure we received your original message.
You should receive a response within 72 hours. If for some reason you do not, please follow up via email to ensure we received your original message.

After the initial reply to your report, the security team will endeavor to keep
you informed of the progress being made towards a fix and full announcement,
Expand Down
2 changes: 1 addition & 1 deletion cpp/3rd_party/json/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ project(json)

file(GLOB_RECURSE SOURCES "*.cpp" "*.c" "*.hpp")

add_library(json ${SOURCES})
add_library(json ${SOURCES})
4 changes: 2 additions & 2 deletions cpp/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ if (${AL_PG})
find_package(PkgConfig)

pkg_check_modules(liburing IMPORTED_TARGET GLOBAL liburing>=2.0)

include(CheckCSourceCompiles)
set(CMAKE_REQUIRED_DEFINITIONS -D_GNU_SOURCE)
check_c_source_compiles("
Expand All @@ -140,7 +140,7 @@ if (${AL_PG})
return 0;
}
" HAVE_STATX)

if(liburing_FOUND AND HAVE_STATX)
set(HAVE_LIBURING TRUE)
add_compile_definitions(HAVE_LIBURING=1)
Expand Down
2 changes: 1 addition & 1 deletion cpp/vcpkg-configuration.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@
"overlay-triplets": [
"./vcpkg-triplets"
]
}
}