-
Notifications
You must be signed in to change notification settings - Fork 3.2k
Feat: Firestore support #5088
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ScottMansfield
wants to merge
36
commits into
google:main
Choose a base branch
from
ScottMansfield:feat/firestore
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Feat: Firestore support #5088
Changes from all commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
96d1dca
Oneshot attempt at adding firestore support for memory and sessions
ScottMansfield d2d2231
Formatting and fixing the bucket name handling
ScottMansfield 8312dc8
Correct imports for firestore
ScottMansfield 7760b70
Add firestore to test dependencies
ScottMansfield b29afb7
Fix tests
ScottMansfield 31ffb86
Fix mypy errors
ScottMansfield 565cc61
Undo unintended changes
ScottMansfield 9387cb3
Sorting imports
ScottMansfield 03910e9
Fix async mocks
ScottMansfield 49c7bf5
Fixing tests again again again
ScottMansfield bf77d28
Merge branch 'main' into feat/firestore
ScottMansfield fbd16eb
Empty commit
ScottMansfield 5645fe8
Fixing one more test to use the firestore mock client
ScottMansfield 9d8bb5d
Move firestore integration into integrations package
ScottMansfield 49580f6
Merge branch 'main' into feat/firestore
ScottMansfield bf47b5c
Updating session service to address various concerns
ScottMansfield 28a571e
pyink and isort
ScottMansfield 7d0b4cc
Addressing mypy errors
ScottMansfield 9a27b7e
Addressing deprecation warnings for firestore query syntax
ScottMansfield cb9e4d1
Make memory actually work by implementing add_session_to_memory.
ScottMansfield 1f01174
Formatting
ScottMansfield 72cc9d2
Remove unnecessary firestore runner
ScottMansfield 16a4a86
Merge branch 'main' into feat/firestore
ScottMansfield bef661c
formatting
ScottMansfield e18a1f8
Adding more detailed comments for both session and memory services
ScottMansfield b652669
Much improved unit tests
ScottMansfield 9c19c02
Adding app to document hierarchy
ScottMansfield 486c21a
Improving tests and handling one failure mode in memory
ScottMansfield 2daa18d
Append all event data in a single transaction
ScottMansfield e05dd5b
Remove dead code
ScottMansfield d7458b7
Ensure memory generation does not go over the firestore batch limit
ScottMansfield 6a48d0e
Aligning firestore storage with other session implementations
ScottMansfield ca724d2
Hardening firestore against concurrent modification
ScottMansfield 72e7abf
Merge branch 'main' into feat/firestore
ScottMansfield 78dd7a1
Mypy errors
ScottMansfield 99efca5
Adding test for temp state handling
ScottMansfield File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # Copyright 2026 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| """Firestore integrations for ADK.""" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| # Copyright 2026 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| DEFAULT_STOP_WORDS = { | ||
| "a", | ||
| "about", | ||
| "above", | ||
| "after", | ||
| "again", | ||
| "against", | ||
| "all", | ||
| "am", | ||
| "an", | ||
| "and", | ||
| "any", | ||
| "are", | ||
| "as", | ||
| "at", | ||
| "be", | ||
| "because", | ||
| "been", | ||
| "before", | ||
| "being", | ||
| "below", | ||
| "between", | ||
| "both", | ||
| "but", | ||
| "by", | ||
| "can", | ||
| "could", | ||
| "did", | ||
| "do", | ||
| "does", | ||
| "doing", | ||
| "don", | ||
| "down", | ||
| "during", | ||
| "each", | ||
| "else", | ||
| "few", | ||
| "for", | ||
| "from", | ||
| "further", | ||
| "had", | ||
| "has", | ||
| "have", | ||
| "having", | ||
| "he", | ||
| "her", | ||
| "here", | ||
| "hers", | ||
| "herself", | ||
| "him", | ||
| "himself", | ||
| "his", | ||
| "how", | ||
| "i", | ||
| "if", | ||
| "in", | ||
| "into", | ||
| "is", | ||
| "it", | ||
| "its", | ||
| "itself", | ||
| "just", | ||
| "may", | ||
| "me", | ||
| "might", | ||
| "more", | ||
| "most", | ||
| "must", | ||
| "my", | ||
| "myself", | ||
| "no", | ||
| "nor", | ||
| "not", | ||
| "now", | ||
| "of", | ||
| "off", | ||
| "on", | ||
| "once", | ||
| "only", | ||
| "or", | ||
| "other", | ||
| "our", | ||
| "ours", | ||
| "ourselves", | ||
| "out", | ||
| "over", | ||
| "own", | ||
| "s", | ||
| "same", | ||
| "shall", | ||
| "she", | ||
| "should", | ||
| "so", | ||
| "some", | ||
| "such", | ||
| "t", | ||
| "than", | ||
| "that", | ||
| "the", | ||
| "their", | ||
| "theirs", | ||
| "them", | ||
| "themselves", | ||
| "then", | ||
| "there", | ||
| "these", | ||
| "they", | ||
| "this", | ||
| "those", | ||
| "through", | ||
| "to", | ||
| "too", | ||
| "under", | ||
| "until", | ||
| "up", | ||
| "very", | ||
| "was", | ||
| "we", | ||
| "were", | ||
| "what", | ||
| "when", | ||
| "where", | ||
| "which", | ||
| "who", | ||
| "whom", | ||
| "why", | ||
| "will", | ||
| "with", | ||
| "would", | ||
| "you", | ||
| "your", | ||
| "yours", | ||
| "yourself", | ||
| "yourselves", | ||
| } | ||
195 changes: 195 additions & 0 deletions
195
src/google/adk/integrations/firestore/firestore_memory_service.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,195 @@ | ||
| # Copyright 2026 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import asyncio | ||
| import logging | ||
| import os | ||
| import re | ||
| from typing import Any | ||
| from typing import Optional | ||
| from typing import TYPE_CHECKING | ||
|
|
||
| from google.cloud.firestore_v1.base_query import FieldFilter | ||
| from typing_extensions import override | ||
|
|
||
| from ...events.event import Event | ||
| from ...memory import _utils | ||
| from ...memory.base_memory_service import BaseMemoryService | ||
| from ...memory.base_memory_service import SearchMemoryResponse | ||
| from ...memory.memory_entry import MemoryEntry | ||
| from ._stop_words import DEFAULT_STOP_WORDS | ||
|
|
||
| if TYPE_CHECKING: | ||
| from google.cloud import firestore | ||
|
|
||
| from ...sessions.session import Session | ||
|
|
||
| logger = logging.getLogger("google_adk." + __name__) | ||
|
|
||
| DEFAULT_EVENTS_COLLECTION = "events" | ||
| DEFAULT_MEMORIES_COLLECTION = "memories" | ||
|
|
||
|
|
||
| class FirestoreMemoryService(BaseMemoryService): # type: ignore[misc] | ||
| """Memory service that uses Google Cloud Firestore as the backend. | ||
|
|
||
| It uses the existing session data to create memories in a top-level memory collection. | ||
| """ | ||
|
|
||
| def __init__( | ||
| self, | ||
| client: Optional[firestore.AsyncClient] = None, | ||
| events_collection: Optional[str] = None, | ||
| stop_words: Optional[set[str]] = None, | ||
| memories_collection: Optional[str] = None, | ||
| ): | ||
| """Initializes the Firestore memory service. | ||
|
|
||
| Args: | ||
| client: An optional Firestore AsyncClient. If not provided, a new one | ||
| will be created. | ||
| events_collection: The name of the events collection or collection group. | ||
| Defaults to 'events'. | ||
| stop_words: A set of words to ignore when extracting keywords. Defaults to | ||
| a standard English stop words list. | ||
| memories_collection: The name of the memories collection. Defaults to | ||
| 'memories'. | ||
| """ | ||
| if client is None: | ||
| from google.cloud import firestore | ||
|
|
||
| self.client = firestore.AsyncClient() | ||
| else: | ||
| self.client = client | ||
| self.events_collection = events_collection or DEFAULT_EVENTS_COLLECTION | ||
| self.memories_collection = ( | ||
| memories_collection or DEFAULT_MEMORIES_COLLECTION | ||
| ) | ||
| self.stop_words = ( | ||
| stop_words if stop_words is not None else DEFAULT_STOP_WORDS | ||
| ) | ||
|
|
||
| @override | ||
| async def add_session_to_memory(self, session: Session) -> None: | ||
| """Extracts keywords from session events and stores them in the memories collection.""" | ||
| batch = self.client.batch() | ||
| count = 0 | ||
|
|
||
| for event in session.events: | ||
| if not event.content or not event.content.parts: | ||
| continue | ||
|
|
||
| text = " ".join([part.text for part in event.content.parts if part.text]) | ||
| if not text: | ||
| continue | ||
|
|
||
| keywords = self._extract_keywords(text) | ||
| if not keywords: | ||
| continue | ||
|
|
||
| doc_ref = self.client.collection(self.memories_collection).document() | ||
| batch.set( | ||
| doc_ref, | ||
| { | ||
| "appName": session.app_name, | ||
| "userId": session.user_id, | ||
| "keywords": list(keywords), | ||
| "author": event.author, | ||
| "content": event.content.model_dump( | ||
| exclude_none=True, mode="json" | ||
| ), | ||
| "timestamp": event.timestamp, | ||
| }, | ||
| ) | ||
| count += 1 | ||
| if count >= 500: | ||
| await batch.commit() | ||
| batch = self.client.batch() | ||
| count = 0 | ||
|
|
||
| if count > 0: | ||
| await batch.commit() | ||
ScottMansfield marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| def _extract_keywords(self, text: str) -> set[str]: | ||
| """Extracts keywords from text, ignoring stop words.""" | ||
| words = re.findall(r"[A-Za-z]+", text.lower()) | ||
| return {word for word in words if word not in self.stop_words} | ||
|
|
||
| async def _search_by_keyword( | ||
| self, app_name: str, user_id: str, keyword: str | ||
| ) -> list[MemoryEntry]: | ||
| """Searches for events matching a single keyword.""" | ||
| query = ( | ||
| self.client.collection(self.memories_collection) | ||
| .where(filter=FieldFilter("appName", "==", app_name)) | ||
| .where(filter=FieldFilter("userId", "==", user_id)) | ||
| .where(filter=FieldFilter("keywords", "array_contains", keyword)) | ||
| ) | ||
|
|
||
| docs = await query.get() | ||
| entries = [] | ||
| for doc in docs: | ||
| data = doc.to_dict() | ||
| if data and "content" in data: | ||
| try: | ||
| from google.genai import types | ||
|
|
||
| content = types.Content.model_validate(data["content"]) | ||
| entries.append( | ||
| MemoryEntry( | ||
| content=content, | ||
| author=data.get("author", ""), | ||
| timestamp=_utils.format_timestamp(data.get("timestamp", 0.0)), | ||
| ) | ||
| ) | ||
| except Exception as e: | ||
| logger.warning(f"Failed to parse memory entry: {e}") | ||
|
|
||
| return entries | ||
|
|
||
| @override | ||
| async def search_memory( | ||
| self, *, app_name: str, user_id: str, query: str | ||
| ) -> SearchMemoryResponse: | ||
| """Searches memory for events matching the query.""" | ||
| keywords = self._extract_keywords(query) | ||
| if not keywords: | ||
| return SearchMemoryResponse() | ||
|
|
||
| tasks = [ | ||
| self._search_by_keyword(app_name, user_id, keyword) | ||
| for keyword in keywords | ||
| ] | ||
| results = await asyncio.gather(*tasks, return_exceptions=True) | ||
|
|
||
| seen = set() | ||
| memories = [] | ||
| for result_list in results: | ||
| if isinstance(result_list, BaseException): | ||
| logger.warning(f"Memory keyword search partial failure: {result_list}") | ||
| continue | ||
| for entry in result_list: | ||
| content_text = "" | ||
| if entry.content and entry.content.parts: | ||
| content_text = " ".join( | ||
| [part.text for part in entry.content.parts if part.text] | ||
| ) | ||
| key = (entry.author, content_text, entry.timestamp) | ||
| if key not in seen: | ||
| seen.add(key) | ||
| memories.append(entry) | ||
|
|
||
| return SearchMemoryResponse(memories=memories) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.