Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
- **Remote GFQL sends the resolved strictness level (#1916)**: `gfql_remote()` previously hardcoded its client-side preflight to `strict=False` and sent the server nothing, so the same query was strict locally and loose remotely. The preflight now honors the resolved level, and the request body carries a new `strictness` field (`"strict"` / `"warn"` / `"quiet"`) alongside the existing `engine` field. **Server-side honoring is a server change and is not in this repository**: a server that does not read `strictness` applies its own default, so a non-default level requested remotely warns once, in the same shape as the existing Let/DAG compatibility warning (#1955). A client holding only a `dataset_id` can now also preflight names when `bind(schema=...)` supplied them, since a declared schema is names without data.

### Fixed
- **Explicit unsupported remote engines now decline before side effects (#1957 completion)**: `gfql_remote` and `python_remote` preserve explicit `pandas` and `cudf` requests on the wire, while unsupported requests such as `polars` and `polars-gpu` raise typed `GFQLRemoteError` `E405` before credential refresh, upload, or POST. This is the remote service boundary, distinct from the release's local `polars`/`polars-gpu` GFQL engines; the existing `engine='auto'` policy is unchanged.
- **Strict GFQL validation now rejects relationship types that the edge schema proves absent (#1916)**: a strict binder previously deferred any relationship type when its catalog listed no known types, even when the edge schema had no generic `type` carrier, so `gfql_validate()` passed a query that execution rejected. It now raises typed `E301` when absence is provable. A generic `type` carrier with no declared catalog remains unjudgeable without scanning values and still defers, while an explicitly empty declared catalog is judgeable and rejects. Focused validator/executor and binder tests pin all three boundaries.
- **Cross-kind `WITH` whole-entity rebinds now fail early with typed `E108` (#1937)**: the local Cypher compiler guarded node-to-node and edge-to-edge rebinds but let a bare MATCH-bound node alias take a live edge alias's name, or the reverse, which could resolve rows and properties against different bindings. The guard now rejects any bare entity alias renamed onto another live pattern alias at compile time. Carries and self-renames, fresh targets, scalar/property shadows, terminal `RETURN` renames, `WITH`-to-`MATCH` reentry, and earlier, more specific validation errors keep their existing behavior. Focused tests pin both cross-kind error directions and the adjacent valid and precedence boundaries.
- **All-null Boolean `sum()` on `engine='polars-gpu'` now returns integer zero instead of null (#1997)**: cudf-polars 26.02 reports null for an all-null Boolean reduction, while GFQL's documented Boolean aggregate extension follows the Cypher `sum()` empty-input identity and returns `0`. The result normalization now fills only Boolean `sum` before the shared Int64 cast; Boolean `count` and non-Boolean `sum` keep their existing null behavior at this helper boundary. Direct boundary tests pin the positive cell and both negative controls.
Expand Down
23 changes: 10 additions & 13 deletions graphistry/compute/chain_remote.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import warnings
import zipfile

from graphistry.Engine import Engine, EngineAbstractType, resolve_input_engine
from graphistry.Engine import EngineAbstractType
from graphistry.Plottable import Plottable
from graphistry.client_session import DatasetInfo
from graphistry.compute.ast import ASTLet, ASTObject
Expand All @@ -22,7 +22,10 @@
from graphistry.io.metadata import deserialize_plottable_metadata
from graphistry.compute.exceptions import ErrorCode, GFQLSyntaxError, GFQLTypeError
from graphistry.compute.remote_df_io import (
require_supported_frame_library, resolve_csv_reader, validate_csv_import_args)
require_supported_frame_library,
resolve_csv_reader,
resolve_remote_engine,
validate_csv_import_args)
from graphistry.compute.remote_response import (
check_subset_result_bindings,
decode_json_result,
Expand Down Expand Up @@ -150,20 +153,10 @@ def chain_remote_generic(

strict_level = resolve_strict_level(self, strict=strict)

if not api_token:
self._pygraphistry.refresh()
api_token = self.session.api_token

Comment thread
lmeyerov marked this conversation as resolved.
if output_type not in output_types_graph:
raise ValueError(f"Unknown output_type, expected one of {output_types_graph}, got: {output_type}")

# Resolve engine: auto -> pandas/cudf based on graph DataFrame type
engine_resolved = resolve_input_engine(engine, self)
if engine_resolved not in [Engine.PANDAS, Engine.CUDF]:
raise ValueError(f"Remote GFQL only supports 'pandas' or 'cudf' engines (or 'auto' which resolves to one of them). "
f"Got engine='{engine}' which resolved to '{engine_resolved.value}'. "
f"Dask engines are not supported for remote execution.")
engine_str = engine_resolved.value
engine_str = resolve_remote_engine(engine, self, "gfql_remote").value

if format is None:
if output_type == "shape":
Expand Down Expand Up @@ -234,6 +227,10 @@ def chain_remote_generic(
schema=False,
)

if not api_token:
self._pygraphistry.refresh()
api_token = self.session.api_token

if not dataset_id:
dataset_id = self._dataset_id

Expand Down
1 change: 1 addition & 0 deletions graphistry/compute/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ class ErrorCode:
E402 = "remote-response-malformed"
E403 = "remote-format-lossy"
E404 = "remote-unsupported-frames"
E405 = "remote-unsupported-engine"

# Graph constructor errors (E150-E159)
E150 = "duplicate-graph-binding"
Expand Down
20 changes: 8 additions & 12 deletions graphistry/compute/python_remote.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,13 @@
import pandas as pd
import requests

from graphistry.Engine import Engine, EngineAbstractType, resolve_input_engine
from graphistry.Engine import EngineAbstractType
from graphistry.Plottable import Plottable
from graphistry.compute.remote_df_io import (
require_supported_frame_library, resolve_csv_reader, validate_csv_import_args)
require_supported_frame_library,
resolve_csv_reader,
resolve_remote_engine,
validate_csv_import_args)
from graphistry.compute.remote_response import (
decode_json_body,
decode_json_result,
Expand Down Expand Up @@ -132,6 +135,9 @@ def task(g: Plottable) -> Dict[str, Any]:

validate_csv_import_args(df_import_args, "python_remote")
frame_lib = require_supported_frame_library(self._nodes, self._edges, "python_remote")
engine_str = resolve_remote_engine(engine, self, "python_remote").value

assert format in ["json", "csv", "parquet"], f"format should be 'json', 'csv', or 'parquet', got: {format}"

if validate:
if not validate_python_str(code):
Expand All @@ -150,16 +156,6 @@ def task(g: Plottable) -> Dict[str, Any]:

if not dataset_id:
raise ValueError("Missing dataset_id; either pass in, or call on g2=g1.plot(render='g') in api=3 mode ahead of time")

assert format in ["json", "csv", "parquet"], f"format should be 'json', 'csv', or 'parquet', got: {format}"
Comment thread
lmeyerov marked this conversation as resolved.

# Resolve engine: auto -> pandas/cudf based on graph DataFrame type
engine_resolved = resolve_input_engine(engine, self)
if engine_resolved not in [Engine.PANDAS, Engine.CUDF]:
raise ValueError(f"Remote Python execution only supports 'pandas' or 'cudf' engines (or 'auto' which resolves to one of them). "
f"Got engine='{engine}' which resolved to '{engine_resolved.value}'. "
f"Dask engines are not supported for remote execution.")
engine_str = engine_resolved.value

# TODO remove auto-indent when server updated
# workaround parsing bug by indenting each line by 4 spaces
Expand Down
49 changes: 42 additions & 7 deletions graphistry/compute/remote_df_io.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,20 @@
``parquet`` carries an Arrow schema and is the faithful default.
"""
from inspect import getmodule
import typing
import warnings
from typing import BinaryIO, Callable, List, Optional
from typing import BinaryIO, Callable, Optional
from typing_extensions import Literal

from graphistry.Engine import Engine, EngineAbstract, EngineAbstractType, resolve_input_engine
from graphistry.Plottable import Plottable
from graphistry.compute.exceptions import ErrorCode, GFQLRemoteError
from graphistry.compute.typing import DataFrameT
from graphistry.models.compute.chain_remote import DFImportArgs

RemoteAPIName = Literal["gfql_remote", "python_remote"]



CSV_DTYPE_KWARGS = frozenset({'converters', 'dtype'})
CSV_NA_KWARGS = frozenset({'converters', 'keep_default_na', 'na_filter', 'na_values'})
Expand Down Expand Up @@ -46,8 +53,36 @@ def _is_pandas_like(df: Optional[DataFrameT]) -> bool:
return df is None or isinstance(df, pd.DataFrame) or 'unittest.mock' in str(type(df))


def resolve_remote_engine(
engine: EngineAbstractType,
graph: Plottable,
api_name: RemoteAPIName,
) -> Engine:
"""Resolve a supported remote engine before auth, upload, or transport.

:param engine: Requested engine or ``auto``.
:param graph: Graph used to resolve ``auto``.
:param api_name: Public entry point named in the error message.
:return: A pandas or cudf engine.
:raises GFQLRemoteError: When the resolved engine is not supported remotely.
"""
resolved = resolve_input_engine(engine, graph)
if resolved in (Engine.PANDAS, Engine.CUDF):
Comment thread
lmeyerov marked this conversation as resolved.
return resolved

requested = engine.value if isinstance(engine, EngineAbstract) else engine
raise GFQLRemoteError(
ErrorCode.E405,
f"{api_name}: remote execution supports only 'pandas' and 'cudf' engines; "
f"requested {requested!r}, which resolved to {resolved.value!r}.",
field="engine",
value=requested,
suggestion="Use engine='pandas', engine='cudf', or engine='auto' with supported frames.",
)


def require_supported_frame_library(
nodes: Optional[DataFrameT], edges: Optional[DataFrameT], api_name: str
nodes: Optional[DataFrameT], edges: Optional[DataFrameT], api_name: RemoteAPIName
) -> str:
"""Resolve which DataFrame library backs a remote call, before any request is sent.

Expand All @@ -71,7 +106,7 @@ def require_supported_frame_library(

def validate_csv_import_args(
df_import_args: Optional[DFImportArgs],
api_name: str,
api_name: RemoteAPIName,
) -> None:
"""Reject a malformed ``df_import_args`` before any request is sent.

Expand All @@ -89,14 +124,14 @@ def validate_csv_import_args(
)


def ungoverned_csv_axes(df_import_args: Optional[DFImportArgs]) -> List[str]:
def ungoverned_csv_axes(df_import_args: Optional[DFImportArgs]) -> typing.List[str]:
"""Name the lossy csv axes the caller's reader kwargs do not govern.

:param df_import_args: Caller-supplied reader kwargs, or ``None``.
:return: Zero, one, or two axis descriptions; empty means the read is under caller control.
"""
keys = set(df_import_args or {})
axes: List[str] = []
axes: typing.List[str] = []
if not (keys & CSV_DTYPE_KWARGS):
axes.append(CSV_DTYPE_AXIS_WARNING)
if not (keys & CSV_NA_KWARGS):
Expand All @@ -106,7 +141,7 @@ def ungoverned_csv_axes(df_import_args: Optional[DFImportArgs]) -> List[str]:

def resolve_csv_import_args(
df_import_args: Optional[DFImportArgs],
api_name: str,
api_name: RemoteAPIName,
) -> DFImportArgs:
"""Resolve csv reader kwargs, warning per lossy axis the caller left to inference.

Expand All @@ -130,7 +165,7 @@ def resolve_csv_import_args(
def resolve_csv_reader(
read_csv: Callable[..., DataFrameT],
df_import_args: Optional[DFImportArgs],
api_name: str,
api_name: RemoteAPIName,
) -> Callable[[BinaryIO], DataFrameT]:
"""Bind a csv reader that applies the caller's explicit reader kwargs.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@
"test_const_fold_engine_parity.py, which IS in the lane -- and is what caught the "
"engine-blind key in the first place"
),
"graphistry/tests/compute/test_remote_engine_contract.py": (
"remote preflight contract only: 'polars' and 'polars-gpu' are plain engine strings "
"that must be rejected before upload or POST; the module imports no polars runtime "
"and builds no polars frame, so every test runs in the ordinary core lanes"
),
"graphistry/tests/compute/gfql/index/test_index_gpu_edge_match.py": (
"cudf/GPU-gated (module-level importorskip('cudf') + skipif no GPU), not polars-gated; "
"belongs to the separate GPU-lane gap, and the polars CPU lane could not run it"
Expand Down
127 changes: 127 additions & 0 deletions graphistry/tests/compute/test_remote_engine_contract.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
"""Contract tests for explicit engines on remote compute calls."""

import typing
from typing import Optional
from unittest.mock import MagicMock, patch
from typing_extensions import Literal

import pandas as pd
import pytest

from graphistry.Engine import EngineAbstractType
from graphistry.Plottable import Plottable
from graphistry.compute.ast import ASTNode
from graphistry.compute.chain import Chain
from graphistry.compute.chain_remote import chain_remote_generic
from graphistry.compute.exceptions import ErrorCode, GFQLRemoteError
from graphistry.compute.python_remote import python_remote_generic
from graphistry.compute.remote_df_io import RemoteAPIName


TASK = "def task(g):\n return g\n"
QUERY = Chain([ASTNode(filter_dict={"type": "Person"})])
_PostTarget = Literal[
"graphistry.compute.chain_remote.requests.post",
"graphistry.compute.python_remote.requests.post",
]



class Posted(Exception):
"""Stop a test after the request reaches the mocked transport."""


def mock_plottable(dataset_id: Optional[str] = None) -> MagicMock:
"""Build the minimum graph state used by both remote entry points."""
graph = MagicMock()
graph._dataset_id = dataset_id
graph._edges = pd.DataFrame({"s": [0], "d": [1]})
graph._nodes = pd.DataFrame({"id": [0, 1]})
graph._privacy = None
graph._url_params = {}
graph.session.api_token = "refreshed-token"
graph.session.certificate_validation = True
graph.base_url_server.return_value = "https://test.graphistry.com"

def upload(*, validate: bool) -> MagicMock:
graph._dataset_id = "uploaded-dataset"
return graph

graph.upload.side_effect = upload
return graph


def call_remote(
api_name: RemoteAPIName,
graph: Plottable,
engine: EngineAbstractType,
*,
with_creds: bool,
) -> typing.NoReturn:
"""Call one remote entry point with matching mock credentials."""
api_token = "token" if with_creds else None
dataset_id = "dataset" if with_creds else None
if api_name == "gfql_remote":
chain_remote_generic(
graph,
QUERY,
api_token=api_token,
dataset_id=dataset_id,
engine=engine,
format="json",
validate=False,
)
else:
python_remote_generic(
graph,
TASK,
api_token=api_token,
dataset_id=dataset_id,
engine=engine,
format="json",
output_type="json",
validate=False,
)
raise AssertionError("remote call returned before transport")


@pytest.mark.parametrize(
("api_name", "post_target"),
[
("gfql_remote", "graphistry.compute.chain_remote.requests.post"),
("python_remote", "graphistry.compute.python_remote.requests.post"),
],
)
@pytest.mark.parametrize("engine", ["pandas", "cudf"])
def test_explicit_supported_engine_is_sent_unchanged(
api_name: RemoteAPIName, post_target: _PostTarget, engine: EngineAbstractType
) -> None:
graph = mock_plottable("dataset")
with patch(post_target, side_effect=Posted) as post:
with pytest.raises(Posted):
call_remote(api_name, graph, engine, with_creds=True)
assert post.call_args.kwargs["json"]["engine"] == engine


@pytest.mark.parametrize(
("api_name", "post_target"),
[
("gfql_remote", "graphistry.compute.chain_remote.requests.post"),
("python_remote", "graphistry.compute.python_remote.requests.post"),
],
)
@pytest.mark.parametrize("engine", ["polars", "polars-gpu"])
def test_explicit_unsupported_engine_declines_before_side_effects(
api_name: RemoteAPIName, post_target: _PostTarget, engine: EngineAbstractType
) -> None:
graph = mock_plottable()
with patch(post_target) as post:
with pytest.raises(GFQLRemoteError) as excinfo:
call_remote(api_name, graph, engine, with_creds=False)

assert excinfo.value.code == ErrorCode.E405
assert excinfo.value.context["field"] == "engine"
assert excinfo.value.context["value"] == engine
graph._pygraphistry.refresh.assert_not_called()
graph.upload.assert_not_called()
post.assert_not_called()
15 changes: 11 additions & 4 deletions graphistry/tests/test_engine_frame_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,9 +216,10 @@ def test_remote_surfaces_resolve_to_supported_engine(self):
@polars_only
@pytest.mark.parametrize("surface", ["circle", "fa2", "chain_remote", "python_remote", "cluster"])
def test_pandas_computing_surfaces_use_input_resolver(self, surface):
"""The convention itself, pinned per module: a pandas-computing surface
must not call resolve_engine (modern AUTO) -- migrating one to native
polars means deliberately flipping it back and deleting its row here."""
"""Pin pandas-computing surfaces to the input resolver, directly or via
the remote-only wrapper. They must not call modern resolve_engine;
migrating one to native polars means deliberately flipping it back and
deleting its row here."""
import importlib
mod = importlib.import_module({
"circle": "graphistry.layout.circle",
Expand All @@ -229,5 +230,11 @@ def test_pandas_computing_surfaces_use_input_resolver(self, surface):
}[surface])
import inspect
src = inspect.getsource(mod)
assert "resolve_input_engine" in src
if surface in ("chain_remote", "python_remote"):
from graphistry.compute.remote_df_io import resolve_remote_engine

assert "resolve_remote_engine" in src
assert "resolve_input_engine" in inspect.getsource(resolve_remote_engine)
else:
assert "resolve_input_engine" in src
assert "resolve_engine(" not in src.replace("resolve_input_engine(", "")
Loading