-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Add Ecto.Query.label/2 query expression for tagging queries with an S… #4732
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| import Kernel, except: [apply: 2] | ||
|
|
||
| defmodule Ecto.Query.Builder.Label do | ||
| @moduledoc false | ||
|
|
||
| alias Ecto.Query.Builder | ||
|
|
||
| @forbidden ["*/", "/*", <<0>>] | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should let the adapters do the escaping in case some adapters use a different style? |
||
|
|
||
| @doc """ | ||
| Escapes the label text. | ||
|
|
||
| iex> escape(quote(do: "my-query")) | ||
| "my-query" | ||
|
|
||
| """ | ||
| @spec escape(Macro.t()) :: Macro.t() | ||
| def escape(label) when is_binary(label) do | ||
| if String.contains?(label, @forbidden) do | ||
| Builder.error!(forbidden_message(label)) | ||
| end | ||
|
|
||
| label | ||
| end | ||
|
|
||
| def escape({:^, _, [expr]}) do | ||
| quote do | ||
| Ecto.Query.Builder.Label.runtime!(unquote(expr)) | ||
| end | ||
| end | ||
|
|
||
| def escape(other) do | ||
| Builder.error!( | ||
| "`#{Macro.to_string(other)}` is not a valid label. " <> | ||
| "For security reasons, a label must be a literal string or an interpolated string" | ||
| ) | ||
| end | ||
|
|
||
| @doc """ | ||
| Validates a label given at runtime via interpolation. | ||
| """ | ||
| @spec runtime!(term) :: String.t() | ||
| def runtime!(label) when is_binary(label) do | ||
| if String.contains?(label, @forbidden) do | ||
| raise ArgumentError, forbidden_message(label) | ||
| end | ||
|
|
||
| label | ||
| end | ||
|
|
||
| def runtime!(other) do | ||
| raise ArgumentError, "a label must be a string, got: `#{inspect(other)}`" | ||
| end | ||
|
|
||
| defp forbidden_message(label) do | ||
| "a label cannot contain `/*`, `*/`, or null bytes, got: `#{inspect(label)}`. " | ||
| end | ||
|
|
||
| @doc """ | ||
| Builds a quoted expression. | ||
|
|
||
| The quoted expression should evaluate to a query at runtime. | ||
| If possible, it does all calculations at compile time to avoid | ||
| runtime work. | ||
| """ | ||
| @spec build(Macro.t(), Macro.t(), Macro.Env.t()) :: Macro.t() | ||
| def build(query, expr, env) do | ||
| Builder.apply_query(query, __MODULE__, [escape(expr)], env) | ||
| end | ||
|
|
||
| @doc """ | ||
| The callback applied by `build/3` to build the query. | ||
| """ | ||
| @spec apply(Ecto.Queryable.t(), term) :: Ecto.Query.t() | ||
| def apply(%Ecto.Query{} = query, value) do | ||
| %{query | label: value} | ||
| end | ||
|
|
||
| def apply(query, value) do | ||
| apply(Ecto.Queryable.to_query(query), value) | ||
| end | ||
| end | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1581,6 +1581,13 @@ defmodule Ecto.Repo do | |
| in the schema. For more information see the ["Query Prefix"](`m:Ecto.Query#module-query-prefix`) section of the | ||
| `Ecto.Query` documentation. | ||
|
|
||
| * `:label` - A string to tag the generated `UPDATE` with a leading SQL | ||
| comment, such as `/* bump_visits */ UPDATE ...`, to identify the statement | ||
| in database logs and monitoring tools. The string is embedded verbatim into | ||
| the SQL and therefore cannot contain `/*`, `*/`, or null bytes. Prefer a | ||
| stable identifier per call site, as the label becomes part of the query | ||
| cache key. See `Ecto.Query.label/2` to set it on the query itself instead. | ||
|
|
||
| See the ["Shared options"](#module-shared-options) section at the module | ||
| documentation for remaining options. | ||
|
|
||
|
|
@@ -1630,6 +1637,13 @@ defmodule Ecto.Repo do | |
| in the schema. For more information see the ["Query Prefix"](`m:Ecto.Query#module-query-prefix`) section of the | ||
| `Ecto.Query` documentation. | ||
|
|
||
| * `:label` - A string to tag the generated `DELETE` with a leading SQL | ||
| comment, such as `/* purge_stale */ DELETE ...`, to identify the statement | ||
| in database logs and monitoring tools. The string is embedded verbatim into | ||
| the SQL and therefore cannot contain `/*`, `*/`, or null bytes. Prefer a | ||
| stable identifier per call site, as the label becomes part of the query | ||
| cache key. See `Ecto.Query.label/2` to set it on the query itself instead. | ||
|
|
||
| See the ["Shared options"](#module-shared-options) section at the module | ||
| documentation for remaining options. | ||
|
|
||
|
|
@@ -1732,6 +1746,13 @@ defmodule Ecto.Repo do | |
| * `:placeholders` - A map with placeholders. This feature is not supported | ||
| by all databases. See the ["Placeholders" section](#c:insert_all/3-placeholders) for more information. | ||
|
|
||
| * `:label` - A string to tag the generated `INSERT` with a leading SQL | ||
| comment, such as `/* import_users */ INSERT ...`, to identify the statement | ||
| in database logs and monitoring tools. The string is embedded verbatim into | ||
| the SQL and therefore cannot contain `/*`, `*/`, or null bytes. Prefer a | ||
| stable identifier per call site, as the label becomes part of the query | ||
| cache key. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we should support it as a repo option and query option, let's go with one or the other.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It reads more like a query option than a repo option: But Repo.insert/update/delete get a struct/changeset and insert_all gets a list of maps, which is why we ended up with the mixed version. We could switch to the repo option everywhere (same as :prefix). What are your thoughts?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We already have :comment on Postgrex, so if that's your goal, we can add
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The truth is that the comment should be at the beginning of the query. By default in Postgres (for instance) track_activity_query_size == 1024 bytes, long queries are truncated (in pg_stat_statements, pg_stat_activity and, as a result, in GCP Query Insights console). Changing track_activity_query_size requires a DB server restart.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is at the end because that’s what sqlcommenter uses. Does it also work at the beginning? Another option is to have a flag to control it. In any case, the disabling of query caching is not related to the position. If we implement it in Ecto and make it part of query cache, then it can work regardless. One option is to support precomment and postcomment options, and deprecate the Postgres comment one.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did similar change some time ago master...dkuku:ecto:comments_in_query
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. With sqlcommenter, comments are dynamic, which is why the prepared-statement cache must be disabled.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
They are dynamic in some sense but, once the code is compiled, you have a limited set of callers to Ecto.Repo. So it could go through the cache. Unless, of course, you start adding dynamic content to it (such as One option is for us to add two macros: pre_comment and post_comment, that comment before and after (although we store it in a single field called
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How about adding a :cache option ?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did work on this some time ago, and one idea I had was to allow caching the query when the comment is:
This way, by design, you have a limited set of tags for a single query. If you are generating atoms dynamically, you probably have bigger problems in your application, and standard code inspection tools would catch it. |
||
|
|
||
| See the ["Shared options"](#module-shared-options) section at the module | ||
| documentation for remaining options. | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| Code.require_file "../../../support/eval_helpers.exs", __DIR__ | ||
|
|
||
| defmodule Ecto.Query.Builder.LabelTest do | ||
| use ExUnit.Case, async: true | ||
|
|
||
| import Ecto.Query.Builder.Label | ||
| doctest Ecto.Query.Builder.Label | ||
|
|
||
| import Ecto.Query | ||
| import Support.EvalHelpers | ||
|
|
||
| test "label with literal string" do | ||
| query = %Ecto.Query{} |> label("my-report") | ||
| assert query.label == "my-report" | ||
| end | ||
|
|
||
| test "label via keyword syntax" do | ||
| query = from p in "posts", label: "list-posts" | ||
| assert query.label == "list-posts" | ||
| end | ||
|
|
||
| test "label with interpolated string" do | ||
| report = "monthly-report" | ||
| query = %Ecto.Query{} |> label(^report) | ||
| assert query.label == "monthly-report" | ||
| end | ||
|
|
||
| test "overrides on duplicated label" do | ||
| query = %Ecto.Query{} |> label("FOO") |> label("BAR") | ||
| assert query.label == "BAR" | ||
| end | ||
|
|
||
| test "raises on non-string at compile time" do | ||
| assert_raise Ecto.Query.CompileError, ~r"`1` is not a valid label", fn -> | ||
| quote_and_eval(%Ecto.Query{} |> label(1)) | ||
| end | ||
| end | ||
|
|
||
| test "raises on literal containing */" do | ||
| assert_raise Ecto.Query.CompileError, ~r"cannot contain `/\*`, `\*/`, or null bytes", fn -> | ||
| quote_and_eval(%Ecto.Query{} |> label("evil */ DROP TABLE")) | ||
| end | ||
| end | ||
|
|
||
| test "raises on literal containing /*" do | ||
| assert_raise Ecto.Query.CompileError, ~r"cannot contain `/\*`, `\*/`, or null bytes", fn -> | ||
| quote_and_eval(%Ecto.Query{} |> label("evil /* nested")) | ||
| end | ||
| end | ||
|
|
||
| test "raises on literal containing a null byte" do | ||
| assert_raise Ecto.Query.CompileError, ~r"cannot contain `/\*`, `\*/`, or null bytes", fn -> | ||
| quote_and_eval(%Ecto.Query{} |> label("nul\0byte")) | ||
| end | ||
| end | ||
|
|
||
| test "raises on interpolated value containing a null byte" do | ||
| evil = "nul\0byte" | ||
|
|
||
| assert_raise ArgumentError, ~r"cannot contain `/\*`, `\*/`, or null bytes", fn -> | ||
| %Ecto.Query{} |> label(^evil) | ||
| end | ||
| end | ||
|
|
||
| test "raises on interpolated value containing */" do | ||
| evil = "evil */ DROP TABLE" | ||
|
|
||
| assert_raise ArgumentError, ~r"cannot contain `/\*`, `\*/`, or null bytes", fn -> | ||
| %Ecto.Query{} |> label(^evil) | ||
| end | ||
| end | ||
|
|
||
| test "raises on interpolated value containing /*" do | ||
| evil = "evil /* nested" | ||
|
|
||
| assert_raise ArgumentError, ~r"cannot contain `/\*`, `\*/`, or null bytes", fn -> | ||
| %Ecto.Query{} |> label(^evil) | ||
| end | ||
| end | ||
|
|
||
| test "raises on interpolated non-string" do | ||
| not_a_string = 123 | ||
|
|
||
| assert_raise ArgumentError, ~r"must be a string", fn -> | ||
| %Ecto.Query{} |> label(^not_a_string) | ||
| end | ||
| end | ||
|
|
||
| test "exclude resets the label" do | ||
| query = %Ecto.Query{} |> label("FOO") | ||
| assert Ecto.Query.exclude(query, :label).label == nil | ||
| end | ||
| end |

There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe this should be in an admonition block. there have been a few times the past couple of years where we had to add workarounds for people whose cache's blew up without them expecting it.
maybe we should also highlight that if they want to use high cardinality values they can disable the cache for that query. i never used sql comments for logging though so i'm not too sure if this is some kind of extremely bad practice that we shouldn't be encouraging at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@josevalim If comments are always prepended do we need to make them part of the cache? Can we just leave them out and then prepend them to the cached sql in the adapter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They need to be part of the cache, yes!