[SS-69] user-facing docs for Iceberg sink, now with GCP support#36773
[SS-69] user-facing docs for Iceberg sink, now with GCP support#36773ublubu wants to merge 5 commits into
Conversation
ublubu
force-pushed
the
kynan/iceberg-gcp-doc
branch
2 times, most recently
from
3386be7 to
81cac44
Compare
ublubu
force-pushed
the
kynan/iceberg-gcp-doc
branch
from
81cac44 to
f582bfa
Compare
ublubu
force-pushed
the
kynan/iceberg-gcp-doc
branch
from
1f326ad to
07ae4e1
Compare
|
|
||
| The name of the user to connect as. | ||
|
|
||
| - name: "syntax-gcp" |
There was a problem hiding this comment.
Q: How does this affect https://preview.materialize.com/materialize/36773/sql/create-connection/#s3-compatible-object-storage where we mention Google Cloud Storage ... ?
| AWS CONNECTION = <aws_connection>, | ||
| GCP CONNECTION = <gcp_connection>, |
There was a problem hiding this comment.
Kind of like what you did with the examples, I think probably might be cleaner splitting up the syntax and tabbing them.
| As data changes in Materialize, the corresponding Iceberg tables are | ||
| automatically kept up to date. You can sink data from a materialized view, a | ||
| source, or a table. | ||
| Iceberg sinks provide exactly once delivery of updates from Materialize into |
There was a problem hiding this comment.
Could we actually split this page into two?
The current iceberg.md can be a landing page that points people to iceberg-aws and iceberg-gcp or something like that?
All the steps in tabs makes it kind of confusing.
| ```bash | ||
| base64 < key.json | ||
| ``` | ||
|
|
||
| ```mzsql | ||
| CREATE SECRET gcp_service_account_key | ||
| AS decode('<base64-encoded service account key JSON>', 'base64'); | ||
| ``` | ||
|
|
||
| We recommend you base64-encode the key avoid pasting a multi-line JSON literal into SQL. |
There was a problem hiding this comment.
wondering if we can genericize this and include in the syntax table.
And for this specific example ... just make it part of the gcp example below. ... since it references the gcp_service_account_key.
Also, missing the word "to" in front of "avoid"
| specify access parameters to your Iceberg catalog. | ||
| To create an Iceberg sink, you need an [Iceberg catalog | ||
| connection](/sql/create-connection/#iceberg-catalog) that specifies access | ||
| parameters to your Iceberg catalog. |
There was a problem hiding this comment.
In the syntax section below ... maybe add a headless content that blurbs that the syntax has changed to remove the unused USING AWS CONNECTION and we can include it here and in the iceberg sink tutorial (and when the release notes get generated, we can include it in there possibly).
| WAREHOUSE = '<warehouse>', | ||
| AWS CONNECTION = <aws_connection>, | ||
| GCP CONNECTION = <gcp_connection>, | ||
| SCOPE = '<scope>' |
There was a problem hiding this comment.
What's SCOPE? We don't include SCOPE in the syntax_elements.
| base64 < key.json | ||
| ``` | ||
|
|
||
| The `decode(..., 'base64')` form below lets you paste the encoded value |
There was a problem hiding this comment.
Move this sentence to step 2?
| The `decode(..., 'base64')` form below lets you paste the encoded value | ||
| into SQL without worrying about embedded newlines or quotes in the JSON. | ||
|
|
||
| 2. Store the key in a Materialize [secret](/sql/create-secret/): |
There was a problem hiding this comment.
Store the base64-encoded key ...
| ); | ||
| testable: false | ||
|
|
||
| - name: "example-iceberg-catalog-gcp-connection" |
There was a problem hiding this comment.
Purely optional. Since the example in the iceberg tutorial and the create sink page seem to be the same content (just different presentation ... either steps or one big code block), you could just single source from the yaml. (but, we likely won't be making tweaks too frequently so ..eh).
| Use `CREATE SINK ... INTO ICEBERG CATALOG...` to create Iceberg sinks. Iceberg sinks write data from Materialize into an Iceberg table hosted on | ||
| AWS S3 Tables. As data changes in Materialize, your Iceberg tables are | ||
| automatically kept up to date. | ||
| Use `CREATE SINK ... INTO ICEBERG CATALOG...` to create Iceberg sinks. Iceberg |
There was a problem hiding this comment.
Could we also fix the following page: https://github.com/MaterializeInc/materialize/blob/main/doc/user/content/sql/create-sink/_index.md ?
We seem to have broken it when we made changes to the syntax name changes for the append mode.
2 participants
Adding docs for:
Small changes to docs for Iceberg sink.