diff --git a/website/www/site/content/en/documentation/io/managed-io.md b/website/www/site/content/en/documentation/io/managed-io.md index 70b5efe9ed17..28a3be80ee73 100644 --- a/website/www/site/content/en/documentation/io/managed-io.md +++ b/website/www/site/content/en/documentation/io/managed-io.md @@ -59,30 +59,31 @@ and Beam SQL is invoked via the Managed API under the hood. Write Configuration - DELTA - - table (str)
- hadoop_config (map[str, str])
- timestamp (str)
- version (int64)
- - - Unavailable - - - - DELTA_CDC + KAFKA - table (str)
- start_version (int64)
- start_timestamp (str)
- end_version (int64)
- end_timestamp (str)
- hadoop_config (map[str, str])
- include_metadata_columns (list[str])
+ bootstrap_servers (str)
+ topic (str)
+ allow_duplicates (boolean)
+ confluent_schema_registry_subject (str)
+ confluent_schema_registry_url (str)
+ consumer_config_updates (map[str, str])
+ file_descriptor_path (str)
+ format (str)
+ message_name (str)
+ offset_deduplication (boolean)
+ redistribute_by_record_key (boolean)
+ redistribute_num_keys (int32)
+ redistributed (boolean)
+ schema (str)
- Unavailable + bootstrap_servers (str)
+ format (str)
+ topic (str)
+ file_descriptor_path (str)
+ message_name (str)
+ producer_config_updates (map[str, str])
+ schema (str)
@@ -111,34 +112,7 @@ and Beam SQL is invoked via the Managed API under the hood. sort_fields (list[str])
table_properties (map[str, str])
triggering_frequency_seconds (int32)
- - - - KAFKA - - bootstrap_servers (str)
- topic (str)
- allow_duplicates (boolean)
- confluent_schema_registry_subject (str)
- confluent_schema_registry_url (str)
- consumer_config_updates (map[str, str])
- file_descriptor_path (str)
- format (str)
- message_name (str)
- offset_deduplication (boolean)
- redistribute_by_record_key (boolean)
- redistribute_num_keys (int32)
- redistributed (boolean)
- schema (str)
- - - bootstrap_servers (str)
- format (str)
- topic (str)
- file_descriptor_path (str)
- message_name (str)
- producer_config_updates (map[str, str])
- schema (str)
+ write_properties (map[str, str])
@@ -152,22 +126,38 @@ and Beam SQL is invoked via the Managed API under the hood. filter (str)
from_snapshot (int64)
from_timestamp (int64)
+ include_metadata_columns (list[str])
keep (list[str])
poll_interval_seconds (int32)
starting_strategy (str)
streaming (boolean)
to_snapshot (int64)
to_timestamp (int64)
+ watermark_column (str)
+ watermark_column_time_unit (str)
Unavailable - POSTGRES + DELTA + + table (str)
+ hadoop_config (map[str, str])
+ timestamp (str)
+ version (int64)
+ + + Unavailable + + + + SQLSERVER jdbc_url (str)
connection_properties (str)
+ disable_auto_commit (boolean)
fetch_size (int32)
location (str)
num_partitions (int32)
@@ -189,11 +179,10 @@ and Beam SQL is invoked via the Managed API under the hood. - SQLSERVER + POSTGRES jdbc_url (str)
connection_properties (str)
- disable_auto_commit (boolean)
fetch_size (int32)
location (str)
num_partitions (int32)
@@ -265,7 +254,7 @@ and Beam SQL is invoked via the Managed API under the hood. ## Configuration Details -### `DELTA` Read +### `KAFKA` Write
@@ -276,52 +265,85 @@ and Beam SQL is invoked via the Managed API under the hood. + + + + + + + + + + + + + + +
- table + bootstrap_servers str - Identifier of the Delta Lake table. + A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. | Format: host1:port1,host2:port2,...
- hadoop_config + format - map[str, str] + str - Properties passed to the Hadoop Configuration. + The encoding format for the data stored in Kafka. Valid options are: RAW,JSON,AVRO,PROTO
- timestamp + topic str - Timestamp of the Delta Lake table to read. + n/a
- version + file_descriptor_path - int64 + str - Version of the Delta Lake table to read. + The path to the Protocol Buffer File Descriptor Set file. This file is used for schema definition and message serialization. +
+ message_name + + str + + The name of the Protocol Buffer message to be used for schema extraction and data conversion. +
+ producer_config_updates + + map[str, str] + + A list of key-value pairs that act as configuration parameters for Kafka producers. Most of these configurations will not be needed, but if you need to customize your Kafka producer, you may use this. See a detailed list: https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html +
+ schema + + str + + n/a
-### `DELTA_CDC` Read +### `KAFKA` Read
@@ -332,168 +354,156 @@ and Beam SQL is invoked via the Managed API under the hood. -
- table + bootstrap_servers str - Identifier of the Delta Lake table. + A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form `host1:port1,host2:port2,...`
- start_version + topic - int64 + str - Start version of the Delta Lake table to read changes from. Either start_version or start_timestamp must be set. + n/a
- start_timestamp + allow_duplicates - str + boolean - Start timestamp of the Delta Lake table to read changes from. Either start_version or start_timestamp must be set. + If the Kafka read allows duplicates.
- end_version + confluent_schema_registry_subject - int64 + str - End version of the Delta Lake table to read changes up to. + n/a
- end_timestamp + confluent_schema_registry_url str - End timestamp of the Delta Lake table to read changes up to. + n/a
- hadoop_config + consumer_config_updates map[str, str] - Properties passed to the Hadoop Configuration. + A list of key-value pairs that act as configuration parameters for Kafka consumers. Most of these configurations will not be needed, but if you need to customize your Kafka consumer, you may use this. See a detailed list: https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html
- include_metadata_columns + file_descriptor_path - list[str] + str - Metadata columns to include in the output rows. Supported columns are: _change_type, _commit_version, and _commit_timestamp. + The path to the Protocol Buffer File Descriptor Set file. This file is used for schema definition and message serialization.
-
- -### `ICEBERG` Read - -
- - - - - -
ConfigurationTypeDescription
- table + format str - Identifier of the Iceberg table. + The encoding format for the data stored in Kafka. Valid options are: RAW,STRING,AVRO,JSON,PROTO
- catalog_name + message_name str - Name of the catalog containing the table. + The name of the Protocol Buffer message to be used for schema extraction and data conversion.
- catalog_properties + offset_deduplication - map[str, str] + boolean - Properties used to set up the Iceberg catalog. + If the redistribute is using offset deduplication mode.
- config_properties + redistribute_by_record_key - map[str, str] + boolean - Properties passed to the Hadoop Configuration. + If the redistribute keys by the Kafka record key.
- drop + redistribute_num_keys - list[str] + int32 - A subset of column names to exclude from reading. If null or empty, all columns will be read. + The number of keys for redistributing Kafka inputs.
- filter + redistributed - str + boolean - SQL-like predicate to filter data at scan time. Example: "id > 5 AND status = 'ACTIVE'". Uses Apache Calcite syntax: https://calcite.apache.org/docs/reference.html + If the Kafka read should be redistributed.
- keep + schema - list[str] + str - A subset of column names to read exclusively. If null or empty, all columns will be read. + The schema in which the data is encoded in the Kafka topic. For AVRO data, this is a schema defined with AVRO schema syntax (https://avro.apache.org/docs/1.10.2/spec.html#schemas). For JSON data, this is a schema defined with JSON-schema syntax (https://json-schema.org/). If a URL to Confluent Schema Registry is provided, then this field is ignored, and the schema is fetched from Confluent Schema Registry.
@@ -677,10 +687,21 @@ For more information on table properties, please visit https://iceberg.apache.or For a streaming pipeline, sets the frequency at which snapshots are produced. + + + write_properties + + + map[str, str] + + + Properties applied to the underlying file writer (e.g. Parquet write properties like 'write.parquet.bloom-filter-enabled.column.'). + +
-### `KAFKA` Read +### `ICEBERG` Read
@@ -691,162 +712,85 @@ For more information on table properties, please visit https://iceberg.apache.or - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- bootstrap_servers + table str - A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form `host1:port1,host2:port2,...` + Identifier of the Iceberg table.
- topic + catalog_name str - n/a + Name of the catalog containing the table.
- allow_duplicates + catalog_properties - boolean + map[str, str] - If the Kafka read allows duplicates. + Properties used to set up the Iceberg catalog.
- confluent_schema_registry_subject + config_properties - str + map[str, str] - n/a + Properties passed to the Hadoop Configuration.
- confluent_schema_registry_url + drop - str + list[str] - n/a + A subset of column names to exclude from reading. If null or empty, all columns will be read.
- consumer_config_updates - - map[str, str] - - A list of key-value pairs that act as configuration parameters for Kafka consumers. Most of these configurations will not be needed, but if you need to customize your Kafka consumer, you may use this. See a detailed list: https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html -
- file_descriptor_path - - str - - The path to the Protocol Buffer File Descriptor Set file. This file is used for schema definition and message serialization. -
- format - - str - - The encoding format for the data stored in Kafka. Valid options are: RAW,STRING,AVRO,JSON,PROTO -
- message_name + filter str - The name of the Protocol Buffer message to be used for schema extraction and data conversion. -
- offset_deduplication - - boolean - - If the redistribute is using offset deduplication mode. -
- redistribute_by_record_key - - boolean - - If the redistribute keys by the Kafka record key. -
- redistribute_num_keys - - int32 - - The number of keys for redistributing Kafka inputs. -
- redistributed - - boolean - - If the Kafka read should be redistributed. + SQL-like predicate to filter data at scan time. Example: "id > 5 AND status = 'ACTIVE'". Uses Apache Calcite syntax: https://calcite.apache.org/docs/reference.html
- schema + keep - str + list[str] - The schema in which the data is encoded in the Kafka topic. For AVRO data, this is a schema defined with AVRO schema syntax (https://avro.apache.org/docs/1.10.2/spec.html#schemas). For JSON data, this is a schema defined with JSON-schema syntax (https://json-schema.org/). If a URL to Confluent Schema Registry is provided, then this field is ignored, and the schema is fetched from Confluent Schema Registry. + A subset of column names to read exclusively. If null or empty, all columns will be read.
-### `KAFKA` Write +### `ICEBERG_CDC` Read
@@ -857,251 +801,257 @@ For more information on table properties, please visit https://iceberg.apache.or -
- bootstrap_servers + table str - A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. | Format: host1:port1,host2:port2,... + Identifier of the Iceberg table.
- format + catalog_name str - The encoding format for the data stored in Kafka. Valid options are: RAW,JSON,AVRO,PROTO + Name of the catalog containing the table.
- topic + catalog_properties - str + map[str, str] - n/a + Properties used to set up the Iceberg catalog.
- file_descriptor_path + config_properties - str + map[str, str] - The path to the Protocol Buffer File Descriptor Set file. This file is used for schema definition and message serialization. + Properties passed to the Hadoop Configuration.
- message_name + drop - str + list[str] - The name of the Protocol Buffer message to be used for schema extraction and data conversion. + A subset of column names to exclude from reading. If null or empty, all columns will be read.
- producer_config_updates + filter - map[str, str] + str - A list of key-value pairs that act as configuration parameters for Kafka producers. Most of these configurations will not be needed, but if you need to customize your Kafka producer, you may use this. See a detailed list: https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html + SQL-like predicate to filter data at scan time. Example: "id > 5 AND status = 'ACTIVE'". Uses Apache Calcite syntax: https://calcite.apache.org/docs/reference.html
- schema + from_snapshot - str + int64 - n/a + Starts reading from this snapshot ID (inclusive).
-
- -### `ICEBERG_CDC` Read - -
- - - - - - +
ConfigurationTypeDescription
- table + from_timestamp - str + int64 - Identifier of the Iceberg table. + Starts reading from the first snapshot (inclusive) that was created after this timestamp (in milliseconds).
- catalog_name + include_metadata_columns - str + list[str] - Name of the catalog containing the table. + List of top-level metadata columns to include with CDC output rows. Supported columns: +- `_change_type` +- `_row_id` +- `_last_updated_sequence_number` +- `_commit_snapshot_id` +- `_commit_snapshot_sequence_number` +
- catalog_properties + keep - map[str, str] + list[str] - Properties used to set up the Iceberg catalog. + A subset of column names to read exclusively. If null or empty, all columns will be read.
- config_properties + poll_interval_seconds - map[str, str] + int32 - Properties passed to the Hadoop Configuration. + The interval at which to poll for new snapshots. Defaults to 60 seconds.
- drop + starting_strategy - list[str] + str - A subset of column names to exclude from reading. If null or empty, all columns will be read. + The source's starting strategy. Valid options are: "earliest" or "latest". Can be overriden by setting a starting snapshot or timestamp. Defaults to earliest for batch, and latest for streaming.
- filter + streaming - str + boolean - SQL-like predicate to filter data at scan time. Example: "id > 5 AND status = 'ACTIVE'". Uses Apache Calcite syntax: https://calcite.apache.org/docs/reference.html + Enables streaming reads, where source continuously polls for snapshots forever.
- from_snapshot + to_snapshot int64 - Starts reading from this snapshot ID (inclusive). + Reads up to this snapshot ID (inclusive).
- from_timestamp + to_timestamp int64 - Starts reading from the first snapshot (inclusive) that was created after this timestamp (in milliseconds). + Reads up to the latest snapshot (inclusive) created before this timestamp (in milliseconds).
- keep + watermark_column - list[str] + str - A subset of column names to read exclusively. If null or empty, all columns will be read. + Column used to derive the source's output watermark. Must be an existing, required, top-level column of type 'long' or 'timestamp'. If not set, the watermark advances according to snapshot commit timestamp.
- poll_interval_seconds + watermark_column_time_unit - int32 + str - The interval at which to poll for new snapshots. Defaults to 60 seconds. + Time unit used to interpret watermark column of type LONG. One of NANOSECONDS, MICROSECONDS, MILLISECONDS, SECONDS, MINUTES, HOURS, DAYS. Defaults to MICROSECONDS.
+
+ +### `DELTA` Read + +
+ + + + + +
ConfigurationTypeDescription
- starting_strategy + table str - The source's starting strategy. Valid options are: "earliest" or "latest". Can be overriden by setting a starting snapshot or timestamp. Defaults to earliest for batch, and latest for streaming. + Identifier of the Delta Lake table.
- streaming + hadoop_config - boolean + map[str, str] - Enables streaming reads, where source continuously polls for snapshots forever. + Properties passed to the Hadoop Configuration.
- to_snapshot + timestamp - int64 + str - Reads up to this snapshot ID (inclusive). + Timestamp of the Delta Lake table to read.
- to_timestamp + version int64 - Reads up to the latest snapshot (inclusive) created before this timestamp (in milliseconds). + Version of the Delta Lake table to read.
-### `POSTGRES` Read +### `SQLSERVER` Write
@@ -1118,173 +1068,184 @@ For more information on table properties, please visit https://iceberg.apache.or str +
- Connection URL for the JDBC source. + Connection URL for the JDBC sink.
- connection_properties + autosharding - str + boolean - Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;". + If true, enables using a dynamically determined number of shards to write.
- fetch_size + batch_size - int32 + int64 - This method is used to override the size of the data that is going to be fetched and loaded in memory per every database call. It should ONLY be used if the default value throws memory errors. + n/a
- location + connection_properties str - Name of the table to read from. + Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
- num_partitions + location - int32 + str - The number of partitions + Name of the table to write to.
- output_parallelization + password - boolean + str - Whether to reshuffle the resulting PCollection so results are distributed to all workers. + Password for the JDBC source.
- partition_column + username str - Name of a column of numeric type that will be used for partitioning. + Username for the JDBC source.
- password + write_statement str - Password for the JDBC source. + SQL query used to insert records into the JDBC sink.
+
+ +### `SQLSERVER` Read + +
+ + + + + + -
ConfigurationTypeDescription
- read_query + jdbc_url str - SQL query used to query the JDBC source. + Connection URL for the JDBC source.
- username + connection_properties str - Username for the JDBC source. + Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
-
- -### `POSTGRES` Write - -
- - - - + + + @@ -1300,30 +1261,30 @@ For more information on table properties, please visit https://iceberg.apache.or
ConfigurationTypeDescription + disable_auto_commit + + boolean + + Whether to disable auto commit on read. Defaults to true if not provided. The need for this config varies depending on the database platform. Informix requires this to be set to false while Postgres requires this to be set to true. +
- jdbc_url + fetch_size - str + int32 - Connection URL for the JDBC sink. + This method is used to override the size of the data that is going to be fetched and loaded in memory per every database call. It should ONLY be used if the default value throws memory errors.
- autosharding + location - boolean + str - If true, enables using a dynamically determined number of shards to write. + Name of the table to read from.
- batch_size + num_partitions - int64 + int32 - n/a + The number of partitions
- connection_properties + output_parallelization - str + boolean - Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;". + Whether to reshuffle the resulting PCollection so results are distributed to all workers.
- location + partition_column str - Name of the table to write to. + Name of a column of numeric type that will be used for partitioning.
- username + read_query str - Username for the JDBC source. + SQL query used to query the JDBC source.
- write_statement + username str - SQL query used to insert records into the JDBC sink. + Username for the JDBC source.
-### `SQLSERVER` Write +### `POSTGRES` Write
@@ -1423,7 +1384,7 @@ For more information on table properties, please visit https://iceberg.apache.or
-### `SQLSERVER` Read +### `POSTGRES` Read
@@ -1454,17 +1415,6 @@ For more information on table properties, please visit https://iceberg.apache.or Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;". - - - - -
- disable_auto_commit - - boolean - - Whether to disable auto commit on read. Defaults to true if not provided. The need for this config varies depending on the database platform. Informix requires this to be set to false while Postgres requires this to be set to true. -
fetch_size @@ -1556,7 +1506,7 @@ For more information on table properties, please visit https://iceberg.apache.or
-### `BIGQUERY` Read +### `BIGQUERY` Write
@@ -1567,63 +1517,74 @@ For more information on table properties, please visit https://iceberg.apache.or + + + + +
- kms_key + table str - Use this Cloud KMS key to encrypt your data + The bigquery table to write to. Format: [${PROJECT}:]${DATASET}.${TABLE}
- query + drop - str + list[str] - The SQL query to be executed to read from the BigQuery table. + A list of field names to drop from the input record before writing. Is mutually exclusive with 'keep' and 'only'.
- row_restriction + keep - str + list[str] - Read only rows that match this filter, which must be compatible with Google standard SQL. This is not supported when reading via query. + A list of field names to keep in the input record. All other fields are dropped before writing. Is mutually exclusive with 'drop' and 'only'.
- fields + kms_key - list[str] + str - Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in the order specified. If no value is specified, then all fields are returned. Example: "col1, col2, col3" + Use this Cloud KMS key to encrypt your data
- table + only str - The fully-qualified name of the BigQuery table to read from. Format: [${PROJECT}:]${DATASET}.${TABLE} + The name of a single record field that should be written. Is mutually exclusive with 'keep' and 'drop'. +
+ triggering_frequency_seconds + + int64 + + Determines how often to 'commit' progress into BigQuery. Default is every 5 seconds.
-### `BIGQUERY` Write +### `BIGQUERY` Read
@@ -1634,74 +1595,63 @@ For more information on table properties, please visit https://iceberg.apache.or - - - - -
- table + kms_key str - The bigquery table to write to. Format: [${PROJECT}:]${DATASET}.${TABLE} -
- drop - - list[str] - - A list of field names to drop from the input record before writing. Is mutually exclusive with 'keep' and 'only'. + Use this Cloud KMS key to encrypt your data
- keep + query - list[str] + str - A list of field names to keep in the input record. All other fields are dropped before writing. Is mutually exclusive with 'drop' and 'only'. + The SQL query to be executed to read from the BigQuery table.
- kms_key + row_restriction str - Use this Cloud KMS key to encrypt your data + Read only rows that match this filter, which must be compatible with Google standard SQL. This is not supported when reading via query.
- only + fields - str + list[str] - The name of a single record field that should be written. Is mutually exclusive with 'keep' and 'drop'. + Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in the order specified. If no value is specified, then all fields are returned. Example: "col1, col2, col3"
- triggering_frequency_seconds + table - int64 + str - Determines how often to 'commit' progress into BigQuery. Default is every 5 seconds. + The fully-qualified name of the BigQuery table to read from. Format: [${PROJECT}:]${DATASET}.${TABLE}
-### `MYSQL` Read +### `MYSQL` Write
@@ -1718,51 +1668,51 @@ For more information on table properties, please visit https://iceberg.apache.or str @@ -1773,151 +1723,151 @@ For more information on table properties, please visit https://iceberg.apache.or str +
- Connection URL for the JDBC source. + Connection URL for the JDBC sink.
- connection_init_sql + autosharding - list[str] + boolean - Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this. + If true, enables using a dynamically determined number of shards to write.
- connection_properties + batch_size - str + int64 - Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;". + n/a
- disable_auto_commit + connection_init_sql - boolean + list[str] - Whether to disable auto commit on read. Defaults to true if not provided. The need for this config varies depending on the database platform. Informix requires this to be set to false while Postgres requires this to be set to true. + Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this.
- fetch_size + connection_properties - int32 + str - This method is used to override the size of the data that is going to be fetched and loaded in memory per every database call. It should ONLY be used if the default value throws memory errors. + Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
- Name of the table to read from. + Name of the table to write to.
- num_partitions + password - int32 + str - The number of partitions + Password for the JDBC source.
- output_parallelization + username - boolean + str - Whether to reshuffle the resulting PCollection so results are distributed to all workers. + Username for the JDBC source.
- partition_column + write_statement str - Name of a column of numeric type that will be used for partitioning. + SQL query used to insert records into the JDBC sink.
+
+ +### `MYSQL` Read + +
+ + + + + + -
ConfigurationTypeDescription
- password + jdbc_url str - Password for the JDBC source. + Connection URL for the JDBC source.
- read_query + connection_init_sql - str + list[str] - SQL query used to query the JDBC source. + Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this.
- username + connection_properties str - Username for the JDBC source. + Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
-
- -### `MYSQL` Write - -
- - - - - - @@ -1933,24 +1883,24 @@ For more information on table properties, please visit https://iceberg.apache.or
ConfigurationTypeDescription
- jdbc_url + disable_auto_commit - str + boolean - Connection URL for the JDBC sink. + Whether to disable auto commit on read. Defaults to true if not provided. The need for this config varies depending on the database platform. Informix requires this to be set to false while Postgres requires this to be set to true.
- autosharding + fetch_size - boolean + int32 - If true, enables using a dynamically determined number of shards to write. + This method is used to override the size of the data that is going to be fetched and loaded in memory per every database call. It should ONLY be used if the default value throws memory errors.
- batch_size + location - int64 + str - n/a + Name of the table to read from.
- connection_init_sql + num_partitions - list[str] + int32 - Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this. + The number of partitions
- connection_properties + output_parallelization - str + boolean - Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;". + Whether to reshuffle the resulting PCollection so results are distributed to all workers.
- location + partition_column str - Name of the table to write to. + Name of a column of numeric type that will be used for partitioning.
- username + read_query str - Username for the JDBC source. + SQL query used to query the JDBC source.
- write_statement + username str - SQL query used to insert records into the JDBC sink. + Username for the JDBC source.