DefaultSource
Related Doc:
package spark
Write the given DataFrame to the database and return a base relation with which it can subsequently be read.
Write the given DataFrame to the database and return a base relation with which it can subsequently be read.
Arguments are supplied as a collection of (key,value) pairs. They include all of the options described above, as for reading; in particular:
dbtable The name of the target table to write to.Other optional arguments include:
url: A string of the form "IRIS://host:port/namespace" that
specifies the cluster to which the data is to be written.
If omitted, the default cluster specified via the "spark.
iris.master.url" configuration setting is used instead.user: The user account with which to make the connection to the
cluster named in the "url" option above.password: The password for the given user account.batchsize: The number of rows to insert per server round trip.Default = 1000.
isolationlevel: The transaction isolation level.Can be either NONE, REPEATABLE_READ, READ_COMMITTED, READ_UNCOMMITTED, or SERIALIZABLE.
Corresponds to the standard transaction isolation levels specified by the JDBC Connection object.
Default = READ_UNCOMMITTED.
description: An optional description for the newly created table.publicrowid: Specifies whether or not the master row ID column for
the newly created table is to be made publicly visible.shard: Indicates the records of the table are to be distributed
across the instances of the cluster; the optional value
specifies the shard key to use - see IRIS documentation
for CREATE TABLE for more details.autobalance: When writing a dataset to a table that is sharded on a
system assigned shard key, the value true specifies that
the inserted records are to be evenly distributed amongst
the available shards,while the value false specifies that
that they be sent to whichever shard is closest to where
the partitions of the dataset reside.Default = true.
The context in which to create the new relation.
Describes the behavior if the target table already exists.
The arguments with which to initialize the new relation.
The source data frame that is to be written.
A new Relation that can be called upon to execute the given query on the cluster as needed.
Exception if the proposed write operation would certainly fail.
IllegalArgumentException if passed an invalid parameter value.
InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.
Constructs a BaseRelation from the given arguments.
Constructs a BaseRelation from the given arguments. These are supplied as a collection of (key,value) pairs, and must include a value for either:
query: The text of a query to be executed on the cluster, ordbtable: The name of a database table within the cluster, in which
case the entire table is loaded.Other optional arguments include:
url: A string of the form "IRIS://host:port/namespace" that
specifies the cluster from which the data is to be read.
If omitted, the default cluster specified via the "spark.
iris.master.url" configuration setting is used instead.user: The user account with which to make the connection to the
cluster named in the "url" option above.password: The password for the given user account.mfpi: The maximum number of partitions per server instance to
include in any implicit query factorization performed by
the server.Default = 1.
fetchsize: The number of rows to fetch per server round trip.Default = 1000.
partitionColumn, lowerBound, upperBound, and numPartitions:
An explicit description of how to partition the queries
sent to each distinct instance; these have an identical
semantics to the similarly named arguments for the JDBC
data source that is built into Spark.If both mfpi and partionColumn arguments are given, then the explicit
partitioning specification takes precedence.
The context in which to create the new relation.
The arguments with which to initialize the new relation.
A new Relation that can be called upon to execute the given query on the cluster as needed.
IllegalArgumentException if passed an invalid parameter value.
JDBC to Other Databases
for more on the semantics of the column, lo, hi, and partitions
parameters.
A dedicated logger that instances of this class can write to.
A dedicated logger that instances of this class can write to.
The name of the logger that instances of this class can write to.
The name of the logger that instances of this class can write to.
A short user-friendly name for the data source.
A short user-friendly name for the data source.
© 2024 InterSystems Corporation, Cambridge, MA. All rights reserved. Privacy & Terms Guarantee Accessibility
Registers the InterSystems IRIS Spark Connector as a Spark SQL data source provider for the format "com.intersystems.spark", also known by its shorter alias "iris".
This allows clients to execute queries against a cluster by calling Spark's generic load and save functions. For example:
spark.read .format("com.intersystems.spark") .option("query","SELECT * FROM Owls") .load()executes the query
"SELECT * FROM Owls"on the default cluster, and hands its rows back in the form of an appropriately partitioned DataFrame.Here
readmeans 'execute a SELECT statement against the database', whilewritemeans 'execute batch INSERT statements against a database table'.Apache Spark Documentation for more on how to use the generic load and save functions.