DataFrameWriterEx

Instance Constructors

new DataFrameWriterEx(writer: core.DataFrameWriter[α])

writer
A DataFrame writer.

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def address(url: String, user: String = "", password: String = ""): core.DataFrameWriter[α]

Specifies the connection details of the cluster to write to.
Specifies the connection details of the cluster to write to.
Overrides the default cluster specified in the Spark configuration for the duration of this write operation.
url
A string of the form "IRIS://host:port/namespace" that specifies the cluster to read from.
user
The user account with which to make the connection to the cluster named in the "url" option above.
password
The password for the given user account.
returns
The same DataFrameWriter on which this method was invoked.
def address(address: Address): core.DataFrameWriter[α]

Specifies the connection details of the cluster to write to.
Specifies the connection details of the cluster to write to.
Overrides the default cluster specified in the Spark configuration for the duration of this write operation.
address
The connection details of the cluster to write to.
returns
The same DataFrameWriter on which this method was invoked.
final def asInstanceOf[T0]: T0

Definition Classes
Any
def autobalance(value: Boolean = true): core.DataFrameWriter[α]

Specifies whether or not inserted records should be evenly distributed amongst the available shards of the cluster when saving to a table that is sharded on a system assigned shard key.
Specifies whether or not inserted records should be evenly distributed amongst the available shards of the cluster when saving to a table that is sharded on a system assigned shard key.
When explicitly disabled, every dataset partition attempts to write its records directly into the shard on which its Spark executor also runs. This is often faster, since records no longer need to travel across the network to reach their destination shard, but shifts the responsibility of saving roughly the same number of records to each shard to the user, who will now wish to partition the dataset appropriately before writing it to the cluster.
Has no effect if the table is not sharded, or is sharded using a custom shard key.
value
true to evenly distribute records amongst the available shards of the cluster, false to save records into shards that are 'closest' to where the partitions of the dataset reside.
returns
The same DataFrameWriter on which this method was invoked.

See also
InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def description(value: String): core.DataFrameWriter[α]

Specifies a description for the newly created table.
Specifies a description for the newly created table.
Has no effect if the table already exists and the save mode is anything other than OVERWRITE.
value
An arbitrary description for the newly created table.
returns
The same DataFrameWriter on which this method was invoked.

See also
InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
def iris(table: String): Unit

Saves a DataFrame to the given table within the cluster.
Saves a DataFrame to the given table within the cluster.
This enables one to write, for example:
```
df.write.iris("Owls")
```
as a convenient shorthand for the more explicit:
```
df.write.format("com.intersystems.spark")
        .option("dbtable","Owls")
        .save()
```
The following options affect how the operation is performed:
- url: A string of the form "IRIS://host:port/namespace" that specifies the cluster to which the data will be written. If omitted, the default cluster specified via the "spark.iris.master.url" configuration setting is used instead.
- user: The account with which to make the connection to the cluster named in the "url" option above.
- password: The password for the given user account.
- mode: Describes how to behave if the target table already exists.
Can be either OVERWRITE, APPEND, IGNORE, or ERROR.
Default = ERROR.
- batchsize: The number of rows to insert per server round trip.
Default = 1000.
- isolationlevel: The transaction isolation level.
Can be either NONE, REPEATABLE_READ, READ_COMMITTED, READ_UNCOMMITTED, or SERIALIZABLE.
Corresponds to the standard transaction isolation levels specified by the JDBC Connection object.
Default = READ_UNCOMMITTED.
- description: An optional description for the newly created table.
Default = "".
- publicrowid: Specifies that the master RowID field for the newly created table be publicly visible.
Default = false.
- shard: Specifies the shard key for the newly created table.
Can be either true, false, or else a comma separated set of field names.
Default = false.
table
The name of the table to write to.

Exceptions thrown
SQLException if a database access error occurs.
See also
Using JDBC Transaction Isolation Levels
InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def publicRowID(value: Boolean): core.DataFrameWriter[α]

Specifies whether or not the master row ID column of the newly created table is to be made publicly visible.
Specifies whether or not the master row ID column of the newly created table is to be made publicly visible.
Has no effect if the table already exists and the save mode is anything other than OVERWRITE.
value
true for a publicly visible row ID, false otherwise.
returns
The same DataFrameWriter on which this method was invoked.

See also
InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.
def shard(fields: String*): core.DataFrameWriter[α]

Specifies the shard key for the newly created table.
Specifies the shard key for the newly created table.
Has no effect if the table already exists and the save mode is anything other than OVERWRITE.
fields
A (possibly empty) sequence of field names to be used as the user defied shard key. If the sequence is empty then the table will be sharded on the system assigned key.
returns
The same DataFrameWriter on which this method was invoked.

See also
InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.
def shard(value: Boolean): core.DataFrameWriter[α]

Specifies whether or not the newly created table is to be sharded.
Specifies whether or not the newly created table is to be sharded.
Has no effect if the table already exists and the save mode is anything other than OVERWRITE.
value
true to create a sharded table using a shard key assigned by the system, false to create a non sharded table.
returns
The same DataFrameWriter on which this method was invoked.

See also
InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package spark

implicit class DataFrameWriterEx[α] extends AnyRef

Instance Constructors

new DataFrameWriterEx(writer: core.DataFrameWriter[α])

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def address(url: String, user: String = "", password: String = ""): core.DataFrameWriter[α]

def address(address: Address): core.DataFrameWriter[α]

final def asInstanceOf[T0]: T0

def autobalance(value: Boolean = true): core.DataFrameWriter[α]

def clone(): AnyRef

def description(value: String): core.DataFrameWriter[α]

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

def iris(table: String): Unit

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def publicRowID(value: Boolean): core.DataFrameWriter[α]

def shard(fields: String*): core.DataFrameWriter[α]

def shard(value: Boolean): core.DataFrameWriter[α]

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from AnyRef

Inherited from Any

Ungrouped