Class

com.intersystems.spark

DataFrameWriterEx

Related Doc: package spark

Permalink

implicit class DataFrameWriterEx[α] extends AnyRef

Extends the given writer with IRIS specific methods.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DataFrameWriterEx
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DataFrameWriterEx(writer: core.DataFrameWriter[α])

    Permalink

    writer

    A DataFrame writer.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def address(url: String, user: String = "", password: String = ""): core.DataFrameWriter[α]

    Permalink

    Specifies the connection details of the cluster to write to.

    Specifies the connection details of the cluster to write to.

    Overrides the default cluster specified in the Spark configuration for the duration of this write operation.

    url

    A string of the form "IRIS://host:port/namespace" that specifies the cluster to read from.

    user

    The user account with which to make the connection to the cluster named in the "url" option above.

    password

    The password for the given user account.

    returns

    The same DataFrameWriter on which this method was invoked.

  5. def address(address: Address): core.DataFrameWriter[α]

    Permalink

    Specifies the connection details of the cluster to write to.

    Specifies the connection details of the cluster to write to.

    Overrides the default cluster specified in the Spark configuration for the duration of this write operation.

    address

    The connection details of the cluster to write to.

    returns

    The same DataFrameWriter on which this method was invoked.

  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  7. def autobalance(value: Boolean = true): core.DataFrameWriter[α]

    Permalink

    Specifies whether or not inserted records should be evenly distributed amongst the available shards of the cluster when saving to a table that is sharded on a system assigned shard key.

    Specifies whether or not inserted records should be evenly distributed amongst the available shards of the cluster when saving to a table that is sharded on a system assigned shard key.

    When explicitly disabled, every dataset partition attempts to write its records directly into the shard on which its Spark executor also runs. This is often faster, since records no longer need to travel across the network to reach their destination shard, but shifts the responsibility of saving roughly the same number of records to each shard to the user, who will now wish to partition the dataset appropriately before writing it to the cluster.

    Has no effect if the table is not sharded, or is sharded using a custom shard key.

    value

    true to evenly distribute records amongst the available shards of the cluster, false to save records into shards that are 'closest' to where the partitions of the dataset reside.

    returns

    The same DataFrameWriter on which this method was invoked.

    See also

    InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.

  8. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. def description(value: String): core.DataFrameWriter[α]

    Permalink

    Specifies a description for the newly created table.

    Specifies a description for the newly created table.

    Has no effect if the table already exists and the save mode is anything other than OVERWRITE.

    value

    An arbitrary description for the newly created table.

    returns

    The same DataFrameWriter on which this method was invoked.

    See also

    InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.

  10. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  12. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  14. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  15. def iris(table: String): Unit

    Permalink

    Saves a DataFrame to the given table within the cluster.

    Saves a DataFrame to the given table within the cluster.

    This enables one to write, for example:

    df.write.iris("Owls")

    as a convenient shorthand for the more explicit:

    df.write.format("com.intersystems.spark")
            .option("dbtable","Owls")
            .save()

    The following options affect how the operation is performed:

    • url: A string of the form "IRIS://host:port/namespace" that specifies the cluster to which the data will be written. If omitted, the default cluster specified via the "spark.iris.master.url" configuration setting is used instead.
    • user: The account with which to make the connection to the cluster named in the "url" option above.
    • password: The password for the given user account.
    • mode: Describes how to behave if the target table already exists.

    Can be either OVERWRITE, APPEND, IGNORE, or ERROR.

    Default = ERROR.

    • batchsize: The number of rows to insert per server round trip.

    Default = 1000.

    • isolationlevel: The transaction isolation level.

    Can be either NONE, REPEATABLE_READ, READ_COMMITTED, READ_UNCOMMITTED, or SERIALIZABLE.

    Corresponds to the standard transaction isolation levels specified by the JDBC Connection object.

    Default = READ_UNCOMMITTED.

    • description: An optional description for the newly created table.

    Default = "".

    • publicrowid: Specifies that the master RowID field for the newly created table be publicly visible.

    Default = false.

    • shard: Specifies the shard key for the newly created table.

    Can be either true, false, or else a comma separated set of field names.

    Default = false.

    table

    The name of the table to write to.

    Exceptions thrown

    SQLException if a database access error occurs.

    See also

    Using JDBC Transaction Isolation Levels

    InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.

  16. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  17. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  18. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  19. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  20. def publicRowID(value: Boolean): core.DataFrameWriter[α]

    Permalink

    Specifies whether or not the master row ID column of the newly created table is to be made publicly visible.

    Specifies whether or not the master row ID column of the newly created table is to be made publicly visible.

    Has no effect if the table already exists and the save mode is anything other than OVERWRITE.

    value

    true for a publicly visible row ID, false otherwise.

    returns

    The same DataFrameWriter on which this method was invoked.

    See also

    InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.

  21. def shard(fields: String*): core.DataFrameWriter[α]

    Permalink

    Specifies the shard key for the newly created table.

    Specifies the shard key for the newly created table.

    Has no effect if the table already exists and the save mode is anything other than OVERWRITE.

    fields

    A (possibly empty) sequence of field names to be used as the user defied shard key. If the sequence is empty then the table will be sharded on the system assigned key.

    returns

    The same DataFrameWriter on which this method was invoked.

    See also

    InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.

  22. def shard(value: Boolean): core.DataFrameWriter[α]

    Permalink

    Specifies whether or not the newly created table is to be sharded.

    Specifies whether or not the newly created table is to be sharded.

    Has no effect if the table already exists and the save mode is anything other than OVERWRITE.

    value

    true to create a sharded table using a shard key assigned by the system, false to create a non sharded table.

    returns

    The same DataFrameWriter on which this method was invoked.

    See also

    InterSystems IRIS SQL Reference for more information on the options supported by the CREATE TABLE statement.

  23. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  24. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  25. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped

© 2024 InterSystems Corporation, Cambridge, MA. All rights reserved.    Privacy & Terms Guarantee Accessibility