Package org.apache.spark.sql.types

Class StructType

Object
org.apache.spark.sql.types.DataType
org.apache.spark.sql.types.StructType
All Implemented Interfaces:
Serializable, scala.collection.immutable.Iterable<StructField>, scala.collection.immutable.Seq<StructField>, scala.collection.immutable.SeqOps<StructField,scala.collection.immutable.Seq,scala.collection.immutable.Seq<StructField>>, scala.collection.Iterable<StructField>, scala.collection.IterableFactoryDefaults<StructField,scala.collection.Iterable>, scala.collection.IterableOnce<StructField>, scala.collection.IterableOnceOps<StructField,scala.collection.Iterable,scala.collection.Iterable>, scala.collection.IterableOps<StructField,scala.collection.Iterable,scala.collection.Iterable>, scala.collection.Seq<StructField>, scala.collection.SeqOps<StructField,scala.collection.Seq,scala.collection.Seq<StructField>>, scala.Equals, scala.Function1<Object,StructField>, scala.PartialFunction<Object,StructField>, scala.Product
public class StructType extends DataType implements scala.collection.immutable.Seq<StructField>, scala.Product, Serializable
A StructType object can be constructed by 

 StructType(fields: Seq[StructField])
For a StructType object, one or multiple StructFields can be extracted by names. If multiple StructFields are extracted, a StructType object will be returned. If a provided name does not have a matching field, it will be ignored. For the case of extracting a single StructField, a null will be returned.

Scala Example: 


 import org.apache.spark.sql._
 import org.apache.spark.sql.types._

 val struct =
   StructType(
     StructField("a", IntegerType, true) ::
     StructField("b", LongType, false) ::
     StructField("c", BooleanType, false) :: Nil)

 // Extract a single StructField.
 val singleField = struct("b")
 // singleField: StructField = StructField(b,LongType,false)

 // If this struct does not have a field called "d", it throws an exception.
 struct("d")
 // java.lang.IllegalArgumentException: d does not exist.
 //   ...

 // Extract multiple StructFields. Field names are provided in a set.
 // A StructType object will be returned.
 val twoFields = struct(Set("b", "c"))
 // twoFields: StructType =
 //   StructType(StructField(b,LongType,false), StructField(c,BooleanType,false))

 // Any names without matching fields will throw an exception.
 // For the case shown below, an exception is thrown due to "d".
 struct(Set("b", "c", "d"))
 // java.lang.IllegalArgumentException: d does not exist.
 //    ...

A Row object is used as a value of the StructType.

Scala Example: 


 import org.apache.spark.sql._
 import org.apache.spark.sql.types._

 val innerStruct =
   StructType(
     StructField("f1", IntegerType, true) ::
     StructField("f2", LongType, false) ::
     StructField("f3", BooleanType, false) :: Nil)

 val struct = StructType(
   StructField("a", innerStruct, true) :: Nil)

 // Create a Row with the schema defined by struct
 val row = Row(Row(1, 2, true))

Since:
1.3.0
See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from interface scala.Function1

    scala.Function1.UnliftOps<A extends Object,B extends Object>, scala.Function1.UnliftOps$

    Nested classes/interfaces inherited from interface scala.collection.IterableOps

    scala.collection.IterableOps.SizeCompareOps, scala.collection.IterableOps.SizeCompareOps$, scala.collection.IterableOps.WithFilter<A extends Object,CC extends Object>

    Nested classes/interfaces inherited from interface scala.PartialFunction

    scala.PartialFunction.ElementWiseExtractor<A extends Object,B extends Object>, scala.PartialFunction.ElementWiseExtractor$

  • Constructor Summary

    Constructors
    Constructor
    Description
    StructType()
    No-arg constructor for kryo.
    StructType(StructField[] fields)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    StructType
    add(String name, String dataType)
    Creates a new StructType by adding a new nullable field with no metadata where the dataType is specified as a String.
    StructType
    add(String name, String dataType, boolean nullable)
    Creates a new StructType by adding a new field with no metadata where the dataType is specified as a String.
    StructType
    add(String name, String dataType, boolean nullable, String comment)
    Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String.
    StructType
    add(String name, String dataType, boolean nullable, Metadata metadata)
    Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String.
    StructType
    add(String name, DataType dataType)
    Creates a new StructType by adding a new nullable field with no metadata.
    StructType
    add(String name, DataType dataType, boolean nullable)
    Creates a new StructType by adding a new field with no metadata.
    StructType
    add(String name, DataType dataType, boolean nullable, String comment)
    Creates a new StructType by adding a new field and specifying metadata.
    StructType
    add(String name, DataType dataType, boolean nullable, Metadata metadata)
    Creates a new StructType by adding a new field and specifying metadata.
    StructType
    add(StructField field)
    Creates a new StructType by adding a new field.
    StructField
    apply(int fieldIndex)
     
    StructField
    apply(String name)
    Extracts the StructField with the given name.
    StructType
    apply(scala.collection.immutable.Set<String> names)
    Returns a StructType containing StructFields of the given names, preserving the original order of fields.
    String
    catalogString()
    String representation for the type saved in external catalogs.
    int
    defaultSize()
    The default size of a value of the StructType is the total default sizes of all field types.
    boolean
    equals(Object that)
     
    int
    fieldIndex(String name)
    Returns the index of a given field.
    String[]
    fieldNames()
    Returns all field names in an array.
    StructField[]
    fields()
     
    static scala.Option<StructType>
    findMissingFields(StructType source, StructType target, scala.Function2<String,String,Object> resolver)
    Returns a StructType that contains missing fields recursively from source to target.
    static StructType
    fromDDL(String ddl)
    Creates StructType for a given DDL-formatted string, which is a comma separated list of field definitions, e.g., a INT, b STRING.
    int
    hashCode()
     
    scala.collection.Iterator<StructField>
    iterator()
     
    int
    length()
     
    String[]
    names()
    Returns all field names in an array.
    void
    printTreeString()
     
    String
    simpleString()
    Readable string representation for the type.
    String
    sql()
     
    String
    toDDL()
    Returns a string containing a schema in DDL format.
    StructType
    toNullable()
    Returns the same data type but set all nullability fields are true (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).
    String
    toString()
     
    String
    treeString()
     
    String
    treeString(int maxDepth)
     

    Methods inherited from class org.apache.spark.sql.types.DataType

    COLLATIONS_METADATA_KEY, equalsIgnoreCaseAndNullability, equalsIgnoreNullability, equalsStructurally, equalsStructurallyByName, fromJson, json, parseTypeWithFallback, prettyJson, typeName

    Methods inherited from class java.lang.Object

    getClass, notify, notifyAll, wait, wait, wait

    Methods inherited from interface scala.Function1

    apply, apply$mcDD$sp, apply$mcDF$sp, apply$mcDI$sp, apply$mcDJ$sp, apply$mcFD$sp, apply$mcFF$sp, apply$mcFI$sp, apply$mcFJ$sp, apply$mcID$sp, apply$mcIF$sp, apply$mcII$sp, apply$mcIJ$sp, apply$mcJD$sp, apply$mcJF$sp, apply$mcJI$sp, apply$mcJJ$sp, apply$mcVD$sp, apply$mcVF$sp, apply$mcVI$sp, apply$mcVJ$sp, apply$mcZD$sp, apply$mcZF$sp, apply$mcZI$sp, apply$mcZJ$sp, compose

    Methods inherited from interface scala.collection.Iterable

    className, coll, collectionClassName, lazyZip, seq, toIterable

    Methods inherited from interface scala.collection.IterableFactoryDefaults

    empty, fromSpecific, newSpecificBuilder

    Methods inherited from interface scala.collection.IterableOnce

    knownSize, stepper

    Methods inherited from interface scala.collection.IterableOnceOps

    $colon$bslash, $div$colon, addString, addString, addString, aggregate, collectFirst, copyToArray, copyToArray, copyToArray, copyToBuffer, corresponds, count, exists, find, fold, foldLeft, foldRight, forall, foreach, hasDefiniteSize, max, maxBy, maxByOption, maxOption, min, minBy, minByOption, minOption, mkString, mkString, mkString, nonEmpty, product, reduce, reduceLeft, reduceLeftOption, reduceOption, reduceRight, reduceRightOption, reversed, sum, to, toArray, toBuffer, toIndexedSeq, toIterator, toList, toMap, toSet, toStream, toVector

    Methods inherited from interface scala.collection.IterableOps

    $plus$plus, collect, companion, drop, dropRight, dropWhile, filter, filterNot, flatMap, flatten, groupBy, grouped, groupMap, groupMapReduce, head, headOption, init, inits, isTraversableAgain, last, lastOption, map, partition, partitionMap, repr, scan, scanLeft, scanRight, sizeIs, slice, sliding, sliding, span, splitAt, tail, tails, take, takeRight, takeWhile, tapEach, toTraversable, transpose, unzip, unzip3, view, withFilter, zip, zipAll, zipWithIndex

    Methods inherited from interface scala.PartialFunction

    andThen, andThen, applyOrElse, compose, elementWise, isDefinedAt, lift, orElse, runWith, unapply

    Methods inherited from interface scala.Product

    productArity, productElement, productElementName, productElementNames, productIterator, productPrefix

    Methods inherited from interface scala.collection.immutable.Seq

    iterableFactory, toSeq

    Methods inherited from interface scala.collection.Seq

    canEqual, stringPrefix

    Methods inherited from interface scala.collection.SeqOps

    $colon$plus, $colon$plus$plus, $plus$colon, $plus$plus$colon, appended, appendedAll, combinations, concat, contains, containsSlice, corresponds, diff, distinct, distinctBy, endsWith, findLast, indexOf, indexOf, indexOfSlice, indexOfSlice, indexWhere, indexWhere, indices, intersect, isDefinedAt, isEmpty, lastIndexOf, lastIndexOf$default$2, lastIndexOfSlice, lastIndexOfSlice, lastIndexWhere, lastIndexWhere, lengthCompare, lengthCompare, lengthIs, occCounts, padTo, patch, permutations, prefixLength, prepended, prependedAll, reverse, reverseIterator, reverseMap, sameElements, scala$collection$SeqOps$$toGenericSeq, search, search, segmentLength, segmentLength, size, sizeCompare, sizeCompare, sortBy, sorted, sortWith, startsWith, startsWith$default$2, union, updated, view

  • Constructor Details

    • StructType

      public StructType(StructField[] fields)

    • StructType

      public StructType()
      No-arg constructor for kryo.

  • Method Details

    • fromDDL

      public static StructType fromDDL(String ddl)
      Creates StructType for a given DDL-formatted string, which is a comma separated list of field definitions, e.g., a INT, b STRING.

      Parameters:
      ddl - (undocumented)
      Returns:
      (undocumented)
      Since:
      2.2.0

    • findMissingFields

      public static scala.Option<StructType> findMissingFields(StructType source, StructType target, scala.Function2<String,String,Object> resolver)
      Returns a StructType that contains missing fields recursively from source to target. Note that this doesn't support looking into array type and map type recursively.
      Parameters:
      source - (undocumented)
      target - (undocumented)
      resolver - (undocumented)
      Returns:
      (undocumented)

    • fields

      public StructField[] fields()

    • fieldNames

      public String[] fieldNames()
      Returns all field names in an array.

    • names

      public String[] names()
      Returns all field names in an array. This is an alias of fieldNames.

      Returns:
      (undocumented)
      Since:
      2.4.0

    • equals

      public boolean equals(Object that)
      Specified by:
      equals in interface scala.Equals
      Specified by:
      equals in interface scala.collection.Seq<StructField>
      Overrides:
      equals in class Object

    • toString

      public String toString()
      Specified by:
      toString in interface scala.Function1<Object,StructField>
      Specified by:
      toString in interface scala.collection.Iterable<StructField>
      Specified by:
      toString in interface scala.collection.Seq<StructField>
      Overrides:
      toString in class Object

    • hashCode

      public int hashCode()
      Specified by:
      hashCode in interface scala.collection.Seq<StructField>
      Overrides:
      hashCode in class Object

    • add

      public StructType add(StructField field)
      Creates a new StructType by adding a new field. 
      
 val struct = (new StructType)
   .add(StructField("a", IntegerType, true))
   .add(StructField("b", LongType, false))
   .add(StructField("c", StringType, true))
      Parameters:
      field - (undocumented)
      Returns:
      (undocumented)

    • add

      public StructType add(String name, DataType dataType)
      Creates a new StructType by adding a new nullable field with no metadata.

      val struct = (new StructType) .add("a", IntegerType) .add("b", LongType) .add("c", StringType)

      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      Returns:
      (undocumented)

    • add

      public StructType add(String name, DataType dataType, boolean nullable)
      Creates a new StructType by adding a new field with no metadata.

      val struct = (new StructType) .add("a", IntegerType, true) .add("b", LongType, false) .add("c", StringType, true)

      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      Returns:
      (undocumented)

    • add

      public StructType add(String name, DataType dataType, boolean nullable, Metadata metadata)
      Creates a new StructType by adding a new field and specifying metadata. 
      
 val struct = (new StructType)
   .add("a", IntegerType, true, Metadata.empty)
   .add("b", LongType, false, Metadata.empty)
   .add("c", StringType, true, Metadata.empty)
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      metadata - (undocumented)
      Returns:
      (undocumented)

    • add

      public StructType add(String name, DataType dataType, boolean nullable, String comment)
      Creates a new StructType by adding a new field and specifying metadata. 
      
 val struct = (new StructType)
   .add("a", IntegerType, true, "comment1")
   .add("b", LongType, false, "comment2")
   .add("c", StringType, true, "comment3")
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      comment - (undocumented)
      Returns:
      (undocumented)

    • add

      public StructType add(String name, String dataType)
      Creates a new StructType by adding a new nullable field with no metadata where the dataType is specified as a String. 

      
 val struct = (new StructType)
   .add("a", "int")
   .add("b", "long")
   .add("c", "string")
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      Returns:
      (undocumented)

    • add

      public StructType add(String name, String dataType, boolean nullable)
      Creates a new StructType by adding a new field with no metadata where the dataType is specified as a String. 

      
 val struct = (new StructType)
   .add("a", "int", true)
   .add("b", "long", false)
   .add("c", "string", true)
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      Returns:
      (undocumented)

    • add

      public StructType add(String name, String dataType, boolean nullable, Metadata metadata)
      Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String. 
      
 val struct = (new StructType)
   .add("a", "int", true, Metadata.empty)
   .add("b", "long", false, Metadata.empty)
   .add("c", "string", true, Metadata.empty)
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      metadata - (undocumented)
      Returns:
      (undocumented)

    • add

      public StructType add(String name, String dataType, boolean nullable, String comment)
      Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String. 
      
 val struct = (new StructType)
   .add("a", "int", true, "comment1")
   .add("b", "long", false, "comment2")
   .add("c", "string", true, "comment3")
      Parameters:
      name - (undocumented)
      dataType - (undocumented)
      nullable - (undocumented)
      comment - (undocumented)
      Returns:
      (undocumented)

    • apply

      public StructField apply(String name)
      Extracts the StructField with the given name.

      Parameters:
      name - (undocumented)
      Returns:
      (undocumented)
      Throws:
      IllegalArgumentException - if a field with the given name does not exist

    • apply

      public StructType apply(scala.collection.immutable.Set<String> names)
      Returns a StructType containing StructFields of the given names, preserving the original order of fields.

      Parameters:
      names - (undocumented)
      Returns:
      (undocumented)
      Throws:
      IllegalArgumentException - if at least one given field name does not exist

    • fieldIndex

      public int fieldIndex(String name)
      Returns the index of a given field.

      Parameters:
      name - (undocumented)
      Returns:
      (undocumented)
      Throws:
      IllegalArgumentException - if a field with the given name does not exist

    • treeString

      public String treeString()

    • treeString

      public String treeString(int maxDepth)

    • printTreeString

      public void printTreeString()

    • apply

      public StructField apply(int fieldIndex)
      Specified by:
      apply in interface scala.collection.SeqOps<StructField,scala.collection.Seq,scala.collection.Seq<StructField>>

    • length

      public int length()
      Specified by:
      length in interface scala.collection.SeqOps<StructField,scala.collection.Seq,scala.collection.Seq<StructField>>

    • iterator

      public scala.collection.Iterator<StructField> iterator()
      Specified by:
      iterator in interface scala.collection.IterableOnce<StructField>

    • defaultSize

      public int defaultSize()
      The default size of a value of the StructType is the total default sizes of all field types.
      Specified by:
      defaultSize in class DataType
      Returns:
      (undocumented)

    • simpleString

      public String simpleString()
      Description copied from class: DataType
      Readable string representation for the type.
      Overrides:
      simpleString in class DataType

    • catalogString

      public String catalogString()
      Description copied from class: DataType
      String representation for the type saved in external catalogs.
      Overrides:
      catalogString in class DataType

    • sql

      public String sql()
      Overrides:
      sql in class DataType

    • toDDL

      public String toDDL()
      Returns a string containing a schema in DDL format. For example, the following value: StructType(Seq(StructField("eventId", IntegerType), StructField("s", StringType))) will be converted to eventId INT, s STRING. The returned DDL schema can be used in a table creation.

      Returns:
      (undocumented)
      Since:
      2.4.0

    • toNullable

      public StructType toNullable()
      Returns the same data type but set all nullability fields are true (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).

      Returns:
      (undocumented)
      Since:
      4.0.0