Package org.apache.spark.sql.types

Class DataType

Object
org.apache.spark.sql.types.DataType
Direct Known Subclasses:
ArrayType, BinaryType, BooleanType, CalendarIntervalType, DateType, DayTimeIntervalType, MapType, NullType, NumericType, ObjectType, StringType, StructType, TimestampNTZType, TimestampType, UserDefinedType, VariantType, YearMonthIntervalType
public abstract class DataType extends Object
The base type of all Spark SQL data types.

Since:
1.3.0

  • Constructor Details

    • DataType

      public DataType()

  • Method Details

    • COLLATIONS_METADATA_KEY

      public static String COLLATIONS_METADATA_KEY()

    • fromDDL

      public static DataType fromDDL(String ddl)

    • parseTypeWithFallback

      public static DataType parseTypeWithFallback(String schema, scala.Function1<String,DataType> parser, scala.Function1<String,DataType> fallbackParser)
      Parses data type from a string with schema. It calls parser for schema. If it fails, calls fallbackParser. If the fallback function fails too, combines error message from parser and fallbackParser.

      Parameters:
      schema - The schema string to parse by parser or fallbackParser.
      parser - The function that should be invoke firstly.
      fallbackParser - The function that is called when parser fails.
      Returns:
      The data type parsed from the schema schema.

    • fromJson

      public static DataType fromJson(String json)

    • equalsStructurally

      public static boolean equalsStructurally(DataType from, DataType to, boolean ignoreNullability)
      Returns true if the two data types share the same "shape", i.e. the types are the same, but the field names don't need to be the same.

      Parameters:
      ignoreNullability - whether to ignore nullability when comparing the types
      from - (undocumented)
      to - (undocumented)
      Returns:
      (undocumented)

    • equalsStructurallyByName

      public static boolean equalsStructurallyByName(DataType from, DataType to, scala.Function2<String,String,Object> resolver)
      Returns true if the two data types have the same field names in order recursively.
      Parameters:
      from - (undocumented)
      to - (undocumented)
      resolver - (undocumented)
      Returns:
      (undocumented)

    • equalsIgnoreNullability

      public static boolean equalsIgnoreNullability(DataType left, DataType right)
      Compares two types, ignoring nullability of ArrayType, MapType, StructType.
      Parameters:
      left - (undocumented)
      right - (undocumented)
      Returns:
      (undocumented)

    • equalsIgnoreCaseAndNullability

      public static boolean equalsIgnoreCaseAndNullability(DataType from, DataType to)
      Compares two types, ignoring nullability of ArrayType, MapType, StructType, and ignoring case sensitivity of field names in StructType.
      Parameters:
      from - (undocumented)
      to - (undocumented)
      Returns:
      (undocumented)

    • defaultSize

      public abstract int defaultSize()
      The default size of a value of this data type, used internally for size estimation.
      Returns:
      (undocumented)

    • typeName

      public String typeName()
      Name of the type used in JSON serialization.

    • json

      public String json()
      The compact JSON representation of this data type.

    • prettyJson

      public String prettyJson()
      The pretty (i.e. indented) JSON representation of this data type.

    • simpleString

      public String simpleString()
      Readable string representation for the type.

    • catalogString

      public String catalogString()
      String representation for the type saved in external catalogs.

    • sql

      public String sql()