Data metadata (SPSS Modeler)

Data metadata

This section describes how to set up the data model attributes based on pyspark.sql.StructField.

spss.datamodel.Role Objects

This class enumerates valid roles for each field in a data model.

BOTH: Indicates that this field can be either an antecedent or a consequent.

FREQWEIGHT: Indicates that this field is to be used as a frequency weight; this isn't displayed to the user.

INPUT: Indicates that this field is a predictor or an antecedent.

NONE: Indicates that this field is not used directly during modeling.

TARGET: Indicates that this field is predicted or a consequent.

PARTITION: Indicates that this field identifies the data partition.

RECORDID: Indicates that this field identifie the record id.

SPLIT: Indicates that this field splits the data.

spss.datamodel.Measure Objects

This class enumerates measurement levels for fields in a data model.

UNKNOWN: Indicates that the measure type is unknown.

CONTINUOUS: Indicates that the measure type is continuous.

NOMINAL: Indicates that the measure type is nominal.

FLAG: Indicates that the field value is one of two values.

DISCRETE: Indicates that the field value should be interpreted as a collection of values.

ORDINAL: Indicates that the measure type is ordinal.

TYPELESS: Indicates that the field can have any value compatible with its storage.

pyspark.sql.StructField Objects

Represents a field in a StructType. A StructField object comprises four fields:

name (string): name of a StructField
dataType (pyspark.sql.DataType): specific data type
nullable (bool): if the values of a StructField can contain None values
metadata (dictionary): a python dictionary that stores the option attributes

You can use the metadata dictionary instance to store the measure, role, or label attribute for the specific field. The key words for these attributes are:

measure: the key word for measure attribute
role: the key word for role attribute
displayLabel: the key word for label attribute

Example:

from spss.datamodel.Role import Role
from spss.datamodel.Measure import Measure
_metadata = {}
_metadata['measure'] = Measure.TYPELESS
_metadata['role'] = Role.NONE
_metadata['displayLabel'] = "field label description"
StructField("userName", StringType(), nullable=False, 
metadata=_metadata)