Data skipping configuration options

The following configuration options are available:

Types of metadata location

The parameter spark.ibm.metaindex.parquet.mdlocation is interpreted according to the URL type defined in the parameter spark.ibm.metaindex.parquet.mdlocation.type.

The following options are available:

• EXPLICIT_BASE_PATH_LOCATION: This is the default. An explicit definition of the base path to the metadata, which is combined with a data set identifier. This case can be used to configure the MetaIndexManager JVM wide settings and have all of data sets metadata saved under the base path.
• EXPLICIT_LOCATION: An explicit full path to the metadata.

You can set the spark.ibm.metaindex.parquet.mdlocation in two ways:

• By setting a JVM wide configuration

  This configuration is useful for setting the base location once for all data sets and should be used with the EXPLICIT_BASE_PATH_LOCATION type.

  The location of the metadata for each data set will be inferred automatically by combining the base path with a data set identifier.

  • For Scala:

      val jmap = new java.util.HashMap[String,String]()
      jmap.put("spark.ibm.metaindex.parquet.mdlocation", "/path/to/base/metadata/location")
      jmap.put("spark.ibm.metaindex.parquet.mdlocation.type", "EXPLICIT_BASE_PATH_LOCATION")
      MetaIndexManager.setConf(jmap)
    

  • For Python:

      md_backend_config = dict([
      ('spark.ibm.metaindex.parquet.mdlocation', "/path/to/base/metadata/location"),
      ("spark.ibm.metaindex.parquet.mdlocation.type", "EXPLICIT_BASE_PATH_LOCATION")])
      MetaIndexManager.setConf(spark, md_backend_config)
    

• By setting the metadata location for a specific MetaIndexManager instance

  This configuration is useful for setting a specific metadata location for a certain data set and should be used with the EXPLICIT_LOCATION type.

  • For Scala:

    val im = new MetaIndexManager(spark, uri, ParquetMetadataBackend)
    val jmap = new java.util.HashMap[String, String]()
    jmap.put("spark.ibm.metaindex.parquet.mdlocation", "/exact/path/to/metadata/location")
    jmap.put("spark.ibm.metaindex.parquet.mdlocation.type", "EXPLICIT_LOCATION")
    im.setMetadataStoreParams(jmap)
    

  • For Python:

    im = MetaIndexManager(spark, uri, 'com.ibm.metaindex.metadata.metadatastore.parquet.ParquetMetadataBackend')
    md_backend_config = dict([
    ('spark.ibm.metaindex.parquet.mdlocation', "/exact/path/to/metadata/location"),
    ("spark.ibm.metaindex.parquet.mdlocation.type", "EXPLICIT_LOCATION")])
    im.setMetadataStoreParameters(md_backend_config)
    

Limitations

Data skipping might not work if type casting is used in the WHERE clause. For example, given a min/max index on a column with a short data type, the following query will not benefit from data skipping:

select * from table where shortType > 1

Spark will evaluate this expression as (cast(shortType#3 as int) > 1) because the constant 1 is of type Integer.

Note that in some cases Spark can automatically cast the literal to the right type. For example, the previous query works for all other numerical types except for byte type, which would require casting as well.

To benefit from data skipping in such cases, make sure that the literal has the same type as the column type, for example:

select * from table where shortType > cast(1 as short)