0 / 0
Data imputation implementation details for time series experiments
Last updated: Jan 12, 2024
Data imputation implementation details for time series experiments

The experiment settings used for data imputation in time series experiments.

Data imputation methods

Apply one of these data imputation methods in experiment settings to supply missing values in a data set.

Data imputation methods for classification and regression experiments
Imputation method Description
FlattenIterative Time series data is first flattened, then missing values are imputed with the Scikit-learn iterative imputer.
Linear Linear interpolation method is used to impute the missing value.
Cubic Cubic interpolation method is used to impute the missing value.
Previous Missing value is imputed with the previous value.
Next Missing value is imputed with the next value.
Fill Missing value is imputed by using user-specified value, or sample mean, or sample median.

Input Settings

These commands are used to support data imputation for time series experiments in a notebook.

Data imputation methods for time series experiments
Name Description Value DefaultValue
use_imputation Flag for switching imputation on or off. True or False True
imputer_list List of imputer names (strings) to search. If a list is not specified, all the default imputers are searched. If an empty list is passed, all imputers are searched. "FlattenIterative", "Linear", "Cubic", "Previous", "Fill", "Next" "FlattenIterative", "Linear", "Cubic", "Previous"
imputer_fill_type Categories of "Fill" imputer "mean"/"median"/"value" "value"
imputer_fill_value A single numeric value to be filled for all missing values. Only applies when "imputer_fill_type" is specified as "value". Ignored if "mean" or "median" is specified for "imputer_fill_type. (Negative Infinity, Positive Infinity) 0
imputation_threshold Threshold for imputation. The missing value ratio must not be greater than the threshold in one column. Otherwise, results in an error. (0,1) 0.25

Notes for use_imputation usage

  • If the use_imputation method is specified as True and the input data has missing values:

    • imputation_threshold takes effect.
    • imputer candidates in imputer_list would be used to search for the best imputer.
    • If the best imputer is Fill, imputer_fill_type and imputer_fill_value are applied; otherwise, they are ignored.

  • If the use_imputation method is specified as True and the input data has no missing values:

    • imputation_threshold is ignored.
    • imputer candidates in imputer_list are used to search for the best imputer. If the best imputer is Fill, imputer_fill_type and imputer_fill_value are applied; otherwise, they are ignored.

  • If the use_imputation method is specified as False but the input data has missing values:

    • use_imputation is turned on with a warning, then the method follows the behavior for the first scenario.

  • If the use_imputation method is specified as False and the input data has no missing values, then no further processing is required.

For example:

"pipelines": [
      {
        "id": "automl",
        "runtime_ref": "hybrid",
        "nodes": [
          {
            "id": "automl-ts",
            "type": "execution_node",
            "op": "kube",
            "runtime_ref": "automl",
            "parameters": {
              "del_on_close": true,
              "optimization": {
	          "target_columns": [2,3,4],
	          "timestamp_column": 1,
	          "use_imputation": true
              }
            }
          }
        ]
      }
    ]

Parent topic: Data imputation in AutoAI experiments