Passing values from parameter sets into DataStage jobs by command-line interface

When you run a job, you can pass the values from a parameter set into a job run by using the flag --paramset. For example, the following command passes all the parameter set values from the PS_SAMPLE parameter set, which has two fields, DATASET_NAME and FILE_NAME. The values of these two fields are passed in as job parameters to the job run. The example shows the values that are passed into the job run.

$ cpdctl dsjob run --project YongProject --job "TestParameterSet.DataStage job" --wait 300 --paramset PS_SAMPLE 
Job Run ID:  f29d843b-df5d-4b5f-84cc-967f07208fb2
...
...
"PS_SAMPLE.DATASET_NAME"='/ds-storage/BostonProperty-2022.ds'
"PS_SAMPLE.FILE_NAME"='/ds-storage/BostonProperty-2022.csv'

This example shows how you can pass in values from the value set QA that belongs to the parameter set PS_SAMPLE. The job run passes the value set from the value set QA into the job parameter section.

$ cpdctl dsjob run --project YongProject --job "TestParameterSet.DataStage job" --wait 300 --paramset PS_SAMPLE.QA
Job Run ID:  fec9ae5d-df5a-43c9-8bfe-c4cc11493885
...
...
"PS_SAMPLE.DATASET_NAME"='/ds-storage/BostonProperty-2022-QA.ds'
"PS_SAMPLE.FILE_NAME"='/ds-storage/BostonProperty-2022-QA.csv'
"flow_name"='TestParameterSet.DataStage job'

In this example, the parameter set PS_SAMPLE and all its fields are passed into the job run parameters. However, the field values are replaced if they exist in the overriding parameter set PROJDEF. The PROJDEF parameter set has DATASET_NAME and FILE_NAME, whose values are passed into the corresponding fields in the PS_SAMPLE parameter set and sent into job parameters as values. Only fields that are common are overridden. The other fields from the PS_SAMPLE parameter set are passed without changes.

$ cpdctl dsjob run --project YongProject --job "TestParameterSet.DataStage job" --wait 300 --paramset PS_SAMPLE=PROJDEF
Job Run ID:  70c403c5-938d-42be-8b04-dbd4e74947b8
...
...
"PS_SAMPLE.DATASET_NAME"='/ds-storage//ds-storage/BostonProperty-2022-PROJDEF.ds'
"PS_SAMPLE.FILE_NAME"='/ds-storage/BostonProperty-2022-PROJDEF.csv'
"flow_name"='TestParameterSet.DataStage job'

In this example, the values from the value set QA from the parameter set PS_SAMPLE are used in the job run parameters, but are overridden by the values from the parameter set PROJDEF.

$ cpdctl dsjob run --project YongProject --job "TestParameterSet.DataStage job" --wait 300 --paramset PS_SAMPLE.QA=PROJDEF
Job Run ID:  f867cd17-b913-4f5f-8091-720a71668877
...
...
"PS_SAMPLE.DATASET_NAME"='/ds-storage//ds-storage/BostonProperty-2022-PROJDEF.ds'
"PS_SAMPLE.FILE_NAME"='/ds-storage/BostonProperty-2022-PROJDEF.csv'
"flow_name"='TestParameterSet.DataStage job'

You can override a specific parameter by using the --param option. In this example, parameter set values from the value set QA from the parameter set PS_SAMPLE are used to pass in as job parameters but the value of the FILE_NAME field is overridden.

$ cpdctl dsjob run --project YongProject --job "TestParameterSet.DataStage job" --wait 300 --paramset PS_SAMPLE.QA=PROJDEF --param PS_SAMPLE.FILE_NAME=/ds-storage/BostonProperty-2022-OVERRIDE.csv 
Job Run ID:  ee4f233a-8ee1-4f84-a5a5-5c5ec550efd0
...
...

"PS_SAMPLE.DATASET_NAME"='/ds-storage//ds-storage/BostonProperty-2022-PROJDEF.ds'
"PS_SAMPLE.FILE_NAME"='/ds-storage/BostonProperty-2022-OVERRIDE.csv'
"flow_name"='TestParameterSet.DataStage job'