Security for remote engines with DataStage Anywhere
Last updated: Mar 12, 2025
With remote runtime engines for DataStage® Anywhere, you retain full privacy for your sensitive data.
Overview
When you run a DataStage flow, the flow compiles as Orchestrate Shell code (OSH). At runtime, OSH code is converted into JSON data and is pushed to the remote engine where the job execution occurs. The remote engine sits within your VPC, so the data pipelines can be ran behind the secure firewall, without exposing any data to the control plane. You retain full data privacy in compliance with applicable sovereignty regulations. When a DataStage project is tied to a remote engine runtime execution, you cannot revert it back to Cloud Pak for Data as a Service as that might introduce security vulnerabilities.
Communication
DataStage uses API calls to communicate between the control plane and the data plane. Communication always goes from the remote engine to the control plane, including API calls and the basic communication protocol (TLS1.2). Each remote engine instance has its own encryption key (AES 256 encryption algorithm) that is automatically generated for data connections and credentials.
Logs are transmitted back to IBM Cloud to track entitlement (VPC) usage but are also available where the remote engine is located.
Remote Engine and Control Plan communication samples
Remote engine pulls in work type Compile:
[
{
"work_type": "compile",
"metadata": {
"status_id": "ds.runtime:flow.compile.status.details:a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4:e2ed740f-51e6-41b6-80d1-80838085d3b5",
"project_id": "a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4",
"flow_id": "e2ed740f-51e6-41b6-80d1-80838085d3b5"
},
"work": {
"px_executables": "UEsDBBQACAgIAJWTi1gAAAAAAAAAAAAAAAAoAAAAZTJlZDc0MGYtNTFlNi00MWI2LTgwZDEtODA4MzgwODVkM2I1Lm9zaO1Ya1PbOBf+7l/hXd4Zt7NxHOce43iHQtnSUqAE2AtlPLKtBBVHdmWZyzL973skx8TGcSjT3Z2deUsI6PLonKNzk3Q2NjY21N0wulFZdDPDVPejxPWj+M41B6Zpds2OOeiM+g9D7UF/1BqYI7Ot+tE8JiEOVMRVc2i1h1ZnqJqmuhUztd1qd5WNb/2RFNTJydYvry31OLpxf8EUM8Qj5powpx7GWU+Z5ePFUTWKOYloouiJf4nnSGXYj1igvFBUdftw//T9gWtaCWeEzs7NVuticznerhnv1Ix3a8Z7NeP9mvFBzfiwZnxUM262yhMvFT3bfKKaLUURWspUGT4o6ZwEmPIX2iM1ay831U+RN4+oWw9IOJrhfP7oFhAPAO3lhbRJyuOUJ0rLUc8v1Mc0rH1Cr1yzea0pm3+z15wwRJNpxOb4sc/wfGa1z0xDNFNZShWdojlWtRIhTVHWa7GMruiwMl3S4HZheiImFkrco5kO7Sd0WNV3iV+GbAukWTvd+SeMsffqvbsdRmngHnqfsM/dCYguNl62THzrMRLM8ErDKPohI6AmJLqZ3hT9t/f7RywCKCc4UTX759t5qF5jlgBm/FEzm62PmoqpHwUQEzBwerKrm30Y+9mxCwuLK8yPmmNvR3OwmvhPOb7lKr+LMcwSymG2bRuLccc+Q4wg+gDIgg8wwNk2FpOOvYMTn5EYtnOWcVqNr8BASAQSih2fwIKyGLppG6Vpxz7ePipjgCiMOULgbEOnCejNsb3Uv8JVqe0fzrd3tk62zn1hLD2SxtKTzFjihIA2ogFigZ5+Hl1cAOGMkmO/RdfIsd9gFE/In48kfSDb7vXlohwGzWzdDg7JnHAcTO7AwreO/TmNOHZpCp5J/CQn50VRWKRnSmplrGNPCUu4GxKK3UuMAszqlrfk8grcsXOPqVdQyqf6UC7PsY6w36NdTOGYdGUaqSUUII6afnK9EGWBd+w4tyycs2vFLwAd+1d0jd+AhUIp0FYcYxqc7u2sp/AAg3aZwA2D3bjzKKgzaEZgCXNsCn+FLbyl2leuKeAc22cYAYWyV64WtgRdsPOjlPIabuaSm4QtjCKSHlq/poATcYPZdoiS5GCtNeFq1CQefEnS9P2mDKMsihZB1JRpMMuCiyQomZXoy76MU8gzFJAyFRA0d1MW1jO/5DxOLMMAYJPjhGfspTwglyEPGcLvDB5dYSq5Lkg6NoqJe4Xvctoxg4jywZGrTO6DhELmA6//snv0ip68G8S7x9OYXE3OfjJeH37Y5x+OD/p/DJPfY3Y0m7waXg0OeTv43dya97wPuP/u3dbN2cElP6YnUoQFZ8dmOIlS5sNhSEWSEY1gjaIZta7FLQ7NYMaKUy8kvrUqbVmzMPJQaCHD88xRe4h6nSDArWHfH/RxHwWoNRj0MBpNkdXp9vxBr9/S+50+1qHj6cOuh/QpGnb6QzTCI69nWVLsVdI69lrzJJ1mmug3D4Z5JGdmMhTHQTRHhGYYyUtaCBzkeMFzb8GyPq7N3KeqS+RZsHQqY3kKOpoCKV5OwbGr3SvX2fGVn6MNJSQeQ+xu/OMnSNqW9CkPviQxfN+o+rpR9fX9jELjfy92Jm8O379+aTSbxs4ETqcYEhjlieERCtQSH/yi+QmxzaeQK0Ls6WX8krBAJM67MhB+tyavDiBZGNgPSZxgIw5T8DDYYTGyQ3EREUNwpiZ6GM2EE+pms13l/TySbyeHB923QAk0Dp+ndyIV9S0cRdLImnq32fv2HUh6EYPwaYrPOmrgTobwpAdFkkhvP7HmGWaBYxn7QpUrKFb0+HZvshP58uTD7Ksc72sdVOxQfw64DNz8saHk11NwikYelSFJRGSemZswalnfQ/J7SH4PyX8rJMVhuAjKRRwuTkhxdx9X/V/5osH7Fe7UV+UHK0dsBhfe1jNerv/uw/UYi13ANSFv/OdesvlNuXCRUcVNZqnae2VnMvmwL+iO7/N63ths5CWzZbO9bHaXzd6y2V82B8vmcNkcFei2xuaXRsb7iGGfSEMVBGi1CiIsO+1ip1vs9IqdfrEzKHaGxc6oxKcleplQsjIK2q3WRutqnc+tmf7na6MfNRGXa4p59WWramVvPbZU5qvmh6PfVlb66qp3f3d5boI/p+JtiEJXPnrLVTl8G0eMf02BXVXvp4QClUAUQsaYBg1VFmfGQQQPNNxQswmtoTVUmoaCHQ6DsXZwur+vffk/LdALlauaeP11RCEIclcEaV3WVET1XrqJCg8zTmiKnyg/VyxZddSVkEeF/CVmFyBf65uL0vFfUEsHCPBDvuS5BgAAWhoAAFBLAwQUAAgICACVk4tYAAAAAAAAAAAAAAAAFQAAAFRyYW5zZm9ybWVyXzEudHJ4Lm9zaOWT30vDMBDH3/NX3NOqjGirc/5iT6uCMO3YJgpjlJBeu2CTlEsK+/OtbDr2EMVH8eXIfXL3TfIlxzwJ40pLGrgyTeudXKMWQCgtFeyIAYyzyfPjU57cOE/KVMskjle3e34W4OcBPgjwiwAfBvhlgF8F+HWAJ/HhxjFjjNvWf+dF6G2/9egPePGTFf/nW3DcNITOKWtKVSMse+l8+jq3LUlMFa1OF5+jhJQnJ542jJe1qEBa3XQNjO8WBFHV70dfuW08RFwCz4CX04cx8BdjeYHdaVJ4LIDr4aALvjU4qtAgqa5YS20LrEdOi7ruxGpl3vbSH9lW160F7TQ6boRGOLgo44Ui6M0W+X02Se9m71BLBwjc1e+EDgEAABIEAABQSwMEFAAICAgAlZOLWAAAAAAAAAAAAAAAABEAAABUcmFuc2Zvcm1lcl8xLnRyeJ2Qz07DMAzG75X6Dj7CqS27ICreAO0wJK4otJ5mSOPJcTYJxLvjpCvaJnHhkj9f7N/3xU0DI24pIHASoLBP2nBS28BT+IDgJox1VR7yGVp4Mv216+tqrjtX7y7VblZXpmYEKTlPnwhfdQXNhfHAQYU9HJyQe/PZ0sLoPWz4uMF3HBTHtl/EdfL+GfXFSVvQ33mZHAXPvM/0mb84RgTdIciJA8LHXyOrPLeAR+j6pR8PzienaOl8mkKEm8A5aVQxL423sGUpY3o4fd/6jkKKggPLCCXxFb5o/4GvruDdH/AyDBssxV0ehV1/AFBLBwg/DVWC6gAAAOQBAABQSwMEFAAICAgAlZOLWAAAAAAAAAAAAAAAABMAAABvc2hfZGVwZW5kZW5jeS5qc29uq1bKyymOz00siAfROfnJiTmpSlbVSpm5BakVBfHJGYlFxaklSlZKoSFuuhZKOkrFJUWZeelYJDLzCkpLsIjnl5Zgk6jVUcpISiwGWhYdWwsAUEsHCFZRERFRAAAAhgAAAFBLAQIUABQACAgIAJWTi1jwQ77kuQYAAFoaAAAoAAAAAAAAAAAAAAAAAAAAAABlMmVkNzQwZi01MWU2LTQxYjYtODBkMS04MDgzODA4NWQzYjUub3NoUEsBAhQAFAAICAgAlZOLWNzV74QOAQAAEgQAABUAAAAAAAAAAAAAAAAADwcAAFRyYW5zZm9ybWVyXzEudHJ4Lm9zaFBLAQIUABQACAgIAJWTi1g/DVWC6gAAAOQBAAARAAAAAAAAAAAAAAAAAGAIAABUcmFuc2Zvcm1lcl8xLnRyeFBLAQIUABQACAgIAJWTi1hWURERUQAAAIYAAAATAAAAAAAAAAAAAAAAAIkJAABvc2hfZGVwZW5kZW5jeS5qc29uUEsFBgAAAAAEAAQAGQEAABsKAAAAAA==",
"ds_flow_environment": {
"FlowEnvironment": {},
"FlowParameters": {}
}
}
}
]Remote engine pulls in work type Run, example 1:
{
"work_type": "job_run",
"metadata": {
"project_id": "a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4",
"job_run_id": "69f6a270-6912-4321-a913-29277628eed1",
"job_id": "aee11bf2-5be7-4abd-8930-1b0bbba4c6dd",
"accept_language": "en-US,en;q=0.5"
},
"work": {
"FlowEnvironment": {
"S3_BUCKET": "testauth-donotdelete-pr-rs1hqmcbhwuzud",
"AWS_ID": "4102d83e9d19412ca4ffa6a585f7b80b",
"S3_REGION": "us-south",
"AWS_KEY": "9aede861b3cb994bfdc605af10d58c2cf87ccb8e8c8d005c",
"NUM_PARTITIONS": "2",
"$REMOTE_ENGINE": "2220b98e-4f80-49d8-827c-371168a0d920",
"S3_ENDPOINT": "https://s3.us-west.cloud-object-storage.test.appdomain.cloud",
"HARDWARE_SPECIFICATION": "{'datastage':{'compute':{'mem':{'size':'16Gi'},'cpu':{'model':'','units':'4'}},'num_conductors':1,'num_computes':0,'conductor':{'mem':{'size':'4Gi'},'cpu':{'model':'','units':'1'}}}}"
},
"FlowParameters": {
"DSJobRunName": "",
"job_name": "rowgen-cos.DataStage job",
"flow_name": "rowgen-cos",
"project_name": "Test-Auth",
"DSMetricsRepoEnabled": "false"
},
"FlowMessageHandler": {},
"FlowOsh": "UEsDBBQACAgIAJykiFgAAAAAAAAAAAAAAAAoAAAAYjc0OTE1OGItMjI3OS00Nzg4LTgyY2MtNjE2MDdmYTk2OWYwLm9zaO1YW1PbOBR+96/wlp1JmcV2nBvBGHcCKS0UCBBg2wXGI9tKEDiWKylclul/3yPZJgm5UKY73X0oISDJn75zjs5FspaWlpb07Zje6Yze9XFihJTrIR2kJMaRjoReKTvVhlMt6+Wm3koZ9Cs1belHfxSD3j1pfXjv6Mf0zv+AE8yQoMy34ZneSbOe1i/Gx0d1mgpCE64ZPLzCA6QzHFIWaW81Xd/q7J3uH/i2wwUjSf/cLpcv10fjlTnj1TnjtTnj9cnxZc3IdOC6XdY0qWxmUfyk6zmJcCLelp5ZW1pe169pMKCJPx/ABerj4vnhPSCeAKXlS7U0Q5EOBdfKnn5+qT/ncPZIcuPb5m1JW/+Xnbezue9vxXQY+Z3gGofC74JEqe2kH9P7gJGoj2e6UTM6jIBtSHYzYzXj8/7eIaMAFQRzveS+ux/E+i1mHDAbFyXbLF+UdJyENAI3wMDpybZhN2DsneeOTRyfYV+UPHeLDmCx5f9E4Huhi4cUw1OSCHhaca183HPPECMoeQJk/gYMSHat/KHntjEPGUnBnLNM0mz8FAyURKChtPgEJkyqYdiuNfHYc4+3DicxQApjnlQ4M+iUw7p5bjAMb/C01u5v51vt1knrPJTOMqhylsEzZ8mshzZKIsQiY/h17fISiDMmz91Ft8hzP2KUdsnfzzR9oq3UG2pSAYNmNq+NYzIgAkfdB/Dwved+HVKB/WQ4wIyEvKALKI3H+WzFNon13B5hXPgxSbB/hVGE2bzpZTV9Cu65RcTMX6Ch6BlNNb3AetJ/z6zoQYH0EzTA84kiJJAZ8ttclRzvuWnhWaiwC9UfA3run+gWfwQPxUqhVpriJDrdaS9meIJBe5LgjoE1/oBG8xyaEYxgnpvAX+mLYLTsM+eM4Tw3ZBgBw2RUzlZ2ApqLC+kwEXOk2SNpCpY7pUfZAC2eM4aTeYPZVow4P1joTdgUTRLAl3AzDE2VRlkW5UlkqjKYVcG8CCphE/yqr/IU6kwCSFUKCBr4QxbPF34lRModywKgKTAXmXilD+hlqZ2BiAdL0BucKKk5peeilPg3+KHgThlkVAiBPC3kMeIJVD6I+m/bh5vJyafVdPu4l5Kb7tkf1vvO0Z44Oj5o/NXkX1J22O9uNm9WO6ISfbFbg3pwhBufPrXuzg6uxHFyolTIJXsuw5wOWQg7WCKLjGxECxaaJc6t3L9RH5446TCISejMKltOP6YBih1kBYG9VmmiejWKcLnZCFcbuIEiVF5drWO01kNOtVYPV+uNstGoNrABncBo1gJk9FCz2miiNbwW1B1HqT1LW89d6B5eNYfcuHtyzDM9M5ehNI3oAJEkwyhZykMQIMe5zJ1c5Py8touYmp6i9oJRUFmjXdAraVDi1SPYdkuP2m22fRX76IoWk4Ah9rDx5hqKtqNiKoAv4VYYWtOxbk3H+l7GsPL723b3Y2f//bJlmla7C7tTCgUsEdwKSAJsPIS4MK8RW38JOSPFXp4mrgiLZOF8mATCb6u7eQDFwsJhTFKOrTQeQoSBheOZHcuDiByCPZUbMe3LIDRsszIt+3WUu93OQW0XmGDF4fOyJWqhfkSiLBpZ06iZ9R+3QPFRBuljys8iNggnS0bS00ISalRemPMKt8C2jEO5lDMYp9Zxd6fbpqHa+TD7rsD73gCVFhqvAU8C19+saMXxFIJipcjKmHCZmWf2Oow6zq+U/JWSv1LyZ6Wk3AzzpMzzMN8h5dl9Yzr+tW8leH+FM/XN5AurQKwPB97yK95cf+6L6zGWVsAxoWj8795ki5Py2EFGlyeZ0dI+au1u92hP8m48Fjc/G/ZKcdkzalZHzdqoWd+wv61kHIcMh0Qt+BhRuTxGNepUxzu18U5ddjJKdTMFNv73d1MXJRmkC66k5t/hTN9OLcZOXFRNJ8vh5/y6aifJbqvcF2+r/gFQSwcIHQDnraoFAAChFAAAUEsDBBQACAgIAJykiFgAAAAAAAAAAAAAAAATAAAAb3NoX2RlcGVuZGVuY3kuanNvbqtWysspjs9NLIgH0Tn5yYk5qUpW1UqZuQWpFQXxyRmJRcWpJUpWSqEhbroWSjpKxSVFmXnpWCQy8wpKS7CI55eWYJOo1VHKSEosBloWHVsLAFBLBwhWURERUQAAAIYAAABQSwECFAAUAAgICACcpIhYHQDnraoFAAChFAAAKAAAAAAAAAAAAAAAAAAAAAAAYjc0OTE1OGItMjI3OS00Nzg4LTgyY2MtNjE2MDdmYTk2OWYwLm9zaFBLAQIUABQACAgIAJykiFhWURERUQAAAIYAAAATAAAAAAAAAAAAAAAAAAAGAABvc2hfZGVwZW5kZW5jeS5qc29uUEsFBgAAAAACAAIAlwAAAJIGAAAAAA==",
"FlowResourceRequirements": {
"Core": 0,
"Storage": 0,
"Memory": 0,
"Partitions": 0,
"Compute": 0
},
"FlowLimits": {
"WarnLimit": 100,
"RowLimit": 0
},
"FlowId": "b749158b-2279-4788-82cc-61607fa969f0",
"Secrets": [],
"FlowProjectSettings": {
"runMessageHandlerId": "",
"runPriorityQueue": "Medium",
"formatDateString": "",
"formatTimestampString": "",
"formatTimeString": "",
"formatDecimalSeparator": "",
"nlsMap": "",
"collationSequence": "OFF"
}
}Remote engine pulls in work type Run, example 2:
{
"work_type": "job_run",
"metadata": {
"project_id": "a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4",
"job_run_id": "bfdea00d-6ede-45b6-8cf7-540f17e520ee",
"job_id": "eb9de08a-37ef-4c47-9591-abe4c4a57d1b",
"accept_language": "en-US,en;q=0.5"
},
"work": {
"FlowEnvironment": {
"S3_BUCKET": "testauth-donotdelete-pr-rs1hqmcbhwuzud",
"AWS_ID": "4102d83e9d19412ca4ffa6a585f7b80b",
"S3_REGION": "us-south",
"AWS_KEY": "9aede861b3cb994bfdc605af10d58c2cf87ccb8e8c8d005c",
"NUM_PARTITIONS": "2",
"$REMOTE_ENGINE": "2220b98e-4f80-49d8-827c-371168a0d920",
"S3_ENDPOINT": "https://s3.us-west.cloud-object-storage.test.appdomain.cloud",
"HARDWARE_SPECIFICATION": "{'datastage':{'compute':{'mem':{'size':'16Gi'},'cpu':{'model':'','units':'4'}},'num_conductors':1,'num_computes':0,'conductor':{'mem':{'size':'4Gi'},'cpu':{'model':'','units':'1'}}}}"
},
"FlowParameters": {
"DSJobRunName": "",
"job_name": "rowgen-cos_copy_1711141317396_copy_1712769071912.DataStage job",
"flow_name": "rowgen-cos_copy_1711141317396_copy_1712769071912",
"project_name": "Test-Auth",
"DSMetricsRepoEnabled": "false"
},
"FlowMessageHandler": {},
"FlowOsh": "UEsDBBQACAgIAKyKilgAAAAAAAAAAAAAAAAoAAAAZTJlZDc0MGYtNTFlNi00MWI2LTgwZDEtODA4MzgwODVkM2I1Lm9zaO1YW1PbOBR+96/wlp1JmcVxnHuMcQdIaaFAgADbLjAe2VaCwLFdSeGyTP/7Hsl2EuMklOluZx9KCMjSp+8cnZtkraysrKg7QXSv0uh+iEPNi5jjRfGjY7QMw6gbNaNV6zQnXdVWs1NpGR2jqnrRKCYB9lXEVaNlVg3TaKpGRd2MqVqtVOvKyo/+SAa1f7r54b2pnkT3zgccYop4RB0DxtRenDwpw6x/tleNYk6ikCka867xCKkUexH1lbeKqm739s8ODh3DZJyScHhhVCpX69P+6oL+2oL++oL+Rr5/VdESHRjYSVGEssmKgomuF8THIX9berba0uq6ehO5oyh0FgMYR0OcjR89AGICKK1eSdOMeTzmTKnY6sWV+pzD3CfhrWOU70rK+r/svFOKQjaI6Ag/dx3PRua7bhCgoUrHoaKFaITVUo6opCjLrZhHF2xYGM5ZcHtmuC8GUiPuhokNrRdsWLR3Tl6CrP4X1t7dOnC2g2jsOz33Bnvc6YNuYmV508cPLiX+EM+1vKL1KAE7IPGYGEbRPh/sH9EIoJxgppasdw+jQL3DlAFm47JklCuXJRWHXuRD0EPH2emOZjSh751tzUycnWFclmxrOxqBW8T/kOMHrvLHGMMoCTmMVi097betc0QJCieAJLsAA5ItPR20rS5mHiUxLOc8kTQfX4CBkgg0FCs+hQl5NTTD0nPDtnWyfZTHACn02ULhZEFnDOxmW+7Yu8VFra3fLra7m6ebF55wlhZJZ2kscZaoxNBGoY+or42/dq6ugDhhsq09dIds6yNGcZ/8/UzTCW210ZSTMhg0k3ldHJAR4djvP4KHH2zr6zji2AnHEJnEYxmdG0XBLJ8h2fJY2xoQyrgTkBA71xj5mC6aXpHTC3DbyiJmsYHGfKC15fQMawv/PVvFALYjR9aJhUQ+4qjssbtUlRRvW3HmWdjPlqo/A7StP9Ed/ggeCqRCm3GMQ/9st7ucYQKDdp7gnsJqnFHkL3JoQjCF2VYIf4Uv3KnZ586ZwdmWRzEChnxUzlc2B03FedE45AukGVNpEpY6RRQ9tHzODE7kDabbAWLscKk34QhSJi58CSt7XlmmUZJFaRKVZRlMqmBaBKWwHL98lnkKdSYEpCwFBI2cMQ0WC7/mPGamrgOwzDHjiXipD+ily12E8EedR7c4lFJTSttCMXFu8WPGHVPIKA8CuSjkyWchVD6I+m87R1vh6adWvHMyiMlt//wP/X3veJ8fnxw2/2qzLzE9Gva32retHq/6X4zNUcM9xs1Pnzbvzw+v+Ul4KlVIJdsWxSwaUw92u1AUGdHwlxiahuadOC2hIYyY8dgNiGfOK1vmMIhcFJhId12jU22jRs33caXd9FpN3EQ+qrRaDYw6A2TW6g2v1WhWtGatiTV4cLV23UXaALVrzTbq4I7bME2p9jxtbWupe1itPGba/cQxz/RMXIbi2I9GiIQJRsqSHoIAOUll7qYiF+e1kcVUcYrcC6ZBpU93QbukQImXQ7Dtlp6Uu2T7yvbRNSUgLkX0cePNDRRtU8aUC1/CdM/Ti7GuF2N9P2FY+/1tt/+xd/B+VS+X9W4fdqcYCljIme6SENiYB3FRvkF0/SXknBR7eRq/JtQXhfMxD4Tfzf7WIRQLHXsBiRnW42AMEQYrnM3sQBxERBfsqUwLoqEIQs0oV4uyX0e51+8d1veACSwOn5dXIg31IxJF0UiaWr3c+PEVSL6IQvqUxWcZG4STLiJpYkgSadUX5rzCLbAtY0+Ycg5jwY57u/1u5MmdD9PvCrzvDVCxQu014Dxw/c2akh1PISjWsqwMCBOZeW6sQ69p/krJXyn5KyV/VkqKzTBNyjQP0x1SnN03ivGvfCvB+yucqW/zL6wc0SEceCuveHP9uS+uJ1isAo4JWeN/9yabnZRnDjKqOMlMTfukdPv9433Bu/GU3ZttGGvZldu0WZ0269NmY8P4tpZwHFHsEWnwGaJKZYZq+lCdfajPPjTEQ0Ip7wFhjcWbwEU3e6+9Ifzem8DLkgjSJVdXi+9wivdYy7G5S61ishx9nnuvtfiu6h9QSwcIxiK7RwoGAAAzFgAAUEsDBBQACAgIAKyKilgAAAAAAAAAAAAAAAAVAAAAVHJhbnNmb3JtZXJfMS50cngub3NolZBPS8NAEMXv+ynm1ChltNHqQempURCqKW1FoZSwbCbp4v4Jsxvoxzdibekhopdh5sfMe48RkaULlWcLqF3TxqC2ZCUwKc+lOBMA03z2+vxSpHchsnb1Oh2NNvdHftXDr3v4uIffnPJzIQT6Nv4Wqc/iv1H/HAlp1zCFoL2rtCFYD7Ll/H3pW1aUad5crn6+SVykF5F3Aisja1DeNt2BwH3DkNTDYXKYfRMhQQWYA1bzpyngm/NYUuemZKQS0N6OuxJbR5OaHLHulq2yviQzCVYa04kZ7T6O0l/Tt27YSt5rdNxJS3ASVGCpGQaLVfGYz7KHxSdQSwcIKooocfUAAAAVAgAAUEsDBBQACAgIAKyKilgAAAAAAAAAAAAAAAARAAAAVHJhbnNmb3JtZXJfMS50cnhdkM9OwzAMxu+V+g4+wqkdJ0TFGyAOm8QVhdbTDG48Oc4qgXh3nGxFg0v+fLG/3+d0HUy4p4ggWYHiMVsn2XwDpvgBMcyY2qY+lDP08OT662Zom3PdtXrnaikmo8D0ifDVNtD9QYwSTYXhFJTCGxdzx9o9bGXZ4juOhlM/rOJzZt6hvQTtq/V3WeZAkUWOxf3svxITgh0Q9OIDKssvyCuvEfAIm2Htx1PgHAw9Hec5JriJUpImU2dZuoW9aP2Qh8ug3rcoGSqOohPUxP/si1bz+uyUDiWtX38AUEsHCAOe9m/ZAAAAcQEAAFBLAwQUAAgICACsiopYAAAAAAAAAAAAAAAAEwAAAG9zaF9kZXBlbmRlbmN5Lmpzb26rVsrLKY7PTSyIB9E5+cmJOalKVtVKmbkFqRUF8ckZiUXFqSVKVkqhIW66Fko6SsUlRZl56VgkMvMKSkuwiOeXlmCTqNVRykhKLAZaFh1bCwBQSwcIVlEREVEAAACGAAAAUEsBAhQAFAAICAgArIqKWMYiu0cKBgAAMxYAACgAAAAAAAAAAAAAAAAAAAAAAGUyZWQ3NDBmLTUxZTYtNDFiNi04MGQxLTgwODM4MDg1ZDNiNS5vc2hQSwECFAAUAAgICACsiopYKooocfUAAAAVAgAAFQAAAAAAAAAAAAAAAABgBgAAVHJhbnNmb3JtZXJfMS50cngub3NoUEsBAhQAFAAICAgArIqKWAOe9m/ZAAAAcQEAABEAAAAAAAAAAAAAAAAAmAcAAFRyYW5zZm9ybWVyXzEudHJ4UEsBAhQAFAAICAgArIqKWFZRERFRAAAAhgAAABMAAAAAAAAAAAAAAAAAsAgAAG9zaF9kZXBlbmRlbmN5Lmpzb25QSwUGAAAAAAQABAAZAQAAQgkAAAAA",
"FlowResourceRequirements": {
"Core": 0,
"Storage": 0,
"Memory": 0,
"Partitions": 0,
"Compute": 0
},
"FlowLimits": {
"WarnLimit": 100,
"RowLimit": 0
},
"FlowId": "e2ed740f-51e6-41b6-80d1-80838085d3b5",
"Secrets": [],
"FlowProjectSettings": {
"runMessageHandlerId": "",
"runPriorityQueue": "Medium",
"formatDateString": "",
"formatTimestampString": "",
"formatTimeString": "",
"formatDecimalSeparator": "",
"nlsMap": "",
"collationSequence": "OFF"
}
}
}Remote engine pushes job status, example 1:
{
"Job Status": "RUN",
"HB_INSTANCE_HOST_URI": "",
"Project Id": "a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4",
"Queue End": 1712608618451,
"px-instance-id": null,
"send-running-count": 1,
"Job Startup Time": "0",
"Flow Run Id": "69f6a270-6912-4321-a913-29277628eed1",
"message-type": "JobStatus",
"Queue Start": 1712608618449,
"msg_count": 0,
"Flow Id": "aee11bf2-5be7-4abd-8930-1b0bbba4c6dd",
"Job Production Run Time": "0"
}Remote engine pushes job status, example 2:
{
"Job Status": "FOK",
"HB_INSTANCE_HOST_URI": "",
"job_metrics": {
"check_end": 1.712608634152849E9,
"start_end": 1.712608618652393E9,
"finished": 1.712608650705897E9,
"total_stages": 2,
"link_metrics": [
{
"start_time": 1.712608634E9,
"rows_read": 10,
"rows_written": 10,
"stop_time": 1.712608634E9,
"source": "Row_Generator_1",
"state": "finished",
"dest": "IBM_Cloud_Object_Storage_1",
"link_name": "Link_1"
}
],
"total_seconds_cpu": 21.15,
"run_end": 1.712608650705897E9,
"duration": 32.11314010620117,
"total_rows_read": 10,
"check_begin": 1.712608618652393E9,
"run_begin": 1.712608634152849E9,
"stage_metrics": [
{
"stage_seconds_cpu": 21.15,
"start_time": 1.712608648E9,
"rows_read": 10,
"rows_written": 0,
"stop_time": 1.71260865E9,
"stage_type": "cloudobjectstoragePX",
"stage_name": "IBM_Cloud_Object_Storage_1",
"state": "finished"
},
{
"stage_seconds_cpu": 0.0,
"start_time": 1.712608634E9,
"rows_read": 0,
"rows_written": 10,
"stop_time": 1.712608634E9,
"stage_type": "PxRowGenerator",
"stage_name": "Row_Generator_1",
"state": "finished"
}
],
"job_id": "360",
"start_begin": 1.712608618592757E9,
"state": "finished",
"total_rows_written": 10
},
"Project Id": "a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4",
"Queue End": 1712608618451,
"px-instance-id": null,
"send-running-count": 7,
"S3_LOG_LOCATION": "DataStage\/69f6a270-6912-4321-a913-29277628eed1.job.log",
"Job Startup Time": "0",
"Flow Run Id": "69f6a270-6912-4321-a913-29277628eed1",
"message-type": "JobStatus",
"Queue Start": 1712608618449,
"msg_count": 7,
"Flow Id": "aee11bf2-5be7-4abd-8930-1b0bbba4c6dd",
"Job Production Run Time": "32"
}Remote engine pushes job run metrics:
{
totalCpu: 4000
usedCpu: 0.0
totalMemory: 2014576640
usedMemory: 115087176
totalDisk: 250059685888
usedDisk: 160418521088
engineInitialized: 1712608520755
}
computes: null
}Sensitive data
To avoid writing data to logs, avoid using the Peek Stage or the Asset Browser functions. Instead, use sequential files to analyze the actual data and data types that are used in the job design and to parametrize all file names and connections.
Was the topic helpful?
0/1000