Sécurité des moteurs distants avec DataStage Anywhere

Présentation

Lorsque vous exécutez un flux DataStage, celui-ci se compile sous forme de code Orchestrate Shell (OSH). Lors de l'exécution, le code OSH est converti en données JSON et est envoyé au moteur distant où se produit l'exécution du travail. Le moteur distant se trouve dans votre VPC, de sorte que les pipelines de données peuvent être exécutés derrière le pare-feu sécurisé, sans exposer de données au plan de contrôle. Vous conservez l'entière confidentialité de vos données, conformément aux règles de souveraineté applicables. Lorsqu'un projet DataStage est lié à l'exécution d'un moteur à distance, vous ne pouvez pas revenir à Cloud Pak for Data as a Service car cela pourrait introduire des vulnérabilités en matière de sécurité.

Communication

DataStage utilise des appels API pour communiquer entre le plan de contrôle et le plan de données. La communication va toujours du moteur distant au plan de contrôle, y compris les appels API et le protocole de communication de baseTLS1.2 Chaque instance de moteur à distance possède sa propre clé de cryptage (algorithme de cryptage AES 256) qui est automatiquement générée pour les connexions de données et les informations d'identification.

Les journaux sont transmis à IBM Cloud pour suivre l'utilisation des droits (VPC), mais ils sont également disponibles là où se trouve le moteur distant.

Exemples de communication pour le plan de contrôle et de commande à distance

Le moteur à distance tire dans le type de travail Compile :

[
                    {
                        "work_type": "compile",
                        "metadata": {
                            "status_id": "ds.runtime:flow.compile.status.details:a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4:e2ed740f-51e6-41b6-80d1-80838085d3b5",
                            "project_id": "a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4",
                            "flow_id": "e2ed740f-51e6-41b6-80d1-80838085d3b5"
                        },
                        "work": {
                            "px_executables": "UEsDBBQACAgIAJWTi1gAAAAAAAAAAAAAAAAoAAAAZTJlZDc0MGYtNTFlNi00MWI2LTgwZDEtODA4MzgwODVkM2I1Lm9zaO1Ya1PbOBf+7l/hXd4Zt7NxHOce43iHQtnSUqAE2AtlPLKtBBVHdmWZyzL973skx8TGcSjT3Z2deUsI6PLonKNzk3Q2NjY21N0wulFZdDPDVPejxPWj+M41B6Zpds2OOeiM+g9D7UF/1BqYI7Ot+tE8JiEOVMRVc2i1h1ZnqJqmuhUztd1qd5WNb/2RFNTJydYvry31OLpxf8EUM8Qj5powpx7GWU+Z5ePFUTWKOYloouiJf4nnSGXYj1igvFBUdftw//T9gWtaCWeEzs7NVuticznerhnv1Ix3a8Z7NeP9mvFBzfiwZnxUM262yhMvFT3bfKKaLUURWspUGT4o6ZwEmPIX2iM1ay831U+RN4+oWw9IOJrhfP7oFhAPAO3lhbRJyuOUJ0rLUc8v1Mc0rH1Cr1yzea0pm3+z15wwRJNpxOb4sc/wfGa1z0xDNFNZShWdojlWtRIhTVHWa7GMruiwMl3S4HZheiImFkrco5kO7Sd0WNV3iV+GbAukWTvd+SeMsffqvbsdRmngHnqfsM/dCYguNl62THzrMRLM8ErDKPohI6AmJLqZ3hT9t/f7RywCKCc4UTX759t5qF5jlgBm/FEzm62PmoqpHwUQEzBwerKrm30Y+9mxCwuLK8yPmmNvR3OwmvhPOb7lKr+LMcwSymG2bRuLccc+Q4wg+gDIgg8wwNk2FpOOvYMTn5EYtnOWcVqNr8BASAQSih2fwIKyGLppG6Vpxz7ePipjgCiMOULgbEOnCejNsb3Uv8JVqe0fzrd3tk62zn1hLD2SxtKTzFjihIA2ogFigZ5+Hl1cAOGMkmO/RdfIsd9gFE/In48kfSDb7vXlohwGzWzdDg7JnHAcTO7AwreO/TmNOHZpCp5J/CQn50VRWKRnSmplrGNPCUu4GxKK3UuMAszqlrfk8grcsXOPqVdQyqf6UC7PsY6w36NdTOGYdGUaqSUUII6afnK9EGWBd+w4tyycs2vFLwAd+1d0jd+AhUIp0FYcYxqc7u2sp/AAg3aZwA2D3bjzKKgzaEZgCXNsCn+FLbyl2leuKeAc22cYAYWyV64WtgRdsPOjlPIabuaSm4QtjCKSHlq/poATcYPZdoiS5GCtNeFq1CQefEnS9P2mDKMsihZB1JRpMMuCiyQomZXoy76MU8gzFJAyFRA0d1MW1jO/5DxOLMMAYJPjhGfspTwglyEPGcLvDB5dYSq5Lkg6NoqJe4Xvctoxg4jywZGrTO6DhELmA6//snv0ip68G8S7x9OYXE3OfjJeH37Y5x+OD/p/DJPfY3Y0m7waXg0OeTv43dya97wPuP/u3dbN2cElP6YnUoQFZ8dmOIlS5sNhSEWSEY1gjaIZta7FLQ7NYMaKUy8kvrUqbVmzMPJQaCHD88xRe4h6nSDArWHfH/RxHwWoNRj0MBpNkdXp9vxBr9/S+50+1qHj6cOuh/QpGnb6QzTCI69nWVLsVdI69lrzJJ1mmug3D4Z5JGdmMhTHQTRHhGYYyUtaCBzkeMFzb8GyPq7N3KeqS+RZsHQqY3kKOpoCKV5OwbGr3SvX2fGVn6MNJSQeQ+xu/OMnSNqW9CkPviQxfN+o+rpR9fX9jELjfy92Jm8O379+aTSbxs4ETqcYEhjlieERCtQSH/yi+QmxzaeQK0Ls6WX8krBAJM67MhB+tyavDiBZGNgPSZxgIw5T8DDYYTGyQ3EREUNwpiZ6GM2EE+pms13l/TySbyeHB923QAk0Dp+ndyIV9S0cRdLImnq32fv2HUh6EYPwaYrPOmrgTobwpAdFkkhvP7HmGWaBYxn7QpUrKFb0+HZvshP58uTD7Ksc72sdVOxQfw64DNz8saHk11NwikYelSFJRGSemZswalnfQ/J7SH4PyX8rJMVhuAjKRRwuTkhxdx9X/V/5osH7Fe7UV+UHK0dsBhfe1jNerv/uw/UYi13ANSFv/OdesvlNuXCRUcVNZqnae2VnMvmwL+iO7/N63ths5CWzZbO9bHaXzd6y2V82B8vmcNkcFei2xuaXRsb7iGGfSEMVBGi1CiIsO+1ip1vs9IqdfrEzKHaGxc6oxKcleplQsjIK2q3WRutqnc+tmf7na6MfNRGXa4p59WWramVvPbZU5qvmh6PfVlb66qp3f3d5boI/p+JtiEJXPnrLVTl8G0eMf02BXVXvp4QClUAUQsaYBg1VFmfGQQQPNNxQswmtoTVUmoaCHQ6DsXZwur+vffk/LdALlauaeP11RCEIclcEaV3WVET1XrqJCg8zTmiKnyg/VyxZddSVkEeF/CVmFyBf65uL0vFfUEsHCPBDvuS5BgAAWhoAAFBLAwQUAAgICACVk4tYAAAAAAAAAAAAAAAAFQAAAFRyYW5zZm9ybWVyXzEudHJ4Lm9zaOWT30vDMBDH3/NX3NOqjGirc/5iT6uCMO3YJgpjlJBeu2CTlEsK+/OtbDr2EMVH8eXIfXL3TfIlxzwJ40pLGrgyTeudXKMWQCgtFeyIAYyzyfPjU57cOE/KVMskjle3e34W4OcBPgjwiwAfBvhlgF8F+HWAJ/HhxjFjjNvWf+dF6G2/9egPePGTFf/nW3DcNITOKWtKVSMse+l8+jq3LUlMFa1OF5+jhJQnJ542jJe1qEBa3XQNjO8WBFHV70dfuW08RFwCz4CX04cx8BdjeYHdaVJ4LIDr4aALvjU4qtAgqa5YS20LrEdOi7ruxGpl3vbSH9lW160F7TQ6boRGOLgo44Ui6M0W+X02Se9m71BLBwjc1e+EDgEAABIEAABQSwMEFAAICAgAlZOLWAAAAAAAAAAAAAAAABEAAABUcmFuc2Zvcm1lcl8xLnRyeJ2Qz07DMAzG75X6Dj7CqS27ICreAO0wJK4otJ5mSOPJcTYJxLvjpCvaJnHhkj9f7N/3xU0DI24pIHASoLBP2nBS28BT+IDgJox1VR7yGVp4Mv216+tqrjtX7y7VblZXpmYEKTlPnwhfdQXNhfHAQYU9HJyQe/PZ0sLoPWz4uMF3HBTHtl/EdfL+GfXFSVvQ33mZHAXPvM/0mb84RgTdIciJA8LHXyOrPLeAR+j6pR8PzienaOl8mkKEm8A5aVQxL423sGUpY3o4fd/6jkKKggPLCCXxFb5o/4GvruDdH/AyDBssxV0ehV1/AFBLBwg/DVWC6gAAAOQBAABQSwMEFAAICAgAlZOLWAAAAAAAAAAAAAAAABMAAABvc2hfZGVwZW5kZW5jeS5qc29uq1bKyymOz00siAfROfnJiTmpSlbVSpm5BakVBfHJGYlFxaklSlZKoSFuuhZKOkrFJUWZeelYJDLzCkpLsIjnl5Zgk6jVUcpISiwGWhYdWwsAUEsHCFZRERFRAAAAhgAAAFBLAQIUABQACAgIAJWTi1jwQ77kuQYAAFoaAAAoAAAAAAAAAAAAAAAAAAAAAABlMmVkNzQwZi01MWU2LTQxYjYtODBkMS04MDgzODA4NWQzYjUub3NoUEsBAhQAFAAICAgAlZOLWNzV74QOAQAAEgQAABUAAAAAAAAAAAAAAAAADwcAAFRyYW5zZm9ybWVyXzEudHJ4Lm9zaFBLAQIUABQACAgIAJWTi1g/DVWC6gAAAOQBAAARAAAAAAAAAAAAAAAAAGAIAABUcmFuc2Zvcm1lcl8xLnRyeFBLAQIUABQACAgIAJWTi1hWURERUQAAAIYAAAATAAAAAAAAAAAAAAAAAIkJAABvc2hfZGVwZW5kZW5jeS5qc29uUEsFBgAAAAAEAAQAGQEAABsKAAAAAA==",
                            "ds_flow_environment": {
                                "FlowEnvironment": {},
                                "FlowParameters": {}
                            }
                        }
                    }
                ]

Le moteur à distance tire dans le type de travail Run, exemple 1 :

{
    "work_type": "job_run",
    "metadata": {
        "project_id": "a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4",
        "job_run_id": "69f6a270-6912-4321-a913-29277628eed1",
        "job_id": "aee11bf2-5be7-4abd-8930-1b0bbba4c6dd",
        "accept_language": "en-US,en;q=0.5"
    },
    "work": {
        "FlowEnvironment": {
            "S3_BUCKET": "testauth-donotdelete-pr-rs1hqmcbhwuzud",
            "AWS_ID": "4102d83e9d19412ca4ffa6a585f7b80b",
            "S3_REGION": "us-south",
            "AWS_KEY": "9aede861b3cb994bfdc605af10d58c2cf87ccb8e8c8d005c",
            "NUM_PARTITIONS": "2",
            "$REMOTE_ENGINE": "2220b98e-4f80-49d8-827c-371168a0d920",
            "S3_ENDPOINT": "https://s3.us-west.cloud-object-storage.test.appdomain.cloud",
            "HARDWARE_SPECIFICATION": "{'datastage':{'compute':{'mem':{'size':'16Gi'},'cpu':{'model':'','units':'4'}},'num_conductors':1,'num_computes':0,'conductor':{'mem':{'size':'4Gi'},'cpu':{'model':'','units':'1'}}}}"
        },
        "FlowParameters": {
            "DSJobRunName": "",
            "job_name": "rowgen-cos.DataStage job",
            "flow_name": "rowgen-cos",
            "project_name": "Test-Auth",
            "DSMetricsRepoEnabled": "false"
        },
        "FlowMessageHandler": {},
        "FlowOsh": "UEsDBBQACAgIAJykiFgAAAAAAAAAAAAAAAAoAAAAYjc0OTE1OGItMjI3OS00Nzg4LTgyY2MtNjE2MDdmYTk2OWYwLm9zaO1YW1PbOBR+96/wlp1JmcV2nBvBGHcCKS0UCBBg2wXGI9tKEDiWKylclul/3yPZJgm5UKY73X0oISDJn75zjs5FspaWlpb07Zje6Yze9XFihJTrIR2kJMaRjoReKTvVhlMt6+Wm3koZ9Cs1belHfxSD3j1pfXjv6Mf0zv+AE8yQoMy34ZneSbOe1i/Gx0d1mgpCE64ZPLzCA6QzHFIWaW81Xd/q7J3uH/i2wwUjSf/cLpcv10fjlTnj1TnjtTnj9cnxZc3IdOC6XdY0qWxmUfyk6zmJcCLelp5ZW1pe169pMKCJPx/ABerj4vnhPSCeAKXlS7U0Q5EOBdfKnn5+qT/ncPZIcuPb5m1JW/+Xnbezue9vxXQY+Z3gGofC74JEqe2kH9P7gJGoj2e6UTM6jIBtSHYzYzXj8/7eIaMAFQRzveS+ux/E+i1mHDAbFyXbLF+UdJyENAI3wMDpybZhN2DsneeOTRyfYV+UPHeLDmCx5f9E4Huhi4cUw1OSCHhaca183HPPECMoeQJk/gYMSHat/KHntjEPGUnBnLNM0mz8FAyURKChtPgEJkyqYdiuNfHYc4+3DicxQApjnlQ4M+iUw7p5bjAMb/C01u5v51vt1knrPJTOMqhylsEzZ8mshzZKIsQiY/h17fISiDMmz91Ft8hzP2KUdsnfzzR9oq3UG2pSAYNmNq+NYzIgAkfdB/Dwved+HVKB/WQ4wIyEvKALKI3H+WzFNon13B5hXPgxSbB/hVGE2bzpZTV9Cu65RcTMX6Ch6BlNNb3AetJ/z6zoQYH0EzTA84kiJJAZ8ttclRzvuWnhWaiwC9UfA3run+gWfwQPxUqhVpriJDrdaS9meIJBe5LgjoE1/oBG8xyaEYxgnpvAX+mLYLTsM+eM4Tw3ZBgBw2RUzlZ2ApqLC+kwEXOk2SNpCpY7pUfZAC2eM4aTeYPZVow4P1joTdgUTRLAl3AzDE2VRlkW5UlkqjKYVcG8CCphE/yqr/IU6kwCSFUKCBr4QxbPF34lRModywKgKTAXmXilD+hlqZ2BiAdL0BucKKk5peeilPg3+KHgThlkVAiBPC3kMeIJVD6I+m/bh5vJyafVdPu4l5Kb7tkf1vvO0Z44Oj5o/NXkX1J22O9uNm9WO6ISfbFbg3pwhBufPrXuzg6uxHFyolTIJXsuw5wOWQg7WCKLjGxECxaaJc6t3L9RH5446TCISejMKltOP6YBih1kBYG9VmmiejWKcLnZCFcbuIEiVF5drWO01kNOtVYPV+uNstGoNrABncBo1gJk9FCz2miiNbwW1B1HqT1LW89d6B5eNYfcuHtyzDM9M5ehNI3oAJEkwyhZykMQIMe5zJ1c5Py8touYmp6i9oJRUFmjXdAraVDi1SPYdkuP2m22fRX76IoWk4Ah9rDx5hqKtqNiKoAv4VYYWtOxbk3H+l7GsPL723b3Y2f//bJlmla7C7tTCgUsEdwKSAJsPIS4MK8RW38JOSPFXp4mrgiLZOF8mATCb6u7eQDFwsJhTFKOrTQeQoSBheOZHcuDiByCPZUbMe3LIDRsszIt+3WUu93OQW0XmGDF4fOyJWqhfkSiLBpZ06iZ9R+3QPFRBuljys8iNggnS0bS00ISalRemPMKt8C2jEO5lDMYp9Zxd6fbpqHa+TD7rsD73gCVFhqvAU8C19+saMXxFIJipcjKmHCZmWf2Oow6zq+U/JWSv1LyZ6Wk3AzzpMzzMN8h5dl9Yzr+tW8leH+FM/XN5AurQKwPB97yK95cf+6L6zGWVsAxoWj8795ki5Py2EFGlyeZ0dI+au1u92hP8m48Fjc/G/ZKcdkzalZHzdqoWd+wv61kHIcMh0Qt+BhRuTxGNepUxzu18U5ddjJKdTMFNv73d1MXJRmkC66k5t/hTN9OLcZOXFRNJ8vh5/y6aifJbqvcF2+r/gFQSwcIHQDnraoFAAChFAAAUEsDBBQACAgIAJykiFgAAAAAAAAAAAAAAAATAAAAb3NoX2RlcGVuZGVuY3kuanNvbqtWysspjs9NLIgH0Tn5yYk5qUpW1UqZuQWpFQXxyRmJRcWpJUpWSqEhbroWSjpKxSVFmXnpWCQy8wpKS7CI55eWYJOo1VHKSEosBloWHVsLAFBLBwhWURERUQAAAIYAAABQSwECFAAUAAgICACcpIhYHQDnraoFAAChFAAAKAAAAAAAAAAAAAAAAAAAAAAAYjc0OTE1OGItMjI3OS00Nzg4LTgyY2MtNjE2MDdmYTk2OWYwLm9zaFBLAQIUABQACAgIAJykiFhWURERUQAAAIYAAAATAAAAAAAAAAAAAAAAAAAGAABvc2hfZGVwZW5kZW5jeS5qc29uUEsFBgAAAAACAAIAlwAAAJIGAAAAAA==",
        "FlowResourceRequirements": {
            "Core": 0,
            "Storage": 0,
            "Memory": 0,
            "Partitions": 0,
            "Compute": 0
        },
        "FlowLimits": {
            "WarnLimit": 100,
            "RowLimit": 0
        },
        "FlowId": "b749158b-2279-4788-82cc-61607fa969f0",
        "Secrets": [],
        "FlowProjectSettings": {
            "runMessageHandlerId": "",
            "runPriorityQueue": "Medium",
            "formatDateString": "",
            "formatTimestampString": "",
            "formatTimeString": "",
            "formatDecimalSeparator": "",
            "nlsMap": "",
            "collationSequence": "OFF"
        }
    }

Le moteur à distance tire dans le type de travail Exécuter, exemple 2 :

{
    "work_type": "job_run",
    "metadata": {
        "project_id": "a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4",
        "job_run_id": "bfdea00d-6ede-45b6-8cf7-540f17e520ee",
        "job_id": "eb9de08a-37ef-4c47-9591-abe4c4a57d1b",
        "accept_language": "en-US,en;q=0.5"
    },
    "work": {
        "FlowEnvironment": {
            "S3_BUCKET": "testauth-donotdelete-pr-rs1hqmcbhwuzud",
            "AWS_ID": "4102d83e9d19412ca4ffa6a585f7b80b",
            "S3_REGION": "us-south",
            "AWS_KEY": "9aede861b3cb994bfdc605af10d58c2cf87ccb8e8c8d005c",
            "NUM_PARTITIONS": "2",
            "$REMOTE_ENGINE": "2220b98e-4f80-49d8-827c-371168a0d920",
            "S3_ENDPOINT": "https://s3.us-west.cloud-object-storage.test.appdomain.cloud",
            "HARDWARE_SPECIFICATION": "{'datastage':{'compute':{'mem':{'size':'16Gi'},'cpu':{'model':'','units':'4'}},'num_conductors':1,'num_computes':0,'conductor':{'mem':{'size':'4Gi'},'cpu':{'model':'','units':'1'}}}}"
        },
        "FlowParameters": {
            "DSJobRunName": "",
            "job_name": "rowgen-cos_copy_1711141317396_copy_1712769071912.DataStage job",
            "flow_name": "rowgen-cos_copy_1711141317396_copy_1712769071912",
            "project_name": "Test-Auth",
            "DSMetricsRepoEnabled": "false"
        },
        "FlowMessageHandler": {},
        "FlowOsh": "UEsDBBQACAgIAKyKilgAAAAAAAAAAAAAAAAoAAAAZTJlZDc0MGYtNTFlNi00MWI2LTgwZDEtODA4MzgwODVkM2I1Lm9zaO1YW1PbOBR+96/wlp1JmcVxnHuMcQdIaaFAgADbLjAe2VaCwLFdSeGyTP/7Hsl2EuMklOluZx9KCMjSp+8cnZtkraysrKg7QXSv0uh+iEPNi5jjRfGjY7QMw6gbNaNV6zQnXdVWs1NpGR2jqnrRKCYB9lXEVaNlVg3TaKpGRd2MqVqtVOvKyo/+SAa1f7r54b2pnkT3zgccYop4RB0DxtRenDwpw6x/tleNYk6ikCka867xCKkUexH1lbeKqm739s8ODh3DZJyScHhhVCpX69P+6oL+2oL++oL+Rr5/VdESHRjYSVGEssmKgomuF8THIX9berba0uq6ehO5oyh0FgMYR0OcjR89AGICKK1eSdOMeTzmTKnY6sWV+pzD3CfhrWOU70rK+r/svFOKQjaI6Ag/dx3PRua7bhCgoUrHoaKFaITVUo6opCjLrZhHF2xYGM5ZcHtmuC8GUiPuhokNrRdsWLR3Tl6CrP4X1t7dOnC2g2jsOz33Bnvc6YNuYmV508cPLiX+EM+1vKL1KAE7IPGYGEbRPh/sH9EIoJxgppasdw+jQL3DlAFm47JklCuXJRWHXuRD0EPH2emOZjSh751tzUycnWFclmxrOxqBW8T/kOMHrvLHGMMoCTmMVi097betc0QJCieAJLsAA5ItPR20rS5mHiUxLOc8kTQfX4CBkgg0FCs+hQl5NTTD0nPDtnWyfZTHACn02ULhZEFnDOxmW+7Yu8VFra3fLra7m6ebF55wlhZJZ2kscZaoxNBGoY+or42/dq6ugDhhsq09dIds6yNGcZ/8/UzTCW210ZSTMhg0k3ldHJAR4djvP4KHH2zr6zji2AnHEJnEYxmdG0XBLJ8h2fJY2xoQyrgTkBA71xj5mC6aXpHTC3DbyiJmsYHGfKC15fQMawv/PVvFALYjR9aJhUQ+4qjssbtUlRRvW3HmWdjPlqo/A7StP9Ed/ggeCqRCm3GMQ/9st7ucYQKDdp7gnsJqnFHkL3JoQjCF2VYIf4Uv3KnZ586ZwdmWRzEChnxUzlc2B03FedE45AukGVNpEpY6RRQ9tHzODE7kDabbAWLscKk34QhSJi58CSt7XlmmUZJFaRKVZRlMqmBaBKWwHL98lnkKdSYEpCwFBI2cMQ0WC7/mPGamrgOwzDHjiXipD+ily12E8EedR7c4lFJTSttCMXFu8WPGHVPIKA8CuSjkyWchVD6I+m87R1vh6adWvHMyiMlt//wP/X3veJ8fnxw2/2qzLzE9Gva32retHq/6X4zNUcM9xs1Pnzbvzw+v+Ul4KlVIJdsWxSwaUw92u1AUGdHwlxiahuadOC2hIYyY8dgNiGfOK1vmMIhcFJhId12jU22jRs33caXd9FpN3EQ+qrRaDYw6A2TW6g2v1WhWtGatiTV4cLV23UXaALVrzTbq4I7bME2p9jxtbWupe1itPGba/cQxz/RMXIbi2I9GiIQJRsqSHoIAOUll7qYiF+e1kcVUcYrcC6ZBpU93QbukQImXQ7Dtlp6Uu2T7yvbRNSUgLkX0cePNDRRtU8aUC1/CdM/Ti7GuF2N9P2FY+/1tt/+xd/B+VS+X9W4fdqcYCljIme6SENiYB3FRvkF0/SXknBR7eRq/JtQXhfMxD4Tfzf7WIRQLHXsBiRnW42AMEQYrnM3sQBxERBfsqUwLoqEIQs0oV4uyX0e51+8d1veACSwOn5dXIg31IxJF0UiaWr3c+PEVSL6IQvqUxWcZG4STLiJpYkgSadUX5rzCLbAtY0+Ycg5jwY57u/1u5MmdD9PvCrzvDVCxQu014Dxw/c2akh1PISjWsqwMCBOZeW6sQ69p/krJXyn5KyV/VkqKzTBNyjQP0x1SnN03ivGvfCvB+yucqW/zL6wc0SEceCuveHP9uS+uJ1isAo4JWeN/9yabnZRnDjKqOMlMTfukdPv9433Bu/GU3ZttGGvZldu0WZ0269NmY8P4tpZwHFHsEWnwGaJKZYZq+lCdfajPPjTEQ0Ip7wFhjcWbwEU3e6+9Ifzem8DLkgjSJVdXi+9wivdYy7G5S61ishx9nnuvtfiu6h9QSwcIxiK7RwoGAAAzFgAAUEsDBBQACAgIAKyKilgAAAAAAAAAAAAAAAAVAAAAVHJhbnNmb3JtZXJfMS50cngub3NolZBPS8NAEMXv+ynm1ChltNHqQempURCqKW1FoZSwbCbp4v4Jsxvoxzdibekhopdh5sfMe48RkaULlWcLqF3TxqC2ZCUwKc+lOBMA03z2+vxSpHchsnb1Oh2NNvdHftXDr3v4uIffnPJzIQT6Nv4Wqc/iv1H/HAlp1zCFoL2rtCFYD7Ll/H3pW1aUad5crn6+SVykF5F3Aisja1DeNt2BwH3DkNTDYXKYfRMhQQWYA1bzpyngm/NYUuemZKQS0N6OuxJbR5OaHLHulq2yviQzCVYa04kZ7T6O0l/Tt27YSt5rdNxJS3ASVGCpGQaLVfGYz7KHxSdQSwcIKooocfUAAAAVAgAAUEsDBBQACAgIAKyKilgAAAAAAAAAAAAAAAARAAAAVHJhbnNmb3JtZXJfMS50cnhdkM9OwzAMxu+V+g4+wqkdJ0TFGyAOm8QVhdbTDG48Oc4qgXh3nGxFg0v+fLG/3+d0HUy4p4ggWYHiMVsn2XwDpvgBMcyY2qY+lDP08OT662Zom3PdtXrnaikmo8D0ifDVNtD9QYwSTYXhFJTCGxdzx9o9bGXZ4juOhlM/rOJzZt6hvQTtq/V3WeZAkUWOxf3svxITgh0Q9OIDKssvyCuvEfAIm2Htx1PgHAw9Hec5JriJUpImU2dZuoW9aP2Qh8ug3rcoGSqOohPUxP/si1bz+uyUDiWtX38AUEsHCAOe9m/ZAAAAcQEAAFBLAwQUAAgICACsiopYAAAAAAAAAAAAAAAAEwAAAG9zaF9kZXBlbmRlbmN5Lmpzb26rVsrLKY7PTSyIB9E5+cmJOalKVtVKmbkFqRUF8ckZiUXFqSVKVkqhIW66Fko6SsUlRZl56VgkMvMKSkuwiOeXlmCTqNVRykhKLAZaFh1bCwBQSwcIVlEREVEAAACGAAAAUEsBAhQAFAAICAgArIqKWMYiu0cKBgAAMxYAACgAAAAAAAAAAAAAAAAAAAAAAGUyZWQ3NDBmLTUxZTYtNDFiNi04MGQxLTgwODM4MDg1ZDNiNS5vc2hQSwECFAAUAAgICACsiopYKooocfUAAAAVAgAAFQAAAAAAAAAAAAAAAABgBgAAVHJhbnNmb3JtZXJfMS50cngub3NoUEsBAhQAFAAICAgArIqKWAOe9m/ZAAAAcQEAABEAAAAAAAAAAAAAAAAAmAcAAFRyYW5zZm9ybWVyXzEudHJ4UEsBAhQAFAAICAgArIqKWFZRERFRAAAAhgAAABMAAAAAAAAAAAAAAAAAsAgAAG9zaF9kZXBlbmRlbmN5Lmpzb25QSwUGAAAAAAQABAAZAQAAQgkAAAAA",
        "FlowResourceRequirements": {
            "Core": 0,
            "Storage": 0,
            "Memory": 0,
            "Partitions": 0,
            "Compute": 0
        },
        "FlowLimits": {
            "WarnLimit": 100,
            "RowLimit": 0
        },
        "FlowId": "e2ed740f-51e6-41b6-80d1-80838085d3b5",
        "Secrets": [],
        "FlowProjectSettings": {
            "runMessageHandlerId": "",
            "runPriorityQueue": "Medium",
            "formatDateString": "",
            "formatTimestampString": "",
            "formatTimeString": "",
            "formatDecimalSeparator": "",
            "nlsMap": "",
            "collationSequence": "OFF"
        }
    }
}

Le moteur à distance pousse l'état du travail, exemple 1 :

{
    "Job Status": "RUN",
    "HB_INSTANCE_HOST_URI": "",
    "Project Id": "a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4",
    "Queue End": 1712608618451,
    "px-instance-id": null,
    "send-running-count": 1,
    "Job Startup Time": "0",
    "Flow Run Id": "69f6a270-6912-4321-a913-29277628eed1",
    "message-type": "JobStatus",
    "Queue Start": 1712608618449,
    "msg_count": 0,
    "Flow Id": "aee11bf2-5be7-4abd-8930-1b0bbba4c6dd",
    "Job Production Run Time": "0"
}

Le moteur à distance pousse l'état du travail, exemple 2 :

  {
    "Job Status": "FOK",
    "HB_INSTANCE_HOST_URI": "",
    "job_metrics": {
        "check_end": 1.712608634152849E9,
        "start_end": 1.712608618652393E9,
        "finished": 1.712608650705897E9,
        "total_stages": 2,
        "link_metrics": [
            {
                "start_time": 1.712608634E9,
                "rows_read": 10,
                "rows_written": 10,
                "stop_time": 1.712608634E9,
                "source": "Row_Generator_1",
                "state": "finished",
                "dest": "IBM_Cloud_Object_Storage_1",
                "link_name": "Link_1"
            }
        ],
        "total_seconds_cpu": 21.15,
        "run_end": 1.712608650705897E9,
        "duration": 32.11314010620117,
        "total_rows_read": 10,
        "check_begin": 1.712608618652393E9,
        "run_begin": 1.712608634152849E9,
        "stage_metrics": [
            {
                "stage_seconds_cpu": 21.15,
                "start_time": 1.712608648E9,
                "rows_read": 10,
                "rows_written": 0,
                "stop_time": 1.71260865E9,
                "stage_type": "cloudobjectstoragePX",
                "stage_name": "IBM_Cloud_Object_Storage_1",
                "state": "finished"
            },
            {
                "stage_seconds_cpu": 0.0,
                "start_time": 1.712608634E9,
                "rows_read": 0,
                "rows_written": 10,
                "stop_time": 1.712608634E9,
                "stage_type": "PxRowGenerator",
                "stage_name": "Row_Generator_1",
                "state": "finished"
            }
        ],
        "job_id": "360",
        "start_begin": 1.712608618592757E9,
        "state": "finished",
        "total_rows_written": 10
    },
    "Project Id": "a8fe3d27-ddea-4b10-8c04-c687a5eeb3b4",
    "Queue End": 1712608618451,
    "px-instance-id": null,
    "send-running-count": 7,
    "S3_LOG_LOCATION": "DataStage\/69f6a270-6912-4321-a913-29277628eed1.job.log",
    "Job Startup Time": "0",
    "Flow Run Id": "69f6a270-6912-4321-a913-29277628eed1",
    "message-type": "JobStatus",
    "Queue Start": 1712608618449,
    "msg_count": 7,
    "Flow Id": "aee11bf2-5be7-4abd-8930-1b0bbba4c6dd",
    "Job Production Run Time": "32"
}

Le moteur à distance transmet les données relatives à l'exécution du travail :

{
        totalCpu: 4000
        usedCpu: 0.0
        totalMemory: 2014576640
        usedMemory: 115087176
        totalDisk: 250059685888
        usedDisk: 160418521088
        engineInitialized: 1712608520755
    }
    computes: null
}

Données sensibles

Pour éviter d'écrire des données dans les journaux, n'utilisez pas les fonctions Peek Stage ou Asset Browser. Utilisez plutôt des fichiers séquentiels pour analyser les données et les types de données réels utilisés dans la conception du travail et pour paramétrer tous les noms de fichiers et toutes les connexions.