ZenML Release Notes

Breaking Changes

• Resource pools have been reworked to use ZenML Pro Resource Manager v2, and the old OSS resource-pool control-plane objects and related API/CLI/RBAC behavior have been removed. If you used legacy resource pool management in OSS or automation built on those endpoints/commands, you will need to migrate to the new resource request model and update any scripts or integrations accordingly. PR #4977
• ZenML now requires Click >=8.3.3,<=8.5.0. If your environment, plugin, or custom CLI extension pins an older Click version or depends on older Click behavior, update those dependencies to a compatible version before upgrading ZenML. PR #5145
• Dictionary outputs with non-string keys are no longer materialized through the JSON path. If you relied on reading such artifacts back as JSON-compatible data with keys converted to strings, update your code to preserve original key types or change outputs to use string keys for JSON interoperability. PR #5124 …

"description": "Resource pools have been rebuilt on the new ZenML Pro Resource Manager, which now decides scheduling, allocation, and eviction. Eligible dynamic steps request resources through it and wait for an allocation before they start, and Kubernetes is the first backend to apply the allocated CPU, memory, and GPU capacity as pod requests and limits. Running steps keep their allocation alive through step heartbeats and are cancelled gracefully if their resources are reclaimed. Breaking: the previous resource pool implementation has been removed together with its data — existing pools, policies, and resource requests are not migrated and need to be set up again.", "published_at": "2026-09-23T13:30:23Z", "published": true, "audience": "pro", "labels": [ "improvement", "deprecation" ], "docs_url": "https://docs.zenml.io/pro/core-concepts/resource-pools", "should_highlight": true }, { "id": 90, "slug": "prune-unused-pipeline-snapshots",

"description": "Unnamed pipeline snapshots left behind by deleted runs, deployments, and schedules can now be cleaned up. Run zenml pipeline snapshot prune --older-than-days 90 (also available as zenml prune snapshots) to delete snapshots that nothing references anymore, or add --dry-run to see how many would go first. Replacing a named snapshot with replace=True now deletes the old one when it is unused, and snapshot creation is atomic, so a failed attempt no longer leaves a partial snapshot behind.", "published_at": "2026-09-23T13:30:23Z", "published": true, "audience": "oss", "labels": [ "feature" ], "docs_url": "https://docs.zenml.io/user-guides/best-practices/keep-your-dashboard-server-clean", "should_highlight": false }, { "id": 89, "slug": "paginated-logs-for-external-log-stores",

"description": "Reading logs from external log stores no longer means one huge query or hitting the backend's rate limits. A new GET /api/v1/logs/{logs_id}/entries endpoint returns logs in pages with cursors and supports search, level, since, and until filters, and the Datadog log store has been rewritten on top of it. If you maintain a custom log store, update it to the new BaseLogStore.fetch() signature and import LogEntry from zenml.models. For Datadog, the existing run and step log endpoints now return the newest 1,000 entries instead of up to 50,000.", "published_at": "2026-09-23T13:30:23Z", "published": true, "audience": "oss", "labels": [ "improvement", "deprecation" ], "docs_url": "https://docs.zenml.io/stacks/stack-components/log-stores", "should_highlight": false }, { "id": 88, "slug": "pipeline-reliability-fixes-0-97",

"description": "Dynamic pipeline graphs now render correctly even when step records come back out of dependency order, and long-running retry loops no longer crash with an OverflowError after reaching the maximum backoff delay. Dictionaries with non-string keys are no longer stored as JSON, so a step returning {1: 1} now hands {1: 1} to the next step instead of {\"1\": 1}. Large API responses are also cached correctly now, which avoids unnecessary retries after a client timeout.", "published_at": "2026-09-23T13:30:23Z", "published": true, "audience": "oss", "labels": [ "bugfix" ], "should_highlight": false }, { "id": 87, "slug": "webhook-driven-pipeline-automation",

Webhook-driven automation

• Webhook triggers for pipeline snapshots: ZenML now supports first-class webhook-driven automation for secure, project-scoped integrations with GitHub or custom systems PR #5169. You can attach webhook triggers to pipeline snapshots and launch them when matching external events arrive, including typed GitHub filters for merged pull requests, completed workflow runs, pushes, and published releases.
• ClickUp webhook provider: ZenML now includes a built-in ClickUp webhook provider for triggering automation from ClickUp task and list events PR #5216. The provider authenticates deliveries with ClickUp’s raw hex HMAC signature and supports string-based filters, so teams can connect ClickUp activity directly to ZenML workflows.
• Slack webhook provider: ZenML Pro now supports inbound Slack Events API callbacks as a first-class webhook provider PR #5221. Slack apps can authenticate events such as mentions, messages, reactions, and other automation-focused activity to trigger pipeline snapshots or related workflows; this is separate from the existing Slack alerter that sends notifications to Slack. …

"description": "ZenML now supports first-class webhook triggers, so an external event can launch a pipeline snapshot. Attach a trigger to a snapshot and filter on typed GitHub events — merged pull requests, completed workflow runs, pushes, and published releases — or use the new built-in Slack and ClickUp providers to start runs from team activity. Custom webhook providers are supported for everything else, and every integration is project-scoped and signature-authenticated.", "published_at": "2026-09-04T13:32:22Z", "published": true, "audience": "pro", "labels": [ "feature" ], "docs_url": "https://docs.zenml.io/pro/core-concepts/webhooks", "should_highlight": true }, { "id": 86, "slug": "faster-and-more-reliable-pipeline-execution",

"description": "Pipeline runs are now more dependable and start faster. Dynamic pipelines running with ExecutionMode.STOP_ON_FAILURE no longer report success when an isolated step fails — the failed step state now propagates to the overall run status. ZenML also precalculates and reuses Docker build checksums while resolving builds instead of recomputing them, cutting startup overhead for pipelines with many steps.", "published_at": "2026-09-04T13:32:22Z", "published": true, "audience": "oss", "labels": [ "bugfix", "improvement" ] }, { "id": 85, "slug": "aws-rds-iam-database-authentication",

"description": "The MySQL ZenML store can now authenticate to AWS RDS with IAM database authentication instead of a static password. ZenML generates a fresh IAM token for each database connection and enforces TLS hostname and certificate verification in IAM mode, giving self-hosted AWS deployments a passwordless option that matches RDS security best practices. Existing password-based setups keep working unchanged.", "published_at": "2026-09-04T13:32:22Z", "published": true, "audience": "oss", "labels": [ "feature" ] }, { "id": 84, "slug": "workload-scoped-token-hardening",

"description": "Generic ZenML Cloud API tokens are no longer supported: GET /auth/api_token and the GENERIC_TOKEN grant type have been removed, and previously issued generic tokens now fail immediately. These tokens turned a session into a standalone bearer token that outlived the credential it came from. Use personal access tokens, organization service-account API keys, or device-code login instead. The workspace-server endpoint GET /api/v1/api_token is not affected.", "published_at": "2026-08-27T13:00:52Z", "published": true, "audience": "pro", "labels": [ "deprecation", "improvement" ], "docs_url": "https://docs.zenml.io/api-reference/pro-api/getting-started" }, { "id": 82, "slug": "more-flexible-release-management",

"description": "Supported and default ZenML and Kitaru release versions now live in the database instead of static Cloud API configuration, so they can change without a new Cloud API deployment. Anyone can list supported releases and the current default with GET /releases?release_service=<zenml|kitaru>. Superusers on self-hosted ZenML Pro can add, change, and remove versions through the new release management endpoints.", "published_at": "2026-08-27T13:00:52Z", "published": true, "audience": "pro", "labels": [ "improvement" ] }, { "id": 81, "slug": "filter-workspaces-by-type-in-the-api",

Runtime and orchestration

• Multi-pod Kubernetes step operator jobs: Command steps can now run across multiple Kubernetes pods with the Kubernetes step operator. Set pod_count in KubernetesStepOperatorSettings to launch the step as an indexed job, making it easier to distribute command-style workloads across pods. PR #5104
• Local Docker sandbox: ZenML now includes a local Docker sandbox, plus a unified settings model for containerized sandboxes. Local sandbox workflows also support file upload and download, making it easier to test containerized ZenML behavior locally before moving to remote infrastructure. PR #5102

Deployment, security, and artifact integrity

• Enrollment keys from Kubernetes Secrets: The Helm chart now supports server.pro.enrollmentKeySecretRef, so ZenML Pro enrollment keys can be injected from an existing Kubernetes Secret instead of being stored inline in Helm release values. The secret reference is applied consistently to the server, migration, and worker containers. PR #5123 …

"description": "Command steps can now run across multiple pods with the Kubernetes step operator. Set pod_count in KubernetesStepOperatorSettings to launch the step as an indexed Kubernetes job, and ZenML injects coordination environment variables into each pod — making it easier to distribute workloads like torchrun-based training across pods.", "published_at": "2026-08-07T13:24:25Z", "published": true, "audience": "oss", "labels": [ "feature" ], "docs_url": "https://docs.zenml.io/stacks/stack-components/step-operators/kubernetes" }, { "id": 78, "slug": "local-docker-sandbox",

"description": "This release smooths out many rough edges: fewer database-is-locked errors for local SQLite stores, correct project scoping for artifact deletion and prefix lookups, complete exception tracebacks, and no more unnecessary Kubernetes pod retries for already-finished runs. MLflow tracking behaves better on Databricks and managed runtimes, cloudpickle artifacts are validated with a SHA-256 hash on load, service account tokens and adopted API keys keep working through migrations, and Helm deployments can supply the ZenML Pro enrollment key from an existing Kubernetes Secret.", "published_at": "2026-08-07T13:24:25Z", "published": true, "audience": "oss", "labels": [ "bugfix", "improvement" ] }, { "id": 75, "slug": "dynamic-pipelines-are-more-flexible",

Known issues

Artifact version deletion via the API is broken in this release and will be fixed in 0.96.3. Artifact versions can still be deleted using the python SDK:

from zenml.client import Client

Client().delete_artifact_version(..., server_side=False)

Dynamic pipelines

• Explicit start ordering for dynamic steps: Dynamic pipelines now support start_after=... when calling steps, letting you control which concurrently launched steps should wait for others before starting. This makes it easier to model ordering constraints without turning concurrent parts of a dynamic pipeline into fully synchronous execution. Note that start_after is now a reserved step keyword, so steps that previously used a parameter with this name will need to be updated. PR #4995
• More flexible dynamic step inputs: You can now configure whether JSON-serializable raw values passed to steps in dynamic pipelines should be treated as parameters instead of artifacts. The new environment-variable threshold defaults to 0 to preserve existing behavior, while explicit APIs such as with_options(parameters=...) and ExternalArtifact(...) remain available when you want to force either behavior. PR #5079 …

"description": "Dynamic pipelines gained more control and predictability: you can define start ordering between steps, continue execution more gracefully in some failure scenarios, and better handle implicit dependencies and raw values passed between steps. Replay and deployment parameter handling were also cleaned up so runs use the values you expect without confusing or stale configuration showing up.", "published_at": "2026-07-17T11:21:15Z", "published": true, "audience": "oss", "labels": [ "improvement" ], "docs_url": "https://docs.zenml.io/concepts/steps_and_pipelines/dynamic_pipelines" }, { "id": 74, "slug": "digitalocean-integration",