v0.5 includes a few major product and configuration changes that you should be aware of if you are upgrading an existing Polyaxon deployment.
Before upgrading to Polyaxon v0.5, please make sure you are running the latest version v0.4.4. Also please make sure you have a backup of your database.
The latest Polyaxon's Helm chart has up-to-date dependency requirements,
some of these dependencies cannot be upgraded automatically and require a shutdown.
N.B.1 the Postgres dependency is still the same, we also suggest that you always pin the db version,
Polyaxon's built-in dependency points to
N.B.2 Polyaxon supports any version of postgresql from 9.6 and it's recommended that you have this value in your deployment config.
N.B.3 If you are using an external Postgres instance, the chart has been updated to consolidate all external services in one section.
Make sure that your data (database, outputs, logs,...) is stored on some persistent storage (not a pvc created by Helm when you installed polyaxon). Then, in order to upgrade to the latest Polyaxon version, please make sure to teardown your deployment, this will leave your data intact and they will be reused upon the upgrade. (Provided that the storage it resides on is not deleted!)
polyaxon admin teardown
Often times after a Helm teardown, some service accounts or secrets might be left undeleted, please make sure to delete them before proceeding.
In order to upgrade to the new version while reusing your previous data, you need to make sure that your deployment config file is valid, you can use the upgraded CLI to validate your new config file.
polyaxon admin deploy -f config.yaml --check
Make sure to keep a backup of your config file to set the configuration using the web UI.
Several options have been migrated to a web UI settings page, please refer to section Dynamic configuration.
You need to decide which built-in dependencies to use from the chart.
Polyaxon v0.5 comes with built-in dependencies for
That being said, in this version, you can disable all dependencies
and bring your own (either managed by a cloud provider, or hosted on the same cluster but managed by a different chart).
To disable one or several components:
postgresql: enabled: false redis: enabled: false rabbitmq-ha: enabled: false docker-registry: enabled: false
To provide an external connexion for one or several of these dependencies:
externalServices: postgresql: ... redis: ... rabbitmq: ...
For the docker registry, since it's not required for starting the core components, you can use the web UI to set a connexion or several depending on your distribution.
For more details about the HA for each one of these components, please check:
Finally, in the v0.5, Polyaxon allows to use Redis as a broker as well, which means that you can turn-off rabbitmq completely and rely on redis. Please see this for more details.
Several configuration options are now stored in the database, not in a config.yaml file.
Editing the options is now only possible through the web UI. Some configuration are still managed by the config.yaml file, e.g. data, logs, outputs, and will be migrated slowly to the web UI in the following versions.
integrations: slack: hipchat: mattermost: discord: pagerduty: webhooks: auth: github: gitlab: bitbucket: azure: notebookBackend: notebookDockerImage: tensorboardDockerImage: kaniko: image: imageTag: imagePullPolicy: dockerizer: imagePullPolicy nodeSelectors: experiments: jobs: builds: tensorboards: tolerations: resourcesDaemon: experiments: jobs: builds: tensorboards: affinity: experiments: jobs: builds: tensorboards: secretRefs: configmapRefs: privateRegistries: reposAccessToken:
Starting from v0.5, Polyaxon will use one single port for handling API requests, the platform is moving to internal plugins for handling logs, metadata, and other plugins. This will reduce the complexity of securing the platform, and expose more flexibility to extend and create custom plugins.
This change will affect several components:
- In the helm chart there will be only one port exposed from the service.
The CLI will accept only one argument for ports:
If you have a custom configuration please have a look at the service.yaml file and ing.yaml file.
Since most of the configurations are now UI based, the scheduling exposed in the Helm chart is fairly standard, and only applies to the components being deployed, i.e. API, scheduler, events, ...:
nodeSelector: tolerations: affinity:
Additionally, any dependency enabled can be customized in a similar way, e.g.
postgresql: nodeSelector: tolerations: affinity:
For jobs, experiments, builds, notebooks, and tensorboards, a web UI has all information about default values, and an easy way to change them.
After deploying the platform, you can update the default node scheduling. for each primitive through the web UI, and using Polyaxonfiles.
N.B. This change only applies to CE versions, since EE had dynamic configurations.
In order to pull/push to private docker registries, the platform will be using standard credential configuration instead of the previous way of passing user and password. You can still authorize access to private registries by providing user/password, but it should be through a secret with a credential helper.
The credential config can be set in a kubernetes secret and will be attached during the build process to allow pulling and pushing images.
declarations is now deprecated, and
params is used instead.
However the platform will still accept any polyaxonfile containing
Polyaxon v0.5 introduces also new way to declare values in your polyaxonfiles;
These 2 new sections will allow to define types declarations of your polyaxonfiles, and can have default values.
Additionally a user can override or provide a value for an input or output using params with an override file or directly with the CLI:
polyaxon run -f polyaxonfile.yaml -P param1=value1 -P param2=value2
persistence is now deprecated, and
environment is used instead.
The platform will still accept any polyaxonfile containing
configmaps_refs has been renamed to
config_maps_refs, The platform will still accept any polyaxonfile containing
The specficiation requires an extra step:
polyaxon deploy is now moved to
polyaxon admin deploy,
polyaxon admin upgrade, and
polyaxon admin teardown.
declarations is deprecated in favor of
params, this flag is also changed to
--params / -p
declarations is deprecated in favor of
declarations.param1: value1 | value2 is now
params.param1: value1 | value2. The api will still accept requests with
To make the search consistent,
metric is deprecated in favor of
metric.loss: <= 0.1 is now
metrics.loss: <= 0.1. The api will still accept requests with
POLYAXON_DECLARATIONS is now
POLYAXON_PARAMS, if you are using polyaxon-client, your program should work without changes.
log_outputs are now