Polyaxon allows users to connect to one or multiple blobs on Azure Storage to store job outputs and experiment artifacts.

Create an Azure Storage account

You should create a storage account (e.g. plx-storage) and a blob (e.g. outputs). You should then create a file with your access information json object, e.g. az-key.json. This file should include the following information:

{ 
  "AZURE_ACCOUNT_NAME": "plx-storage",
  "AZURE_ACCOUNT_KEY": "your key",
  "AZURE_CONNECTION_STRING": "your connection string",
}

Create a secret on Kubernetes

You should then create a secret with this access keys information on Kubernetes on the same namespace as Polyaxon deployment:

kubectl create secret generic az-secret --from-file=az-secret.json=path/to/az-key.json -n polyaxon

Use the secret name and secret in your outputs persistence definition

persistence:
  outputs:
    [OUTPUTS-NAME-TO-USE]:
      store: azure
      bucket: wasbs://[CONTAINER-NAME]@[ACCOUNT-NAME].blob.core.windows.net/
      secret: [SECRET-NAME]
      secretKey: [SECRET-KEY]

e.g.

persistence:
  outputs:
    outputs:
      store: azure
      bucket: wasbs://[email protected]/
      secret: az-secret
      secretKey: az-secret.json

Update/Install Polyaxon deployment

You can now install/upgrade Polyaxon with access the outputs on Azure.

Storing outputs and artifacts in your experiments/jobs

You can use polyaxon-client to access the outputs in your jobs/experiments.

Polyaxon client does not bundle by default the azure storage requirements to keep the client lightweight:

pip install polyaxon-client[azure]

or to have more control over the version of azure storage:

pip install polyaxon-client
pip install azure-storage

In your experiment/job definition, you can add this step to be available during the run:

build:
  ...
  build_steps:
    ...
    - pip3 install polyaxon-client[azure]

Using the outputs store from tracking

In your experiment/job, Polyaxon exposes the secret related to the outputs as well as the outputs path scheduled for the run as an an env var, and provides an interface to get an authenticated client for each one of these Paths.

from polyaxon_client.tracking import Experiment

experiment = Experiment()
...
experiment.outputs_store.upload_file(file_path)
experiment.outputs_store.upload_dir(dir_path)