Polyaxon allows users to manage all logs generated by jobs, builds, and experiments containers in Amazon S3.

Create an S3 bucket

You should create an S3 bucket (e.g. plx-storage).

In order to use S3 buckets with Polyaxon, you should create a file containing your access information json object, e.g. s3-key.json. This file should include at least the following information:

{
  "AWS_ACCESS_KEY_ID" : "",
  "AWS_SECRET_ACCESS_KEY": ""
}

All possible values:

{
  "AWS_ENDPOINT_URL": "",
  "AWS_ACCESS_KEY_ID": "",
  "AWS_SECRET_ACCESS_KEY": "",
  "AWS_SECURITY_TOKEN": "",
  "AWS_REGION": ""
}

Create a secret on Kubernetes

You should then create a secret with this access keys information on Kubernetes on the same namespace as Polyaxon deployment:

kubectl create secret generic s3-secret --from-file=s3-secret.json=path/to/s3-key.json -n polyaxon

Use the secret name and secret key in your logs persistence definition

persistence:
  logs:
    store: s3
    bucket: s3://[BUCKET-NAME]
    secret: [SECRET-NAME]
    secretKey: [SECRET-KEY]

e.g.

persistence:
  logs:
    store: s3
    bucket: s3://logs-bucket
    secret: s3-secret
    secretKey: s3-key

Update/Install Polyaxon deployment

You can now install/upgrade Polyaxon with access the outputs on S3.