You can use the Elastic Stack to centralise your logging with Polyaxon.

Overview

You can set up EFK (elasticsearch, fluentd/fluentbit, kibana) as a stack to gather logs from Polyaxon core components or experiment and job runs.

If you are running Polyaxon in the cloud, then you can consider a managed service from your cloud provider.

Setup Helm

You need Helm to both setup Polyaxon as well as Elastic stack. You can follow this guide to set up Helm.

Install ElasticSearch

helm install --name elasticsearch elasticsearch --version 7.1.1 --namespace=logging  -f elastic-config.yaml --repo https://helm.elastic.co

You can read more about how to customize your ElasticSearch instance and provide elastic-config.yaml to override the default values.

Install Fluentd

helm install fluentd --name fluentd --namespace=logging -f fluentd-config.yaml --repo https://kiwigrid.github.io

You can read more about how to customize your Fluentd instance and provide fluentd-config.yaml to override the default values.

For instance if you want to log all pods managed by Polyaxon (core and runs), you might want to add:

# Filter to only records with label kubernetes.io/managed-by=polyaxon
<filter kubernetes.**>
@type grep
<regexp>
key $["kubernetes"]["labels"]["app.kubernetes.io/managed-by"]
pattern polyaxon
</regexp>
</filter>

If you want to only monitor the core components:

# Filter to only records with label type=polyaxon-core
<filter kubernetes.**>
@type grep
<regexp>
key $["kubernetes"]["labels"]["type"]
pattern polyaxon-core
</regexp>
</filter>

This filter should be added to your fluentd-config.yaml under extraConfigMaps, e.g:

elasticsearch:
  host: 'elasticsearch-master'

configMaps:
  useDefaults:
    containersInputConf: false
    systemInputConf: false

extraConfigMaps:
  containers.input.conf: |-
    <match fluent.**>
    # Ignore fluentd' log
    @type null
    </match>
    # Docker's containers logs
    <source>
    @type tail
    path /var/log/containers/*.log
    pos_file /var/log/fluentd-containers.log.pos
    tag kubernetes.*
    read_from_head true
    <parse>
    @type json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
    </parse>
    </source>
    # Kubernetes metadata
    <filter kubernetes.**>
    @type kubernetes_metadata
    </filter>
    <match kubernetes.var.log.containers.**kube-system**.log>
    @type null
    </match>
    # Exceptions in logs
    <match raw.kubernetes.**>
    @id raw.kubernetes
    @type detect_exceptions
    remove_tag_prefix raw
    message log
    stream stream
    multiline_flush_interval 5
    max_bytes 50000
    max_lines 1000
    </match>
    # Exclude kube-system
    <match kubernetes.var.log.containers.**kube-system**.log>
    @type null
    </match>
    # Filter to only records with label kubernetes.io/managed-by=polyaxon
    <filter kubernetes.**>
    @type grep
    <regexp>
    key $["kubernetes"]["labels"]["app.kubernetes.io/managed-by"]
    pattern polyaxon
    </regexp>
    </filter>
    # Filter to only records with label kubernetes.io/managed-by=polyaxon
    <filter kubernetes.**>
    @type grep
    <regexp>
    key $["kubernetes"]["labels"]["type"]
    pattern polyaxon-core
    </regexp>
    </filter>

Install Kibana

helm install kibana --version 7.1.1 --name=kibana --namespace=logging -f kibana-config.yaml  --repo https://helm.elastic.co

You can read more about how to customize your ElasticSearch instance and provide kibana-config.yaml to override the default values.

Check the logs on the Kibana UI

You can inspect the logs ont the Kibana UI, by port-forwarding for example.

  1. When Kibana appears click Explore on my own. From the top-left or from the Visualize and Explore Data panel select the Discover item. In the form field Index pattern enter logstash-* It should read “Success!” and Click the > Next step button.

  2. In the next form select timestamp from the dropdown labeled Time Filter field name.

  3. From the bottom-right of the form select Create index pattern. In a moment a list of fields will appear.

  4. From the top-left or the home screen’s Visualize and Explore Data panel, select the Discover item. The logs list will appear.

Notes

Polyaxon will persist by default all runs' logs to a persistent volume or a cloud bucket provided by the user. Please check the Logs storage configuration for more details.