Apache Airavata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

If you want to enable elasticsearch based logging feature for Airavata

Centralized logging for Airavata

This is a new feature we are currently working on 0.17 relea

se so the feature is not yet released. We have added a custom log appender for Airavata and its not registered by default. If you enable it it will push all the Json formatted logs to a Kafka topic. So if you want to enable this

feature you first have to setup a Kafka cluster so that all the logs will be pushed to the Kafka topic created based on the configuration provided. Once you have kafka cluster setup then you can start airavata by simply changing two parameters in airavata-server.properties, but if you want to know

how to setup Kafka cluster please use below related articles for the instructions.

How to enable KafkaAppender 

Overwrite the following property names with your configurations based on your Kafla setup.

# Kafka Logging related configuration
isRunningOnAws= false - Set to true if you are running Airavata on AWS
kafka.broker.list= localhost:9092 - One or more kafka broker node address with the port, Giving one is enough because KafkaProducer will find out the addresses of other nodes
kafka.topic.prefix= local - Topic prefix you want to use because Airavata will create the topic names for you
enable.kafka.logging= true - Enable kafka Appender to register as a log Appender.

How is a sample Json log message look like ?

{
        "serverId" => {
        "serverId" => "192.168.59.3",
        "hostName" => "192.168.59.3",
         "version" => "airavata-0.16-135-gac0cae6",
           "roles" => [
            [0] "gfac"
        ]
    },
           "message" => "Skipping Zookeeper embedded startup ...",
         "timestamp" => "2016-09-09T20:57:08.329Z",
             "level" => "INFO",
        "loggerName" => "org.apache.airavata.common.utils.AiravataZKUtils",
              "mdc"  => {
      				"gateway_id": "21845d02-7d2c-11e6-ae22-562311499611",
      				"experiment_id": "21845d02-7d2c-11e6-ae22-34b6b6499611",
     				"process_id": "21845d02-7d2c-11e6-ae22-56b6b6499611",
      				"token_id": "21845d02-7d2c-11e6-ae22-56b6b6499611"
    			},
        "threadName" => "main",
          "@version" => "1",
        "@timestamp" => "2016-09-09T20:57:11.678Z",
              "type" => "gfac_logs",
              "tags" => [
        [0] "local",
        [1] "CoreOS-899.13.0"
    ],
    "timestamp_usec" => 0
}

http://kafka.apache.org/documentation.html

https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html

https://www.digitalocean.com/community/tutorials/how-to-set-up-a-production-elasticsearch-cluster-on-ubuntu-14-04

https://www.elastic.co/cloud/as-a-service/signup

  • No labels

For an overview of Apache Airavata visit the Website - https://airavata.apache.org/