How can I monitor the performance of Datameer?

Question

How can I monitor the performance of Datameer?

Answer

You can monitor Datameer using Nagios.

Features

Nagios is a popular open source computer system and network monitoring software application. It watches hosts and services, alerting users when things go wrong and again when they get better. Nagios allows you to monitor anything. All hosts and services are monitored through plugins which are simple shell-scripts and programs. Plugins can written in any language. The language you choose needs the ability to print using stdout and return exit codes.

Requirements

  • Linux (or UNIX variant)
  • Configured network (Most of the monitoring plugins works over the network)
  • When using CGI's you also need: 
    • A Webserver (e.g. Apache HTTPD)
    • gd library for "statusmap" and "trends" CGIs

Installation

There are many ways to install Nagios including using a package-manager or building from source. We recommend building from source, because it's easier to understand how Nagios works and where the different files are stored.

For more information see http://www.nagios.org/download/core/thanks/

If you don't want to build from source, read the next sub-section(s).

Install Nagios on Debian5 with apt

Please note, that words like <address> or <port> are placeholders and needs to be replaced with your settings.

  • Install packages using the following command:
    aptitude install nagios3 nagios-plugins
  • Configure Apache to allow access to the Nagios web interface:
    htpasswd -c /etc/nagios3/htpasswd.users nagiosadmin
  • Edit /etc/nagios3/nagios.cfg and set check_external_commands
  • Update permissions for Nagios and Apache:
    /etc/init.d/nagios3 stop 
    dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
    dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3
    /etc/init.d/nagios3 start
  • Then look at the Nagios UI

Install Datameer Job Status plugin

  • Install PHP:
    # aptitude install php5-cli
  • Create /etc/nagios3/conf.d/das_host.cfg
    define host{
    use generic-host ; Inherit default values from a template
    host_name das_server
    ;Insert your environment
     address <address>
    }
  • Create file /etc/nagios3/conf.d/das_service.cfg
    define service{
    use generic-service ; Name of service template to use
    host_name das_server ; The hosts where this service is available
    service_description DAS_JobStatus ; How should Webinterface display this
    service as name ; Insert your environment
    check_command check_das!<user>:<password>@<address>!<port>!<jobConfigurationId>
    ; The command with parameters to get the job status via rest api
    max_check_attempts 1 ; How many retries if state isn't OK
    check_interval 1 ; How long it takes for a one check-interval (minutes)
    retry_interval 1 ; How long it takes for a one recheck-interval (minutes)
    notification_interval 0 ; How long it takes for a one resend -notification interval (minutes) 0 means no resend
    first_notification_delay 0 ; How long to wait before sending a first notification (minutes) 0 means immediately
    notifications_enabled 1 ; Enable/Disable Notification
    }
  • If you need to change your configurations frequently, then look at http://www.ubuntugeek.com/nagios-configuration-tools-web-frontends-or-gui.html.
  • Create /usr/lib/nagios/plugins/check_das file and paste the check_das code from Job Status plugin section of this page.
  • Set execute permissions
    chmod +x /usr/lib/nagios/plugins/check_das
  • Create /etc/nagios-plugins/config/check_das.cfg:
    define command{
     command_name check_das 
    command_line /usr/lib/nagios/plugins/check_das -m $ARG1$ -s $ARG2$ -d $ARG3$
    }
    This is the most useful command for nagios; job-history is more interesting for munin or nagiosgrapher, and job-details just returns the job-configuration details.
  • Restart Nagios:
    /etc/init.d/nagios3 restart

Configuration

See: http://nagios.sourceforge.net/docs/3_0/config.html

Adding a command

A command is a predefined configuration for a shell script which acts as a Nagios plugin. It defines the name for that command and the parameters used. The values for the command parameters are placeholders, which will be replaced later with the correct values. By default, you can find the Command-Configuration-File in <nagios-root>/etc/objects/commands.cfg.

Example

define command{
command_name my_command
command_line $USER1$/myprogram -a $ARG1$ -b $ARG2$ -c $ARG3$
}

In this example, we define a command_name called my_command. The name is used later to describe which command we want to use for monitoring a service. The command_name doesn't need to be the same as the name of the program (it is only usee for identifying the command). Next, define the command_line. This parameter tells Nagios how the program will be used. Nagios supports macros which allows you to avoid editing the command every time you want to use an another parameter for that command. Macros are like variables which are replaced later with the correct values. In this example, $USER1$ contains the path to the Nagios plugins. Then, define the file name of the program and the parameters for that program. Replace the parameter values through macros called $ARGn$ (where n = order number). These "Argument"-macros are replaced later (inside the service-definition) with correct values in the same order described in the command.

Further information about configuration can be found at: http://nagios.sourceforge.net/docs/3_0/objectdefinitions.html#command

Adding a service

A service definition is used to identify a service. The term "service" is used very loosely. It can mean an actual service that runs on the host (POP, SMTP, HTTP, etc.) or some other type of metric associated with the host (such as the response to a ping, number of logged in users, amount of free disk space, etc.). By default, you can find the Service-Configuration-File for the local machine in "<nagios-root>/etc/objects/localhost.cfg"

Example

define service{
use local-service
host_name localhost
service_description My Own Service
check_command my_command!value_a!value_b!value_c
}

In this example, use specifies to use the Service-Template local-service. It's a template containing settings used for all local services. Define host_name to be the names of machines which "runs" or are associated with that service. The service_description is a description displayed inside Nagios.  The check_command option runs the command my_command with some parameters. Those parameters are separated with a "!" (exclamation mark). The service shown in this example uses three parameters (the three we defined in the command definition) for that command. In this case, my_command!value_a!value_b!value_c executes (internally) <path-to-plugins>/myprogram -a value_a -b value_b -c value_c

To learn more, see http://nagios.sourceforge.net/docs/3_0/objectdefinitions.html#service 

Plugins

Job Status

Requires php5-cli installed.

This plugin monitors the status of a job from Datameer. It's getting the JSON value from REST-API through accessing

http://<address>:<port>/rest/job-configuration/job-status/<jobConfigurationId>

It writes a string to stdout which can be parsed by Nagios and returns an exit code which is used as the current status of the service you are monitoring.

  • Syntax:

    # /usr/lib/nagios/plugins/check_das -m <address> -s <port> -d <jobConfigurationId>
    
  • Command-Line-Parameters:

    Parameter

    Default-Value

    Description

    -b

    2048

    Read buffer length for the JSON-Output, which contains the JobStatus information.

    -d

    0

    The job ConfigurationID from the job you want to monitor

    -h

     

    Shows the Help for this script

    -m

    127.0.0.1

    The machine where the Datameer application has been installed

    -s

    8080

    The port which Datameer uses

    -v

     

    Shows the version of this script

  • Execution examples:

    You can execute each nagios plugin manually, they all return a standardized line. Maybe useful for testing.

    Command

    Description

    /usr/lib/nagios/plugins/check_das

    Monitor Job-ConfigurationID "0" from "127.0.0.1:8080"

    /usr/lib/nagios/plugins/check_das -d 42

    Monitor JobConfigurationID "42" from "127.0.0.1:8080"

    /usr/lib/nagios/plugins/check_das -m mydomain.com -s 8043 -d 61

    Monitor JobConfigurationID "61" from "mydomain.com:8043"

    /usr/lib/nagios/plugins/check_das -m myusername:mypwd@mydomain.com -s 443 -d 89

    Monitor JobConfigurationID "89" from "mydomain.com:443" with User "myusername" and Password "mypwd"

  • Nagios Command
    define command{
    command_name check_das # The Name of the Command
    command_line $USER1$/check_das -m $ARG1$ -s $ARG2$ -d $ARG3$ # The Command with Parameters
    }
  • Nagios Service:

    define service{      
      use local-service ; Name of service template to use
      host_name localhost ; The hosts where this service is available
      service_description DAS_JobStatus ; How should Webinterface display this service as a name
      check_command check_das!127.0.0.1!8080!1 ; The command with parameters
      max_check_attempts 1 ; How many retries if state isn't OK
      check_interval 1 ; How long it takes for a one check-interval (minutes)
      retry_interval 1 ; How long it takes for a one recheck-interval (minutes)
      notification_interval 0 ; How long it takes for a one resend-notification interval (minutes) 0 means no resend
      first_notification_delay 0 ; How long to wait before sending a first notification (minutes) 0 means immediately
      notifications_enabled ; Enable/Disable Notification
     }      

The parameters below "check_command" are used to send the failure notification only one time.

Weblinks

Attachement