nAttrMon kill after an amount of minutes

On a nAttrMon configuration you might have several inputs, validations and outputs. You might use the waitForFinish parameter to ensure another instance of the same plug doesn't start while the previous is still running.

This is specially true usually on input plugs that depend on the reply of external systems (e.g. database, sftp, a REST service, etc...).

But what if there isn't any timeout in place or you simply don't want that plug to run forever keeping nAttrMon waiting? You can use the setting killAfterMinutes.

Example

input:
  name            : Big queries
  # running every hour
  cron            : "0 */1 * * *"
  onlyOnEvent     : true
  # make sure that just one is executing
  waitForFinish   : true
  # make sure if doesn't execute for more than 30 minutes
  killAfterMinutes: 30
  execFrom        : nInput_DB
  execArgs        :
    key : myBigDatabase
    sqls:

      Database/Big query 1: |
        SELECT stuff 
        FROM something 
        -- [...]

In this case, every hour, nAttrMon will try to execute the "Big queries" but since killAfterMinutes is defined it will interrupt/terminate this input execution if it's longer than 30 minutes.

Using buffers in nAttrMon

If you have a large number of nAttrMon inputs and probably a smaller number of outputs the probability of having sets of inputs that all run at the same time is high.

There is no problem in this as nAttrMon will handle each input execution in parallel but you might find, by debugging nAttrMon, that your outputs and/or validations might be "overwhelmed".

The symptoms are usually big spikes on cpu usage (when all inputs start) and an overall delay of execution of plugs (plugs that should execute afterwards take longer than expected to start).

So if you have n inputs and all of them change attributes you will have n changes to nattrmon::cvals. If your outputs subscribe nattrmon::cvals changes each of those outputs will be triggered also n times to process each change. Of course the larger number of inputs the more threads will be executing in parallel.

This is usually the reason for larger than expected cpu usage spikes and delay of execution of other plugs because nAttrMon is busy executing outputs.

The same applies between inputs & validations and validations & outputs where the nattrmon::warnings channel is also used.

So what can be done to aliviate the load on outputs and validations?

nAttrMon buffers

The answer is buffering changes to nattrmon::cvals and nattrmon::warnings.

OpenAF provides a special channel type called 'buffer'. This channel implementation can receive a set of operations (e.g. set, setall, unset, unsetall) from a source channel and only trigger those operations, as batch operations (e.g. setall and unsetall), on a target channel on two conditions: maximum time after the first operation was buffered and maximum number of operations buffered.

The end result in nAttrMon is a single output execution for n inputs/validations executed at the same time and a single validation execution for _n_inputs executed at the same time. This solves part of the cpu usage spikes and potential delays on other plug executions when dealing with multiple simultaneous (or near simultaneous) inputs or validations.

What needs to be changed to use nAttrMon buffers

First activate the following flags on the nattrmon.yaml configuration file (based on the provided nattrmon.yaml.sample):

BUFFERCHANNELS: true
BUFFERBYNUMBER: 100
BUFFERBYTIME  : 1000

This creates two new channels: nattrmon::cvals::buffer and nattrmon::warnings::buffer. These channels will only change after nattrmon::cvals and nattrmon::warnings have 100 changes or more than 1000ms passed since the first change not reflected on the buffer channels.

Inputs

The inputs will continue to use internally nattrmon::cvals. So no changes are needed on inputs.

Validations

The validations can now use the nattrmon::cvals::buffer since they will also benefit from the buffering as the outputs do:

validation:
   name            : Set of generic validations
   chSubscribe     : nattrmon::cvals::buffer
   waitForFinish   : false
   killAfterMinutes: 5
   execFrom        : nValidation_Generic
   execArgs        :
   [...]

Note: you can use killAfterMinutes in any plug to ensure that if it goes over a specific amount of minutes nAttrMon will terminate the execution of that plug as a sanity measure. Use with care.

Note for custom validations: your validation should be able to handle a single change (args.op == "set") or an array of changes (args.op == "setall").

Outputs

The outputs can now use the nattrmon::cvals::buffer and nattrmon::warnings::buffer:

output:
    name         : Output Warnings by email
    chSubscribe  : nattrmon::warnings::buffer
    waitForFinish: false
    onlyOnEvent  : false
    [...]

output:
    name          : Output ES values
    chSubscribe   : nattrmon::cvals::buffer
    considerSetAll: true
    waitForFinish : true
    onlyOnEvent   : true

Note for custom outputs: your output should be able to handle a single change (args.op == "set") or an array of changes (args.op == "setall").

Simple nAttrMon debugging

One of the common questions when using nAttrMon is: "has my plug even executed?"

There are several approaches to find the answer being, one of the most common ones, just adding a print statement on your nAttrMon plug or turning the specific plug debug flag, if it exists. There also functions to just test specific plugs inside and outside nAttrMon.

Nevertheless, has your nAttrMon's configuration grows it might become difficult to understand: what is running at which moment? what is taking so much CPU? why it was fast and now it's slow?

Almost always the answer relies on what is being executed at each time.

Channel 'ps' and others

Wouldn't you like to just execute a "ps" (similar to the unix 'ps' command) on nAttrMon and just see what's running? Well, you can.

Just add the channels output plug (check on outputs.disabled for *channels.yaml).

Now, on an openaf-console, just execute:

> $ch("ps").createRemote("http://nattrmon:nattrmon@your.server:8090/chs/ps");
> table $ch("ps").getAll();
    name    | type |                uuid                |         start
------------+------+------------------------------------+------------------------
Input Test 1|inputs|1fc61235-97d6-068f-e705-5693712aba75|2019-10-01T09:42:10.027Z

In this example there was only one plug executing but it will return as many as there is currently running. It's useful to understand which plugs take too long to execute. The plugs that you find here for a long period of time probably need to be rethinked: is the plug taking longer than expected? is it ocasional?

There is another channel that comes also to the rescue to answer these questions:

> $ch("plugs").createRemote("http://nattrmon:nattrmon@your.server:8090/chs/plugs");
> $ch("plugs").getAll();
[...]
+-----+----------+--------------------------+----------------+
| [1] |    meta: |                    name: | Input Test 1   |
|     |          |             description: |                |
|     |          |                    type: | inputs         |
|     |          |                category: | uncategorized  |
|     |          |                    cron: | */10 * * * * * |
|     |          |            timeInterval: | -1             |
|     |          |           waitForFinish: | true           |
|     |          |             onlyOnEvent: | true           |
+-----+----------+--------------------------+----------------+
|     |    args: |            {}            | -              |
+-----+----------+--------------------------+----------------+
|     |    last: | 2019-10-01T09:49:02.521Z |                |
|     | created: | 2019-10-01T09:46:25.237Z |                |
+-----+----------+-------------------------------------------+
|     |   stats: |        lastExecTimeInMs: | 2501           |
|     |          |         avgExecTimeInMs: | 2500.5         |
|     |          |           numberOfExecs: | 16             |
|     |          |    numberOfExecsInError: | 0              |
|     |          |         numberOfRunning: | 1              |
+-----+----------+--------------------------+----------------+
[...]

Note: if you have the http output activated you can also see this info in the "Plugs" tab on your browser

This information allows you to see how the plug is registered, when was the "last" time it executed, how much time it took (yes, I added a sleep on the dummy test input), the average execution time, how many times it executed without and with errors and how many of them are currently being executed.

Debug flag

There is also another way: if you edit or create a nattrmon.yaml configuration file (based on the nattrmon.yaml.sample) there are three options that might help you while debugging:

LOGCONSOLE: true
LOG_ASYNC: false
DEBUG: true

Note: do uncomment them if they are commented. And don't forget to comment again after debugging.

The first flag, LOGCONSOLE, will not create log files and just print the log to the stdout. Of course this isn't mandatory but you probably don't want to fill up your logs with debug information.

The second flag, LOG_ASYNC, determines if nAttrMon's logs should be "written when possible without stopping nAttrMon" (option true, the default) OR if nAttrMon's logs should be "written immediatelly stopping nAttrMon if needed".

The third flag, DEBUG, will turn on all the logging debug messages. It will show you a lot more information, including which plug is being executed.

2019-10-01 09:57:18.344 | INFO | DEBUG | nAttrMon monitor plug
2019-10-01 09:57:18.367 | INFO | DEBUG | Added plug system monitor
2019-10-01 09:57:18.377 | INFO | DEBUG | Starting at fixed rate for system monitor - 30000
2019-10-01 09:57:18.393 | INFO | DEBUG | Creating a thread for system monitor with uuid = ff7c708c-ba70-4371-b764-96f2d1c44eed
2019-10-01 09:57:18.408 | INFO | DEBUG | nAttrMon start load plugs
2019-10-01 09:57:18.417 | INFO | DEBUG | Ignore list: []
2019-10-01 09:57:18.857 | INFO | Loading inputs: /openaf/nAttrMon/config/inputs/01.test.yaml
2019-10-01 09:57:18.870 | INFO | DEBUG | Added plug Input Test 1
2019-10-01 09:57:18.875 | INFO | DEBUG | nAttrMon exec input plugs
2019-10-01 09:57:18.904 | INFO | DEBUG | nAttrMon exec output plugs
2019-10-01 09:57:18.907 | INFO | DEBUG | nAttrMon exec validation plugs
2019-10-01 09:57:18.910 | INFO | DEBUG | nAttrMon restoring snapshot
2019-10-01 09:57:18.917 | INFO | nAttrMon started.
2019-10-01 09:57:20.010 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:57:30.005 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:57:40.007 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:57:48.395 | INFO | DEBUG | Executing 'system monitor' (ff7c708c-ba70-4371-b764-96f2d1c44eed)
2019-10-01 09:57:50.009 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:58:54.626 | INFO | nAttrMon stopped.

On this example we just added a single input plug "Input Test 1" to be executed every 10 seconds. You can see those executions and also the "system monitor". The "system monitor" is a special plug to ensure nAttrMon is running (if not it will restart itself).

nAttrMon multiple plugs in one file

When adding any nAttrMon plug (either inputs, validations or outputs), the advisable setup is to have a structured set of files on a folder hierarchy. In this structure each file is usually a single plug definition.

Nevertheless once you get more advanced setups there are cases where you might want to have more than one plug per file taking advantage of the plug "ignore"/"disabling" mechanisms.

Let's take a simple example of a nAttrMon input plug:

input:
  name         : Database queries
  cron         : "*/10 * * * *"
  waitForFinish: true
  onlyOnEvent  : true
  execFrom     : nInput_DB
  execArgs     :
     key: MYDB
     sqls:
        Database/Status XYZ: >
           SELECT control "Control", value "Value"
           FROM status
           WHERE control in ('X', 'Y', 'Z')

        Database/Status ABC: >
           SELECT control "Control", value "Value"
           FROM status
           WHERE control in ('A', 'B', 'C')

In this example both queries will execute every 10 minutes. But you might want different cron schedules temporarily to find some issue on ABC controls.

Usually you would create two files, comment the ABC query on the first and just have the ABC on the second. But now if you have to quickly ignore all database queries you will have to remember the extra temporary plug you created before. Could there be a better alternative?

The answer is yes. You can keep both SQLs on the same file and temporarily have different cron settings:

input:
  #--------------------------------
  - name         : Database queries
    cron         : "*/10 * * * *"
    waitForFinish: true
    onlyOnEvent  : true
    execFrom     : nInput_DB
    execArgs     :
       key: MYDB
       sqls:
          Database/Status XYZ: >
             SELECT control "Control", value "Value"
             FROM status
             WHERE control in ('X', 'Y', 'Z')

          #Database/Status ABC: >
          #   SELECT control "Control", value "Value"
          #   FROM status
          #   WHERE control in ('A', 'B', 'C')

  #--------------------------------------------
  - name         : Database queries (temporary)
    cron         : "*/1 * * * *"
    waitForFinish: true
    onlyOnEvent  : true
    execFrom     : nInput_DB
    execArgs     :
       key: MYDB
       sqls:
          Database/Status ABC : >
             SELECT control "Control", value "Value"
             FROM status
             WHERE control in ('A', 'B', 'C')             

What's the difference? Instead of defining a map, if you define an array nAttrMon it will consider two plugs instead of one. So now you can control the temporary different cron setting by just commenting lines while keeping everything in the same file.

Note: always have different plug names, otherwise the last will overwrite the previous.

Using ElasticSearch with nAttrMon

nAttrMon captures monitoring inputs to be validated and output to where deem necessary. But you start having more than one instance or you are monitoring a lot of inputs you start having the need to build specific dashboards, visualizations, etc.
Additionally althougth nAttrMon lets you store input history, it's limited since it's not intended to be used for longer than a couple of days.
For all these challenges output and use ElasticSearch & Kibana provide you the answer:

• Elasticsearch let's you store more than just a couple of days of nAttrMon's input
• Kibana let's you visualize the data anyway you want from ElasticSearch.

How to set it up

So how to get nAttrMon to output to ElasticSearch? Simply create a new output (example in YAML):

output:
  name         : Output ES values
  chSubscribe  : nattrmon::cvals
  waitForFinish: true
  onlyOnEvent  : true
  execFrom     : nOutput_ES
  execArgs     : 
    url      : http://127.0.0.1:9200
    #user     : nouser
    #pass     : nopass
    #considerSetAll: false 
    #funcIndex: "ow.ch.utils.getElasticIndex('nattrmon-attrs',\"yyyy.ww\")" 
    ## note: Check getElasticIndex function, If a specific format is needed
    ## you can provide it as aFormat (see ow.format.fromDate)"
    funcIndex: "ow.ch.utils.getElasticIndex('nattrmon-attrs', 'yyyy.ww')"
    #index: nattrs
    stampMap : 
      environment: EnvA
      region     : EU
    #include  :
    #  - test/test 1
    # exclude  :

In most of the cases you only need to specify the ElasticSearch URL. But there are more parameters:

Exec arguments	Description
index	The ElasticSearch index where to store the current values (channel nattrmon::cvals).
funcIndex	In most of the cases you want to use a different ElasticSearch index on different times. You can specify a function for that.
considerSetAll	The nOutput_ES is based on channel subscription (chSubscribe) so it can potential receive a setAll operation. In case you use nAttrMon channel buffers you need to set this argument to true. Otherwise you are probably fine with false.
stamp	Since you might have several nAttrMon's instances dumping attribute values to the same ElasticSearch index you can "stamp" each output entries with specific entries (e.g. in the example environment and region)
include	You can specify that only some attributes' values will be sent to ElasticSearch.
exclude	Or you can specifiy that only some attributes' values shouldn't be sent to ElasticSearch.
user	The user credential if needed.
pass	The password credential if needed.

Since in most ElasticSearch configuration cases it's advisable to have different indexes per time you can provide nAttrMon with a function on funcIndex. Using the helper function ow.ch.utils.getElasticIndex:

ow.ch.utils.getElasticIndex('nattrmon-attrs', 'yyyy.ww')

How the output looks in ElasticSearch?

For each attribute value change a new entry will be created in ElasticSearch:

The key is id and it's a hash. Then you will have the name of the attribute and the date it was checked. The value will be in a key with the same name as the attribute. If the attribute has categories the '/' will be converted to a ''. In the example above: _Random_Number and Random_Dice. The stamp keys will also, of course, be part of the ElasticSearch entry.

And the warnings?

To add warnings just create a new similar output changing it's name, the ElasticSearch index where the warnings will be kept and replace on the chSubscribe to use nattrmon::warnings instead of nattrmon::cvals.

Example of an OpenAF channel accessing the ElasticSearch's nattrmon-warns index

The warnings instead of the name of the attribute have the title of the warning, level (e.g. High, Medium, Low, Info, Closed), the description, the date of the last update and date of creation. The stamp map will also be part of the ElasticSearch entry as expected.

Example of a Kibana dashboard using sample warnings from nAttrMon