Modifying XML using E4X

OpenAf uses the Java Mozilla Rhino javascript engine and one of the features that, despite being obsolete, has still been kept is E4X.

E4X simply allows for easy interaction and also modification of XML content in Javascript.

Let's start by the XML example used in "Accessing XML using E4X":

<shopcart>
  <!-- Client id -->
  <client type="personal" id="12345">
    <name>Joe Smith</name>
  </client>

  <!-- Store id -->
  <store>
    <id>12345</id>
    <type>diy</type>
  </store>

  <!-- Items on the shopcart -->
  <items>
    <item id="123">
       <qty>1</qty>
       <name>Item 1</name>
    </item>
    <item id="456">
       <qty>10</qty>
       <name>Item 2</name>
    </item>
  </items>
</shopcart>

Then you can load it to a xml javascript variable through various ways:

var xml = io.readFileXML("myxml.xml");
var xml = new XMLList(myXMLString);
var xml = <shopcart><!-- Client id --><client type="personal" id="12345">...

Modifying data using the xml variable in javascript

Any element that you can access directly, you can change it:

> xml.client.name.toString()
"Joe Smith"
> xml.client.name = "Scott Tiger"
"Scott Tiger"
> xml.client.name.toString()
"Scott Tiger"

Changing attributes:

> xml.client.@id.toString()
"12345"
> xml.client.@id = "67890"
"67890"
> xml.client.@id.toString()
"67890"

And any comments are kept:

> xml.toString()
<shopcart>
  <!-- Client id -->
  <client id="12345" type="personal">
    <name>Scott Tiger</name>
  </client>
  <!-- Store id -->
  <store>
    <id>12345</id>
...    

Adding and removing children elements

To add you can simply think like an array:

> xml.items.item[2] = <item id="789"><qty>2</qty><name>Item 3</name></item>
> xml.items.item.length()
3

And to remove a children element you can use delete:

> delete xml.items.item[2]
> xml.items.item.length()
2

There are more methods, listed on the end of this section, to add or remove elements like appendChild(child), prependChild(child), insertChildAfter(), insertChildBefore()_

XML Object methods

Method	Description
appendChild(child)	Adds the child XML object at the end of the corresponding XML element children.
prependChild(child)	Inserts a child XML object prior to the beginning of the corresponding XML element children.
insertChildAfter(childRef, childNew)	Inserts childNew after childRef.
insertChildBefore(childRef, childNew)	Inserts childNew before childRef.
copy()	Returns a deep copy of a XML object starting on the corresponding XML element children (without the parent).

Accessing XML using E4X

OpenAf uses the Java Mozilla Rhino javascript engine and one of the features that, despite being obsolete, has still been kept is E4X.

E4X simply allows for easy interaction of XML content in Javascript.

Let's start by the following XML example:

<shopcart>
  <!-- Client id -->
  <client type="personal" id="12345">
    <name>Joe Smith</name>
  </client>

  <!-- Store id -->
  <store>
    <id>12345</id>
    <type>diy</type>
  </store>

  <!-- Items on the shopcart -->
  <items>
    <item id="123">
       <qty>1</qty>
       <name>Item 1</name>
    </item>
    <item id="456">
       <qty>10</qty>
       <name>Item 2</name>
    </item>
  </items>
</shopcart>

Loading XML to OpenAF

You can load this xml from a file:

var xml = io.readFileXML("myxml.xml");

From a string:

var xml = new XMLList(myXMLString);

Or directly:

var xml = <shopcart><!-- Client id --><client type="personal" id="12345">...

All of them will result in a xml javascript variable of type "xml".

Accessing data using the xml variable in javascript

To access any element simply think like you would access it if it was a JSON map:

> xml.client.name.toString()
"Joe Smith"
> xml.store.id.toString()
"12345"

Each element has several methods from which we are using the toString method. There is a list of methods on the end of this section.

For attributes it's the same but add the prefix "@":

> xml.client.@type.toString()
"personal"

For multiple items you can think of it now as an array:

> xml.items.item[0].@id.toString()
123
> xml.items.item[1].qty.toString()
10

But this, is also valid (think of it as the first "store" tag element):

> xml.store[0].id.toString()
"12345"

Accessing all children nodes

> xml.children()[1].name.toString()
"Joe Smith"

Why is the children element on position 1 and not in 0? Because comments are also nodes:

> xml.children()[0].toString()
"<!-- Client id -->"

Using for cycles

Don't forget that you are in javascript, so you can do for cycles:

var totalQty = 0;
for(var i in xml.items.item) {
    totalQty += Number(xml.items.item[i].qty);
}
// 11

You can also search all descendants in a for cycle:

for(var i in xml..name) {
    print(xml..name[i].toString());
}
// Joe Smith
// Item 1
// Item 2

Using namespaces

You can also use namespaces:

var xml = <links><a xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://www.google.com">Test link</a></links>;

var ns = new Namespace("http://www.w3.org/1999/xlink");
var url = xml..a;
var link = url.@ns::href;
// http://www.google.com

XML Object methods

Method	Description
attribute(attributeName)	Returns the attribute attributeName from the corresponding XML element node.
attributes()	Returns a list of the attributes associated with the corresponding XML element node.
child(propertyName)	Returns the children elements named propertyName associated with the corresponding XML element node.
children()	Returns all children elements associated with the corresponding XML element node.
descendants(name)	Equivalent to "xml..name".
elements(name)	Equivalent to children() if no name is provided or equivalent to child if a name is provided.
parent()	Returns the parent element associated with the corresponding XML element node.
comments()	Returns the children elements of type 'comment' associated with the corresponding XML element node.
text()	Returns the text component of the corresponding XML element node.
name()	Returns a map with the name and local name associated with the corresponding XML element node.
localName()	Returns the local name associated with the corresponding XML element node.
namespace(prefix)	Returns the namespace for the provided prefix of the namespace associated with the corresponding XML element node.
namespaceDeclarations()	Returns an array of namespaces associated with the corresponding XML element node.
childIndex()	Returns the index number on the childrens list of the corresponding XML element node.
length()	Returns the size of the children elements of the corresponding XML element node.
nodeKind()	Returns if the corresponding XML element node is a element of a comment.
toString()	Returns the string of the current XML element node.
toXMLString()	Returns the XML encoded string of the current XML element node.

Appending to files

Let's start with a simple question: how to append content to an existing file?

The answer is: check the "help" for the io.writeFile* function you need. Most of them already support an extra parameter to append to existing files.

io.writeFileString example

To write to a custom log you would do something like:

io.writeFileString("mylog.log", new Date() + " | Just did something\n");

But using this example the file mylog.log would get overwritten every time you execute it. To get around it simple add an extra flag:

io.writeFileString("mylog.log", new Date() + " | Just did something\n", void 0, true);

And everytime you run it, it will add a new line to the mylog.log file.

Appending several files into one

A one_liner that is usually helpfull is:

> $from(io.listFilenames("my/path")).ends(".js").select(r => { io.writeFileString("all.js", io.readFileString(r), void 0, true); });

This will actually create all.js file with the contents of all files in my/path with suffix ".js". Of course you now can change the $from conditions to whatever special conditions you might have.

nAttrMon kill after an amount of minutes

On a nAttrMon configuration you might have several inputs, validations and outputs. You might use the waitForFinish parameter to ensure another instance of the same plug doesn't start while the previous is still running.

This is specially true usually on input plugs that depend on the reply of external systems (e.g. database, sftp, a REST service, etc...).

But what if there isn't any timeout in place or you simply don't want that plug to run forever keeping nAttrMon waiting? You can use the setting killAfterMinutes.

Example

input:
  name            : Big queries
  # running every hour
  cron            : "0 */1 * * *"
  onlyOnEvent     : true
  # make sure that just one is executing
  waitForFinish   : true
  # make sure if doesn't execute for more than 30 minutes
  killAfterMinutes: 30
  execFrom        : nInput_DB
  execArgs        :
    key : myBigDatabase
    sqls:

      Database/Big query 1: |
        SELECT stuff 
        FROM something 
        -- [...]

In this case, every hour, nAttrMon will try to execute the "Big queries" but since killAfterMinutes is defined it will interrupt/terminate this input execution if it's longer than 30 minutes.

Using OpenAF thread "boxes"

What is an OpenAF thread-box? It's basically being able to run a block of OpenAF code (a function) on a separate Java thread that will be interrupted either due to a timeout or the specific result of a control function.

It's specially useful for controlling the execution of custom code that might run for longer than expected (for example: executing a block of code that should run in seconds taking more than 5 minutes) or until a specific condition happens (for example: executing a block of code while no Ctrl-C is hit on the keyboard by the user).

Example with timeout

Let's examine an example:

> var res = $tb(() => {
    print("Start...");
    sleep(5000, true);
    print("End.");
})
.timeout(2500)
.exec();

Start...
> res
timeout

So what happened? The code should print Start wait 5 seconds and then print End. But since the thread-box should timeout after 2.5 seconds it just printed Start and then finished returning the string timeout.

If you remove the timeout:

> var res = $tb(() => {
    print("Start...");
    sleep(5000, true);
    print("End.");
})
.exec();

Start...
End.
> res
true

Now, without a timeout, it prints Start and End and return res.

Example with stopWhen

If you need to control the execution with a custom function you can use the stopWhen function:

> var c = 0;
> var res = $tb(() => {
    for(var ii = 0; ii < 10; ii++) {
        print(c);
        c++;
        sleep(100);
    }
})
.stopWhen(() => {
    return c > 5;
})
.exec();

0
1
2
3
4
5
> res
stop

Whenever the function returns true, the execution of the thread-box block of code will be interruped. To increase the interval between calls to the stopWhen function, 25ms, you can add extra time with a sleep call on body of the function.

Of course you can mix stopWhen and timeout if necessary.

Note: you can use threadBoxCtrlC function as a stopWhen function to exit whenever a Ctrl-C is hit on the keyboard by a user.

How to run OpenAF code from shell script

To run an OpenAF script you just need to execute: 'openaf -f myScript.js'. But what if you wanted to include some OpenAF code without having to create a separate file. There are several ways to achived this with OpenAF.

Inline code

The simplest way it's using the '-c' option:

$ openaf -c 'var res = $rest().get("https://uinames.com/api"); print(printMap(res, void 0, "plain", true))'
+----------+---------------+
|    name: | John          |
| surname: | Lawrence      |
|  gender: | male          |
|  region: | United States |
+----------+---------------+

Incorporating into a shell script:

#!/bin/sh

name=$(openaf -c 'print($rest().get("https://uinames.com/api").name)')
echo Hi $name

Running:

$ sh sayHi.sh
Hi Andrea

Multiline code

For simple code inline code works well but for multi line code it may become hard to read. Another option is using the '-p' option that will receive input from stdin.

#!/bin/bash

output=$(openaf -i script -p << __endScript

  // OpenAF javascript code
  var res = \$rest().get("https://uinames.com/api/?ext");
  print(res.name + " " + res.surname + ";" + res.phone);

__endScript
)

# interpreting result in bash
IFS='\;'
read -a strarr <<< "$output"

echo Name : ${strarr[0]}
echo Phone: ${strarr[1]}

And the result of mixing shell script and OpenAF:

Name : Angela Johnston
Phone: (134) 382 2457

Using buffers in nAttrMon

If you have a large number of nAttrMon inputs and probably a smaller number of outputs the probability of having sets of inputs that all run at the same time is high.

There is no problem in this as nAttrMon will handle each input execution in parallel but you might find, by debugging nAttrMon, that your outputs and/or validations might be "overwhelmed".

The symptoms are usually big spikes on cpu usage (when all inputs start) and an overall delay of execution of plugs (plugs that should execute afterwards take longer than expected to start).

So if you have n inputs and all of them change attributes you will have n changes to nattrmon::cvals. If your outputs subscribe nattrmon::cvals changes each of those outputs will be triggered also n times to process each change. Of course the larger number of inputs the more threads will be executing in parallel.

This is usually the reason for larger than expected cpu usage spikes and delay of execution of other plugs because nAttrMon is busy executing outputs.

The same applies between inputs & validations and validations & outputs where the nattrmon::warnings channel is also used.

So what can be done to aliviate the load on outputs and validations?

nAttrMon buffers

The answer is buffering changes to nattrmon::cvals and nattrmon::warnings.

OpenAF provides a special channel type called 'buffer'. This channel implementation can receive a set of operations (e.g. set, setall, unset, unsetall) from a source channel and only trigger those operations, as batch operations (e.g. setall and unsetall), on a target channel on two conditions: maximum time after the first operation was buffered and maximum number of operations buffered.

The end result in nAttrMon is a single output execution for n inputs/validations executed at the same time and a single validation execution for _n_inputs executed at the same time. This solves part of the cpu usage spikes and potential delays on other plug executions when dealing with multiple simultaneous (or near simultaneous) inputs or validations.

What needs to be changed to use nAttrMon buffers

First activate the following flags on the nattrmon.yaml configuration file (based on the provided nattrmon.yaml.sample):

BUFFERCHANNELS: true
BUFFERBYNUMBER: 100
BUFFERBYTIME  : 1000

This creates two new channels: nattrmon::cvals::buffer and nattrmon::warnings::buffer. These channels will only change after nattrmon::cvals and nattrmon::warnings have 100 changes or more than 1000ms passed since the first change not reflected on the buffer channels.

Inputs

The inputs will continue to use internally nattrmon::cvals. So no changes are needed on inputs.

Validations

The validations can now use the nattrmon::cvals::buffer since they will also benefit from the buffering as the outputs do:

validation:
   name            : Set of generic validations
   chSubscribe     : nattrmon::cvals::buffer
   waitForFinish   : false
   killAfterMinutes: 5
   execFrom        : nValidation_Generic
   execArgs        :
   [...]

Note: you can use killAfterMinutes in any plug to ensure that if it goes over a specific amount of minutes nAttrMon will terminate the execution of that plug as a sanity measure. Use with care.

Note for custom validations: your validation should be able to handle a single change (args.op == "set") or an array of changes (args.op == "setall").

Outputs

The outputs can now use the nattrmon::cvals::buffer and nattrmon::warnings::buffer:

output:
    name         : Output Warnings by email
    chSubscribe  : nattrmon::warnings::buffer
    waitForFinish: false
    onlyOnEvent  : false
    [...]

output:
    name          : Output ES values
    chSubscribe   : nattrmon::cvals::buffer
    considerSetAll: true
    waitForFinish : true
    onlyOnEvent   : true

Note for custom outputs: your output should be able to handle a single change (args.op == "set") or an array of changes (args.op == "setall").

Simple nAttrMon debugging

One of the common questions when using nAttrMon is: "has my plug even executed?"

There are several approaches to find the answer being, one of the most common ones, just adding a print statement on your nAttrMon plug or turning the specific plug debug flag, if it exists. There also functions to just test specific plugs inside and outside nAttrMon.

Nevertheless, has your nAttrMon's configuration grows it might become difficult to understand: what is running at which moment? what is taking so much CPU? why it was fast and now it's slow?

Almost always the answer relies on what is being executed at each time.

Channel 'ps' and others

Wouldn't you like to just execute a "ps" (similar to the unix 'ps' command) on nAttrMon and just see what's running? Well, you can.

Just add the channels output plug (check on outputs.disabled for *channels.yaml).

Now, on an openaf-console, just execute:

> $ch("ps").createRemote("http://nattrmon:nattrmon@your.server:8090/chs/ps");
> table $ch("ps").getAll();
    name    | type |                uuid                |         start
------------+------+------------------------------------+------------------------
Input Test 1|inputs|1fc61235-97d6-068f-e705-5693712aba75|2019-10-01T09:42:10.027Z

In this example there was only one plug executing but it will return as many as there is currently running. It's useful to understand which plugs take too long to execute. The plugs that you find here for a long period of time probably need to be rethinked: is the plug taking longer than expected? is it ocasional?

There is another channel that comes also to the rescue to answer these questions:

> $ch("plugs").createRemote("http://nattrmon:nattrmon@your.server:8090/chs/plugs");
> $ch("plugs").getAll();
[...]
+-----+----------+--------------------------+----------------+
| [1] |    meta: |                    name: | Input Test 1   |
|     |          |             description: |                |
|     |          |                    type: | inputs         |
|     |          |                category: | uncategorized  |
|     |          |                    cron: | */10 * * * * * |
|     |          |            timeInterval: | -1             |
|     |          |           waitForFinish: | true           |
|     |          |             onlyOnEvent: | true           |
+-----+----------+--------------------------+----------------+
|     |    args: |            {}            | -              |
+-----+----------+--------------------------+----------------+
|     |    last: | 2019-10-01T09:49:02.521Z |                |
|     | created: | 2019-10-01T09:46:25.237Z |                |
+-----+----------+-------------------------------------------+
|     |   stats: |        lastExecTimeInMs: | 2501           |
|     |          |         avgExecTimeInMs: | 2500.5         |
|     |          |           numberOfExecs: | 16             |
|     |          |    numberOfExecsInError: | 0              |
|     |          |         numberOfRunning: | 1              |
+-----+----------+--------------------------+----------------+
[...]

Note: if you have the http output activated you can also see this info in the "Plugs" tab on your browser

This information allows you to see how the plug is registered, when was the "last" time it executed, how much time it took (yes, I added a sleep on the dummy test input), the average execution time, how many times it executed without and with errors and how many of them are currently being executed.

Debug flag

There is also another way: if you edit or create a nattrmon.yaml configuration file (based on the nattrmon.yaml.sample) there are three options that might help you while debugging:

LOGCONSOLE: true
LOG_ASYNC: false
DEBUG: true

Note: do uncomment them if they are commented. And don't forget to comment again after debugging.

The first flag, LOGCONSOLE, will not create log files and just print the log to the stdout. Of course this isn't mandatory but you probably don't want to fill up your logs with debug information.

The second flag, LOG_ASYNC, determines if nAttrMon's logs should be "written when possible without stopping nAttrMon" (option true, the default) OR if nAttrMon's logs should be "written immediatelly stopping nAttrMon if needed".

The third flag, DEBUG, will turn on all the logging debug messages. It will show you a lot more information, including which plug is being executed.

2019-10-01 09:57:18.344 | INFO | DEBUG | nAttrMon monitor plug
2019-10-01 09:57:18.367 | INFO | DEBUG | Added plug system monitor
2019-10-01 09:57:18.377 | INFO | DEBUG | Starting at fixed rate for system monitor - 30000
2019-10-01 09:57:18.393 | INFO | DEBUG | Creating a thread for system monitor with uuid = ff7c708c-ba70-4371-b764-96f2d1c44eed
2019-10-01 09:57:18.408 | INFO | DEBUG | nAttrMon start load plugs
2019-10-01 09:57:18.417 | INFO | DEBUG | Ignore list: []
2019-10-01 09:57:18.857 | INFO | Loading inputs: /openaf/nAttrMon/config/inputs/01.test.yaml
2019-10-01 09:57:18.870 | INFO | DEBUG | Added plug Input Test 1
2019-10-01 09:57:18.875 | INFO | DEBUG | nAttrMon exec input plugs
2019-10-01 09:57:18.904 | INFO | DEBUG | nAttrMon exec output plugs
2019-10-01 09:57:18.907 | INFO | DEBUG | nAttrMon exec validation plugs
2019-10-01 09:57:18.910 | INFO | DEBUG | nAttrMon restoring snapshot
2019-10-01 09:57:18.917 | INFO | nAttrMon started.
2019-10-01 09:57:20.010 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:57:30.005 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:57:40.007 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:57:48.395 | INFO | DEBUG | Executing 'system monitor' (ff7c708c-ba70-4371-b764-96f2d1c44eed)
2019-10-01 09:57:50.009 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:58:54.626 | INFO | nAttrMon stopped.

On this example we just added a single input plug "Input Test 1" to be executed every 10 seconds. You can see those executions and also the "system monitor". The "system monitor" is a special plug to ensure nAttrMon is running (if not it will restart itself).

Test latency

The usual latency test most of us use if simply execute the ping operating system command from the source to the target we want to measure. But nowadays, ICMP packets (what ping really transmits) might be blocked by a firewall or similar.

With OpenAF's function ow.format.testLatency that isn't a problem since it will try to open a TCP socket to the desired host and port and measure the time taken for the socket connection to be created (and then it closes it).

It's not perfect but is usually enough to understand the relative latency to another host and port.

First I will present a test function a later show how to use it as an one-liner.

The test function

Here is a test function that will taken several measures and return you: each measure, the average and a chart representation of the variation between each measure.

ow.loadFormat();

var tl = (host, port, times) => { 
    times = _$(times).isNumber().default(3); 
    var tries = [], sum = 0, max = 0; 

    for(var ii = 0; ii < times; ii++) {
        tries.push({
            sample: ii+1, 
            latency: ow.format.testPortLatency(host, port)
        });
    }; 
    tries.forEach((v) => {
        sum += v.latency; 
        max = (v.latency > max) ? v.latency : max; 
        v.chart = ow.format.string.progress(v.latency, max, 0, 50, "=", " ");
    }); 
    tries.push({ 
        sample: "avg", 
        latency: Math.floor(sum/times) + "ms"
    });

    return tries;
}

The one-liner

Now convert it to a one-liner and write on an openaf-console, for example:

> ow.loadFormat();
> var tl = (host, port, times) => { times = _$(times).isNumber().default(3);var tries = [], sum = 0, max = 0; for(var ii = 0; ii < times; ii++) { tries.push({ sample: ii+1, latency: ow.format.testPortLatency(host, port)});}; tries.forEach((v) => { sum += v.latency; max = (v.latency > max) ? v.latency : max; v.chart = ow.format.string.progress(v.latency, max, 0, 50, "=", " "); }); tries.push({ sample: "avg", latency: Math.floor(sum/times) + "ms" }); return tries; }

And test it:

> table tl("www.yahoo.com", 443);
sample|latency|                      chart                       
------+-------+--------------------------------------------------
1     |63     |==================================================
2     |60     |================================================  
3     |63     |==================================================
avg   |62ms   

Of course, the chart would only help you when there are slight or significant variations in latency between tests:

> table tl("dynamodb.ap-southeast-2.amazonaws.com", 443, 15);
sample|latency|                      chart                       
------+-------+--------------------------------------------------
1     |354    |==================================================
2     |364    |==================================================
3     |357    |================================================= 
4     |351    |================================================  
5     |352    |================================================  
6     |357    |================================================= 
7     |406    |==================================================
8     |358    |============================================      
9     |354    |============================================      
10    |348    |===========================================       
11    |361    |============================================      
12    |358    |============================================      
13    |369    |=============================================     
14    |354    |============================================      
15    |366    |=============================================     
avg   |360ms  

From the command-line

If you want to run it from the command-line:

$ openaf -c 'ow.loadFormat();var tl = (host, port, times) => { times = _$(times).isNumber().default(3);var tries = [], sum = 0, max = 0; for(var ii = 0; ii < times; ii++) { tries.push({ sample: ii+1, latency: ow.format.testPortLatency(host, port)});}; tries.forEach((v) => { sum += v.latency; max = (v.latency > max) ? v.latency : max; v.chart = ow.format.string.progress(v.latency, max, 0, 50, "=", " "); }); tries.push({ sample: "avg", latency: Math.floor(sum/times) + "ms" }); return tries; }; print(printTable(tl(  "www.google.com", 443, 15)));'
sample|latency|                      chart                       
------+-------+--------------------------------------------------
1     |35     |==================================================
2     |30     |===========================================       
3     |30     |===========================================       
4     |26     |=====================================             
5     |30     |===========================================       
6     |29     |=========================================         
7     |29     |=========================================         
8     |28     |========================================          
9     |28     |========================================          
10    |29     |=========================================         
11    |33     |===============================================   
12    |30     |===========================================       
13    |32     |==============================================    
14    |33     |===============================================   
15    |33     |===============================================   
avg   |30ms   

And replace the function arguments on the end of the one-liner.

Checking arguments

In OpenAF whenever you built a simple function, or you have a piece of code on a oJob that expects arguments or any code block you will eventually need to ensure the type of the input variables/arguments and enforce mandatory arguments.

Let's say that you build a function to add two values and return a verbose result. It receives arguments a and b that must be numbers, but if b is not provided it assumes 0. Argument c should be a string and if not provided a default value is provided. All this in plain javascript is:

function sum(a, b, c) {
    if (typeof a != "number") throw "a needs to be a number";
    if (typeof b == "undefined") b = 0;
    if (typeof b != "number") throw "b needs to be a number";
    if (typeof c == "undefined") c = "the result is ";
    if (typeof c != "string") throw "c needs to be a string";

    return c + String(a + b);
}

If you inspect it carefully you need to have the "if" statements in the right order to achieve the intended result. The more arguments you have the harder to read.

OpenAF as a shortcut to help with readability. It's not intended to be like TypeScript but just a small helper for OpenAF scripting. So the previous function would look like this with these OpenAF shortcuts:

function sum(a, b, c) {
    _$(a, "a").isNumber().$_();

    b = _$(b, "b").isNumber().default(0);
    c = _$(c, "c").isString().default("the result is ");

    return c + String(a + b);
}

Each declaration starts always with _$(variable, stringName). The stringName is actually optional if you want to become "more verbose" (you will see how in a second).

The declaration is then followed by conditions like isNumber and isString. You can provide an argument to each of these "conditions" to have more verbose exceptions than "a needs to be a number.".

The declaration ends with default(aValue) or $_(stringName). If no value is provided default will ensure that the variable will always default to some value while $_ doesn't return any value but will throw an exception indicating that the variable value wasn't provided.

So a more verbose type checking could be:

function sum(a, b, c) {
    _$(a).isNumber("The argument 'a' needs to be a number.').$_('The argument 'a' is mandatory for the sum function.');

    b = _$(b).isNumber("The argument 'b', if defined, must be a number.").default(0);
    c = _$(c).isString("The argument 'c', if defined, must be a string.").default("the result is ");

    return c + String(a + b);
}

You can see the list of all available conditions by executing desc _$() on an openaf-console.

Executing a remote SSH command in background

Whenever you execute a SSH command on a remote host it will wait until that process ends. Even with Unix targets if you add a "&" on the end of the command and/or add "nohup" on the beginning you will still wait. Same obviously applies in OpenAF:

var aTarget = { 
    host : "some.host",
    port : 22,
    login: "me",
    pass : "change"
    //id: "~/.ssh/mykey_rsa"
}

var res = $ssh(aTarget)
.sh("nohup wget https://download.dokuwiki.org/src/dokuwiki/dokuwiki-stable.tgz &")
.get();

// res will have all the output from wget as if '&' was never there in the first place

The reason is related to how SSH works. While any of the stdout, stderr or stdin are "open" the execution command won't end. So a simple way to solve it is to change the execution command to:

var res = $ssh(aTarget)
.sh("nohup wget https://download.dokuwiki.org/src/dokuwiki/dokuwiki-stable.tgz 2> wget.out > /dev/null < /dev/null &")
.get();

// res.stdout and res.stderr now don't shown anything since the process was
// left running on the server

Later you can check the execution command output:

var res = $ssh(aTarget).sh("cat wget.out").get(0).stdout;

What about promises?

Yes, you could code something like:

var res;
var promise = $do(() => {
    res = $ssh(aTarget)
          .sh("wget https://download.dokuwiki.org/src/dokuwiki/dokuwiki-stable.tgz")
          .get();
});

But if you are running on a cloud function (e.g. AWS Lambda) payed by execution time you don't want to wait for the promise to be fullfilled (specially if the download will or might take some time).

How to connect directly to a local JVM via JMX

OpenAF's JMX client enables the creation of JMX "client" scripts. To connect the "client" to the target JVM you need an JMX URL.

But if the JVMs are local to the same host you can request a "special" URL that will connect to them without the need to open any remote JMX ports and change the target Java startup arguments.

To do this first list the "recognized" running JVMs on your host:

> plugin("JMX");
> af.fromJavaMap(jmx.getLocals());
+---------+-----+-------+-----------------------------------+
| Locals: | [0] | name: | /openaf/openaf.jar --console      |
|         |     |   id: | 24720                             |
+---------+-----+-------+-----------------------------------+
|         | [1] | name: | /myApp/myapp.jar                  |
|         |     |   id: | 15004                             |
+---------+-----+-------+-----------------------------------+

Using the id number, for the local Java JVM you want to target, now call attach2Local like this, creating a new JMX object instance:

> var localJMX = new JMX(String(jmx.attach2Local(15004).get("URL")));

If successfull you will now be able to use the localJMX JMX client object:

> var obj = jmx.getObject("java.lang:type=Runtime");
> String(obj.get("ClassPath"));
"/myApp/myapp.jar"

How to access remote JMX data

JMX, the Java Management Extensions, allows you to monitor/manage any Java based process. Since OpenAF is also a Java process it can be used as a "client" to check other java processes (including other OpenAF processes).

To set it up you will need to enable remote JMX access on the target Java process. Typically:

java -D"com.sun.management.jmxremote.port=9999" -D"com.sun.management.jmxremote.authenticate=false" -D"com.sun.management.jmxremote.ssl=false" -jar myApp.jar

Note: is not recommended to disable authentication and SSL if you weren't on a controlled environment.

Now, in OpenAF, you can check remotely some JMX beans like the java.lang.Runtime and java.lang.Memory.

Let's start by creating a JMX object instance:

> plugin("JMX");
> var jmx = new JMX("service:jmx:rmi:///jndi/rmi://127.0.0.1:9999/jmxrmi");

Note: It's also possible to attach directly to running JVMs on the same host without having to change their startup arguments using JMX.attach2Local.

Checking existing attributes

Let's check the available attributes for java.lang.Memory:

> var obj = jmx.getObject("java.lang:type=Runtime");
> af.fromJavaMap(obj.getAttributes());
+-------------+------+----------------+------------------+------------------+
| attributes: |  [0] |      openType: |       className: | java.lang.String |
|             |      |                |     description: | java.lang.String |
|             |      |                |        typeName: | java.lang.String |
+-------------+------+----------------+------------------+------------------+
|             |      | attributeType: | java.lang.String |                  |
|             |      |       isWrite: | false            |                  |
|             |      |        isRead: | true             |                  |
|             |      |            is: | false            |                  |
|             |      |          name: | Name             |                  |
|             |      |   description: | Name             |                  |
+-------------+------+----------------+------------------+------------------+
|             |  [1] |      openType: |       className: | java.lang.String |
|             |      |                |     description: | java.lang.String |
|             |      |                |        typeName: | java.lang.String |
+-------------+------+----------------+------------------+------------------+
|             |      | attributeType: | java.lang.String |                  |
|             |      |       isWrite: | false            |                  |
|             |      |        isRead: | true             |                  |
|             |      |            is: | false            |                  |
|             |      |          name: | ClassPath        |                  |
|             |      |   description: | ClassPath        |                  |
+-------------+------+----------------+------------------+------------------+
...

Retrieving string values

So we now know there are several attributes and their corresponding type: "Name", "ClassPath", etc... To retrieve the current value of each just execute:

> String(obj.get("Name"))
"12345@openaf"
> String(obj.get("ClassPath"))
"/openaf/openaf.jar"

Retrieving composite values

But not all attributes have simple type values like String. If you check java.lang.Memory you will find that the specific values are wrapped on a Java class:

> var obj = jmx.getObject("java.lang:type=Memory");
> af.fromJavaMap(obj.getAttributes()).attributes[1].name
"HeapMemoryUsage"
> af.fromJavaMap(obj.getAttributes()).attributes[1].attributeType
"javax.management.openmbean.CompositeData"
> af.fromJavaMap(obj.getAttributes()).attributes[1].openType.typeName
"java.lang.management.MemoryUsage"

To retrieve composite values just get the first composite value and use the corresponding class methods to retrieve "sub-values":

> Number(o.get("HeapMemoryUsage").get("max"))
1821376512
> Number(o.get("HeapMemoryUsage").get("init"))
130023424
> Number(o.get("HeapMemoryUsage").get("used"))
13376936
> Number(o.get("HeapMemoryUsage").get("committed"))
124780544

Note: The class and type should be available on the OpenAF's classpath if not standard.

Setting a proxy

Usually the default java proxy settings cover pretty much all cases. But there are a few cases where it would be helpful to programatically set a proxy.

Let's start with the basic proxy settings.

Setting a HTTP/HTTPs proxy

To set a HTTP/HTTPs proxy you will need the proxy host and the proxy port:

ow.loadObj();
ow.obj.setHTTPProxy("a.host", 1234);
ow.obj.setHTTPSProxy("a.host", 1234)

After this all HTTP/HTTPs communications in OpenAF will use the provided proxy.

Keep in mind that any external processes/scripts executed from OpenAF (e.g. executing sh("someCommand")) won't inherit these proxy settings.

Setting a SOCKS proxy

Nevertheless the most interesting case is connecting to a SOCKS proxy. If you are on machine A but you need to access resources through machine B running OpenAF on machine A like it was running on machine B you can establish a simple dynamic port forwarding with SSH. To establish this simple execute on machine A:

ssh -D 12345 myuser@machine.b

Afterwards, in OpenAF, just execute:

ow.loadObj();
ow.obj.setSOCKSProxy("127.0.0.1", 12345);

Now all network connections, from OpenAF, will go through th socks proxy actually feeling like you are executing OpenAF on machine B.

Note: if a SOCKS proxy user & password is needed you can add it as extra parameters.

Keep in mind that any external processes/scripts executed from OpenAF (e.g. executing sh("someCommand")) won't inherit these proxy settings.

Setting a FTP proxy

Although less used you can also set a proxy for FTP connections:

ow.loadObj();
ow.obj.setFTPProxy("a.host", 1234);

Encrypt/Decrypt with public/private keys

There are built-in functions to easily encrypt/decrypt text using a single password key (using functions af.encrypt and af.decrypt). But if you need better security you might need to encrypt/decrypt using a public and a private keys.

This can be achieved using the ow.java.cipher. It enables the generation or loading of existing public and private keys and use them to encrypt/decrypt text.

To create an object instance:

ow.loadJava();
var cipher = new ow.java.cipher();

Generating public/private keys

You can generate a public and a private key with a specific key length (by default 2048 bits). You can generate for 3072, 4096, 7680 and 15360 bits (the bigger, the longer to generate):

var keys = cipher.genKeyPair(4096);

To save the private and public keys in files:

cipher.saveKey2File("key.pub", keys.publicKey, false);
cipher.saveKey2File("key.priv", keys.privateKey, true);

To load the keys from the files:

var kpPUB  = cipher.readKey4File("key.pub", false);
var kpPRIV = cipher.readKey4File("key.priv", true);

Encrypting/Decrypting text

Now that you have a pair of public and private keys you can easily encrypt/decrypt text:

var cipheredText = cipher.encrypt2Text(plainText, kpPUB);

var plainText = cipher.decrypt4Text(cipheredText, kpPRIV);

The functions with the "Text" suffix automatically convert to a Base64 text for easier transport. But there are also functions for encrypting/decrypting to/from arrays of bytes and streams.

Dynamically adding a custom JDBC driver

The latest OpenAF versions enable the dynamic loading of custom JDBC drivers.

Let's take, for example, the CSV JDBC driver. After downloading the jar file to an empty folder let's also create a sample CSV file (test.csv):

"key";"value"
1; "Item 1"
2; "Item 2"
3; "Item 3"

Then, on an OpenAF script or console execute:

> loadExternalJars(".")
> var db = new DB("org.relique.jdbc.csv.CsvDriver", "jdbc:relique:csv:.?separator=;&fileExtension=.csv", "sa", "sa");
> sql db select * from test
key|  value
---+---------
1  | "Item 1"
2  | "Item 2"
3  | "Item 3"
[#3 rows]

So, what happened:

• The loadExternalJars function tries to dynamically load all .jar files on the path provider (using the OpenAF's custom class loader).
• When creating the DB object instance, OpenAf will try to find the provided JDBC class driver and will load it through an internal "proxy", if needed.
• The returned DB object instance will work as the H2, PostgreSQL, etc... drivers (within the limits of the JDBC driver).

Check current OpenAF Java memory

Keeping in mind that OpenAF runs on a JVM so you might want to keep an eye on the current heap size.

Checking current memory size

You can do this easily with the included ow.java library. Just load it:

ow.loadJava();

And call ow.java.getMemory:

> ow.java.getMemory()
+--------+------------+
|   max: | 2048917504 |
| total: | 46137344   |
|  used: | 12635344   |
|  free: | 33502000   |
+--------+------------+

If you want to convert it to the corresponding byte abbrevation just use the first boolean argument:

> ow.java.getMemory(true)
+--------+---------+
|   max: | 1.91 GB |
| total: | 44.0 MB |
|  used: | 12.4 MB |
|  free: | 31.6 MB |
+--------+---------+

Forcing the Java garbage collector

If you fill the heap size temporarially and, afterwards, no longer need the assigned memory Java will eventualy free up once it's corresponding Java garbabe collector runs:

> var list = io.listFiles("c:/windows/system32");
> ow.java.getMemory(true)
+--------+---------+
|   max: | 1.91 GB |
| total: | 44.0 MB |
|  used: | 16.7 MB |
|  free: | 27.3 MB |
+--------+---------+
> ow.java.gc();
> ow.java.getMemory(true)
+--------+---------+
|   max: | 1.91 GB |
| total: | 44.0 MB |
|  used: | 12.1 MB |
|  free: | 31.9 MB |
+--------+---------+

Using the function parallel4Array

OpenAF allows you to code in javascript while using Java functionality underneath. But there are obvious differences between the two languages.

Example

Let's take a simple example for parallel processing. First you create an array with the information of several files:

var files = listFilesRecursive("/usr");

Running it sequentially

Next you want to capture the result of executing a "not so fast" operation of performing an external shell command and capture the result:

var res = [];
for(var i = 0; i < files.length; i++) {
    res.push(sh("ls " + files[i]));
}

Using this simple for cycle the time elapsed, on a specific environment, was 1 second and 686 ms.

Running it in parallel

But this was a rather sequential processing in nature. OpenAF includes a function, parallel4Array, that wlil automate much of the work of using Java threads to perform the same exact functionality:

var res = parallel4Array(files, (v) => {
    return sh("ls " + v.filename);
})

Changing the for cycle by parallel4Array, and testing on the same environment, the result was a exactly equal res array in around 842ms. So what happenend?

Breaking it down:

• The number of cpu cores was detected.
• The files array was split evenly between the number of cpu cores detected.
• Each "splitted array" was executed sequentially on separate executing thread (Java thread).
• OpenAF kept an eye on the operating system reported machine load introducing delays when cpu cores got "overloaded".

This last step is helpful when executing on machines with a large number of cpu cores.

"Can I tune it?"

Yes, of course. A third optional argument of parallel4Array let's you override the cpu detection and "machine load" control. A fourth argument of parallel4Array gives you direct access to the Java threads objects, unique ids, etc… Check the help information in openaf-console: "help parallel4Array".

For advanced cases you can use the function parallelArray which let's you control how the end result will be built letting you provide a function to "unify" all the threads results.

There is even the function parallel that will launch multiple threads to execute the same provided function without the need to provide an array.

nAttrMon multiple plugs in one file

When adding any nAttrMon plug (either inputs, validations or outputs), the advisable setup is to have a structured set of files on a folder hierarchy. In this structure each file is usually a single plug definition.

Nevertheless once you get more advanced setups there are cases where you might want to have more than one plug per file taking advantage of the plug "ignore"/"disabling" mechanisms.

Let's take a simple example of a nAttrMon input plug:

input:
  name         : Database queries
  cron         : "*/10 * * * *"
  waitForFinish: true
  onlyOnEvent  : true
  execFrom     : nInput_DB
  execArgs     :
     key: MYDB
     sqls:
        Database/Status XYZ: >
           SELECT control "Control", value "Value"
           FROM status
           WHERE control in ('X', 'Y', 'Z')

        Database/Status ABC: >
           SELECT control "Control", value "Value"
           FROM status
           WHERE control in ('A', 'B', 'C')

In this example both queries will execute every 10 minutes. But you might want different cron schedules temporarily to find some issue on ABC controls.

Usually you would create two files, comment the ABC query on the first and just have the ABC on the second. But now if you have to quickly ignore all database queries you will have to remember the extra temporary plug you created before. Could there be a better alternative?

The answer is yes. You can keep both SQLs on the same file and temporarily have different cron settings:

input:
  #--------------------------------
  - name         : Database queries
    cron         : "*/10 * * * *"
    waitForFinish: true
    onlyOnEvent  : true
    execFrom     : nInput_DB
    execArgs     :
       key: MYDB
       sqls:
          Database/Status XYZ: >
             SELECT control "Control", value "Value"
             FROM status
             WHERE control in ('X', 'Y', 'Z')

          #Database/Status ABC: >
          #   SELECT control "Control", value "Value"
          #   FROM status
          #   WHERE control in ('A', 'B', 'C')

  #--------------------------------------------
  - name         : Database queries (temporary)
    cron         : "*/1 * * * *"
    waitForFinish: true
    onlyOnEvent  : true
    execFrom     : nInput_DB
    execArgs     :
       key: MYDB
       sqls:
          Database/Status ABC : >
             SELECT control "Control", value "Value"
             FROM status
             WHERE control in ('A', 'B', 'C')             

What's the difference? Instead of defining a map, if you define an array nAttrMon it will consider two plugs instead of one. So now you can control the temporary different cron setting by just commenting lines while keeping everything in the same file.

Note: always have different plug names, otherwise the last will overwrite the previous.