Search This Blog

Showing posts with label medium. Show all posts
Showing posts with label medium. Show all posts

Sunday, January 26, 2020

Using arrays with parallel

OpenAF is a mix of Javascript and Java, but "pure" javascript isn't "thread-safe" in the Java world. Nevertheless being able to use Java threads with Javascript is very useful starting with performance.

One common pitfall is not paying attention to what is "thread-safe" and what isn't. Meaning, what is "aware" that it can be running on a multi-thread environment where several threads might be competing for access to a resource and what isn't.

Let's see a pratical example of a common pitfall with arrays.

Example of what should NOT be done

In this example we will create a simple array and push new elements in parallel. So we would expect that it would have the same number of elements always:

var targetArray = [];
// making up a source array with 10 elements to push to targetArray
var sourceArray = repeat(9, '-').split(/-/);
parallel4Array(source, v => {
    targetArray.push(v);
    return v;
});

print(sourceArray.length);  // 10
print(targetArray.length);  // 10

Great, so for a source array and target array are equal. It works, right? Let's increase the "thread competition" to 10000 and compare again:

var targetArray = [];
var sourceArray = repeat(9999, '-').split(/-/);
parallel4Array(source, v => {
    targetArray.push(v);
    return v;
});

print(sourceArray.length);  // 10000
print(targetArray.length);  // 9961

There are no longer equal. And if you run it again and increase the number of elements on the source array the difference will increase.

This is what happens when you forget that a javascript array is not thread-safe (and it's actually a good thing because being thread-safe is usually slower when you are not using threads).

Examples of what should be done

Using sync

The first immediate way to solve this is to ensure that just one thread will access the targetArray at a time. You can do this in OpenAF with the sync function:

var targetArray = [];
var sourceArray = repeat(9999, '-').split(/-/);
parallel4Array(source, v => {
    sync(() => {
        targetArray.push(v);
    }, targetArray);
    return v;
});

print(sourceArray.length);  // 10000
print(targetArray.length);  // 10000

The sync function will use Java underneath (synchronize) to ensure that only one thread at a time accessing targetArray will execute the provided function.

So the problem is solved, right? Well if you measure the performance of using sync and not using sync you will notice that it can be up to twice as bad.

Why? Because when one thread is inserting a value into targetArray all other threads have to stop and wait. That will slow down everything and probably make it not so much as effective as running it sequentially (depending on the processing outside the sync function).

Using syncArray

On the OpenAF library ow.obj there is a wrapper for you to use a "thread-safe" java array: ow.obj.syncArray

ow.loadObj();
var targetArray = new ow.obj.syncArray();
var sourceArray = repeat(99999, '-').split(/-/);
parallel4Array(source, v => {
    targetArray.add(v);
    return v;
});

print(sourceArray.length);            // 100000
print(targetArray.toArray().length);  // 100000

The java array version is optimized to be faster in these conditions. So the bigger is the sourceArray the bigger the benefits with ow.obj.syncArray.

The ow.obj.syncArray has more methods that you can explore from the OpenAF's help including: addAll, clear, get, indexOf, length, remove and even getJavaObject that will let you iteract with the original java object directly.

Comparison

Changing the above examples to perform something "harder" for each array element like calculating the Math.sin of each value and then comparing the performance you would get something similar to:

Strategy Source size Target size Average time
Parallel 100000 <100000 2.11s
Sync 100000 100000 2.44s
ow.obj.syncArray 100000 100000 2.06s

But what's the time if it's done sequentially? In this simple example: a lot better (~1.1s). Keep in mind that you only gain a performance advantage when the time spent dealing with threads and concurrent access is lot lower relatively to the time spent processing each array element.

So, in conclusion, there is no right or wrong answer. You need to test to get the best for your case.

Sunday, January 12, 2020

Using channel peers

If you use OpenAF channels there are some cases where you would like to connect them across OpenAF scripts.

To achieve this you can use the $ch.expose and the $ch.createRemote functions that allow you to expose an internal OpenAF to be accessible by other OpenAF scripts remotely.

But if the original OpenAF script which exposed the channel "dies" all the others will no longer be able to access the corresponding data. This might not be the desired scenario in some cases.

Peering

But making a mix of expose and createRemote functions between several OpenAF scripts you can achieve what is provided by the \$ch.peer function.

The \$ch.peer will expose an OpenAF channel and exchange sync data with a set of "peer" OpenAF scripts that just have to execute the same similar command:

$ch("myChannel").peer(12340, 
                      "/myURI", 
                      [ "http://script01:12340/myURI", 
                        "http://script02:12341/myURI", 
                        "http://script03:12343/myURI"]);

The signature is:

$ch.peer(aLocalPortOrServer, aPath, aRemoteURL, aAuthFunc, aUnAuthFunc)

The parameters are similar to the $ch.expose function:

Parameter Type Description
aLocalPortOrServer Object/Number The local port or a HTTPServer object to hold the HTTP/HTTPs server.
aPath String The URI where the channel interaction will be performed.
aRemoteURL Array An array of peer URLs
aAuthFunc Function This function will be called with user and password. If returns true the authentication is successful.
aUnAuthFunc Function This function will be called with the http server and the http request.

If you call it, after the first time, with a different array of aRemoteURL it will replace the existing. If the array references itself it will be ignored.

Unpeering

To remove a peering you can call:

$ch.unpeer(aRemoteURL)

Testing

Setting up data channels in each peer

First script (on host h1):

$ch("data").peer(9080, "/data", [ "http://h1:9080/data", "http://h1:9090/data", "http://h2:9080/data" ]);

Second script (on host h1):

$ch("data").peer(9090, "/data", [ "http://h1:9080/data", "http://h1:9090/data", "http://h2:9080/data" ]);

Third script (on host h2):

$ch("data").peer(9080, "/data", [ "http://h1:9080/data", "http://h1:9090/data", "http://h2:9080/data" ]);

Changing and retrieving data

  1. On host h2:
> $ch("data").size()
0
  1. On host h1:
> $ch("data").size()
0
> $ch("data").setAll(["canonicalPath"], io.listFiles(".").files);
> $ch("data").size()
25
  1. On host h2:
> $ch("data").size()
25

Sunday, December 8, 2019

Logging to log files

There are several benefits of using the OpenAF's log* functions. One of them is to quickly have a logs folder with automated housekeeping with just a couple of lines of code.

Let's start with a simple script with log functions:

log("Init");

try {
    log("Writing file...");
    io.writeFileString("test.txt", "test");
    log("File test.txt written.");
} catch(e) {
    logErr("Couldn't write file test.txt. Error: " + String(e));
}

log("Done");

Executing you will get the expected standard output:

2019-12-08 09:19:00.540 | INFO | Init
2019-12-08 09:19:00.750 | INFO | Writing file...
2019-12-08 09:19:00.780 | INFO | File test.txt written.
2019-12-08 09:19:00.785 | INFO | Done

Log to a folder

Let's log to a folder:

ow.loadCh();
// Setting default logging to the logs folder
ow.ch.utils.setLogToFile({ logFolder: "logs" });
// Ensuring logs are written synchronously
setLog({ async: true });

log("Init");

try {
    log("Writing file...");
    io.writeFileString("test.txt", "test");
    log("File test.txt written.");
} catch(e) {
    logErr("Couldn't write file test.txt. Error: " + String(e));
}

log("Done");

Executing again the result seems similar but a new "logs" folder was created with a daily log file containing all the log messages. When creating a new daily log file it will automatically gzip all old daily log files.

But that's the standard behaviour that can, of course, be customizable. Let's add housekeeping to delete files older than one week:

ow.loadCh();
ow.ch.utils.setLogToFile({ 
    logFolder: "logs",
    HKhowLongAgoInMinutes: 7200
});
//...

p.s.: you can specify a different folder to hold the previous compressed log files using _backupFolder. You can also control if old log files should be compressed or not with dontCompress._

Logging to files by hour

The default is logging to daily files but you can customize to any date time grouping. To group files by hour can simply add a specific fileDateFormat. Additionally let's change the log filenames generated using filenameTemplate (that uses handlebars notation).

Since we are changing the log filenames for the housekeeping process to be able to identify the new log filenames you need to provide a HKRegExPattern regular expression pattern also.

ow.loadCh();
ow.ch.utils.setLogToFile({ 
    logFolder: "logs",
    filenameTemplate: "hourly-logs-{{timedate}}.log",
    fileDateFormat: "yyyy-MM-dd-HH",
    HKhowLongAgoInMinutes: 7200,
    HKRegExPattern: "hourly-log-\\d{4}-\\d{2}-\\d{2}-\\d{2]\\.log"
});
//...

Logging as CSV files

To log into CSV files you just need to change the filenameTemplate and the lineTemplate:

ow.loadCh();
ow.ch.utils.setLogToFile({ 
    logFolder: "logs",
    filenameTemplate: "log-{{timedate}}.csv",
    lineTemplate: "\"{{timedate}}\";\"{{type}}\";\"{{{message}}}\"\n",
    HKhowLongAgoInMinutes: 7200
});
//...

Logging as NDJSON files

To log as NDJSON:

ow.loadCh();
ow.ch.utils.setLogToFile({ 
    logFolder: "logs",
    filenameTemplate: "log-{{timedate}}.csv",
    lineTemplate: "{d:\"{{timedate}}\",t:\"{{type}}\",m:\"{{{message}}}\"}\n",
    HKhowLongAgoInMinutes: 7200
});

Keeping log entries only on files

You can also specify that log entries should only be recorded in the log files and not shown on the standard output console:

ow.loadCh();
ow.ch.utils.setLogToFile({
    logFolder: "logs",
    setLogOff: true
});

And more...

You can check all the available options by executing:

> help ow.ch.utils.setLogToFile

on an openaf-console prompt.

Thursday, November 7, 2019

Handling oJob deps failure

When executing an oJob each job can depend on the successfull execution of other jobs. That means that if a depending job fails the current job execution will fail or stall (if it's an ojob.sequential = true).

Dependency timeout

If ojob.sequential != true any failed job dependency will keep the oJob waiting for a successfull execution of the dependent job. You can avoid that using a dependency timeout:

ojob:
  # timeout of 2500ms
  depsTimeout: 2500 

In this case whenever a job doesn't execute because another failed it will wait just the amount of specific ms for another successfull execution. Otherwise it will terminate with an error indicating the a dependency timeout has occurred.

Individual dependency

You can also execute code to decide what should be done if a dependent job fails when ojob.sequential != true:

todo:
  - Init
  - Test 1
  - Test 2

jobs:
  #-----------
  - name: Init
    exec: |
      // Initialize error flag
      global.inError = false;

  #-------------
  - name: Test 1
    exec: |
      print("Test 1");
      // Throw an error if args.error is defined
      if (args.error) throw("Problem with test 1");

  #-------------
  - name: Test 2
    deps:
      - name  : Init
      - name  : Test 1
        onFail: |
          // if the dependent job fails, 
          // change the global error flag and proceed
          global.inError = true;
          return true;
    exec: |
      if (!global.inError)
        print("Test 2");
      else
        printErr("Can not execute Test 2");

In this example there are three jobs:

  • Init - initializes a global inError flag.
  • Test 1 - generates an error if the error argument is defined during this oJob execution.
  • Test 2 - depends on the Init and Test 1 jobs. If Test 1 job fails it sets the global inError flag and proceeds with the execution that checks that same flag.

The onFail entry on the list of dependencies for job Test 2 is actually the code of a function that receives three parameters:

  • args - the current job arguments
  • job - the current job definition
  • id - the current job execution id

If this onFail functions returns true the job execution will proceed. If it returns false the job execution stalls as it does by default.

Wednesday, October 30, 2019

Modifying XML using E4X

OpenAf uses the Java Mozilla Rhino javascript engine and one of the features that, despite being obsolete, has still been kept is E4X.

E4X simply allows for easy interaction and also modification of XML content in Javascript.

Let's start by the XML example used in "Accessing XML using E4X":

<shopcart>
  <!-- Client id -->
  <client type="personal" id="12345">
    <name>Joe Smith</name>
  </client>

  <!-- Store id -->
  <store>
    <id>12345</id>
    <type>diy</type>
  </store>

  <!-- Items on the shopcart -->
  <items>
    <item id="123">
       <qty>1</qty>
       <name>Item 1</name>
    </item>
    <item id="456">
       <qty>10</qty>
       <name>Item 2</name>
    </item>
  </items>
</shopcart>

Then you can load it to a xml javascript variable through various ways:

var xml = io.readFileXML("myxml.xml");
var xml = new XMLList(myXMLString);
var xml = <shopcart><!-- Client id --><client type="personal" id="12345">...

Modifying data using the xml variable in javascript

Any element that you can access directly, you can change it:

> xml.client.name.toString()
"Joe Smith"
> xml.client.name = "Scott Tiger"
"Scott Tiger"
> xml.client.name.toString()
"Scott Tiger"

Changing attributes:

> xml.client.@id.toString()
"12345"
> xml.client.@id = "67890"
"67890"
> xml.client.@id.toString()
"67890"

And any comments are kept:

> xml.toString()
<shopcart>
  <!-- Client id -->
  <client id="12345" type="personal">
    <name>Scott Tiger</name>
  </client>
  <!-- Store id -->
  <store>
    <id>12345</id>
...    

Adding and removing children elements

To add you can simply think like an array:

> xml.items.item[2] = <item id="789"><qty>2</qty><name>Item 3</name></item>
> xml.items.item.length()
3

And to remove a children element you can use delete:

> delete xml.items.item[2]
> xml.items.item.length()
2

There are more methods, listed on the end of this section, to add or remove elements like appendChild(child), prependChild(child), insertChildAfter(), insertChildBefore()_

XML Object methods

Method Description
appendChild(child) Adds the child XML object at the end of the corresponding XML element children.
prependChild(child) Inserts a child XML object prior to the beginning of the corresponding XML element children.
insertChildAfter(childRef, childNew) Inserts childNew after childRef.
insertChildBefore(childRef, childNew) Inserts childNew before childRef.
copy() Returns a deep copy of a XML object starting on the corresponding XML element children (without the parent).

Accessing XML using E4X

OpenAf uses the Java Mozilla Rhino javascript engine and one of the features that, despite being obsolete, has still been kept is E4X.

E4X simply allows for easy interaction of XML content in Javascript.

Let's start by the following XML example:

<shopcart>
  <!-- Client id -->
  <client type="personal" id="12345">
    <name>Joe Smith</name>
  </client>

  <!-- Store id -->
  <store>
    <id>12345</id>
    <type>diy</type>
  </store>

  <!-- Items on the shopcart -->
  <items>
    <item id="123">
       <qty>1</qty>
       <name>Item 1</name>
    </item>
    <item id="456">
       <qty>10</qty>
       <name>Item 2</name>
    </item>
  </items>
</shopcart>

Loading XML to OpenAF

You can load this xml from a file:

var xml = io.readFileXML("myxml.xml");

From a string:

var xml = new XMLList(myXMLString);

Or directly:

var xml = <shopcart><!-- Client id --><client type="personal" id="12345">...

All of them will result in a xml javascript variable of type "xml".

Accessing data using the xml variable in javascript

To access any element simply think like you would access it if it was a JSON map:

> xml.client.name.toString()
"Joe Smith"
> xml.store.id.toString()
"12345"

Each element has several methods from which we are using the toString method. There is a list of methods on the end of this section.

For attributes it's the same but add the prefix "@":

> xml.client.@type.toString()
"personal"

For multiple items you can think of it now as an array:

> xml.items.item[0].@id.toString()
123
> xml.items.item[1].qty.toString()
10

But this, is also valid (think of it as the first "store" tag element):

> xml.store[0].id.toString()
"12345"

Accessing all children nodes

> xml.children()[1].name.toString()
"Joe Smith"

Why is the children element on position 1 and not in 0? Because comments are also nodes:

> xml.children()[0].toString()
"<!-- Client id -->"

Using for cycles

Don't forget that you are in javascript, so you can do for cycles:

var totalQty = 0;
for(var i in xml.items.item) {
    totalQty += Number(xml.items.item[i].qty);
}
// 11

You can also search all descendants in a for cycle:

for(var i in xml..name) {
    print(xml..name[i].toString());
}
// Joe Smith
// Item 1
// Item 2

Using namespaces

You can also use namespaces:

var xml = <links><a xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="http://www.google.com">Test link</a></links>;

var ns = new Namespace("http://www.w3.org/1999/xlink");
var url = xml..a;
var link = url.@ns::href;
// http://www.google.com

XML Object methods

Method Description
attribute(attributeName) Returns the attribute attributeName from the corresponding XML element node.
attributes() Returns a list of the attributes associated with the corresponding XML element node.
child(propertyName) Returns the children elements named propertyName associated with the corresponding XML element node.
children() Returns all children elements associated with the corresponding XML element node.
descendants(name) Equivalent to "xml..name".
elements(name) Equivalent to children() if no name is provided or equivalent to child if a name is provided.
parent() Returns the parent element associated with the corresponding XML element node.
comments() Returns the children elements of type 'comment' associated with the corresponding XML element node.
text() Returns the text component of the corresponding XML element node.
name() Returns a map with the name and local name associated with the corresponding XML element node.
localName() Returns the local name associated with the corresponding XML element node.
namespace(prefix) Returns the namespace for the provided prefix of the namespace associated with the corresponding XML element node.
namespaceDeclarations() Returns an array of namespaces associated with the corresponding XML element node.
childIndex() Returns the index number on the childrens list of the corresponding XML element node.
length() Returns the size of the children elements of the corresponding XML element node.
nodeKind() Returns if the corresponding XML element node is a element of a comment.
toString() Returns the string of the current XML element node.
toXMLString() Returns the XML encoded string of the current XML element node.

Friday, October 18, 2019

How to run OpenAF code from shell script

To run an OpenAF script you just need to execute: 'openaf -f myScript.js'. But what if you wanted to include some OpenAF code without having to create a separate file. There are several ways to achived this with OpenAF.

Inline code

The simplest way it's using the '-c' option:

$ openaf -c 'var res = $rest().get("https://uinames.com/api"); print(printMap(res, void 0, "plain", true))'
+----------+---------------+
|    name: | John          |
| surname: | Lawrence      |
|  gender: | male          |
|  region: | United States |
+----------+---------------+

Incorporating into a shell script:

#!/bin/sh

name=$(openaf -c 'print($rest().get("https://uinames.com/api").name)')
echo Hi $name

Running:

$ sh sayHi.sh
Hi Andrea

Multiline code

For simple code inline code works well but for multi line code it may become hard to read. Another option is using the '-p' option that will receive input from stdin.

#!/bin/bash

output=$(openaf -i script -p << __endScript

  // OpenAF javascript code
  var res = \$rest().get("https://uinames.com/api/?ext");
  print(res.name + " " + res.surname + ";" + res.phone);

__endScript
)

# interpreting result in bash
IFS='\;'
read -a strarr <<< "$output"

echo Name : ${strarr[0]}
echo Phone: ${strarr[1]}

And the result of mixing shell script and OpenAF:

Name : Angela Johnston
Phone: (134) 382 2457

Wednesday, October 16, 2019

Simple nAttrMon debugging

One of the common questions when using nAttrMon is: "has my plug even executed?"

There are several approaches to find the answer being, one of the most common ones, just adding a print statement on your nAttrMon plug or turning the specific plug debug flag, if it exists. There also functions to just test specific plugs inside and outside nAttrMon.

Nevertheless, has your nAttrMon's configuration grows it might become difficult to understand: what is running at which moment? what is taking so much CPU? why it was fast and now it's slow?

Almost always the answer relies on what is being executed at each time.

Channel 'ps' and others

Wouldn't you like to just execute a "ps" (similar to the unix 'ps' command) on nAttrMon and just see what's running? Well, you can.

Just add the channels output plug (check on outputs.disabled for *channels.yaml).

Now, on an openaf-console, just execute:

> $ch("ps").createRemote("http://nattrmon:nattrmon@your.server:8090/chs/ps");
> table $ch("ps").getAll();
    name    | type |                uuid                |         start
------------+------+------------------------------------+------------------------
Input Test 1|inputs|1fc61235-97d6-068f-e705-5693712aba75|2019-10-01T09:42:10.027Z

In this example there was only one plug executing but it will return as many as there is currently running. It's useful to understand which plugs take too long to execute. The plugs that you find here for a long period of time probably need to be rethinked: is the plug taking longer than expected? is it ocasional?

There is another channel that comes also to the rescue to answer these questions:

> $ch("plugs").createRemote("http://nattrmon:nattrmon@your.server:8090/chs/plugs");
> $ch("plugs").getAll();
[...]
+-----+----------+--------------------------+----------------+
| [1] |    meta: |                    name: | Input Test 1   |
|     |          |             description: |                |
|     |          |                    type: | inputs         |
|     |          |                category: | uncategorized  |
|     |          |                    cron: | */10 * * * * * |
|     |          |            timeInterval: | -1             |
|     |          |           waitForFinish: | true           |
|     |          |             onlyOnEvent: | true           |
+-----+----------+--------------------------+----------------+
|     |    args: |            {}            | -              |
+-----+----------+--------------------------+----------------+
|     |    last: | 2019-10-01T09:49:02.521Z |                |
|     | created: | 2019-10-01T09:46:25.237Z |                |
+-----+----------+-------------------------------------------+
|     |   stats: |        lastExecTimeInMs: | 2501           |
|     |          |         avgExecTimeInMs: | 2500.5         |
|     |          |           numberOfExecs: | 16             |
|     |          |    numberOfExecsInError: | 0              |
|     |          |         numberOfRunning: | 1              |
+-----+----------+--------------------------+----------------+
[...]

Note: if you have the http output activated you can also see this info in the "Plugs" tab on your browser

This information allows you to see how the plug is registered, when was the "last" time it executed, how much time it took (yes, I added a sleep on the dummy test input), the average execution time, how many times it executed without and with errors and how many of them are currently being executed.

Debug flag

There is also another way: if you edit or create a nattrmon.yaml configuration file (based on the nattrmon.yaml.sample) there are three options that might help you while debugging:

LOGCONSOLE: true
LOG_ASYNC: false
DEBUG: true

Note: do uncomment them if they are commented. And don't forget to comment again after debugging.

The first flag, LOGCONSOLE, will not create log files and just print the log to the stdout. Of course this isn't mandatory but you probably don't want to fill up your logs with debug information.

The second flag, LOG_ASYNC, determines if nAttrMon's logs should be "written when possible without stopping nAttrMon" (option true, the default) OR if nAttrMon's logs should be "written immediatelly stopping nAttrMon if needed".

The third flag, DEBUG, will turn on all the logging debug messages. It will show you a lot more information, including which plug is being executed.

2019-10-01 09:57:18.344 | INFO | DEBUG | nAttrMon monitor plug
2019-10-01 09:57:18.367 | INFO | DEBUG | Added plug system monitor
2019-10-01 09:57:18.377 | INFO | DEBUG | Starting at fixed rate for system monitor - 30000
2019-10-01 09:57:18.393 | INFO | DEBUG | Creating a thread for system monitor with uuid = ff7c708c-ba70-4371-b764-96f2d1c44eed
2019-10-01 09:57:18.408 | INFO | DEBUG | nAttrMon start load plugs
2019-10-01 09:57:18.417 | INFO | DEBUG | Ignore list: []
2019-10-01 09:57:18.857 | INFO | Loading inputs: /openaf/nAttrMon/config/inputs/01.test.yaml
2019-10-01 09:57:18.870 | INFO | DEBUG | Added plug Input Test 1
2019-10-01 09:57:18.875 | INFO | DEBUG | nAttrMon exec input plugs
2019-10-01 09:57:18.904 | INFO | DEBUG | nAttrMon exec output plugs
2019-10-01 09:57:18.907 | INFO | DEBUG | nAttrMon exec validation plugs
2019-10-01 09:57:18.910 | INFO | DEBUG | nAttrMon restoring snapshot
2019-10-01 09:57:18.917 | INFO | nAttrMon started.
2019-10-01 09:57:20.010 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:57:30.005 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:57:40.007 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:57:48.395 | INFO | DEBUG | Executing 'system monitor' (ff7c708c-ba70-4371-b764-96f2d1c44eed)
2019-10-01 09:57:50.009 | INFO | DEBUG | Executing 'Input Test 1' (0b7787a8-1135-a823-705b-1574a2eed64e)
2019-10-01 09:58:54.626 | INFO | nAttrMon stopped.

On this example we just added a single input plug "Input Test 1" to be executed every 10 seconds. You can see those executions and also the "system monitor". The "system monitor" is a special plug to ensure nAttrMon is running (if not it will restart itself).

Thursday, October 10, 2019

Executing a remote SSH command in background

Whenever you execute a SSH command on a remote host it will wait until that process ends. Even with Unix targets if you add a "&" on the end of the command and/or add "nohup" on the beginning you will still wait. Same obviously applies in OpenAF:

var aTarget = { 
    host : "some.host",
    port : 22,
    login: "me",
    pass : "change"
    //id: "~/.ssh/mykey_rsa"
}

var res = $ssh(aTarget)
.sh("nohup wget https://download.dokuwiki.org/src/dokuwiki/dokuwiki-stable.tgz &")
.get();

// res will have all the output from wget as if '&' was never there in the first place

The reason is related to how SSH works. While any of the stdout, stderr or stdin are "open" the execution command won't end. So a simple way to solve it is to change the execution command to:

var res = $ssh(aTarget)
.sh("nohup wget https://download.dokuwiki.org/src/dokuwiki/dokuwiki-stable.tgz 2> wget.out > /dev/null < /dev/null &")
.get();

// res.stdout and res.stderr now don't shown anything since the process was
// left running on the server

Later you can check the execution command output:

var res = $ssh(aTarget).sh("cat wget.out").get(0).stdout;

What about promises?

Yes, you could code something like:

var res;
var promise = $do(() => {
    res = $ssh(aTarget)
          .sh("wget https://download.dokuwiki.org/src/dokuwiki/dokuwiki-stable.tgz")
          .get();
});

But if you are running on a cloud function (e.g. AWS Lambda) payed by execution time you don't want to wait for the promise to be fullfilled (specially if the download will or might take some time).

Wednesday, October 9, 2019

How to access remote JMX data

JMX, the Java Management Extensions, allows you to monitor/manage any Java based process. Since OpenAF is also a Java process it can be used as a "client" to check other java processes (including other OpenAF processes).

To set it up you will need to enable remote JMX access on the target Java process. Typically:

java -D"com.sun.management.jmxremote.port=9999" -D"com.sun.management.jmxremote.authenticate=false" -D"com.sun.management.jmxremote.ssl=false" -jar myApp.jar

Note: is not recommended to disable authentication and SSL if you weren't on a controlled environment.

Now, in OpenAF, you can check remotely some JMX beans like the java.lang.Runtime and java.lang.Memory.

Let's start by creating a JMX object instance:

> plugin("JMX");
> var jmx = new JMX("service:jmx:rmi:///jndi/rmi://127.0.0.1:9999/jmxrmi");

Note: It's also possible to attach directly to running JVMs on the same host without having to change their startup arguments using JMX.attach2Local.

Checking existing attributes

Let's check the available attributes for java.lang.Memory:

> var obj = jmx.getObject("java.lang:type=Runtime");
> af.fromJavaMap(obj.getAttributes());
+-------------+------+----------------+------------------+------------------+
| attributes: |  [0] |      openType: |       className: | java.lang.String |
|             |      |                |     description: | java.lang.String |
|             |      |                |        typeName: | java.lang.String |
+-------------+------+----------------+------------------+------------------+
|             |      | attributeType: | java.lang.String |                  |
|             |      |       isWrite: | false            |                  |
|             |      |        isRead: | true             |                  |
|             |      |            is: | false            |                  |
|             |      |          name: | Name             |                  |
|             |      |   description: | Name             |                  |
+-------------+------+----------------+------------------+------------------+
|             |  [1] |      openType: |       className: | java.lang.String |
|             |      |                |     description: | java.lang.String |
|             |      |                |        typeName: | java.lang.String |
+-------------+------+----------------+------------------+------------------+
|             |      | attributeType: | java.lang.String |                  |
|             |      |       isWrite: | false            |                  |
|             |      |        isRead: | true             |                  |
|             |      |            is: | false            |                  |
|             |      |          name: | ClassPath        |                  |
|             |      |   description: | ClassPath        |                  |
+-------------+------+----------------+------------------+------------------+
...

Retrieving string values

So we now know there are several attributes and their corresponding type: "Name", "ClassPath", etc... To retrieve the current value of each just execute:

> String(obj.get("Name"))
"12345@openaf"
> String(obj.get("ClassPath"))
"/openaf/openaf.jar"

Retrieving composite values

But not all attributes have simple type values like String. If you check java.lang.Memory you will find that the specific values are wrapped on a Java class:

> var obj = jmx.getObject("java.lang:type=Memory");
> af.fromJavaMap(obj.getAttributes()).attributes[1].name
"HeapMemoryUsage"
> af.fromJavaMap(obj.getAttributes()).attributes[1].attributeType
"javax.management.openmbean.CompositeData"
> af.fromJavaMap(obj.getAttributes()).attributes[1].openType.typeName
"java.lang.management.MemoryUsage"

To retrieve composite values just get the first composite value and use the corresponding class methods to retrieve "sub-values":

> Number(o.get("HeapMemoryUsage").get("max"))
1821376512
> Number(o.get("HeapMemoryUsage").get("init"))
130023424
> Number(o.get("HeapMemoryUsage").get("used"))
13376936
> Number(o.get("HeapMemoryUsage").get("committed"))
124780544

Note: The class and type should be available on the OpenAF's classpath if not standard.

Thursday, October 3, 2019

Check current OpenAF Java memory

Keeping in mind that OpenAF runs on a JVM so you might want to keep an eye on the current heap size.

Checking current memory size

You can do this easily with the included ow.java library. Just load it:

ow.loadJava();

And call ow.java.getMemory:

> ow.java.getMemory()
+--------+------------+
|   max: | 2048917504 |
| total: | 46137344   |
|  used: | 12635344   |
|  free: | 33502000   |
+--------+------------+

If you want to convert it to the corresponding byte abbrevation just use the first boolean argument:

> ow.java.getMemory(true)
+--------+---------+
|   max: | 1.91 GB |
| total: | 44.0 MB |
|  used: | 12.4 MB |
|  free: | 31.6 MB |
+--------+---------+

Forcing the Java garbage collector

If you fill the heap size temporarially and, afterwards, no longer need the assigned memory Java will eventualy free up once it's corresponding Java garbabe collector runs:

> var list = io.listFiles("c:/windows/system32");
> ow.java.getMemory(true)
+--------+---------+
|   max: | 1.91 GB |
| total: | 44.0 MB |
|  used: | 16.7 MB |
|  free: | 27.3 MB |
+--------+---------+
> ow.java.gc();
> ow.java.getMemory(true)
+--------+---------+
|   max: | 1.91 GB |
| total: | 44.0 MB |
|  used: | 12.1 MB |
|  free: | 31.9 MB |
+--------+---------+

Sunday, September 15, 2019

Function profiling

When coding/scripting there are two important things to do depending on the use the code is going to have: debugging and profiling.

In OpenAF there is an included mini test library that it's actually also used to perform the OpenAF's own build automated tests: ow.test.

In this article we are going to show how to use the ow.test to perform quick code profiling.

Example

Let's take a simple exercise. Imagine you have a 10K entries array and you don't know what to use: $from(array).select() or array.map().

Let's create the array and load ow.test:

ow.loadTest();
var ar = [];
for(let i = 0; i < 10000; i++) {
    ar.push(i);
}

Let's create now the sample function for $from:

function fromTest() {
    $from(ar)
    .select((r) => {
        return r + 1;
    });
}

And map:

function mapTest() {
    ar
    .map((r) => {
        return r + 1;
    });
}

We have the array and we have the test functions now let's use ow.test to understand how each function behaves during 400 executions:

for(let i = 0; i < 400; i++) {
    ow.test.test("$from", fromTest);
    ow.test.test("map", mapTest);
}

Now for the results:

print("Results:\n" + printMap(ow.test.getProfile()));
print("Averages:\n" + printMap(ow.test.getAllProfileAvg()));

So, clearly, in this case, map won to the $from. The average execution time is better and even the minimum time for $from is worse than the max for map.

Of course each case is different and that's the reason that there never is a "silver bullet" solution. You just have to test it.

Tuesday, September 10, 2019

Using ElasticSearch with nAttrMon

nAttrMon captures monitoring inputs to be validated and output to where deem necessary. But you start having more than one instance or you are monitoring a lot of inputs you start having the need to build specific dashboards, visualizations, etc.
Additionally althougth nAttrMon lets you store input history, it's limited since it's not intended to be used for longer than a couple of days.
For all these challenges output and use ElasticSearch & Kibana provide you the answer:
  • Elasticsearch let's you store more than just a couple of days of nAttrMon's input
  • Kibana let's you visualize the data anyway you want from ElasticSearch.

How to set it up

So how to get nAttrMon to output to ElasticSearch? Simply create a new output (example in YAML):
output:
  name         : Output ES values
  chSubscribe  : nattrmon::cvals
  waitForFinish: true
  onlyOnEvent  : true
  execFrom     : nOutput_ES
  execArgs     : 
    url      : http://127.0.0.1:9200
    #user     : nouser
    #pass     : nopass
    #considerSetAll: false 
    #funcIndex: "ow.ch.utils.getElasticIndex('nattrmon-attrs',\"yyyy.ww\")" 
    ## note: Check getElasticIndex function, If a specific format is needed
    ## you can provide it as aFormat (see ow.format.fromDate)"
    funcIndex: "ow.ch.utils.getElasticIndex('nattrmon-attrs', 'yyyy.ww')"
    #index: nattrs
    stampMap : 
      environment: EnvA
      region     : EU
    #include  :
    #  - test/test 1
    # exclude  :
In most of the cases you only need to specify the ElasticSearch URL. But there are more parameters:

Exec arguments Description
index The ElasticSearch index where to store the current values (channel nattrmon::cvals).
funcIndex In most of the cases you want to use a different ElasticSearch index on different times. You can specify a function for that.
considerSetAll The nOutput_ES is based on channel subscription (chSubscribe) so it can potential receive a setAll operation. In case you use nAttrMon channel buffers you need to set this argument to true. Otherwise you are probably fine with false.
stamp Since you might have several nAttrMon's instances dumping attribute values to the same ElasticSearch index you can "stamp" each output entries with specific entries (e.g. in the example environment and region)
include You can specify that only some attributes' values will be sent to ElasticSearch.
exclude Or you can specifiy that only some attributes' values shouldn't be sent to ElasticSearch.
user The user credential if needed.
pass The password credential if needed.

Since in most ElasticSearch configuration cases it's advisable to have different indexes per time you can provide nAttrMon with a function on funcIndex. Using the helper function ow.ch.utils.getElasticIndex:
ow.ch.utils.getElasticIndex('nattrmon-attrs', 'yyyy.ww')

How the output looks in ElasticSearch?

For each attribute value change a new entry will be created in ElasticSearch:


The key is id and it's a hash. Then you will have the name of the attribute and the date it was checked. The value will be in a key with the same name as the attribute. If the attribute has categories the '/' will be converted to a ''. In the example above: _Random_Number and Random_Dice. The stamp keys will also, of course, be part of the ElasticSearch entry.

And the warnings?

To add warnings just create a new similar output changing it's name, the ElasticSearch index where the warnings will be kept and replace on the chSubscribe to use nattrmon::warnings instead of nattrmon::cvals.
Example of an OpenAF channel accessing the ElasticSearch's nattrmon-warns index


The warnings instead of the name of the attribute have the title of the warning, level (e.g. High, Medium, Low, Info, Closed), the description, the date of the last update and date of creation. The stamp map will also be part of the ElasticSearch entry as expected.
Example of a Kibana dashboard using sample warnings from nAttrMon

Saturday, September 7, 2019

Quick XML to/from JSON conversion

At a first glance XML and JSON have some similiarities but XML is actually document oriented while JSON is data-oriented. Nevertheless for some cases, specially when processing in Javascript, it's useful to convert XML into JSON and JSON into XML. Let's check some basic examples.

Simple example XML to JSON

Consider the following example:

<orders>
    <orderId type="normal">1234</orderId>
    <items>
        <item>
            <id>1234</id>
            <qty>1</qty>
        </item>
        <item>
            <id>5678</id>
            <qty>2</qty>
        </item>
    <items>
</orders>

To simply convert it to JSON in OpenAF just execute:

> af.fromXML2Obj("<orders>\n\t<orderId type=\"normal\">1234</orderId>\n\t<items>\n\t\t<item>\n\t\t\t<id>1234</id>\n\t\t\t<qty>1</qty>\n\t\t</item>\n\t\t<item>\n\t\t\t<id>5678</id>\n\t\t\t<qty
>2</qty>\n\t\t</item>\n\t</items>\n</orders>")
{
  "orders": {
    "orderId": "1234",
    "items": {
      "item": [
        {
          "id": "1234",
          "qty": "1"
        },
        {
          "id": "5678",
          "qty": "2"
        }
      ]
    }
  }
}

The result is pretty much similar to the original XML with just one small detail: the orderId's type attribute. It's missing from the final JSON object because JSON doesn't have "an array of attributes per key".

But there is a workaround:

> af.fromXML2Obj("<orders>\n\t<orderId type='normal'>1234</orderId>\n\t<items>\n\t\t<item>\n\t\t\t<id>1234</id>\n\t\t\t<qty>1</qty>\n\t\t</item>\n\t\t<item>\n\t\t\t<id>5678</id>\n\t\t\t<qty>2</qty>\n\t\t</item>\n\t</items>\n</orders>", ["orderId"])
{
  "orders": {
    "orderId": {
      "_type": "normal",
      "_": "1234"
    },
    "items": {
      "item": [
        {
          "id": "1234",
          "qty": "1"
        },
        {
          "id": "5678",
          "qty": "2"
        }
      ]
    }
  }
}

The last optional argument of af.fromXML2Obj is actually an array of keys meaning that if the XML tag name is found on the XML OpenAF will make it a map with keys prefixed with "_".

In this case, "_" is the XML tag associated value (e.g. 1234) and "_type" is the tag attribute type with its corresponding value (e.g. "normal").

Simple example JSON to XML

The reverse of the previous example is also possible:

> var obj = af.fromXML2Obj("<orders>\n\t<orderId type='normal'>1234</orderId>\n\t<items>\n\t\t<item>\n\t\t\t<id>1234</id>\n\t\t\t<qty>1</qty>\n\t\t</item>\n\t\t<item>\n\t\t\t<id>5678</id>\n\t\t\t<qty>2</qty>\n\t\t</item>\n\t</items>\n</orders>");
> af.fromObj2XML(obj);
<orders><orderId>1234</orderId><items><item><id>1234</id><qty>1</qty></item><item><id>5678</id><qty>2</qty></item></items></orders>

Note: Keep in mind that af.fromXML2Obj and af.fromObj2XML are just "simplifiers" to handle XML/JSON conversion. For full support of XML you should use OpenAF's XML plugin.

RSS example

One pratical use for these functions is the hability to easily convert RSS feeds into JSON and JSON into a RSS feed:

> mapArray(af.fromXML2Obj($rest().get("http://feeds.reuters.com/reuters/technologyNews")).rss.channel.item, ["title", "pubDate"])
[
  {
    "title": "Apple says Uighurs targeted in iPhone attack but disputes Google findings",
    "pubDate": "Fri, 06 Sep 2019 20:14:45 -0400"
  },
  {
    "title": "U.S. states launch antitrust probes of tech companies, focus on Facebook, Google",
    "pubDate": "Fri, 06 Sep 2019 18:05:56 -0400"
  },
  {
    "title": "Alphabet says received civil investigative demand from U.S. DoJ",
    "pubDate": "Fri, 06 Sep 2019 17:28:58 -0400"
  },
  {
[...]

Friday, September 6, 2019

Sending emails

The OpenAF included plugin Email allows to easily send emails from any OpenAF script. Let's check it using a simple example:

plugin("Email");
var email = new Email("smtp.gmail.com", "my.email@gmail.com", true, true, false);
email.login("my.email@gmail.com", "myAppPassword");
email.send("Something is wrong", "Something was detected to be very wrong.", [ "someone@somewhere.com" ], [], [], "my.email@gmail.com");

Step by step

Let's translate each line. After including the plugin Email we created a new Email instance for the SMTP server "smtp.gmail.com" for the email account "my.email@gmail.com", turned SSL and TLS on and specified the email wasn't going to contain any HTML.

// Email(aSMTPServer, theFromEmailAddress, useSSL, useTLS, containsHTML)
var email = new Email("smtp.gmail.com", "my.email@gmail.com", true, true, false);

The Email plugin will try to "guess" the right ports to access the SMTP server but if you need to force it you can do it:

email.setPort(12345);

The next step was authenticating with the SMTP server:

email.login(aLogin, aPassword);

And then finally we sent the simple email:

// email.send(aSubjectString, aMessageString, anArrayOfTOs, anArrayOfCCs, anArrayOfBCCs, aFromEmailAddress)
email.send("Something is wrong", "Something was detected to be very wrong.", [ "someone@somewhere.com" ], [], [], "my.email@gmail.com");

How to send a HTML email

To send a HTML email first you will need to specify it on the Email instance creation:

var email = new Email("smtp.gmail.com", "my.email@gmail.com", true, true, true);

Afterwards you add the HTML using the .setHTML function:

email.setHTML("<h1>BIG NEWS</h1>Everything is <b>okay</b>.");

So, if we just defined the email contents why should we defined aMessageString on the email.send function? In case the email client doesn't support HTML emails the aMessageString parameter of the email.send function will be used.

Adding attachments

To add an attachment use the function email.addAttachment:

email.addAttachment("/my/folder/with/attach1.pdf");

Adding images for the HTML content

If you use HTML content you will probably also want to include images. These aren't the usual regular attachments and there is actually 3 options available:

Embed image files

email.setHTML("<html>...<img src=\"cid:myimage.png\"/>...</html>");
email.embedFile("/some/path/myimage.png", "myimage.pn");

Embed image URLs

email.setHTML("<html>...<img src=\"cid:myimage\"/>...</html>");
email.embedURL("https://some.server/some/image.jpg", "myimage");

Automatic embedding images and reference them by URL

email.setHTML("<html>...<img src="https://some.server/some/image.jpg"/>...</html>");
email.addExternalImage("https://some.server/some/image.jpg");

How to debug

After creating the new Email instance just add:

email.getEmailObj().setDebug(true);

Once the email sending operation starts all communication with the SMTP server will be output to stdout/stderr.

Tuesday, September 3, 2019

Using the SNMP plugin

In OpenAF there are two main SNMP plugins: SNMP and SNMP server. In this article we are going to focus on the SNMP plugin that provide SNMP client functionality.

SNMP has different versions (e.g. 1, 2, 3) that require different settings. Starting with version 1 and 2 all you need is the SNMP connection details and, optinionally, a community:

plugin("SNMP");
var snmp = new SNMP("udp:demo.snmplabs.com/161", "public");  // version 1/2

Checking an OID value

To get the value associated with an OID:

snmp.get("1.3.6.1.2.1.1.3.0");
// {
//  "1.3.6.1.2.1.1.3.0": "36 days, 12:34:56.51"
//}

Sending a trap/inform

To send a trap simply:

snmp.trap("1.3.6.1.4.1.20408.4.1.1.2", [
    { OID: "1.2.3.4.5.6.7.8", type: "s", value: "My error message." }
])

You just need to provide the trap OID and an array of OID based values. Each value can have a different type. The supported types are:

Type Description
i Integer
u Unsigned
c Counter32
s String
x Hex String
d Decimal String
n A null object
o An object id
t Timeticks
a An ip address

To inform it's exactly the same but it will return you a Java response object:

var response = snmp.inform("1.3.6.1.4.1.20408.4.1.1.2", [
    { OID: "1.2.3.4.5.6.7.8", type: "s", value: "My error message." }
])

In contrast sending a trap will return immediately and there won't be any acknowledgement.

Version 3

On version 3 you need to provide a little more information:

plugin("SNMP");
var aTimeout = 3000, aNumberOfRetries = 3;
var snmp = new SNMP("udp:demo.snmplabs.com/161", "public", aTimeout, aNumberOfRetries, 3, {
    engineId      : "8000000001020304",
    authPassphrase: "authKey1",
    privPassphrase: "privKey1",
    authProtocol  : "MD5",
    privProtocol  : "DES",
    securityName  : "usr-md5-des"
})

But all the rest is same as shown previously.

Friday, August 30, 2019

Applying selectors to an array

Whenever using an array of maps, and specially when analyzing them on OpenAF-console, you probably want to focus just on some map entries and not have all the other map keys on the screen. To make it easier there is a function mapArray in OpenAF for this case and others.

Let's say you have an array of maps and submaps like this:

> $rest().get("https://jsonplaceholder.typicode.com/users");
[
  {
    "id": 1,
    "name": "Leanne Graham",
    "username": "Bret",
    "email": "Sincere@april.biz",
    "address": {
      "street": "Kulas Light",
      "suite": "Apt. 556",
      "city": "Gwenborough",
      "zipcode": "92998-3874",
      "geo": {
        "lat": "-37.3159",
        "lng": "81.1496"
      }
    },
    "phone": "1-770-736-8031 x56442",
    "website": "hildegard.org",
    "company": {
      "name": "Romaguera-Crona",
      "catchPhrase": "Multi-layered client-server neural-net",
      "bs": "harness real-time e-markets"
    }
  },
  {
    "id": 2,
    "name": "Ervin Howell",
    "username": "Antonette",
    "email": "Shanna@melissa.tv",
    "address": {
[...]

Let's say that you want to compare just the name, the city on the address and the company name. With mapArray that's easy:

> mapArray( $rest().get("https://jsonplaceholder.typicode.com/users"), [ "name", "address.city", "company.name" ] )
[
  {
    "name": "Leanne Graham",
    "address.city": "Gwenborough",
    "company.name": "Romaguera-Crona"
  },
  {
    "name": "Ervin Howell",
    "address.city": "Wisokyburgh",
    "company.name": "Deckow-Crist"
  },
  {
    "name": "Clementine Bauch",
    "address.city": "McKenziehaven",
[...]

Since now you just have the fields you want you can also use table on openaf-console:

> table mapArray( $rest().get("https://jsonplaceholder.typicode.com/users"), [ "name", "address.city", "company.name" ] )
          name          | address.city |   company.name
------------------------+--------------+------------------
Leanne Graham           |Gwenborough   |Romaguera-Crona
Ervin Howell            |Wisokyburgh   |Deckow-Crist
Clementine Bauch        |McKenziehaven |Romaguera-Jacobson
Patricia Lebsack        |South Elvis   |Robel-Corkery
[...]

If you have an array on a map element you can just reference it directly:

> mapArray( someMap, ["id", "items[0].subId"])
[
  {
    "id": 1,
    "items[0].subId": 1
  },
  {
    "id": 2,
    "items[0].subId": 4
  },
  {
    "id": 3,
    "items[0].subId": 43
  }
]

Note: mapArray uses ow.obj.getPath and ow.obj.setPath internally that you can use in other situations directly.

Tuesday, August 27, 2019

Searching map keys and values

The main data structure in OpenAF is, of course, javascript maps. We already cover how to easily search on arrays using $from. But what about maps?

Let's take an example:

> var weather = $rest().get("http://www.metaweather.com/api", { location: 742676 });

The "weather" variable now should have a map with the weather forecast for Lisbon. If you inspect it has sub-arrays with the forecast for the next days, map entries to provide info about sun rise and sun set, an array with the weather sources, etc...

Searching keys

If you now the exact path it's easy to build the map "path" to get the values that you want but the OpenAF function searchKeys will actually help you find those paths. Let's try to find those sun rise and sun set entries:

> searchKeys(weather, "sun")
{
  ".sun_rise": "2019-08-27T07:00:52.671773+01:00",
  ".sun_set": "2019-08-27T20:14:52.385796+01:00"
}

The string search parameter (e.g. sun) is actually a case insensitive regular expression. And it just found you all map entries with the word "sun" anywhere. Now you know that weather.sun_rise and weather.sun_set will get you the info you need. Let's try to find now temperatures:

> searchKeys(weather, "temp")
{
  ".consolidated_weather[0].min_temp": 17.98,
  ".consolidated_weather[0].max_temp": 23.674999999999997,
  ".consolidated_weather[0].the_temp": 24.23,
  ".consolidated_weather[1].min_temp": 17.405,
  ".consolidated_weather[1].max_temp": 23.05,
  ".consolidated_weather[1].the_temp": 23.39,
  ".consolidated_weather[2].min_temp": 16.985,
  ".consolidated_weather[2].max_temp": 24.47,
  ".consolidated_weather[2].the_temp": 24.625,
[...]

Now the temperatures are found within the consolidate_weather array. So you now have a list of all the paths to get the temperatures even within an array.

Searching values

Searching keys of a map wouldn't be complete without being able to search for values. On the initial request we used the Lisbon location code "742676". Let's search for that:

> searchValues(weather, "742676")
{
  ".woeid": 742676
}

As expected the result is the path on where that value was found. Let's now use a regular expression to try to find if there are any lat/log coordinates:

> searchValues(weather, "-?\\d+\\.\\d+,-?\\d+\\.\\d+");
{
  ".parent.latt_long": "39.557919,-7.844810",
  ".latt_long": "38.725670,-9.150370"
}

Other advanced uses

Where these two functions searchKeys and searchValues really shine when you are using the openaf-console to investigate a big and/or complex javascript map. They will quickly filter and find you the data that you need to help you write your scripts.

The previous examples show you the basic use of these functions. There are other options (check, in the openaf-console, "help searchKeys" and "help searchValues") to make the search case sensitive and even to execute a callback function whenever keys or values are found (for example: search & replace).

Monday, August 26, 2019

Using an in-memory database

OpenAF comes with the H2 database embeeded. The main objective is, of course, to interact with H2 databases. But it also provides other perks (e.g. like the MVStore). One of them is an "in-memory" H2 database.

Why would you want or need an "in-memory" database? There are a lot of uses that we won't elaborate now but you can read them in wikipedia. But since H2 is also a relational database engine it can be helpful even if it just for testing.

Creating

Creating the H2 in-memory database it's easy. Just provide a name and execute:

> var db = createDBInMem("testDB");

Why a name? Because you can have several databases providing you have the memory capacity for them. The OpenAF's createDBInMem function returns an already instatiated db object that you can now use.

Using it

> db.u("CREATE TABLE test (id NUMBER(10), desc VARCHAR2(255))");
0
> db.q("SELECT * FROM test");
{
  "results": []
}

Let's insert some data:

> db.us("INSERT INTO test (id, desc) VALUES (?, ?)", [ 0, "Result 0" ]);
1
> db.q("SELECT * FROM test")
{
  "results": [
    {
      "ID": 0,
      "DESC": "Result 0"
    }
  ]
}

Let's make it a hundred dummy data records:

> var ar = []; for(var ii = 1; ii < 99; ii++) { ar.push([ ii, String("Result " + ii)]); }
98
> db.usArray("INSERT INTO test (id, desc) VALUES (?, ?)", ar)
98
> db.commit();
> 
> db.q("SELECT COUNT(1) c FROM test")
{
  "results": [
    {
      "C": "99"
    }
  ]
}

Persisting

If you exit OpenAF the in-memory database will, of course, lose all it's data. But what if you want to persist it? There are some helper functions for that: persistDBInMem and loadDBInMem:

> persistDBInMem(db, "test.sql");

Then on another OpenAF execution:

> var db = createDBInMem("testDB");
> loadDBInMem(db, "test.sql")
5
> db.q("select count(1) C from test")
{
  "results": [
    {
      "C": "99"
    }
  ]
}

Note: the generated files are SQL files. There isn't any intention of supporting large volumes of data databases.

Tuesday, August 20, 2019

Setting DB auto commit

By default OpenAF opens all database connections in "autocommit=false" mode. But in some special cases you might need to turn it on. Let's see one of those cases:

Creating a PostgreSQL tablespace example

If you open a PostgreSQL connection and try to create a tablespace "TEMP" as you would usually do, this will happen:
> var db = new DB("jdbc:postgresql://some.host:5432/postgres", "adminUser", "adminPass");
> db.u("create tablespace temp location '/data'");
-- JavaException: org.postgresql.util.PSQLException: ERROR: CREATE TABLESPACE cannot run inside a transaction block
That's because PostgreSQL considers anything with "autocommit=false" to be a transaction block.

How to set auto commit to on?

To bypass this situation, on a dedicated DB connection, execute the following:
> var db = new DB("jdbc:postgresql://some.host:5432/postgres", "adminUser", "adminPass");
> db.getConnect().setAutoCommit(true);
> db.u("create tablespace temp location '/data'");
1
The getConnect method will return you the JDBC connection instance object which, among others, allows you to execute the setAutoCommit method. That's enough to then execute the "create tablespace" statement without being in a transaction block.

Using arrays with parallel

OpenAF is a mix of Javascript and Java, but "pure" javascript isn't "thread-safe" in the Java world. Nevertheless be...