Relaxed JSON parser

In OpenAF, when using the jsonParser function, the parsing sticks to the strict JSON definition.

For example the following behaves as expected:

> jsonParser("{ \"a\": 1 }");
{
   "a": 1
}
> JSON.parse("{ \"a\": 1 }");
{
   "a": 1
}

But using a more "relaxed" JSON definition, the same functions will fail:

> jsonParser("{ a: 1 }");
{ a: 1 }
> JSON.parse("{ a: 1 }");
-- SyntaxError: Unexpected token in object literal

The jsonParser function will return the text string representation as it's unable to parse the JSON string. The native JSON.parse will actually throw an execption.

Using GSON

OpenAF includes the GSON library which can parse more "relaxed" JSON definitions. Since OpenAF version 20200108, the OpenAF jsonParser function can also use the GSON library. There is a new second boolean argument that if true alternates to GSON for parsing the string provided on the first argument:

> jsonParse("{ a: 1 }", true);
{
   "a": 1
}

oJob exception handling

When you create an OpenAF's oJob each job runs independently and you are responsible for handling exceptions on each.

But couldn't we have a global try/catch function for all jobs? Yes, you can add a function with ojob.catch.

oJob throwing exceptions example

todo:
  - Normal job
  - Bug job
  - Another Bug job

jobs:
  #-----------------
  - name: Normal job
    exec: |
      print("I'm a normal job.");

  #--------------
  - name: Bug job
    exec: |
      print("I'm a buggy job.");
      throw "BUG!";

  #----------------------
  - name: Another Bug job
    exec: |
      print("I'm another buggy job.");
      throw "BUG!";

This ojob has 3 jobs. "Normal job" will execute without throwing any exceptions. But "Bug job" and "Another Bug job" will throw two exceptions when executed.

Executing you will see the three jobs executing with two of them failing. The entire oJob process will finish with exit code 0.

oJob general exception example

todo:
  - Normal job
  - Bug job
  - Another Bug job

ojob:
  catch: |
    var msg = "Error executing job '" + job.name + "'\n";
    msg += "\nArguments:\n" + stringify(args, void 0, "") + "\n";
    msg += "\nException:\n" + stringify(exception, void 0, "") + "\n";
    logErr(msg, { async: false });

    if (String(exception) == "BUG!") exit(1);

jobs:
  #-----------------
  - name: Normal job
    exec: |
      print("I'm a normal job.");

  #--------------
  - name: Bug job
    exec: |
      print("I'm a buggy job.");
      throw "BUG!";

  #----------------------
  - name: Another Bug job
    exec: |
      print("I'm another buggy job.");
      throw "BUG!";

This example is equal to the previous one but it adds ojob.catch. The catch function receives several arguments:

Argument	Type	Description
args	Map	The current args map at the point of the exception.
job	Map	The job map where the exception occurred.
id	Number	If the job was executed within a specific sub-id.
deps	Map	A map of job dependencies.
exception	Exception	The exception itself.

If the function returns true the exception will be recorded as usual and the job will be registered has failed. If the function returns false the exception will be ignored.

In this example the function will actually stop the entire oJob process with exit code 1.

oJob Check for stall

When running an oJob there might be situations that you want to ensure that the entire process won't enter into a stall (e.g. being stopped on a "dead-end" waiting for some service or lock or whatever).

In oJob there is actually a feature to allow you to ensure, no matter what, your oJob won't run pass a specific timeout or for a function to be executed to determine if the oJob is at a stall situation.

Killing after x seconds

The easiest configuration is ensuring that there is a general timeout for the entire oJob:

ojob:
  checkStall:
    # check for a stall every x seconds (default 60)
    everySeconds    : 1
    # kill the entire process after x seconds
    killAfterSeconds: 4

todo:
  - Test job

jobs:
  #---------------
  - name: Test job
    exec: |
      args.wait = _$(args.wait).default(5000);

      log("Waiting for " + args.wait + "ms...");
      sleep(args.wait, true);
      log("Done");

Executing this oJob you will get different results depending on the amount of time the "Test job" takes. It's configured to "kill it self" if it takes longer than 4 seconds and it will check for that every second (e.g. on real situations you should use the default of 60 seconds).

For 1,5 seconds:

$ ojob test.yaml wait=1500
>> [Test job] | STARTED | 2019-11-11T12:13:17.199Z------------------------
2019-11-11 12:13:17.230 | INFO | Waiting for 1500ms...
2019-11-11 12:13:18.736 | INFO | Done

<< [Test job] | Ended with SUCCESS | 2019-11-11T12:13:18.740Z ============

For 5 seconds:

$ ojob test.yaml wait=5000
>> [Test job] | STARTED | 2019-11-11T12:22:00.058Z -----------------------
2019-11-11 12:22:00.085 | INFO | Waiting for 5000ms...
oJob: Check stall over 4000
2019-11-11 12:22:03.878 | ERROR | oJob: Check stall over 4000

Killing depending on a function

If you have certain conditions that can be easily checked to determine if the oJob is stalled you can use a function:

ojob:
  checkStall:
    everySeconds    : 1
    checkFunc       : |
      print("checking for stall...");
      if (global.canDie) {
        print("should die.");
        return true;
      }

todo:
  - Init
  - Test job

jobs:
  #-----------
  - name: Init
    exec: |
      global.canDie = false;

  #---------------
  - name: Test job
    exec: |
      log("Waiting for 2500ms...");
      sleep(2500, true);

      log("Setting canDie to true...");
      global.canDie = true;

      log("Waiting for another 2500ms...");
      sleep(2500, true);

      log("Done");

In this case a global variable canDie is only set to true after the first 2,5 seconds of execution of the Test job job. As soon as the checkFunc is executed and confirms the conditions by returning a true value the oJob is immediatelly stopped.

$ ojob test2.yaml
checking for stall...
>> [Init] | STARTED | 2019-11-11T12:32:02.828Z -----------------------------
<< [Init] | Ended with SUCCESS | 2019-11-11T12:32:02.857Z ==================
>> [Test job] | STARTED | 2019-11-11T12:32:02.893Z -------------------------
2019-11-11 12:32:02.924 | INFO | Waiting for 2500ms...
checking for stall...
checking for stall...
2019-11-11 12:32:05.429 | INFO | Setting canDie to true...
2019-11-11 12:32:05.430 | INFO | Waiting for another 2500ms...
checking for stall...
should die.

You can see the several checkFunc executions by the output "checking for stall…" and once the global variable canDie was true all the oJob stopped it's execution.

Checking at the job level

All the previous options checked for stall for the entire oJob execution but you can specify the same at the job level using typeArgs.timeout and typeArgs.stopWhen that are available for all types of jobs in oJob.

Example with typeArgs.timeout

In this example the Test job job is set to timeout after 1,5 seconds:

todo:
  - Init
  - Test job
  - Done

jobs:
  #-----------
  - name: Init
    exec: |
      global.canDie = false;

  #-----------
  - name: Done
    exec: |
      log("Everything is done.");

  #-------------------
  - name    : Test job
    typeArgs:
      timeout: 1500
    exec    : |
      log("Waiting for 2500ms...");
      sleep(2500, true);

      log("Setting canDie to true...");
      global.canDie = true;

      log("Waiting for another 2500ms...");
      sleep(2500, true);

      log("Done");

Executing it the job will actually end in error after the specified timeout:

>> [Init] | STARTED | 2019-11-11T12:47:25.568Z ----------------------------
<< [Init] | Ended with SUCCESS | 2019-11-11T12:47:25.624Z =================
>> [Test job] | STARTED | 2019-11-11T12:47:25.662Z ------------------------
2019-11-11 12:47:25.684 | INFO | Waiting for 2500ms...

!! [Test job] | Ended in ERROR | 2019-11-11T12:47:27.197Z =================
- id: 8ebbf961-822d-3b95-ca09-8dfb335ab6cb
  error: Job exceeded timeout of 1500ms

===========================================================================
>> [Done] | STARTED | 2019-11-11T12:47:27.277Z ----------------------------
2019-11-11 12:47:27.297 | INFO | Everything is done.

<< [Done] | Ended with SUCCESS | 2019-11-11T12:47:27.300Z =================

Example with typeArgs.stopWhen

In this example the Test job job is set stop whenever the stopWhen function returns a true value:

todo:
  - Init
  - Test job
  - Done

jobs:
  #-----------
  - name: Init
    exec: |
      global.canDie = false;

  #-----------
  - name: Done
    exec: |
      log("Everything is done.");

  #-------------------
  - name    : Test job
    typeArgs:
      stopWhen: |
        if (global.canDie) {
           print("should die...");
           return true;
        }
    exec    : |
      log("Waiting for 2500ms...");
      sleep(2500, true);

      log("Setting canDie to true...");
      global.canDie = true;

      log("Waiting for another 2500ms...");
      sleep(2500, true);

      log("Done");

Executing the job will actually stop without any error if the stopWhen function returns the a true value. To end the job with an error simply throw an exception on the stopWhen function.

>> [Init] | STARTED | 2019-11-11T12:38:54.232Z -----------------------------
<< [Init] | Ended with SUCCESS | 2019-11-11T12:38:54.263Z ==================
>> [Test job] | STARTED | 2019-11-11T12:38:54.298Z -------------------------
2019-11-11 12:38:54.330 | INFO | Waiting for 2500ms...
2019-11-11 12:38:56.837 | INFO | Setting canDie to true...
should die...
2019-11-11 12:38:56.838 | INFO | Waiting for another 2500ms...

<< [Test job] | Ended with SUCCESS | 2019-11-11T12:38:56.857Z ==============
>> [Done] | STARTED | 2019-11-11T12:38:56.012Z =============================
2019-11-11 12:38:56.025 | INFO | Everything is done.

<< [Done] | Ended with SUCCESS | 2019-11-11T12:38:56.098Z ==================

How to convert arrays to maps and vice-versa

When working with javascript maps of maps or arrays of maps you might sometimes think: "if only it was an array instead of an object" or "if only it was an object instead of an array".

Usually this happens when you know the perfect method/library but it would only work with arrays or objects.

In OpenAF there are two functions to try to make it easier to convert arrays to maps and vice-versa: ow.obj.fromArray2Obj and ow.obj.fromObj2Array.

Converting a map into an array

Let's take, for example, the map that it's returned by the getRemoteOPackDB():

{
    "OpenAF-Templates": {
        "version": "20190906",
        "files": [
            // ...
        ],
        "description": "..."
    },
    "OpenAFLambdaLayers": {
        "version": "20190809",
        "files": [
            //...
        ],
        "description": "..."
    },
    ...
}

You can quickly convert into an array:

> ow.loadObj();
> var ar = ow.obj.fromObj2Array(getRemoteOPackDB(), "name")
[
    {
        "name": "OpenAF-Templates",
        "version": "20190906",
        "files": [
            // ...
        ],
        "description": "..."
    },
    {
        "name": "OpenAFLambdaLayers",
        "version": "20190809",
        "files": [
            // ...
        ],
        "description": "..."
    }
]
> printTable(mapArray(ar, ["name", "version"]));
       name        |version
-------------------+--------
OpenAF-Templates   |20190906
OpenAFLambdaLayers |20190809
// ...

The second parameter for ow.obj.fromObj2Array is the attribute that the array of maps will include if you want to map each key into an entry (e.g. in this case name).

Converting an array into a map

Let's take the example of an array with the list of files:

> var files = io.listFiles("/my/folder").files;
> var obj = ow.obj.fromArray2Obj(files, "canonicalPath");
{
    "/my/folder/a.js": {
        "isDirectory": false,
        "isFile": true,
        "filename": "a.js",
        "filepath": "/my/folder/a.js",
        "lastModified": 1534542464558,
        "createTime": 1534541960321,
        "lastAccess": 1562153514605,
        "size": 16384,
        "permissions": "xrw"
    },
    "/my/folder/b.js": {
        // ...
    }
}

The second argument provided to the function ow.obj.fromArray2Obj will be each map's entry that will be used as key on the resulting map.

Optionally you can also add a third boolean argument to indicate if you want, or not, the entry remote from each map detail.

Appending to files

Let's start with a simple question: how to append content to an existing file?

The answer is: check the "help" for the io.writeFile* function you need. Most of them already support an extra parameter to append to existing files.

io.writeFileString example

To write to a custom log you would do something like:

io.writeFileString("mylog.log", new Date() + " | Just did something\n");

But using this example the file mylog.log would get overwritten every time you execute it. To get around it simple add an extra flag:

io.writeFileString("mylog.log", new Date() + " | Just did something\n", void 0, true);

And everytime you run it, it will add a new line to the mylog.log file.

Appending several files into one

A one_liner that is usually helpfull is:

> $from(io.listFilenames("my/path")).ends(".js").select(r => { io.writeFileString("all.js", io.readFileString(r), void 0, true); });

This will actually create all.js file with the contents of all files in my/path with suffix ".js". Of course you now can change the $from conditions to whatever special conditions you might have.

nAttrMon kill after an amount of minutes

On a nAttrMon configuration you might have several inputs, validations and outputs. You might use the waitForFinish parameter to ensure another instance of the same plug doesn't start while the previous is still running.

This is specially true usually on input plugs that depend on the reply of external systems (e.g. database, sftp, a REST service, etc...).

But what if there isn't any timeout in place or you simply don't want that plug to run forever keeping nAttrMon waiting? You can use the setting killAfterMinutes.

Example

input:
  name            : Big queries
  # running every hour
  cron            : "0 */1 * * *"
  onlyOnEvent     : true
  # make sure that just one is executing
  waitForFinish   : true
  # make sure if doesn't execute for more than 30 minutes
  killAfterMinutes: 30
  execFrom        : nInput_DB
  execArgs        :
    key : myBigDatabase
    sqls:

      Database/Big query 1: |
        SELECT stuff 
        FROM something 
        -- [...]

In this case, every hour, nAttrMon will try to execute the "Big queries" but since killAfterMinutes is defined it will interrupt/terminate this input execution if it's longer than 30 minutes.

Checking arguments

In OpenAF whenever you built a simple function, or you have a piece of code on a oJob that expects arguments or any code block you will eventually need to ensure the type of the input variables/arguments and enforce mandatory arguments.

Let's say that you build a function to add two values and return a verbose result. It receives arguments a and b that must be numbers, but if b is not provided it assumes 0. Argument c should be a string and if not provided a default value is provided. All this in plain javascript is:

function sum(a, b, c) {
    if (typeof a != "number") throw "a needs to be a number";
    if (typeof b == "undefined") b = 0;
    if (typeof b != "number") throw "b needs to be a number";
    if (typeof c == "undefined") c = "the result is ";
    if (typeof c != "string") throw "c needs to be a string";

    return c + String(a + b);
}

If you inspect it carefully you need to have the "if" statements in the right order to achieve the intended result. The more arguments you have the harder to read.

OpenAF as a shortcut to help with readability. It's not intended to be like TypeScript but just a small helper for OpenAF scripting. So the previous function would look like this with these OpenAF shortcuts:

function sum(a, b, c) {
    _$(a, "a").isNumber().$_();

    b = _$(b, "b").isNumber().default(0);
    c = _$(c, "c").isString().default("the result is ");

    return c + String(a + b);
}

Each declaration starts always with _$(variable, stringName). The stringName is actually optional if you want to become "more verbose" (you will see how in a second).

The declaration is then followed by conditions like isNumber and isString. You can provide an argument to each of these "conditions" to have more verbose exceptions than "a needs to be a number.".

The declaration ends with default(aValue) or $_(stringName). If no value is provided default will ensure that the variable will always default to some value while $_ doesn't return any value but will throw an exception indicating that the variable value wasn't provided.

So a more verbose type checking could be:

function sum(a, b, c) {
    _$(a).isNumber("The argument 'a' needs to be a number.').$_('The argument 'a' is mandatory for the sum function.');

    b = _$(b).isNumber("The argument 'b', if defined, must be a number.").default(0);
    c = _$(c).isString("The argument 'c', if defined, must be a string.").default("the result is ");

    return c + String(a + b);
}

You can see the list of all available conditions by executing desc _$() on an openaf-console.

Encrypt/Decrypt with public/private keys

There are built-in functions to easily encrypt/decrypt text using a single password key (using functions af.encrypt and af.decrypt). But if you need better security you might need to encrypt/decrypt using a public and a private keys.

This can be achieved using the ow.java.cipher. It enables the generation or loading of existing public and private keys and use them to encrypt/decrypt text.

To create an object instance:

ow.loadJava();
var cipher = new ow.java.cipher();

Generating public/private keys

You can generate a public and a private key with a specific key length (by default 2048 bits). You can generate for 3072, 4096, 7680 and 15360 bits (the bigger, the longer to generate):

var keys = cipher.genKeyPair(4096);

To save the private and public keys in files:

cipher.saveKey2File("key.pub", keys.publicKey, false);
cipher.saveKey2File("key.priv", keys.privateKey, true);

To load the keys from the files:

var kpPUB  = cipher.readKey4File("key.pub", false);
var kpPRIV = cipher.readKey4File("key.priv", true);

Encrypting/Decrypting text

Now that you have a pair of public and private keys you can easily encrypt/decrypt text:

var cipheredText = cipher.encrypt2Text(plainText, kpPUB);

var plainText = cipher.decrypt4Text(cipheredText, kpPRIV);

The functions with the "Text" suffix automatically convert to a Base64 text for easier transport. But there are also functions for encrypting/decrypting to/from arrays of bytes and streams.

Using the function parallel4Array

OpenAF allows you to code in javascript while using Java functionality underneath. But there are obvious differences between the two languages.

Example

Let's take a simple example for parallel processing. First you create an array with the information of several files:

var files = listFilesRecursive("/usr");

Running it sequentially

Next you want to capture the result of executing a "not so fast" operation of performing an external shell command and capture the result:

var res = [];
for(var i = 0; i < files.length; i++) {
    res.push(sh("ls " + files[i]));
}

Using this simple for cycle the time elapsed, on a specific environment, was 1 second and 686 ms.

Running it in parallel

But this was a rather sequential processing in nature. OpenAF includes a function, parallel4Array, that wlil automate much of the work of using Java threads to perform the same exact functionality:

var res = parallel4Array(files, (v) => {
    return sh("ls " + v.filename);
})

Changing the for cycle by parallel4Array, and testing on the same environment, the result was a exactly equal res array in around 842ms. So what happenend?

Breaking it down:

• The number of cpu cores was detected.
• The files array was split evenly between the number of cpu cores detected.
• Each "splitted array" was executed sequentially on separate executing thread (Java thread).
• OpenAF kept an eye on the operating system reported machine load introducing delays when cpu cores got "overloaded".

This last step is helpful when executing on machines with a large number of cpu cores.

"Can I tune it?"

Yes, of course. A third optional argument of parallel4Array let's you override the cpu detection and "machine load" control. A fourth argument of parallel4Array gives you direct access to the Java threads objects, unique ids, etc… Check the help information in openaf-console: "help parallel4Array".

For advanced cases you can use the function parallelArray which let's you control how the end result will be built letting you provide a function to "unify" all the threads results.

There is even the function parallel that will launch multiple threads to execute the same provided function without the need to provide an array.

Converting an array to a CSV file and vice-versa

The OpenAF's included CSV object provides some basic functionality for handling CSV files. Althought not intended for handling "huge" CSV files some of these functions are usually handy to quickly convert javascript arrays (composed of javascript map entries) to and from a CSV file.

The main CSV functions to use are: CSV.fromArray2File and CSV.fromFile2Array.

Converting from an Array to a CSV file

Here is an example:

// Let's prepare a sample array with all the file info from the current folder
var myArray = io.listFiles(".").files;

// Let's create a CSV object instance
var csv = new CSV();

// Now simply write a file based on the existing array
csv.fromArray2File(myArray, "mycsv.csv");

Now if you check the newly created (or overwritten) CSV file it will look similar to:

"isDirectory","isFile","filename","filepath","canonicalPath","lastModified","createTime","lastAccess","size","permissions"
"false","true","opack","./opack","/openaf/opack",1569823332000,1569823332000,1569823332000,353,"xrw"
"false","true","openaf","./openaf","/openaf/openaf",1569823332000,1569823332000,1569823332000,342,"xrw"

which maps the original array content:

> myArray
[{
    isDirectory: false,
    isFile: true,
    filename: "opack",
    filepath: "./opack",
    canonicalPath: "/openaf/opack",
    lastModified: 1569823332000,
    createTime: 1569823332000,
    lastAccess: 1569823332000,
    size: 353,
    permissions: "xrw"
}, {
    isDirectory: false,
    isFile: true, 
    filename: "openaf",
    filepath: "./openaf",
    canonicalPath: "/openaf/openaf",
    lastModified: 1569823332000,
    createTime: 1569823332000,
    lastAccess: 1569823332000,
    size: 342,
    permissions: "xrw"
}
...

If you examine it carefully the boolean values, from the javascript map, were converted to strings. This is expected since the CSV format will only map javascript Number and String trying to convert any other unknown type to String.

Note: Javascript maps can contain sub-maps and sub-arrays. These won't be correctly converted to CSV.

Specific array fields

As an optional third argument of the CSV.fromArray2File function you can also limit the fields used or provide a specific order to use:

csv.fromArray2File(myArray, "mycsv.csv", ["canonicalPath", "size"]);

Converting a CSV file into an array

Using the previous example where we converted a javascript array into a CSV file, it's easy to convert back to an array:

var myNewArray = csv.fromFile2Array("mycsv.csv");

But, as previously warned, some types will be represented as strings:

> myNewArray
[{
  isDirectory: "false",
  isFile: "true",
  filename: "opack",
  filepath: "./opack",
  canonicalPath: "/openaf/opack",
  lastModified: "1569823332000",
  createTime: "1569823332000",
  lastAccess: "1569823332000",
  size: "353",
  permissions: "xrw"
}, {
...

Nevertheless these functions provide a quick and easy way to convert javascript arrays to and from CSV files which can be useful when you are trying to use the javascript data with other tools.

Creating a ZIP file

OpenAF includes a specific plugin to group all the ZIP related functionality trying to make it easy to use it.

In this case we will show how easy is to create a ZIP file. We have two local files that we want to add to a new zip file: myclass.java and myclass.class.

plugin("ZIP");

zip.putFile("src/myclass.java", "myclass.java");
zip.putFile("bin/myclass.class", "myclass.class");

zip.generate2File("myclass.zip", { compressionLevel: 9 });
zip.close();

If you look carefully we are taking two local files, from the same folder, but storing them in different folders inside the ZIP file (e.g. zip.putFile(target, source)).

The ZIP file will get written when you call the zip.generate2File(aFilePath, mapOptions) function. Besides creating the "myclass.zip" we are also specifying that we want the maximum compression possible (e.g. level 9).

You can explore the contents of this newly created zip file using zip.list(aZipFile) function:

> plugin("ZIP");
> var zip = new ZIP();
> zip.list("myclass.zip");
+--------------------+-----------------+-------------------+
|  src/myclass.java: | compressedSize: | 17                |
|                    |           size: | 15                |
|                    |            crc: | 3010494688        |
|                    |           name: | src/myclass.java  |
|                    |        comment: | null              |
|                    |           time: | 1569369872000     |
+--------------------+-----------------+-------------------+
| bin/myclass.class: | compressedSize: | 18                |
|                    |           size: | 16                |
|                    |            crc: | 626008697         |
|                    |           name: | bin/myclass.class |
|                    |        comment: | null              |
|                    |           time: | 1569369872000     |
+--------------------+-----------------+-------------------+

Note: for bigger ZIP files you can use zip.streamPutFile*

Testing a TCP port

When performing TCP connections you always have to deal with the eventual connection failure. For example, if you write a script that will connect to a specific server, you should deal with the issue that when executing the script might not have connectivity to the desire target.

So how to handle these events? The typical answer is waiting for the connection error exception and deal with it. For example:

try {
    // make the TCP call
} catch(e) {
    // handle the exception
}

You should always handle the exception but you can avoid even going into the try/catch block with a quick TCP connectivy check:

ow.loadFormat();

var host = "my.service.host";
var port = 1234;

if (ow.format.testPort(host, port)) {
    var result;
    try {
        result = callService(host, port, params);
        // process result
    } catch(e) {
        logErr("Problem calling service on " + host + ":" + port);
    }
} else {
    logErr("No connectivity to " + host + ":" + port);
}

The ow.format.testPort function allows for quick socket connection tests. By default it timeouts after 1.5 seconds but you can change that using a third parameter:

// Wait 5 seconds before declaring farAway service not reachable (false)
ow.format.testPort(farAwayHost, farAwayPort, 5000);

Introduction to streams in OpenAF

Some of the functions you find in OpenAF have the "stream version" and the "non stream version". For example: io.readFileBytes and io.readFileStream; io.writeFileBytes and io.writeFileStream; HTTP.getBytes and HTTP.getStream; etc.

So what's the difference?

Roughly the non stream version will get the relevant contents to memory from another source or write them from memory to another source. It's fast but the bigger the contents, the bigger memory you will need.

The stream version provides an object that let's other functions retrieve small sub-sets of the content (from another source or memory) while handling that content.

For example, reading a file:

var contents = io.readFileBytes("myFile.bin");

// vs

var istream = io.readFileStream("myFile.bin");

The variable contents will hold all the bytes on the myFile.bin file while the variable istream will be an object allowing other functions to read parts of myFile.bin.

Input and output streams

These OpenAF streams are actually nothing more than the Java's InputStream and OutputStream objects. Input for streams that let you read content from another source and Output for streams that let you write content to some other source.

In the file example you have an input stream: istream. So how can you now, for example, write the contents you get from the input stream to another file?

var istream = io.readFileStream("myFile.bin");
var ostream = io.writeFileStream("myNewFile.bin");

ioStreamCopy(ostream, istream);

Usually these stream objects need to be closed once used by calling the method .close(). But this OpenAF's ioStreamCopy does everything for you.

There are more methods like ioStreamRead, ioStreamReadLines, ioStreamReadBytes, ioStreamWrite, ioStreamWriteBytes, etc…

Handling input streams

Let's assume that you are reading a huge csv file and you want to process each line:

var istream = io.readFileStream("mycsv.csv");
ioStreamReadLines(istream, (line) => {
    // Handle the line

    return result;
});
istream.close();

In this example the function ioStreamReadLines will read contents from the istream input stream until it finds the defined separator, new line ('\n') by default in this case. Then it calls the callback function with one argument: the entire line. When it ends the istream is closed since it's no longer needed.

What about the returned result? One of the benefits is that you don't have to read to the end of the stream/file. You can stop at any time. If 'result = true' the function ioStreamReadLines will stop reading from the input stream and return.

Note: there is an argument on the ioStreamReadLines function to provide a different separator than '\n'. Check "help ioStreamReadLines" on an openaf-console.

gzip/gunzip functionality in OpenAF

There are built-in functionality, in OpenAF, to apply gzip or gunzip to files or arrays of bytes. The functionality is available on the object io with the functions: io.gzip, io.gunzip, io.readFileGzipStream and io.writeFileGzipStream. They divide into two ways of using it:

To/From an array of bytes

The simplest way is to gzip/gunzip to/from an array of bytes:

var theLog = io.gunzip(io.readFileBytes("abc.log.gz"));
var gzLog  = io.writeFileBytes("new.log.gz", io.gzip(theLog));

The only issue is that the array of bytes (e.g. theLog, gzLog) will be kept in memory.

To/From streams

To address larger sets of bytes without occuring into memory spending, specially for large files, the are the stream based functions:

var rstream = io.readFileGzipStream("abc.log.gz");
// use the rstream and change it's content
// when ready to write just create a write stream
var wstream = io.writeFileGzipStream("new.log.gz");
// and use it to write to the new gzip file

Compress/Uncompress

To help store big javascript objects (even in memory) OpenAF provides two functions: compress and uncompress.

Of course the gains will be greater the bigger, and more compressable, the object is. Let's see some examples:

> var out = compress(io.listFiles("."));
> out.length
1959
> stringify(io.listFiles("."), void 0, 22).length
11674
> stringify(uncompress(out), void 0, "").length
11674

Of course objects are not stored in memory as their stringify version but, you get the idea. It's specific for cases when you need to keep an object in memory that you won't be acesssing on the medium/long term of the execution of your OpenAF script. Of course, it's also easy to save/load from a binary file:

> io.writeFileBytes("myObj.gz", compress(io.listFiles(".")));
> var theLog = uncompress(io.readFileBytes("myObj.gz"));

How to upload/download files from a Window/SMB share folder

The OpenAF's SMB plugin (or the SaMBa plugin, or the Server Message Block plugin) can be added by installing the "plugin-smb" oPack. It allows scripts to upload, download, remove and list files on a remote Windows/Samba share (depending on the user's permission, of course).

(note: it supports SMBv3)

How to install it

Just execute:

opack install plugin-smb

How to use it

After installing you need to include the SMB plugin on your code:

plugin("SMB");

Now you can create any javascript object instance to access a SMB URL on a given domain with the corresponding user and password:

var smb = new SMB("smb://my.server/myShare", "mydomain", "myuser", "mypassword");

Listing files on a share folder

To list the files on a specific folder you can use the .listFiles function:

> smb.listFiles("smb://my.server/myShare/aFolder");
+--------+-----+---------------+--------------------------------------------------------+
| files: | [0] |     filename: | a/                                                     |
|        |     |     filepath: | smb://my.server/myShare/aFolder/a/                     |
|        |     |         size: | 0                                                      |
|        |     |  permissions: | <all permissions>                                      |
|        |     | lastModified: | 1552671846146                                          |
|        |     |   createTime: | 1302948583597                                          |
|        |     |  isDirectory: | true                                                   |
|        |     |       isFile: | false                                                  |
+--------+-----+---------------+--------------------------------------------------------+
|        | [1] |     filename: | Thumbs.db                                              |
|        |     |     filepath: | smb://my.server/myShare/aFolder/Thumbs.db              |
|        |     |         size: | 99328                                                  |
|        |     |  permissions: | <all permissions>                                      |
|        |     | lastModified: | 1555318143997                                          |
|        |     |   createTime: | 1379322293251                                          |
|        |     |  isDirectory: | false                                                  |
|        |     |       isFile: | true                                                   |
+--------+-----+---------------+--------------------------------------------------------+

Notice that the filepath entry is the prebuilt URL you can use to list another sub-folder, for example.

Downloading a file from a share folder

To download a file you just need to provide the SMB URL:

> smb.getFile("smb://my.server/myShare/aFolder/Thumbs.db", "theOtherThumbs.db");
99328

It will return you the number of bytes transfered.

You can also use .getFileBytes to handle the download content as an internal array of bytes instead of saving to a file and .getInputStream to receive a stream to handle the download. Check the corresponding help information on the openaf-console.

Uploading a file to a share folder

To upload a file it's similar to the download, reversing the arguments for the .putFile function:

> smb.putFile("readme.txt", "smb://my.server/myShare/aFolder/readme.txt");
1235

It will also return you the number of bytes transfered.

As with the .getFile function you also have the functions .writeFileBytes to upload content directly from an internal array of bytes instead of a local file and .writeFileStream to upload directly from a stream. Check the corresponding help information on the openaf-console. There is also an extra argument to append content to an existing file on the remote share folder.

Removing a file from a remote share folder

To delete a file from a remote share folder just use the .rm function:

smb.rm("smb://my.server/myShare/aFolder/readme.txt")

Is defined or undefined

Two of the most common functions used in OpenAF are: isDef and isUnDef. The reason is because javascript variables, when created, are "undefined" and only become defined after a value is assigned to it. So, before using that javascript variable is common to test if it's defined or not.

> var abc
> isDef(abc)
false
> abc = 123
123
> isDef(abc)
true

Ok, what happens if it's undefined?

> var xyz
> isUnDef(xyz)
true
> String(xyz + 123)
NaN

Comparing maps and arrays

In javascript it's easy to compare simple types but the same doesn't apply to maps or arrays. Let's take some examples:

var a = {
  "name": "Line",
  "points": {
    "start": {
      "x": 5,
      "y": 10
    },
    "end": {
      "x": 15,
      "y": 25
    }
  }
};

var b = {
  "name": "Line",
  "points": {
    "start": {
      "x": 5,
      "y": 10
    },
    "end": {
      "x": 15,
      "y": 25
    }
  }
}

The map a and b are almost equal but if you compare them:

> a == b
false
> 1 == 1
true
> "a" = "a"
true

The reason is that a and b are different objects despite having the same keys, strutucture and values. The only way to compare is to compare key by key and value by value. In OpenAF the function compare does just that:

> compare(a, b)
true
>
> p.points.start.x = 0
> p.points.start.y = 0
>
> compare(a, b)
false

Using the openaf console you can actually see the difference:

> diff a with b
 -      "x": 5,
 -      "y": 10
 +      "x": 0,
 +      "y": 0

Comparing arrays

What about arrays? Well, they are also objects. Alone (e.g. [1, 2, 3]) or mixed (e.g. { a: 1, b: [ 1, 2, 3]}). So you can use the compare function in the same way:

> compare([1, 2, 3], [1, 2, 3])
true
> compare([0, 1, 2, 3, 4], [1, 2, 3])
false

Using OpenAF’s merge

OpenAF includes a base function to help on the task of merging maps. Let’s consider a simple example:

var source = {
   x: 1,
   y: -1
}

And you want to merge it with a map with the extra keys you need:

var extra = {
   z: -5
}

Now to merge the two maps simply execute:

> merge(source, extra)
{ 
   x: 1,
   y: -1,
   z: -5
}

Merging into arrays

Now imagine that instead of one map you have an array of maps? No problem, merge will merge with every single map part of the array of maps.

var source = [{
   x: 1,
   y: -1
}, {
   x: -5,
   y:-5
}];

var extra = { 
   z: -10
};

sprint(merge(source, extra))

// The result will be:
//[{
//   x: 1,
//   y -1,
//   z: -10
//},{
//   x: -5,
//   y: -5,
//   z: -10
//}]

Note: the keys of the second parameter map will always override the equivalent keys on the source (either in map or array mode).

Handling map keys spacing in templates

If you ever came across a similar JSON map to this:

{
   "Update"      : "2019-08-24T12:34:56.789Z",
   "Stats"       : {
      "Memory stats": {
         "Free memory" : 1234,
         "Total memory": 5678
      }
   }
}

You know that to access JSON map keys with space in javascript you need to do something like this:

print("Update: " + myMap.Update);
print("Free mem: " + myMap.Stats["Memory stats"]["Free memory"] + "MB");

To replicate in a HandleBars template you would write for the first line:

Update: {{Update}}

But, how to access the keys with space in HandleBars?

Update: {{Update}}
Free mem: {{Stats.[Memory stats].[Free memory]}}MB

So the final code would be, for example:

tprint("Update: {{Update}}", myMap);
tprint("Free mem: {{Stats.[Memory stats].[Free memory]}}MB", myMap);

Encode/Decode base64 in OpenAF

Base64 is a enconding scheme used to represent binary data in ASCII strings (using only 6-bit instead of the full 8-bit). It's used in several cases: LDAP's LDIF files character representation; encoding binary email attachments; embeeding images and fonts in HTML/CSS; avoiding delimiters been interpreted as delimiters inside a sequence of characters on a field: etc...

So, how to quickly encode or decode a string or an array of bytes to and from Base 64 in OpenAF:

> af.fromBytes2String(af.toBase64Bytes("This is a test"))
"VGhpcyBpcyBhIHRlc3Q="

> af.fromBytes2String(af.fromBase64("VGhpcyBpcyBhIHRlc3Q="))
"This is a test"

Keep in mind that the af.toBase64Bytes & af.fromBase64 return always a byte array. So in order to display it you need to use af.fromBytes2String to convert the byte array back to a string.

Of course, to convert a binary content you can just provide the array of bytes:

> af.fromBytes2String(af.toBase64Bytes(io.readFileBytes("openaf.ico")))
AAABAAQAgIAAAAEAIAAoCAEARgAAAEBAAAABACAAKEIAAG4IAQAgIAAAAQAgAKgQAACWS...