Total.js framework v4

The big day is here! Ten years of experience with Node.js and more than five months of hard work brings this great new version of the most powerful Node.js framework in the world. This big day is not about the framework only, and we are getting new documentation and new versions of some Total.js products.

• Website: www.totaljs.com
• Documentation

https://www.youtube.com/watch?v=Gc2tStqKtBA

As the author of the framework - I wanted to keep utterly backward compatibility, so I created a new NPM repository called total4. I want to maintain the older version of Total.js because it's part of many projects. Total.js 4 is based on the Total.js 3 core with more significant modifications, so many older projects could work without any changes.

NPM installation which you will love:

What's new?

• New documentation
• New version of Total.js CMS
• New version of Total.js Code
• New version of Total.js SuperAdmin
• New version of Total.js Cloud
• New version of Total.js OpenPlatform (coming soon)
• New Total.js API services
• Redesigned website

Cleaned code

I removed many useless code lines, and some of the existing functionality has been much improved and simplified.

• removed ton of older code
• much simple coding
• much faster coding
• improved performance
• improved security
• completely rewritten NoSQL embedded database (engine is called TextDB)
• improved routing
• new documentation
• new version doesn't affect existing Total.js apps
• improved SMTP server
• new Total.js documentation

Unified routing

I unified Total.js routing for all types of routes like classic Routes, Files, or WebSocket. Now it is much simple to create routes.

• Documentation

ROUTE('GET     /',      action);
ROUTE('POST    /',      action);
ROUTE('PUT     /',      action);
ROUTE('PATCH   /',      action);
ROUTE('DELETE  /',      action);
ROUTE('SOCKET  /',      action);
ROUTE('FILE    /*.jpg', action);

Improved routing to schemas:

ROUTE('GET     /api/users/        *Users --> query');
ROUTE('POST    /api/users/        *Users --> check insert (response)');
ROUTE('GET     /api/users/{id}/   *Users --> read');
ROUTE('PUT     /api/users/{id}/   *Users --> query');
ROUTE('DELETE  /api/users/{id}/   *Users --> remove');

New API endpoint

I have designed a new API endpoint that will work with our client-side library called jComponent or between Total.js applications. New API endpoint is targeted for better handling data with web applications, it's faster, and the significant benefit is encryption/decryption between server-side <-> client-side.

• Documentation

ROUTE('API    /api/    -users_query       *Users --> query');
ROUTE('API    /api/    +users_insert      *Users --> check insert (response)');
ROUTE('API    /api/    +users_update/id   *Users --> update');

Improved routing algorithm

This new version has completely replaced the routing algorithm with a new one. The new algorithm is much faster and more accurate. It's used for classic routing and WebSocket routing.

Improved schemas

Also, Total.js Schemas have been updated and improved too. A lot of functionality is retained. IMPORTANT: now, the model doesn't contain any internal methods or hidden values. It has only values prepared according to the schema.

Removed:

• transformations
• hooks
• operations

Added:

• operations connected to NEWOPERATION()
• tasks connected to NEWTASK()
• a model as the second argument in all operations with excepts setQuery() and setRead(), example:

NEWSCHEMA('Users', function(schema) {

    schema.define('name', 'Name(40)');

    schema.setInsert(function($, model) {
        model.id = UID();
        DBMS().insert('tbl_user', model).callback($.done(model.id));
    });

});

I have added a new EXEC() method, which can call schema operation. It's a fresh alternative to the older $ACTION() method.

• Documentation

EXEC('+Products --> check save (response)', { name: 'Total.js' }, function(err, response) {
    console.log(err, response);
});

EXEC('-Products --> read', function(err, response) {
    console.log(err, response);
});

Or another example:

var controller = EXEC('-Users --> query', { name: 'Total.js' }, function(err, response) {
    console.log(err, response);
});

controller.query = { fields: 'id,name' };
controller.user = { name: 'Peter', sa: true };

Total.js builds

I have added support for creating Total.js application by Total.js AppBuilder - visual interface for creating server-side Total.js applications. We will bring new information in a separate blog post soon.

Embedded TextDB

Big changes contain Total.js NoSQL embedded database. I have renamed it to TextDB, but I keept method names NOSQL() and TABLE(). This little big canon contains a new core and by default is processed in its own child-process (another thread) if the app opens a web server.

TextDB is more stable than the older NoSQL embedded database and finally supports INNER JOIN. I have removed a lot of useless functionality from TextDB compared to NoSQL. You will love it.

The big new is internal an in-memory mode. The database is loaded into the memory if the database's size is less than the defined limit in CONF.textdb_inmemory option (by default, this behavior is disabled). This behavior can speed up the performance of reading.

New FileStorage

I completely rewrote FileStorage. New FileStorage is fully optimized for the Cluster and now the FileStorage store's files according to Total.js UIDs. Files are stored in directories in which names are generated from UID identifier. A main FileStorage directory can contain maximum of 9999 directories, where each directory can contain thousands of (uploaded/stored) files.

• Documentation

var mynewfileid = UID();

FILESTORAGE('uploads').save(fileid, YOUR_READSTREAM_OR__BUFFER__OR__FILENAME, function(err, meta) {
    console.log('STORED');
});

New unit-testing mechanism

I thought about creating a better unit-testing as never before, and I have added the so-called Route testing algorithm. With route testing, you can quickly check if your route is working correctly or returning unexpected output. Therefore, I have extended the controller by adding a new controller.runtest() method, which performs route tests without creating of real request.

• Documentation

// Example
// File: /controllers/test.js

exports.install = function() {

    ROUTE('GET /customers/         *Customers --> query');
    ROUTE('GET /customers/{id}/    *Customers --> read');

    // Testing route
    ROUTE('GET /test/', test);

};

// Action which performs unit-testing
function test() {
    // this === Total.js Controller
    var self = this;
    self.runtest('GET /customers/', 'Customers --> @query');
    self.runtest('GET /customers/123/', 'Customers --> @read'); // Invalid Id
};

Output:

Replaced algorithm for file upload

I replaced the existing algorithm for uploading files. The older version of Total.js uses Formidable algorithm, so my new algorithm called "Total Siro" increases the performance by around +15% in the processing of multipart/data.

Good to know: In the view of development is nothing changed.

Updated cluster

I have updated Total.js Cluster by adding an exciting feature called auto scale mode. This mode can automatically create new threads when the application is overloaded (if the thread doesn't respond under 70 ms, then the main tread opens a new thread). If some threads are idle, then the main thread removes them.

I have improved communication between threads. Now you can an emit event within each thread via EMIT2(event_name, [arg1], [argN]) method, example:

// Captures event
ON('myevent', function() {
    console.log('SOM TU');
});

// Emits the event in each thread
EMIT2('myevent', 'arg1', 'arg2', 'argN');

You can enable cluster mode very easily. Just use our prepared starting script and uncomment the cluster option:

• Starting script: index.js
• Example: Total.js Cluster

Improved Threads

Total.js Threads are targeted for high-performance web applications, and they are our vision of Total.js server-less architecture. We have added full support of Threads to the Total.js framework without using reverse proxy per each thread (now the main process is used as a reverse proxy). Internally communication is provided via Unix Socket so that threads won't work on Windows.

Structure of threaded app:

Threads support cluster with our awesome auto-scaling feature or fixed count of threads. That means if some thread endpoint is overloaded, then the main process will launch another thread for the thread which is overloaded. Total.js framework kills unused threads automatically.

• Documentation
• Example

The performance is really unreal. In our test, we transferred around 12 GB of data in 2 minutes between 4 computers, and threads open only six threads per endpoint. The main thread took only 70 MB of memory (the main thread is used as an internal reverse proxy), and other threads took between 150-300 MB of memory.

Replaced algorithm of FlowStream

The new FlowStream algorithm is similar to Total.js Flow. I have added a parser for components declared in HTML syntax. Look to example:

<script total>

    // Server-side code for FLOWSTREAM

    exports.name = 'Battery';
    exports.icon = 'fal fa-battery-full';
    exports.config = {};
    exports.inputs = [{ id: 'message', name: 'Message' }];
    exports.outputs = [{ id: 'output', name: 'Output' }];

    exports.make = function(instance) {

        instance.message = function($) {
            var data = $.data;
            var value = data.percentage || 0;
            $.send('output', value);
            $.status(value);
            $.dashboard(value);
        };

    };

</script>

<settings>
    UI for settings
</settings>

<style>
    /* Custom styles */
</style>

<body>
    Component body for UI
</body>

<script>
    // Client-side script
</script>

• FlowStream documentation

Versioning

We are moving to partial updates in versioning like in our client-side library - jComponent (not breaking changes if everything will work correctly). That means that we will increment a version of Total.js 4 continuously when performing some changes.

Migration

I have prepared several steps for migration from older versions of the Total.js framework. But my personal opinion is: don't change older projects if they work correctly. We will upgrade Total.js 3 too, with fixing of new bugs or security issues.

• Migration to Total.js 4

What's the future?

I believe (as the author of the framework) that this version of Total.js framework will be like LTS version for the next 3-5 years. But never say never, and we will see.

Total.js Apps Monitoring

Total.js 4 offers better and precise statistics than Total.js v3 for Total.js Apps Monitoring. Total.js Apps Monitor monitors primary data of the process - like memory consumption with usage in % and all Total.js internal statistics like requests per minute, errors, overloads, WebSocket connections, etc..

Apps Monitoring:

• Follow install simple instructions
• Open Total.js Apps Monitoring

What's next?

Enjoy the new version of the Total.js framework, and let's code something extraordinary.

• Visit new webiste
• New documentation
• Sign in to Total.js Platform
• Commercial support
• Join Total.js Telegram

• Follow Total.js on Twitter
• Follow Total.js on LinkedIn
• Follow Total.js on Facebook
• Join Total.js newsletter