Update: Prototype code for Phase 1 of HotForestGreen almost done, implementing authentication

Posted on June 1, 2011


About 1 month ago I started coding the HotForestGreen framework, first from a Smart spaces point of view, to expand that to the use of any distributed system.

Below you find an update and some (more) backgrounds.

HotForestGreen: what and why?

First of all, lets start with the what and why.

In 2009 I built the first version of the “Roomware Socket Server”. This framework allowed me to create distributed applications, like: “using the phone as a remote control for applications running in a smart space”.

In the years that followed, my attention went elsewhere, new stuff happened and new stuff emerged and right now I consider the original “Roomware Socket Server” to be very limited and outdated.

“HotForestGreen” is a framework to simplify the creation of distributed applications. Using HotForestGreen you can – for instance – create multi-player games online, Smart Spaces you can remote-control using your laptop, tablet or smartphone.

Instead of one computer taking care of everything, you can run one or more computer applications on multiple computers, where each computer can have a very specific role in the Cluster that is formed by all.

Why do I do it – on a personal level?

I simply want the best, most simple to use kick-ass framework to create great, kick-ass, super scalable and mind blowing distributed systems.

I think it is great to build such systems and I think it will be the next big thing in the coming years.

As there is nothing available and nothing available that covers all my needs, I am building that kick-ass framework myself.

Why is this relevant? –  Solving the basic stuff for “the internet of things”

When you automate a building or create a multi-player game, your application is no longer running on one machine in one single environment. Multiple sensors take care of sensing stuff, including people entering doors and swiping RFID cards. Multiple devices take care of multiple things like opening doors and changing the light and temperature in a room.

So how do you deal with all the information requested, required and produced by all these things? How do you manage updates and requests from one machine to another or to many, sent over the network? How do you isolate stuff from tempering by users while still responding to input?

Currently – and as far as I know – there is no framework to deal with the complexity of such distributed systems in a way that is simple and scalable and not a “one trick pony”.

What is my goal with this project?

My goal is to:

  1. Create and release an open-source framework – Which can be used free and freely for non-commercial purposes which:
    1. Is easy to implement – So that you can have stuff up and running in 30 minutes
    1. Is guarded against all basic attacks – So that whatever you build at least will take (a lot of) effort to break and hijack by others
    2. Solves all basic issues for you – Like user authentication and sending updates on Remote Objects from one machine to all others sharing that same Object
  2. Reel in some commercial work to implement the framework – And have fun
  3. Make it easy for others – To start building simple and complex Distributed Systems

State of the project, June 1, 2011

I have been coding like crazy in the past weeks, using the off-hours and weekends and things start to take shape.

Phase 1: Building the basis – In progress

Before June 10, 2011 I hope to finish Phase 1, which covers the basics on data, data conversion and Client/Server and Client/Client messaging. This includes:

  1. Remote Objects – Sending and updating Remote Objects to all Remote clients
    1. Update: June 1, 2011
      1. Basic code is written. Not tested yet
  2. Server feedback – Including IP numbers of Servers serving specific sandboxes
    1. Update: June 1, 2011
      1. Implemented in very basic way. Returns its own IP number.
  3. Heartbeats and auto-reconnect – Including auto-discovery of services and sandboxes and auto-reconnects when a Client connection drops or a Server drops out
    1. Update: June 1, 2011
      1. Implemented, but very unclear process due to change of course. Not tested yet.
  4. User Authentication – Including a basic “plug and play” scenario for logging into SandBoxes and assuring messages sent from a Client are “signed” by the user Credentials
    1. Update: June 1, 2011
      1. Implemented of Messages started. Implementation of user authentication (login) process implemented in very basic steps. Not tested yet
  5. Data manipulation – Using a “plug and play” approach via: “Transparant Data Access Layers” to access, change and store data in databases
    1. Update: June 1, 2011
      1. Implemented and tested with direct database round trip within same Client. Tested with 4000 records, reading from database and injecting back into database with changed data.
      2. Implemented and tested injecting new records.
  6. Data Conversion – Including the conversions of Objects to “Comma Separated Values” vice versa, in order to:
    1. Send updates – On Remote Objects through the network
      1. Update: June 1, 2011
        1. Not tested
    2. Reduce code for working with Databases – As we can inject data from databases into objects and database records
      1. Update: June 1, 2011
        1. Implemented and tested with first test runs
When these main items are done, I can release the code and start building some proof of concepts, showing the possibilities of the framework.

Phase 2: Securing the framework – Not started yet

Phase 2 is about making the Framework more solid and more protected against people who want to hijack data and events.

  1. Server to Server updates – Including: “I am currently serving Sandboxes A, B and C with a total of N connections”
  2. Load Balancing – Including a smart distribution of Sandboxes so that the load to serve Clients is shared by many
  3. Server to Server security – Using handshakes and shared keys. Assuring that when a user plugs in a “Server” to intercept messages and sandboxes, this “Server” is refused by the network
  4. SSL encryption – Currently all data sent is in Clear Text, making it extremely easy to hijack and abuse the system

When these items are done, the Framework can withstand most basic attacks, can be scaled up by hot-plugging new Servers in the network and will continue to work even when multiple servers drop out.

Phase 3: Opening up the framework – Not started yet

While the basis is for specific use in a controlled environment, some things are missing to make the framework even more useful. These include:

  1. The use of Web-sockets – For Socket-communication over port 80 in environments behind firewalls
  2. Porting the code from C# to Java and ActionScript – To provide a broader use base, allowing you to run the framework on:
    1.  Android, Apple, Linux and in Flash
    2. Intel and ARM-based devices
Other possibly needed ports I will leave up to the community that will hopefully emerge. I simply do not have the time.

Phase 4: Testing and cleaning up – Not started yet

In Phase 4 all crap and remaining leaks will be filtered out of the system. During phase 2, most issues have been tested and addressed, but as a part of the process.

The work includes:

  1. Testing specific use-cases for performance – By running as much connections and messages as possible, until the system collapses
  2. Testing possible ways to hijack users and data – By scrutinizing the security model and implementing code that should break it
  3. Improving performance – By taking away data-hogs and

Real-time + Non-linear + distributed + asynchronous = mind bending

In the past 3 weeks I have been coding and designing much of the basics related to messaging from Client to Client, using the Server as a pass through station.

In this coding and design, the following aspects have to be covered.

  1. Real-time – All data flowing through the system can be real time data from sensors and devices and applications to anything, including databases
  2. Non-linear – Any event can happen any time. For instance: clients can already start firing data to sandboxes while sandboxes are not up yet. Still, data should arrive without any loss
  3. Distributed – There can be many clients and many servers. Working together, the servers spread the work-load and step in when one of their peers drops out.
  4. Asynchronous – While we are doing stuff, we do not wait for the answer, but continue for the next. Only when we receive some response back, we take action on the old request again. This include remote calls to remote methods.
What happens in the system as a total is basically this:
  1. A mess of messages are flowing around – Wired by the Servers and the references to the clients who sent them
  2. Applications and Clients observing specific messages in specific sandboxes – Making it easy to dispatch events and messages over the network
  3. Client responding to the content of messages – Which can be data converted to Remote Objects or Requests for a Reply – like in the case of a login or a remote request for data

Loss and lossless

The first version of the HotForestGreen framework will not be lossless. This means that in the following scenario “Client A” can lose data:

  1. “Client A” sends a request to “Application B”
  2. While waiting on the response from “Application B”, “Client A” loses its connection to the network
  3. “Application B” sends a response to the “old” “Client A”, which does not exist anymore
  4. “Client A” does not receive the data it requested for
To create a lossless data flow, data has to be cached somewhere. The first approach will very likely be Server Side. So when a Client drops out and reconnects, the missed packages can be (re) sent to the client.
Posted in: update