Remote Objects within HotForestGreen are:

1. Shared – Over all clients who are connected to the same Application Cluster and stated their interest in them
2. Automatically updated – When one or more Remote Objects on some other Remote Client send their update to the Application Cluster

The Remote Objects are created and sent with the following starting points and concepts to take into mind:

1. Remote Clients -Each Application connected to the Framework is called a Remote Client
2. Distributed Applicatons – Instead of one application performing all tasks and containing all code, you can create many small applications, each performing specific tasks and each running on multiple systems. The collection of all these local, small and independent applications is called a “Distributed Application” where “distributed” is used in the sense of “distributed over multiple places”. Within these Distributed Applications, all independent applications work together as a team: where each team member performs a specific task, does not specifically needs to know who else is there (limited awareness) and the team as a whole delivers the total end result
3. Application Clusters – Each of your Applications or Remote Clients run in an Application Cluster. The Application Cluster
4. Remote Data Manager – All Remote Objects are managed using the Remote Data Manager. This manager is explained later in this document
5. Bi-directional Socket Connection – Remote Objects are sent over bi-directional Socket Connections
6. Message based – Each update on a Remote Object is sent in a Message over the bi-directional Socket Connection. This message has a Message Header and a Message Body.
1. Message Header – The Message Header is a serialized String Array with several parameters, put in a specific and fixed order of appearance. The header in this setup has a relatively low overhead for de-serialization
2. Message Body – The Message Body can contain any content in any structure of your liking. By default the Framework will use its internal Serialization Format which is optimized to reduce overhead on serialization, de-serialization and the number of bytes sent over the network. More about this later
7. Local Data Repositories – Each Remote Client will keep a Local Data Repository with all Data Objects
8. Data Hooks – To define which local Object Class of objects will receive what data, you register that physical Object Class to a Data Hook to the Server. Each event on that Data Hook will be pushed to objects in that Object Class
9. Distributed Event Model – Each Remote Client in the Application Cluster registers liseners to specific Data Hooks to specific Application Sandboxes in specific Application Clusters. When some Remote Client sends an update on a specific Remote Object over a Socket Connection to the Application Cluster, the Application Server will dispatch this update to all other Remote Clients registered against that Data Hook. Each registered Remote Client will then automatically receive a Data Push
10. Serialization and de-serialization – To send data over the Socket connection, we serialize and de-serialize that data using a common and shared format. Internally the Framework uses “Solid Comma Separated Values” compareable to CSV but with a more solid separator for items/fields/columns and rows. Apart from the serialization format used in the Framework you can use your own via your own custom Data Serializers
11. Data Serializers – As stated HotForestGreen uses an internal format to Serialize and de-serialize data. This process is done via a Data Serializer which is pluggable: meaning that you can create your own Serializer/De-serializer. For the Framework, data is data and as long as all your Remote Objects of that specific class use the same Serialization / De-serialization format it does not matter what fomat you use.
12. Binary data – Bottom line: all data sent over whatever type of Socket connecton is binary data. For most Socket connections and for the Framework this is without any problem
13. WebSockets – HotForestGreen supports WebSockets. WebSockets from version XX support binary data when sent in a separate package.

The Remote Data Manager

Basis:

1. Application Clusters – Application Clusters are the sum of all Remote Clients running in a specific Application SandBox
   
2. Application SandBoxes – All Remote Data and Remote Objects run in specific SandBoxes. Each SandBox is a closed environment in which objects and data runs independantly from other Application SandBoxes in your Application and Remote Client. Assumed is that (Remote) Objects in “SandBox 1″ are not the the same as (Remote) Objects in “SandBox 2″. This means that when “Object 1″ in “SandBox 2″ receives an update, “Object 1″ in “SandBox 2″ will remain unaffected
3. Multiton Instances – The Remote Data Manager is a Multiton, where each Instance of a Remote Data Manager manager the data of the specific SandBox represented by that Remote Data Manager
4. Persistancy of Remote Objects – Each Remote Object is by default a Persistent Object. This means that when “Person 1″ in “Sandbox 1″  is sent to all Remote Clients — instead of creating a new “Person 1″  each time it is received– all local Objects on all Remote Clients in “Sandbox 1″  in your Application Cluster representing “Person 1″ will automatically be updated by the Remote Data Manager. To do this, each Remote Data Manager holds a Local DataMap
5. Local Data Repository and Simple Object Database – Each instance of the Remote Data Manager holds a Local Data Repository of all Remote Objects registered against a specific SandBox. This Local Data Repository forms the bridge between your local objects and the Remote Data that is received over your Socket Connections. So when the Remote OBject “Person 1″ is received, it will first be resolved against the Local Data Repository. If it already exists, that specific object will be updated. If it does not exists, a new Local Object will be created and added to the Local Data Repository for all future updates
6. Messages, Observers and Remote Event Dispatches – As stated before, the Framework uses Messages and Remote Event Dispatches. Each Remote Client and each Local Data Repository is in the end an Observer on specific Events dispatched against the HotForestGreen Servers
7. Local Events and Local Event Dispatches – To create a simple basis for updates, the Remote Data Manager offers several hooks to Observe Events on Remote Objects. These hooks are:
1. Added – A Remote Object is added to the collection
2. Updated – The Remote Object is updated somewhere in the Application Cluster: either locally or on another Remote Client
3. Removed – The Remote Object is removed from the stack

Serialization and de-serialization of Object Data

This topic is also covered in other posts:

1. Formats to serialize to – Here. About considerating and dropping the use of CSV, JSON and XML and moving towards the SCSV (Solid CSV) format used in HotForestGreen

Basis of serialization and de-serialization

1. Sending “objects” – When you send “objects” over any connection, what is really sent is information within and about that object. This information includes:
1. A reference to the object/object class – In order to de-serialize, you need to know what your basis is.
2. The content of that object – Being values in variables and other objects nested within that object
2. Serialization – When you serialize an object and object structure, you basically iterate through that object (and any sub-object) and read and store all found values into some kind of format. XML could be one. JSON another, your own format yet another. In the end, all objects store basic primitives like Strings, Numbers and Boolean values and your object structure is nothing more than that: a structure to hold your data
3. De-serialization – When you de-serialize data, you translate your “flattened” data structure into a (new) object structure that resembles the original you serialized. To do so you need:
1. Object Classes – That mirror the Classes (containing the structure) of the objects you serialized. For instance: the class “Person”
2. Shared Class References – To resolve the right Object Class to instantiate when re-creating the object structure that was serialized. In most cases this happens by mapping: MyClassMap.addClass(“PersonClass”, my.classes.Person.class() ). When we serialize the data, we can include info on the object Class we serialized. For instance: <object classreference=”PersonClass” >. Where “PersonClass” can be resolved into the Class: Person
3. References to map the variables – To inject the values from the serialized dataset into your object, you need to know where to put that data. For instance: <firstName>John</firstName> will do that trick, or: <object classreference=”PersonClass” firstName=”John”>
4. The code to do the de-serialization – In most cases this is done by iteration and reflection, using the structure of your serialized data as starting point: person[ dataItem.name ] = dataItem.value

Data Maps and Persistancy of Remote Objects

1. One single object for one Date Item – When you send updates “person 1″ to all Remote Clients, you want those updates to happen in the one single Data Object you created for “person 1″ on each Client
2. Using Data Maps and Object IDs – To find objects already created, HFG uses Data Maps to store the objects we create and receive on each type (“Person”, “Car”). This Data Map has the following layers:
1. SandBox – The SandBox in which the data is running. “Person 1″ in “SandBox A” is not the same “Person 1″ as we find it in “SandBox B”
2. HFG Class Reference – When we store the Object Reference, we do that based on the HotForestGreen ClassReference. For instance: the class “Person” can be stored under “PersonClass” for “SandBox A”. “Car” can be referred to as “CarClass”
3. Object ID – This is the unique ID per object, either created by HotForestGreen or by the Data Provider (a database or your code)
3. Mapping Remote Objects – A Remote Object is mapped on the following parameters:
1. Class Reference – We need to know to which Class it has to be de-serialized
2. Object Identifier – The name of the variable that contains the Object ID  Record ID
4. Resolving Remote Objects – When a Remote Object is received, we use the SandBox ID to access the datamap containing separate Data Maps for each HFG Class Map within that SandBox. Once we have resolved that, we have the final Data Map with references to all existing Data Objects, using the Object ID as the Key to resolve that object from the Data Map
5. No object found – When no object with those keys are found, a new instance will be created by the Remote Data Manager and injected into the Data Map. That new object will then be offered for de-serialization of the data
6. De-serializing the data – When the DataMap Manager returns an Object corresponding the given keys, the de-serialization process can start and the values from the Remote Object can be injected into our local representative

Sending any (registered) object

1. No specific Extentions needed – In an older version of the Framework you had to extend the Remote Object Class to make any object a Remote Object. This process has been removed.
2. Mapping Local Classes against Remote Object Class References – As your Class Structure can differ between projects using and Sharing the same Remote Objects you do need some kind of common name to make the mapping / injection possible. This is done by mapping your local Class agains the Remote Object Class Reference. For instance: “your.project.classes.Person” is mapped against “PersonClass” in “SandBox A”. When you receive Remote Object Data with Class Reference “PersonClass”, your mapping will resolve that into your Class: “your.project.classes.Person”
3. Knowing your SandBox – One thing you need to do and continue to do is keep tabs on your SandBoxes. There are two ways to do that:
1. Manually – You code it into your application. This is the most effecient way for your resources, as you do not need extra processing time to resolve the SandBox
2. From a Map – Each Object you receive and create can be used as a key to map the SandBox it was created for. The advantage is that your application will keep track of where what Object belongs without your further help. The disadvantage is that this costs resources in memory and time to resolve the SandBox from the Map you use
4. Registering persistant objects against a Object/Sandbox Map – When you want to use maps to register where your object belongs, you use:
1. RemoteDataManager.mapObject( yourObject, yourSandBox )
5. Sending your object to your SandBox – Your object can be sent in the following ways:
1. Using all parameters – This is the fastest way as the Framework does not have to waste extra time resolving all implicit data:
1. RemoteDataManager.sendUpdate( yourObject, theSandBox_ObjectClass_ReferenceName, yourSandBox).
2. Using the SandBox name – When you only use the SandBox name the Framework will try and resolve the SandBox ObjectClass Reference Name from the Class/Class Reference Name Map you made for that Object, for that SandBox.
1. RemoteDataManager.sendUpdate( yourObject, yourSandBox).
3. Using the Object – When you built an Object/SandBox Map, the call is even shorter in code:
1. RemoteDataManager.sendUpdate( yourObject )
4. Using an Extention on the RemoteObject Class – Using / extending this Class, you can tell each Object explicitly to which SandBox it belongs and with which Remote Object Class Reference Name it is associated. The call will be moved to the Remote Object itself, to keep coding short and simple:
1. yourObject.sendUpdate()

Sending lists of Remote Objects

1. Using an Array or Collection – To indicate which Remote Objects you want to send in a List, you place them inside a Array or Collection
2. Sending the list via the RemoteDataManager – As the Array itself does not hold info on where or how to send the objects you use the RemoteDataManager directly, even when your Remote Objects are Extended form the RemoteObject Class. Like this:
1. RemoteDataManager.sendList( yourCollection, theSandBoxClassReferenceID, yourSandBox)
3. All objects of the same Type – You can not send a list of mixed types as the Framework will serialize the objects as if they are from the same Class

Serializing nested objects

1. Any type – Nested object can be of any type, including collections of other Objects
2. Single Objects – Any object nested in another object is serialized as a sub-set of that parent object.
3. Collection of primitives – Primitives like String values and Numeric values are serialized as a sub-set of that Data Object and placed inside the DataRow of the object containing that collection of primitives
4. Collections of Objects – Collections of objects inside an object are not serialized by the Framework. If you need to send collections, you can do two things:
1. Hide the actual Collection and share references – Store the actual objects into a private Collection and share a pubic reference list using String values or numbers to resolve the actual objects on the Remote Client side.
2. Create / download your own Serializer – As the Serializer/de-serializers are pluggable, you can create, inject and use your own versions.

De-serializing nested objects

1. Limited implementation – The Framework will offer you a very limited implementation of Serialization/de-serialization of Nested Objects. The main reason is that:
1. In most cases more is not needed – In most cases you do not a very elaborate Deep Serialization
2. In those cases you need it: it is not good enough – In the cases where you DO need Deep Serialization/Deep de-serialization, your own solutions will be much better than what the Framework might be able to provide
3. Plug and play: download or write your own – The Serializer / De-serializer is a pluggable system. You can inject your own Serializers / De-serializers, state in the Message Headers which specific one you used and skip whatever the Framework might have provided you with your own (downloaded) awesomness
2. Single Objects – Using a object- and class reference inside the Main Object, the Remote Object / Data Object that is the result of de-serializin the sub-set belonging to that nested object is put into the Main Object. This process can repeat itself for nested objects inside nested objects
3. Collections of Objects – Collections are de-serialized as a sub-set of that parent object and placed in the collection object of the Main Object. Any existing data in that Collection is replaced in that process. Object references to Data Objects / Remote Objects like “Person 1″ will be pointing to the exact same object “Person 1″ as before the purge and replacement of the existing Collection
1. Use references to resolve the Objects – If you sent references, the Framework can use the Local Data Map can use to resolve the actual objects. You will need to write your own setters or mechanisms in your Remote Objects to do the actual work, as the Framework will not do that for you
2. Use your own process – See “Plug and play: download or write your own” in item 1: “Limited implementation” of this list
4. Maps – Maps can not be serialized by the Framework. As they can be built up in any way possible, with any type of key possible, they form a potential danger for overhead in both coding and processing. Short and simple: avoid Maps.
5. Collection of primitives – Primitives like String values and Numeric values are de-serialized and will replace the existing Collection. If your programming language treats primitives as Objects, like Java does, the object references to the String with value “String 1″ will not be the same as the Object reference of the previous value “String 1″

Serializing arrays of primitives like Dates, numbers and Strings

1. End points – Any object that can be considered an “end point” will be translated to a String value representing that value.
1. Date/Time – For date/time a standard encoding will be used that can be de-serialized in any coding language. The de-serialization is part of the Framework
2. String values – In most languages, Strings are Strings and not Objects. The framework will simply translate Strings to a String value
3. Numeric values – Numbers can be translated to String values and will be so. The decimal-separator will be a dot

Receiving and processing Remote Objects

1. SandBoxes and Sockets – For security reasons and scalebility, each SandBox runs in its own and explicit Socket Connection.
2. Messages – Each update on Remote Objects is sent in a Message. This Message contains:
1. A Message header – Which has been discussed earlier and contains all meta-data for the Server and the Remote Clients
2. A Message Body – Which contains for SCSV:
1. One Header with field names – Used to map the values to the variables on the local instances of the Remote Objects
2. One or more DataRows – As a Message can contain data for one or more Remote Objects, we package them in DataRows. Each DataRow contains data for one instances of one specific Remote Object
3. Headers and Meta Data – Each message we receive on our Sockets contain Meta-Data. This meta-data includes:
1. Remote Object Class Reference Name -  This is a generic and shared key over all applications using these Remote Objects, to resolve the Serialized data to the right Class
2. SandBox name – Even though we run data within SandBox specific Sockets, we send also the SandBox name with the header.
4. Automated process - An Extened Remote Object is received and handled automatically. Using the local SandBox and the Remote Object Class Reference Name, the Framework will (try and) resolve the local instance of the Remote Object and once created or found, it will inject the values as sent in the Message Body into the corresponding Local instance of the Remote Object

Optimizing sending lists by using Challenge/Response

1. Challenge: Remote Client makes generic Data Query – For instance: “give me all people with first name ‘John’.”
   
2. Response: Sending IDs only – As some, or even most of all Data Objects covering that Query might already be in possession of the Remote Client, sending all Objects again might be overkill. Instead, the first Response can be to send only the IDs of each Data Object, so the Remote Client can refine its request for those Objects not already in the Local DataMap
3. Challenge 2: Requesting specific DataSets – As soon as the Remote Client has resolved which IDs are not yet present in its Local DataMap, it will request the specific dataset by stating the specific Object IDs it requires.
4. Response 2: Sending requested data – Based on the given Object IDs, the Receiver of the Request will send the missing Data Objects
5. Resolving the Data Objects – Once the data is received, the Remote Client will resolve the Data Objects into its local set of Remote Objects

Optimizing Remote Object Handling

1. Explicit coding – The more explicit your coding is, the less the Framework will have to resolve and the faster your processing will be
2. Extending the Remote Object Class – This class allows you to state explicitly per Object to which Class and SandBox it belongs, keeping your code short and simple
1. Instantiating an Extended Remote Object – A Remote Object is instantiated in the following way:
1. myRemoteObject = new ExtendedRemoteObject( sandBoxName, sandBboxClassReferenceName )
2. Sending an Extended Remote Object – An Extened Remote Object is sent in the following way:
1. myRemoteObject.sendUpdate() ;

Here we go:

HotForestGreen seems to sit on a perfect intersection between the Maker Movement/DIY shops and the explosion of low cost Android based devices.  Can you briefly speak to the vision behind the project and why you decided to go the Open Source route?

I will start with the vision.

When I started developing “HotForestGreen 0.1 Alpha” in 2008 I believed that: “within 5 years, the ‘tangible web’ and interactive spaces will be the next big thing”, using simple RFID tags like the Oystercard and the Dutch “Public Transportation Chip Card” as one of the means to start interaction.

In 2008 I felt strongly that that time was now. That this new thing we read about and talked about for years was about to happen. Mainly due to the emergence more and more of low cost hardware and things like RFID cards for things like public transport and due to the increasing availability of all parts (like RFID readers) to fill in the gaps. Because all parts needed to make it work had become mainstream or were about to become mainstream or at least available for consumers like me and you.

I also had this chain of thoughts unrolling like a movie where all unwanted consequences unfolded of a society where your every action and interaction with any of these systems become traceable and recordable.

Rather than to wait for big parties to pick this up and develop centralized, closed and closed-sourced connected solutions to a closed set of private parties and for big parties (think “Real World Facebook”) I would love to see something more open and more available for anyone to build on and use.

Apart from the sheer boredom and dread I feel related to the limitation of a centralized vision, I shun away from the lack of innovation any centralized vision has as a byproduct. There are simply too many people involved with too many different agendas.

To give you a simple example: when the Web emerged, AOL and the likes assumed that Catered Portals were the start and endpoint of your Internet experience. These portals would be managed by their people, with the content they thought would be interesting for you. To keep it short: If the web would have stayed like that sites and services like Amazon, Facebook, WikiPedia and Twitter would never have emerged on the web the way they did now: because the internet as we know it has been created in an open garden.

I want the ‘tangible web’ (where whatever happens online in your personal data-cloud can be presented in a physical space) to become reality by the power of the people. To become reality with things people created in their own homes. I want to see the ideas that really kick ass emerge into the mainstream with the potential to really change and improve part of our lives in ways we can not imagine now.

Android is only one point of entry, as it meets most of the items on the list below:

The most important goal to me is to make the project and the experiments that will follow easy and even inviting for others to reproduce and to use as a starting point for whatever you want to do yourself. Android, with more and more phones and tablets moving towards the 100 euro threshold is one point of entry. Low cost Netbooks (200 euro) is another.

RaspberryPI is yet another project I am watching. When you see that tiny, low cost $25 ARM11 computer running Quake 3 under Linux on a big screen (see here), that is quite impressive.

More than Android devices I think RaspberryPI will be an exploding supernova among tinkerers.

When you look at my priorities in development and platform choice, I use a simple funnel: Most people understand MS Windows and Android and Netbooks and phones and tablets and what you can (potentially) do with that. Lesser people get Linux. Even lesser amounts of people understand projects like Arduino and how to overcome the limitations of that platform. Apple for me is more and more a closed platform with (for the iDevices) a programming language I do not have the time to learn. For visual stuff, Flash still is the best choice to go for me, as I used that platform for several years and it runs on Android, Windows and Linux. So that is what my focus and prioritization is based on in the development of the HotForestGreen platform and the HFG demos that will emerge in the next months.

The project is Open Source for several reasons. One is related to the vision above. I simply want that “open garden” to emerge, as unstoppable as it happened for HTML and web development. Another is my lack of business skills. I tried to start several companies in the past 20 years and all that is needed to run a business simply does not match with my mental DNA. Instead of trying to keep things closed and secret and try to become one of the “five” major players in this field, I simply build, experiment, document, bring it to the open and see what emerges and whatever like-minded people I find.

HFG can be used for a wide range of Internet of Things developments, but can you give some examples of  projects or people that this framework would ideally be suited for?

Home automation is one. Think of DIY Home State monitoring where every Event and State from doors and windows and stuff inside the house (fridges, heating, room temperature, lighting) can be checked and recorded and changed. Where you can check if the light in the bathroom is still on from the bedroom and switch it of using either your phone, tablet or laptop as all connect to the same shared interface.

Another is Real Time Collaboration in Application Clusters where what I do can be shared with what you do and where we can work together on the same things from different locations. One practical example in this is the care-industry (dentists, home doctors, hospitals) where people at the reception can prepare your workstation at the place where you treat your client so that you can focus on the client without the distractions of the system.

On a bigger scale you can think about care-centers where people with disabilities and/or of old age can be monitored so that — when something goes wrong — helpers are immediately notified when something seems wrong and help is only 30 minutes away instead of: “whenever you can reach that panic button at the other side of the room”.

Also on this and even bigger scale can be construction sites and factories where multitudes of low cost sensors observe the environment and potential hazards and in which you are alerted when your current or future location can become an (immediate) hazard due to some changes or (system) failure or due to some vehicle rushing unseeingly your way.

On a small scale, it can help people to setup very scalable Real Time systems in a very short time. Think of a chat-box as one of the most primary and simple implementations and any of the examples above when you needs or plans are more ambitous.

Looking at the target audience: the framework is designed to cater beginners with simple “Hello World” Real Time applications running on a Netbook to the very advanced users who need Load Balancing and multiple Clusters running on multiple locations all over the world, in several layers over encrypted connections with fallback-scenarios and alternative routes for “Plan B” and “Plan C” and fallback scenarios for several HFG Server Failures where not even one byte of data should get lost during those events.

You have recently been working on implementing WebSockets in the project. Do you have any other favorite or important project features that you could share more about? 

One of the most important next steps is to create Virtual Representants of Arduino and XBee modules. A Virtual Representants represents the real object on any remote client “as if” it is that real object. They are something like the marriage of a Proxy and a Mediator.  Once the framework is finished it has to be very easy to start working with either Arduino and XBee (and any other device I find time to represent in code). What these Virtual Proxies involve is the following:

What’s next?  What would be the best way for readers to find out about the latest developments?

I follow Engadget and Hack a Day at this moment. They represent about 80% of all stuff that bubbles up of what I might find interesting. Apart from that, I use Google. Depending on my need at that point and the things I want to build, I search the web until I find what I need.

Could you go into anymore detail about the next stage in HFG development?  i.e: possible release dates or if you are looking for any contributions from other programmers, etc?

Flu struck me this week. Other priorities – when returning from Italy to Holland – last week. Holiday the week for that.

In the holiday I studied Design Patterns, Intellectual Property and Open Source licences to increase my knowledge on two fields I know just a little about.

Just before holiday I finialized the first proof of concept for WebSockets, establishing a two way Socket connection with Remote Objects in JavaScript, running in Chrome.

Design Patterns

Whether I use them or not, to know what they are and what makes what pattern what is quite relevant I think. It will improve my own coding and the results you will download when you want to contribute on this project.

To assure I will own the knowledge I started to map them and write blog posts on Design Patterns.

Intellectual Property and Open Source Licenses

Choosing which Open Source License will apply will also determine how useful the code will be for others and how many people will not contribute.

The goal is to go for a licence that is as open as possible, allowing both open and closed projects and both commercial and non-commercial projects to happen, where the only restriction is that you respect the authors and the work itself.

Git or SVN?

Many projects I know have moved from SVN to Git repositories. Apparently Git is the new shit.

Instead of sticking to what I know, I decided to drop SVN for Git.

I have been considering maintaining two repositories, by mirroring the base libraries to a Git and SVN repository, but SVN for sure will freak out when files suddenly disappear while they are still registered in the local .svn lists. So Git it will be. And ZIP files for those who want to just download the code base.

WebSockets

As you might have read, I consider WebSockets to be of importance for the framework. First of all, because it allows you to build applications in the browser, without the need of any compiler or special IDE.

Second, WebSockets might help solve the issue of specific settings on Firewalls, blocking any traffic that is not HTTP or FTP.

I run some successful tests in Chrome with a simple chat and tracing system, showing me connection state and events and activity on the Browser side.

Next steps

I hope to pick up steam again in the next week. The total of three weeks not being productive on HotForestGreen was not foreseen.

The new deadlines are now set to:

1. September 1st – Release of the C# and JavaScripr code base and several demos to get started
2. October -

Older Status Updates

• June 1: Prototype code for first phase of HotForestGreen almost done, implementing authentication
• June 13: First round trip successful
• June 19: Gettign a grip on all things considered. After a round of high-impact refactorings, things got to the point of functional again. Also contains a lot of background info on SandBoxes, the new security model, Systems of Trust and the revised Auto Connect system.
• July 5: First tests with Remote Objects in the refactored code. After a thorough refactoring, I tested if the basis is still working.
• Agust 3 – This post

What is this Framework about?

See the post of June 19. See also the “About” page here.

Update on roadmap

Phase 1: Building the basis – In progress per May 4, 2011

Before June 10, 2011 I hope to finish Phase 1, which covers the basics on data, data conversion and Client/Server and Client/Client messaging. This includes:

1. Remote Objects – Sending and updating Remote Objects to all Remote clients
1. Status: June 19, 2011
1. Basis is tested last week. Found some bugs in retrieving persistant objects, leading to creation of news ones instead of retrieving the old
2. Need to be retested. Will be done in the next 2 weeks.
2. Status: July 5, 2011
1. Bug: Echo from own data - Remote Objects seem to receive an echo from their own sent update. This is unwanted behavior.
2. Bug: Application crashes - As Visual Studio rather crashes than show me which exception is not handled, it is hard to find what is going on. It is probably related to the Remote Objects.
3. Test: Synchronized Browsers succesful - I have tested the Remote Objects in a simple experiment with synchronized browsers.
4. Improvement of Event Model - The Remote Objects are receiving a more and more sophisticated Event Model, making it easier to monitor what is going on.
2. Server feedback – Including IP numbers of Servers serving specific sandboxes
1. Status: June 19, 2011
1. Tested and in revision now. Refactored to make the message in standard BaseMessage format. Will do more refactoring today to revise the process of: “knock knock, who’s there?”
2. Status: July 5, 2011 
1. No progress.
3. Heartbeats and auto-reconnect – Including auto-discovery of services and sandboxes and auto-reconnects when a Client connection drops or a Server drops out
1. Status: June 19, 2011
1. Revised to be independent of Servers running. To be tested in a Server Cluster.
2. Status: July 5, 2011
1. Bug fix: Static timer - The timer used for the heartbeat of the SandBoxes only dealt with one of all registered sandboxes, as it was a static instead of a local variable.
2. Preparation for multi-tier servers - To support multiple port numbers the SandBox connector is updated. Tests today to make it work properly have been cancelled. To be picked up later as it is only a priority to test Server to Server communication.
3. Test - Auto connect seems to work fine. Within 20 seconds of a Server becoming available, the separate applications are synchronizing Remote objects.
4. User Authentication – Including a basic “plug and play” scenario for logging into SandBoxes and assuring messages sent from a Client are “signed” by the user Credentials
1. Status: June 19, 2011
1. No further progress
2. Status: July 5, 2011
1. No progress
5. Data manipulation – Using a “plug and play” approach via: “Transparant Data Access Layers” to access, change and store data in databases
1. Status: June 19, 2011
1. Tested first version three weeks ago. To be revisited after finalizing items 1 to 4.
2. Status: July 5, 2011
1. No progress
6. Data Conversion – Including the conversions of Objects to “Comma Separated Values” vice versa, in order to:
1. Send updates – On Remote Objects through the network
2. Reduce code for working with Databases – As we can inject data from databases into objects and database records

Phase 2: Securing the framework – Started per June 16

Phase 2 is about making the Framework more solid and more protected against people who want to hijack data and events.

1. Server to Server updates – Including: “I am currently serving Sandboxes A, B and C with a total of N connections”
1. Status: June 19, 2011
1. Started implementation of Secured SandBoxes to make this possible and simple
2. Implemented Server Sate Manifest as a Remote Object that contains the data per Server and automatically updates all other Servers when the state on that Server changes
2. Status: July 5, 2011
1. Bug fix: Secret sandboxes not registered properly - Due to a bug in the refactored code, allowing for the creation of three different sandboxes: Open/Public, Trusted/Public  and Hidden, the Internal Server Manafest was not sent and received properly. This bug has been fixed yesterday.
2. Other progress - Briefly tested, except from the bug fix, no progress made
2. Load Balancing – Including a smart distribution of Sandboxes so that the load to serve Clients is shared by many
1. Status: June 19, 2011
1. Implemented, not tested yet. When a Client requests an IP number, the Client Socket will provide the IP of the Server that is least loaded.
2. Status: July 5, 2011
1. No progress made
3. Server to Server security – Using handshakes and shared keys. Assuring that when a user plugs in a “Server” to intercept messages and sandboxes, this “Server” is refused by the network
1. Status: June 19, 2011
1. Work in progress. Servers now build up a Secured SandBox using the exact same mechanism as Clients, with the difference that they use an internal Shared Keyword to test if they can Trust the other Server
2. Status: July 5, 2011
1. Trust relationships seem to work - The basic principle has been tested and trust-relationships seem to work. Connections are refused when a Client Server can not be trusted and three different type of SandBox Connections have been defined. See “Server to Server updates” under the update of July 5, 2011.
2. More testing required - To be sure that the code is solid, more testing is required.
4. SSL encryption – Currently all data sent is in Clear Text, making it extremely easy to hijack and abuse the system
1. Status: June 19, 2011
1. Not started yet
2. Status: July 5, 2011
1. Not started yet

When these items are done, the Framework can withstand most basic attacks, can be scaled up by hot-plugging new Servers in the network and will continue to work even when multiple servers drop out.

Phase 3: Opening up the framework – Started 

While the basis is for specific use in a controlled environment, some things are missing to make the framework even more useful. These include:

1. The use of Web-sockets – For Socket-communication over port 80 in environments behind firewalls
1. Status: August 3
1. WebSocket Server is working for current implementation / protocol.
2. Porting the code from C# to Java and ActionScript – To provide a broader use base, allowing you to run the framework on:
1.  Android, Apple, Linux and in Flash
2. Intel and ARM-based devices

Phase 4: Testing and cleaning up – Not started yet

In Phase 4 all crap and remaining leaks will be filtered out of the system. During phase 2, most issues have been tested and addressed, but as a part of the process.

The work includes:

1. Testing specific use-cases for performance - By running as much connections and messages as possible, until the system collapses
2. Testing possible ways to hijack users and data – By scrutinizing the security model and implementing code that should break it
3. Improving performance – By taking away data-hogs and

From my point of view – using Java, ActionScript 3 and C# – JavaScript is not an Object Oriented Programming language.

You can not create explicit classes, explicitly define static variables and methods and the definition and instantiation of an Object is… …awkward.

Base article

1. Javascript and OOP – by Tim Scarfe:
1. Creating “Classes” and extending them with methods –  See here
2. Object orienting programming in JavaScript – By Mike Koss.
1. Super and sub-classes – Describing the object prototype-chain and how to use and “extend” the prototype of an object.  See here

Packages

What I found missing is the use of Class Packages and Package Chains. So I created my own library for that kind of stuff.

Functions, function body and function scope

One way to pretend OOP in JavaScript is by using a method or function as the constructor and “Class” body.

Like this:

// JavaScript "class" Cat
function cat(name) {
 // Returns an object that we can extend with methods
   this.name = name;
   // Methods are defined like this
   this.talk = function() {
	alert( this.name + " say meeow!" )
   }
} 

// Example code of use
cat1 = new cat("felix")
cat1.talk() //alerts "felix says meeow!"

Extending objects and creating Classes via prototyping

Prototyping is another way to extend a “Class”.

Instead of cramming the constructor method/body with other methods, we extend the base method by extending its prototype.

Unlike Java and C# there is no limit or limitation where and when you do this. As long as you are inside the same (root) scope as the method or object you extend.

// JavaScript "class" Cat
function cat(name) {
   // Returns an object that we can extend with methods
   this.name = name;
}
// Extend the base constructor with methods via the prototype
cat.prototype.talk=function(){
 // Due to the prototype we are in scope 
 // of the object made via the cat-method
   alert( this.name + " say meeow!" ) 
}

Super and sub-Classes

Each object returned by a function is of base type Object.

When you create a method, this method implies a new (derived) object which can be extended via its prototype as we did above.

Knowing this we can do the following with “cat”:

// Constructor function Cat( name )
{
    this.name=name;
} 
// Extending the prototype of cat by injecting Animal into the prototype Cat.prototype=new Animal();
// Making sure the constructor on the object is Cat Cat.prototype.constructor = Cat;

Creating packages

By default JavaScript uses a flat structure, without the Package Chains we know from OOP languages like Java or C#.

As I need this for a better structuring of the HotForestGreen JavaScript Library, I started the creation of a small OOP library for that today.

It extends the basic JavaScript Object with the method: getPackage( packageName);

As any object – including the JavaScript object delimiterd by <script></script> – is a JavaScript Object, the call to resolve a package looks like this:

// Get package reference - this is similar to "include" or "using"
var package = this.getPackage("peter.kaptein.package")

// Create the "Class" and constructor in the package
package.MyClass = function () {
	alert("MyMethod constructor")
}

// Define a method in our "Class"
package.MyClass.prototype.findMe = function () {
	alert("found me")
}

// Creating a new object with our nested Class
var object = new package.MyClass()
object.findMe(); // returns : "found me"

Result – still awkward, but better

Defining classes still feels very awkward, as I am used to state something like this:

// ActionScript package definition package package.chain.somepackage {
 // Class definition
     public class MyClass      {
            // Constructor and Class Body go here
    }
}

Using packages – which are retrieved and referred to using a simple method – however, makes it easier to store specific classes in specific place holders and create cleaner code.

It also prevents me from accidently overwriting other Classes which might do the same but nor really are the same.

Package naming and reference

Using meaningful names helps creating a second layer of neatness.

// Not really meaningful is the following
var bla = this.getPackage("bli.bla.bloo"); 
// which represent out super awesome Socket Stuff

Is less meaningful than:

// A clear naming 
var sockethandlerPackage = this.getPackage("network.sockets.handlers");
// So we know what we talk about

Reflection and lack of reflection

Unfortunately JavaScript does not support something like Strong Typing.

Everything we use is an generic object, and can change its type on run time.

This means that:

1. Limited reflection – The editor you are using – and if it supports some sort of reflection – will only know what is in scope of a specific object due to its prototypes.
2. No sanity checks on pre-compiling – Whatever methods and variables you call on an object: only when the application runs you will know if your code will break.

This means whatever we create based on a improvised OOP approach can only be tested on run time.

Assuring clean Package Chains

We want “some.nice.class.package.mypackage” to use the exact same Package Chain as ”some.nice.class.package”.

To assure a clean package chain we use:

1. Global lookup table with Class Packages – getPackage uses a global lookup table (which is a OOPackage object itself)
2. Assure Package instead of “Create Package” – The Assure Package method calles “assurePackage” to check the existing Package Chains and assures we do not create multiple instances of parts of the same package chain.

HotForestGreen has an auto-detection process running to assure connections to be re-established when they are broken and new Servers to be found for the Data Streams when the old ones collapse, crash or die.

Creation and refactoring phases

Phase 1: Making it work

Auto Discovery - Phase 1

This first model assumed we wanted to keep a list of found “Hubs” based on the pings we did within the local network.

As the project progressed, the benefits of Hubs turned out to be minimal, as each time we re-connected or connected a SandBox, we would do a network wide scan anyway.

Also, there were too many steps and too many classes involved, leading to spaghetti code with a lot of entanglements which were close to impossible to understand on first sight.

Phase 2: Cleaning up and cutting out the fluff

Phase 2: Cutting out the fluff

In the second phase, I removed the Hub and simplified the model to find and recognize a Server.

There is one Class less and the relationship between objects is more clear.

Once a Hub was found, we would immediately ask it for a Server Manifest. Instead of the Server Manifest registering the ServerConnection to a Hub, we simply registered the ServerConnection to the Sandbox.

Instead of using three different type of connections and protocoles, we would only use two:

1. The initial ping – Which is the simplest protocol to find a live machine
2. The ServerConnection – Using Messages to receive the Server Manifest, requesting for an IP address and initializing the Server Side SandBox

Phase 3: Expanding the connection types

Phase 3: Expanding the connection types

As we require to support more (including connections to a remote IP address) than just Local Connections and standard Socket Connections (we also want to support Web Sockets and HTTP calls) I started a third phase: adding ConnectionParameters.

From only a Port and scanning the Local Area Network (LAN), it now needed:

1. A port – To connect to
2. An IP Address – Being a static constant: IP_LOCALNETWORK  or any IP online.
3. The Connecton Type – Being WebSockets, Normal Sockets, HTTP Connections and anything you can imagine.

Explicitly instantiating a specific type of SandBox Connection

I added methods to ConnectionParameters to instantiate the ApplicationSandBox for a specific type of Connection, to reduce coding, to reduce the required knowledge of all involved parameters and combinations you can use and to clarify what options you have when you want to create a SandBox connection.  It also dramatically reduces repetition of actions and the required boilerplate code to create a specific connection.

Introduction of the ServerConnectionManager

As I was unhappy with the role and responsibility of the SandBox in keeping its connections alive I started moving the responsibility to check for connections to a separate Class, called the ServerConnectionManager.

Phase 4: Making it ServerConnection Centric

Phase 4: Making it ServerConnection Centric

Three things happened due to the addition of more conneection options:

1. Shift of weight away from Application SandBox – As the Framework started as a SandBox Centric system, the added options for specific type of connections moved the weight or importance more and more towards the ServerConnection.
2. Change in the role of Application SandBox – The role of the Application SandBox started to shift. As stated before, the code inside the SandBox related to keep the connections alive, started to feel more and more misplaced. Where it once instantiated the connection, it now seemed more logical to have the Connection instantiate the Application SandBox.
3. Reduction of complexity – To maintain the old setup, where the Sandbox carried the responsibility of connecting to Servers and keep those connections alive (by re-wiring when connections got broken) included more and more complex thinking to keep things sane.

The solution was to make the Connection the center of the Auto Connection process.

The steps in the new setup are as follows:

1. You define and create a Server Connection – Which can run in the Local Area Network, or can point to a specific location.
2. The Server Connection Object:
1.  Instantiates the SandBox – Which is stored in a Lookup Table, so if it is already existing, nothing is done and the connection is added
3. You start ServerConnectionManager – To start the auto-discovery process. If the process is already running, this request is queed and started when the previous scan is done.
4. The ServerConnectionManager – Is a Singleton object with a heartbeat of 20 seconds. When triggered it will start:
1. The DisposableNodeScanner – Which will:
1. Check – Existing active  connections per SandBox if we already have a Connection running in that IP range.
2. Ping – When connections are broken to all given IP ranges of Local Networks
3. Try – And establish a Server Connection via the ServerConnection for each responding IP address

By Extending the ServerConnection with the ConnectionParameters we simplify the model and chain of Classes. As each ServerConnection is specifically linked to the Connection Parameters this is a logical step.

Phase 5: Finalizing it

Phase 5: Finalizing it

To extend the ConnectionParameters did not work out as Local Area Connections can run over multiple IP numbers. Therefore one ConnectionParameters object can relate to multiple ServerConnections.

Enter the SandBoxServerConnector: a general purpose definition object in which we can set the type of connection and use to establish and keep connections to all Sandboxes on all Servers.

What changed from the model in Step 3 and 4 is the following:

1. Sandbox ServerConnector - Is now the central element and abstracts and shields the actual Server Connections
1. Starts the Server Connection Manager – Via an explicit call made at the end of all configurations
2. Can instantiate the Appication SandBox – Through local methods, saving you the work of setting the SandBox name again
2. The Applicaton SandBox – Which was hidden by abstraction
1. Has become visible again – To use specific methods
2. Lost its methods to connect to Servers – As this has become the full responsibility of the Server Connection Manager
3. No longer stores references to Connections – As this is moved to the objects themselves, following my own rules on refactoring

Separately initializing the SandBox

To avoid explicit methods for all types of connections and SandBoxes and to avoid the implementation of all kind of exceptions (we have LAN connections and explicit connections. Then – per connection type – we can request a dominantly WebSocket type connection or a normal one. Then we can ask for public, trusted and secured SandBoxes) we extract the initialization methods for each specific Sandbox.

It adds one extra line of code to your initialization process, which becomes like this (for a standard Socket Connection in our LAN):

// Define the connector
SandBoxServerConnector connector =
     SandBoxServerConnector.standardSocketInLAN("sandBox", 821);

// Initialize a public sandBox called "sandBox"  // on the given port in our connector
connector.initializeOpenPublicSandBox(onconnectedHandler, null);

// Start the Connecton Manager to create the connections
SandBoxServerConnector.startConnectionManager();

Manual start of the Server Connection Manager

As the Server Connection Manager will do a forced scan of all IP addresses within the range of your LAN connection, we want to reduce the overhead as much as possible.

Initially the scan would be performed automatically on registration of a SandBox.

The disadvantage of this is clear: when you register multiple SandBoxes, multiple scans will be performed, unless some extra mechanism is added to the scanning process – leading to a lot of added complexity and never being completely fail safe.

Instead we use the most simple of solutions:

When you are done creating and registering your SandBoxes, you start the discovery process of the Server Connection Manager.

The HotForestGreen base design started with Standard type Sockets. As support of as many different platforms (HTML, Flash, Java, Android, Windows, Mac, Linux) as possible is a requirement WebSockets were on the list for a long time.

I started implementing them in the past days: the first week of July, 2011.

Based on some online sources like the jWebsocket article and the code in the WebSockets server by Bert Mortheamer  I was able to build a specific handler.

Multi-purpose server

As Sockets are the base objects for all communication, and protocols are what are added, I moved away from Protocol Specific Servers: which was the second incarnation of my code after refactoring.

Mutation of Class Structure

Below you will find the mutation of the Class Structure to support the new directions. Each phase took at least one refactoring, breaking the entire project until it was fully implemented.

Original setup

As we only had one type of Socket, the structure was simple.

Original Class Structure to handle Client Sockets

WebSockets are not the same as Standard Sockets

Web Sockets implement a different protocol and a specific handshake, based on HTPP. This means that we need to extend the Standard Socket and also have to find a way to direct the incoming Socket to the correct handler.

Strategy – Introducing different type of Sockets

So we need to choose which we will use. In the Class Structure as shown below, the main assumption was that we would tell the Server what kind of Sockets it would support. Based on that it would choose the Socket Type from a Lookup Table and instantiate that specific Socket Type to deal with the incoming data and Socket Type related protocols.

Second implementation: Strategy to chosse specific Socket Type

Socket Handlers – Making the model more flexible

What I did not like about the second model, implemented on a Saturday, was the rigid nature. Each SocketServer would only serve a specific type of Sockets, making mixing of Socket Types harder to do.

After starting the implementation of a specific connection of type: Bi-Directional Gateway, I decided to do a third refactoring: moving towards Socket Handlers instead of Socket Types.

Using Socket Handlers instead of Socket Types

The difference is how the Socket Encapsulation is done.

1. Socket Types: decision before reading the content – With Socket Types, I have to decide beforehand which Class I instantiate. Only then I will start the process of receiving data.
2. Socket Handlers: first read content, then decide how to handle it – With the Socket Handlers, I first start receiving data (the Header) and based on what I find inside that Header, I will instantiate a Handler which will deal with Socket Type Specific protocols.

Encapsulation and the ISocket Interface

To serve specific Socket Types, we need to cater specific communication protocols. The protocol to encapsulate data, as used in the Standard Type Socket is different form a Web Socket. Still, the basic activities we want to perform remain the same.

These actions are:

1. Send and receive Messages – Which are (should be) always formatted the same way, containing a header for the Server and a message Body to be processed by the receiving Client.
2. Check if the Socket is still connected – Of which the method might vary per Socket Type, but the outcome is always the same: “yes” or “no”
3. Start and stop the Socket – Including its disconnection
4. Auto-connect the Socket when connection is broken – Done via the Socket Handler

Benefits and results – simpler model, more options

1. It does not matter anymore what Type of  Socket is connecting –  WebSocket, HTML http request, Standard Socket, your own protocol: if we recognize it, we will assign a specific Handler to it to deal with
2. Any Type of Socket can communicate data to any other Type of Socket – As long as they are in the same Scope, defined by the Port and SandBox
3. The Class Structure stays simple while options grow - Very important. There is not just one way to do things and not just one technology to get it done.
4. It is easier to expand the model with new (your own) Socket Types – Using injection. See: “Creating and adding new Socket Types”

Creating and adding new Socket Types

Using the ISocket interface, you can add any Type of Socket to the collection of existing Sockets.

What you need is the following:

1. A Socket Type Identifier – This is for internal use.
2. The implementation of ISocket – As the Socket Handler expects a specific standardized Interface, it will expect you to implement and use that specific contract.
3. A Header Validator – Which is a public method in your Class. The Header Validator is used by the Socket Handler to verify if the Header is specific to a Socket Type.

Socket Type Auto Detection

For each Socket Connection, the header will be verified against the different Socket Type Handlers, using a public method.

If the validation process returns a “true” then that Socket Type Handler is the one to be used.

Allowing fox mixed Socket Types on one single port

Instead of separating Socket Types per port (as is usual for HTPP, FTP and Socket type connections) we want to allow any Socket Type to be able to connect. In the end they all are Base Sockets and in the end they all connect because they need the data flowing within one or more Application Clusters.

These are the reasons:

1. The method is less important than the purpose – The purpose is to exchange data between Clients. As these Clients can include HTML pages with Web Sockets, and more standard protocols might be added, we should accept any protocol IF we can recognize it.

Plug and Play – Extending the model with a HTPP handler

Now that we have a Plug and Play type of model, where we inject Handlers into the Framework, we can start expanding it. One of the next modules is to handle standard HTTP requests.

This allows for the Servers to offer a web page and web site, allowing for a very simple low-effort way to build Remote Controls for public use.

Web Based Remote Controls

Web Based Remote Controls are small web sites, offering you the tools to communicate with an Application Cluster using a simple Web Browser.

The standards I am thinking of using for Web Based Remote Controls are those:

1. HTML – Allowing for a broader support and the lack of need for a specific plugin or compiler (Flash, Java). Might not work on all browsers. Specifically when WebSockets are used but not supported.
1. WebSockets – For real time interactions via JavaScript
2. HTTP calls – When your browser does not allow for more complex stuff including Web Sockets. This will be implemented in the second phase of the Simple Web Browser Implementation in the Framework.
3. Intercepted calls in a custom browser – When you can deploy a application for general use which includes a Web Browser that intercepts calls of specific types, like: “socketcall:<your call to the HotForestGreen Application Cluster>”
2. Flash – allowing for a more stable platform and browser independent implementation. Might not run on all systems (old Android phones, specific tablets, Mac iOS)
1. Normal Sockets – As supported by flash
2. Web Sockets – When normal Sockets are blocked due to their (lack of) protocol

HotForestGreen uses a specific set of Socket connections with specific qualities for specific purposes.

Overview

1. WebSockets – For HTML Clients. Read more about them in: “Web Sockets and Web Clients“.
1. Requires a specifif IP Address - To connect to a HotForestGreen Server, the WebSocket needs to connect to a Known Server on a Known Web Address
2. Are not binary – Even though WebSockets can support binary data, this is currently not supported by the HotForestGreen Server
3. Streams per data package – Each data package or Message is streamed. Each Message is held Server Side until all data is in and parsed only then to all observers. this will create a minimal latency, based on package size and transfer speed.
4. Uses an event based model – Each message sent is comparable with an Event.
2. StandardSockets – As the name suggests, these are your run of the mill Sockets without any extra protocol
1. Requires a specific IP Address – If you want to connect to a location online
2. Allows for AutoConnect – Inside a local network: by scanning all local IP addresses for the existence of one or more HotForestGreen Socket Servers
3. Are not binary - Even though standard Sockets can support binary data, this is currently not supported by the HotForestGreen Server
4. Streams per data package - Each data package or Message is streamed. Each Message is held Server Side until all data is in and parsed only then to all observers. This will create a minimal latency, based on package size and transfer speed.
5. Uses an event based model - Each message sent is comparable with an Event.
3. StreamSockets – These Sockets are optimized to stream continuous data.
1. Requires a specific IP Address - If you want to connect to a location online
2. Allows for AutoConnect - Inside a local network: by scanning all local IP addresses for the existence of one or more HotForestGreen Socket Servers
3. Is binary – To allow for binary streams
4. Streams continuously - Each set of N bytes is immediately streamed to all observers, allowing for minimal latency
5. Not implemented yet – Streaming Sockets are not implemented yet.

See also

Why is binary data not the standard?

HotForestGreen uses a very simple protocol, designed to work on most, if not all, platforms including Android, HTML, Flash, Java and C#.

The packages sent contain clear text data for the following reasons:

1. Simplicity of design – Regardless of your platform, you can read and handle Socket Data that is clear text. Handling binary data can become very complex.
1. Clear text format - Most data handled in systems are Strings and Numbers.
2. ASCII Separators – To define the start and stop of the headers and the data sent within a package, ASCII separators are used to mark:
1. <02> Start of text – Indicates where the Message Body starts
2. <04> End of Transmission – To signal where the block of data ends.
3. <30> Record separator – To state where a data row or record ends and a new one starts (see the next item: “Serialization and de-serialization of data”)
4. <31> Unit Separator – To separate values and fields within a data row or record
3. Serialization and de-serialization of data – Data is serialized and de-serialized using Lists (Arrays), ASCII based separators (see previous item) and operations on Lists (and Arrays) to Split and Join this data. This allows for very short and relatively fast code.

Everything is binary, no?

Everything we send via Sockets is binary data, with values from 0×00 to 0xFF. So far so good.

Reserved codes

But like WebSockets, some binary codes are reserved for specific use.

Sending truly binary data, like sound files or images, could lead to accidental occurrences of specific codes we use to separate data.

Fast serialization and de-serialization

In our case to separate data and allow for fast and simple Serialization and De-serialization.

Holding state

Currently, all packages sent to a Server are put in a “Holding state” until all data is in. Then they are distributed to all registered observers.

Socket for streaming data 

As binary data can be large (think files and images) or continuous (think audio and video streams) the best solution might be to have a Socket specifically for streaming data.

There is more to HotForestGreen than Java, Flash and C#.

As I started working on the (new) road map for this Framework in 2010, Web Sockets (and a simple integrated HTTP Web Server) became a very important part of it.

Why WebSockets?

Web Sockets in a Web Client, connected to a Server

There are several reasons for the implementation of Web Sockets:

1. Allow for communication where Low Level Sockets might not be - In some environments, anything that is not a specific Socket with a specific protocol (in most cases: “is not HTTP”) is blocked.
2. HTML-based Web Clients – Which can be build by anyone with a text editor and allows for quick development and a very low point of entry for anyone who wants to start playing with HotForestGreen.
3. No compiler needed – As they run in JavaScript inside HTML pages inside a browser, the only thing you need is a browser that supports Web Sockets
4. Can be used by anyone – With a basic knowledge of HTML and JavaScript
5. Allow for a simple distribution model: the web – With any non-web based standard, your application needs to be downloaded and installed to work. Wwith Web Sockets, the only thing your users have to do is connect to the right location, with a browser that supports Web Sockets

HTML based Web Clients

Web Clients can be of any type including: Flash, Java and HTML.

HTML based Web Clients are applications that run (in most cases) in a Web Browser, communicate to a Server and use HTML (and JavaScript) to process and present data and interact with the user and the Server.

Useful links on WebSockets

1. jWebsocket for Android – This article by Predrag Stojadinovic describes how the WebSoeckets and handshakes are set up

The ABC of a WebSocket

Let’s assume we have a web socket connection like this:

var host = "ws://localhost:80/mySandBox?myVar1=1&myVar2=2";
var ws = new WebSocket(host);

Header

A WebSocket Header will be sent on connect and can contain the following base data, separated by a LF or CR/LF.

The header as received on the Server can look like this:

GET /test?x=1&y=2 HTTP/1.1\r\nUpgrade: WebSocket\r\nConnection: Upgrade\r\nHost: localhost:821\r\nOrigin: null\r\nSec-WebSocket-Protocol: someprotocol\r\nSec-WebSocket-Key1: &q 19 6 54 c6 C& & 533 <a7 (\r\nSec-WebSocket-Key2: 27W 72 00 0×5 94\r\n\r\n����J��g” object {string}

When we replace the \r\n for enters we get this:

GET /mySandBox?myVar1=1&myVar2=2 HTTP/1.1
Upgrade: WebSocket
Connection: Upgrade
Host: localhost:821
Origin: null
Sec-WebSocket-Protocol: null
Sec-WebSocket-Key1: &q 19 6 54 c6 C& & 533  <a7  (\r\nSec-WebSocket-Key2: 27W  72  00 0x5 94
\r\n\r\n<8 byte long binary data for key 3>

Connecting to a SandBox

If you look at the red part of the URL, you will see the sandbox name:

"ws://localhost:80/mySandBox?myVar1=1&myVar2=2";

The requested SandBox is passed to the Server as if it is a folder on that Server.

Web Socket Server Response

The Server Response we send back is mostly an echo from what we received, shuffled around.

For protocol V76 we add the secret keys and a check sum in our response, which is based on the three keys we sent.

HTTP/1.1 101 WebSocket Protocol Handshake
Upgrade: WebSocket
Connection: Upgrade
Sec-WebSocket-Origin: http://null
Sec-WebSocket-Location: ws://localhost:821/mySandBox

8jKS'y:G*Co,Wxa-

WebSocket Header Parts

Below is a full overview of the header, with a short overview of each part. As we only use a very small sub set of the call we receive (marked in red), all parts we do not use for the framework will be marked grey. The parts vital for the returning hand-shake are marked green. Other parameters used in the return set to the browser include our red ones and the ones marked blue.

1. GET/POST – The beginning string of our header reflecting the way the data is sent
1. URI – The part right after our base connection information. In our example this is: “/mySandBox?myVar1=1&myVar2=2″
2. HTTP version – The verison of the HTTP client/call. Unused
2. Upgrade: WebSocket – The type of connection. We use this as the identifier for the Type of Socket Handler
3. Connection: Upgrade - The type of connection as defined in Upgrade
4. Host: localhost:80 – Our IP and port number to which we connected. As we already known this from our Socket Connection, we do not need this.
5. Origin: null – Unknown purpose. Might contain the IP of the caller
6. Sec-WebSocket-Protocol: <your definition> – Contains the protocol as you set it in the second variable when you create a WebSocket. Unused for HotForestGreen
7. Sec-WebSocket-Key1: <a key value> – Contains the first web socket key from the client
8. Sec-WebSocket-Key2: <a key value> - Contains the second web socket key from the client
9. 8 byte binary data for key 3 - Contains the 3rd key, to be read as binary

Everything is binary data over a Standard Socket

When you dive into Web Sockets and how a Web Socket Server is created, at the end of the ride you will find a standard issue Data Socket.

Even the data itself is standard data sent as a binary stream over a binary connection

What differs from a “normal” socket is the following:

1. The header – With each connection attempt from a Client to a Server, the Client sends a HTTP WebSocket header with specific data to establish a hand schake
2. The hand shake – The Client expects a specific reply from the Server, to establish a low level type of trust: that it is connected to a Web Socket Server
3. Information about packages and (in case of binary data) package size – For Web Sockets, these are mainly used for binary data
4. Start and End Bytes – Each data package we send to a Server or Client starts with a Start byte (<0×00>) and ends with an End byte (<0xFF>)

Everything is Clear Text data with specific separators

HotForestGreen works primarily with Clear Text Data, separated into specific blocks using specific ASCII codes.

The main reason is to keep things open and simple when we translate and de-serialize the incoming packages.

If you need to send binary data…

Where binary data is required, we send two packages:

1. Header package – A header package indicating the Server to expect a binary stream of a specific length
2. Binary stream – The binary stream itself.

WebSocket Start and end bytes

Each WebSocket data package is wrapped in a Start and End Byte. Like this: “<StartByte>My Socket Message<EndByte>”.

More concrete, these bytes are: ”<0×00>My message<0xFF>”.

Network Streams and WebSocket Messages

A WebSocket Client requres an HTTP complient protocol to send Socket Data to thee client.

For this we use a (C#) NetworkStream instead of a bare Socket.

These streams state their lenght in the first parts of the data stream and as such have a slightly different build up from a generic Socket Connection which basically just streams data when there is data to stream and has no terminator at the end of the stream – except for 0×00 values as there is no data in the buffer used to inject the Socket Data in – when done writing.

Limited implementation for Web Sockets

WebSockets only have a limited implementation due to their character of running in an open space and using JavaScript code exposed and retrievable as clear text.

1. No auto connect – You need to specify the IP Address to which you want to connect via a Web socket
2. No Trusted or Secured Sandboxes – As these require a shared key on Client and Server and as HTML is uncompiled clear text, we rather not.
3. No binary data – As stated before, this is not a priority yet, as we have no use cases yet where Web Clients need to send binary data to an Application Cluster

No auto connect

The auto detection process as used on the C# and Java Clients runs in several threads, to allow for parallel scans over multiple ports. If done in a sequential order, the process might take several seconds to complete.

Short and simple: I wanted to keep things simple and avoided this for Web Sockets

Scope of Web Sockets

WebSocket Clients are designed for use in:

1. HTML clients – To allow for simple and fast designs and simple distribution of applications using the Real Time Interaction models of HotForestGreen over the web
2. C#, Flash and Java clients – To connect to Web Socket based Application Access Points
3. Public Sandboxes – Which do not require a secret hand shake and accept any client trying to connect

Public SandBoxes – and what they are

There are three types of SandBoxes:

1. Public SandBoxes – Allowing anyone to connect, if their header and stated protocol is recognized
2. Public Trusted SandBoxes – Allowing only Trusted Clients to connect: using a secret handshake with passwords both Trusted Client and Trusted Server know. If either of them fails to give the proper response, no Trust can be established and the connection is broken.
3. Secured SandBoxes – These are SandBoxes hidden under a specific key and secured via a Trust Relationship. For instance: the secured Sandbox: “mySecuredSandBox” with key: “my secret key” will be hidden under “my secret key::mySecuredSandBox”. This makes scanning and attempting to connect to Secured SandBoxes a bit harder.

Any basic HotForestGreen Client does auto-detection when “switched on”. This Auto Detection scans the local network the Client is connected to (via the device it runs on and the network that device is connected to).

It will automatically connect to an Application Cluster once it:

1. Has found one or more Servers.
2. Is allowed to join.

Images

Below you will find four images with some extra explanations visualizing the process and the full implementation.

Purpose

The purpose of the Automated Process is to:

1. Plug and Play – Instead of you having to do a lot of configuration on Networks and IP Addresses, the Framework will take care of that. Allowing you to focus your energy on what is relevant for you.
2. Simplify building Distributed Applications - To make building Distributed Applications, running over multiple machines, and within one or more Application Clusters, containing it all, as easy as cake.
3. Simplify connecting to Distributed Applications - To allow you to build (Stray) Clients to your Distributed Applications and work with whatever goes on inside.
4. Allow for fail safe- and fallback scenarios - By allowing you to add any number of Servers within your Local Network and do automated Load Balancing and automated Re-routing of communication streams when Servers drop out / crash

Targets

The targets for auto-connection are:

1. Servers and Server Clusters – Finding each other and communicating work load and number of connections
2. Trusted Clients – Running inside the network and the Application Cluster and being known
3. Stray Clients – Like mobile phones and tablets hooking into the Network and coming from anywhere

Scope: Local Network

Step 1: Pinging the entire local Network

Pinging the local network

1. Client IP – Based on the IP address the Client is connected to, the IP range is determined in which the Client could find the Servers. This can be one or more IP addresses, depending on how the computer is connected.
2. Simulatious ping – Per IP range a simultanious ping is done to all numbers inside that range, including the machine itself.
3. Silence and response – Each IP address will either respond on the ping, or not respond at all.
1. Response – There is a machine on the other side. We do not know yet if it is a Server.
2. No response – There is nothing there.

Step 2: Who is a Server? Requesting the Server Manifest

Sending the Server Manifest Request

The next step is to find out who of our present machines is a Server or not. We do this as follows:

Connecting to the Found Servers

For each Server we found in Step 2, we will attempt and receive the following:

1. Connect – The Client will not attempt to connect to the machines / IP Addresses that responded with a Server Manifest.
2. Refusal and Acceptance – Based on the security levels set on the Server, the Server might expect the client to provide a specific handshake.
1. Accepted – Either the Server and Client share the same secret keywords, or are open to anyone. A persistent Socket Connection is created and maintained as long a s possible.
2. Refused – Something did not match in the secret handshake between Client and Server. The Server or Client refuses to connect and no connection is established.

End result of the Discovery and Elimination process

At the end of the Discovery and Elimination process starting with all IP addresses, filtering out:

1. No response – All who give the silence of “no-one here
2. Not a Server – Then all who are not a Server
3. Not trusted – Then who are not trusted or do not trust you

If all went well we are now connected to one or more Servers in the Network, able to send and receive real time data to other connected Clients in that Network.

Time needed: less than 1 second

The whole Discovery and Elimination Process is time-optimized and can take place in less than 1 second, as all 256 IP numbers are scanned simultaneously.

Saving more system resources

As most Servers will probably linger in the IP range from xxx.xxx.xxx.001 to – let’s say: xxx.xxx.xxx.o50, due to the way DNS numbers are issued to “who comes first gets the lowest number”, and most Servers where there before most Clients, a lot of overhead can be saved by scanning IP ranges in blocks of 20 to 40 successive addresses.

If this improvement will be implemented depends on negative experiences while running the framework and possible future developments to optimize this process even more.

Plug and play

No human interaction was needed:

1. To find and define the IP address(es) of the found Server(s) – as this was derived from the IP address assigned to the computer when it entered the network
2. To find and to connect to the Application Cluster – As the load-balancing and the internal handshakes take care of connecting Clients to SandBoxes and exchange data within these SandBoxes between the Clients.

Once connected: Running multiple Sandboxes, Servers, Ports and Layers

The auto-discovery process as shown above is only part of the whole story. In the image below I attempted to show all the options you have after a Server is discovered.

They will occur:

1. When you have multiple Servers – Arranging Load Balancing and Auto-wiring you to the proper Server, serving your requested SandBox
2. Running your application over multiple ports – When you connect to other Ports in your network.

Why is this relevant?

Multiple SandBoxes, Servers and Layers visualized

Multiple Sanboxes, Servers, Ports and Layers

1.  Multiple Sandboxes over Multiple Servers – HotForestGreen Servers within the same network are interconnected and share their work-load. So when you connect to “a” Server, you might actually be re-directed to another IP address which is best to serve you for that specific SandBox.
1. Multiple Servers – The moment you need a more robust solution with fall-back scenarios in case of Server Failure, you will have two or more Servers in your network. All of these Servers communicate to each other, sharing their current state and work load in real time.
2. Multiple SandBoxes – Each SandBox is a closed environment in which data is protected and shielded from accidental contamination. You will find that you will – soon enough – run two or more SandBoxes to host the data streams related to your application.
2. Multiple Layers by using Multiple Ports – One Cluster running over one single port over multiple machines can be considered one “Layer”.
1. Layers – A Layer is like a sheet of paper on which your network draws its data.
2. Separating Layers – It is possible you want to run separate Layers and create “hidden layers” for specific tasks. For instance to separate your common and public data streams from the one that are more sensitive.
3. Ports – A port is like a channel on which your broadcast your data. When you connect to that port, you will be able to receive the data. If you do not, the data will be invisible for you.
4. Clients and Servers running on one Layer – Are able to discover each other and exchange information.
5. Clients and Servers running on multiple Layers – Can only see and discover other LAyers by explicitly connecting to it.

Fail saves

The system has several fail saves built in for when machines drop out of the Network.

Heartbeat and connection attempts

When a Server is not up yet, the Client falls back on a constant heart beat of (by default) 20 seconds, in which all connections of all Sandboxes are checked. If no connection is there where it should be, the Client will attempt to (re) connect to a Server.

Load balancing: pointing Clients to the right direction

While Load Balancing might – at first – not be perceived that way, it is (and allows for) a fail save.

Here is how:

Each Server in the Network is aware of the State of each other Server. So when one Server drops out, all other Servers will know almost immediately and will be able to define which Server is most fit to jump in the gap and take over the services it provided.

When clients, who were connected to the disappeared Server, start requesting for new a new connection for a specific SandBox, the Load Balancing takes care of pointing them to the proper new Server that will handle their requests.

Client- and Server side Caching: Preventing loss of real time data

In this first version of the framework, data loss due to a Server Dropout, is not covered.

This will be covered by Server and Client side Data Caching. When Clients lose a connection, the data for that specific SandBox(es) can be buffered until a new connection is established, making sure no data is lost.

On the Server side, data received can be cached for all Clients who come in later: updating them on the events that happened in the past N seconds.

Automated re-wiring of data connections

This allows for scenarios where Servers drop out, crash or die, where Servers kick in later, or Servers are replaced/hot swapped. All while the application at large continues to function and run with no visible effect apart from the small hiccups when Clients and Servers start actively re-discovering and re-wiring their connections to other Servers.

Due to other priorities I wans not able to do much since June 19.

Today I did two things:

1. Test with synchronized Browsers – This test included the following elements:
1. Synchronization of URLs – When one of the browsers navigates to another URL, all other browsers follow.
2. Synchronization of scrolling – When one browser scrolls up and down the page, all other browsers follow.
2. Fix of bug in heartbeat – I removed a bug in the heart-beat of the Application SandBox. This lead to only one sandbox scanning and updating connections  when no connections were provided.

The synchronized browser is one of the first proofs of concept and – as a project – will be cleaned up for reference.

Primary target: make implementation of Synchronized Objects as easy as possible

HotForestGreen is all about Synchronized (Remote) Objects.

Building this framework, I strive to create a solution that takes all the hard way out of the process, and only leaves the very basal things left for the coders using HotForestGreen.

This is slowly becoming reality. Each next iteration (there were 3 major ones in the past 2 months) makes it simpler to understand, without having to know the exact mechanics.

30 minutes

At the end of the ride, novice programmers should be able to have their first project with Distributed (Remote) Objects up and running in 30 minutes:

1. Changing colors of screens of other computers
2. Browsing web pages simultaneous on multiple screens
3. Turning leds on and off on Arduino boards and via XBee modules by addressing (setting values on) Virtual Objects or Mediators

Open for improvement

1. Layering applications over multiple ports fails - In the period of June 19 I implemented the options for multi-layered applications, living on several ports “on top of each other”. The idea is a bit like Flatland, where several universes can exist on top of each other. There are some bugs in the connection scheme which – right now – make connections from a Sandbox to one port fail if another is connected.
2. Remote Objects seem te receive an echo of their data – When testing the Synchronized Browser, the Browser Object who sends an update to the Application Cluster seems to receive an echo of that update from the Server. This echo should not happen, as the object should be excluded from the recipient list.

Next steps

Since I was not able to do anything since June 19, this list has not changed.

In the next weeks I will:

1. Finalize the Security layer, part of Phase 2
2. Test, debug and clean up the prototype code for Phase 1 and 2.
3. Test and debug the framework with some basic applications

Main focus for the next days

The goal is to have something demo-able before August.

Finalizing Database access and working with databases is moved – for now – after getting the basic framework running and tested to satisfaction.

Older Status Updates

• June 1: Prototype code for first phase of HotForestGreen almost done, implementing authentication
• June 13: First round trip successful
• June 19: Gettign a grip on all things considered. After a round of high-impact refactorings, things got to the point of functional again. Also contains a lot of background info on SandBoxes, the new security model, Systems of Trust and the revised Auto Connect system.
• July 5: This post

What is this Framework about?

See the post of June 19. See also the “About” page here.

Update on roadmap

Phase 1: Building the basis – In progress per May 4, 2011

Before June 10, 2011 I hope to finish Phase 1, which covers the basics on data, data conversion and Client/Server and Client/Client messaging. This includes:

1. Remote Objects – Sending and updating Remote Objects to all Remote clients
1. Status: June 19, 2011
1. Basis is tested last week. Found some bugs in retrieving persistant objects, leading to creation of news ones instead of retrieving the old
2. Need to be retested. Will be done in the next 2 weeks.
2. Status: July 5, 2011
1. Bug: Echo from own data – Remote Objects seem to receive an echo from their own sent update. This is unwanted behavior.
2. Bug: Application crashes – As Visual Studio rather crashes than show me which exception is not handled, it is hard to find what is going on. It is probably related to the Remote Objects.
3. Test: Synchronized Browsers succesful – I have tested the Remote Objects in a simple experiment with synchronized browsers.
4. Improvement of Event Model – The Remote Objects are receiving a more and more sophisticated Event Model, making it easier to monitor what is going on.
2. Server feedback – Including IP numbers of Servers serving specific sandboxes
1. Status: June 19, 2011
1. Tested and in revision now. Refactored to make the message in standard BaseMessage format. Will do more refactoring today to revise the process of: “knock knock, who’s there?”
2. Status: July 5, 2011 
1. No progress.
3. Heartbeats and auto-reconnect – Including auto-discovery of services and sandboxes and auto-reconnects when a Client connection drops or a Server drops out
1. Status: June 19, 2011
1. Revised to be independent of Servers running. To be tested in a Server Cluster.
2. Status: July 5, 2011
1. Bug fix: Static timer – The timer used for the heartbeat of the SandBoxes only dealt with one of all registered sandboxes, as it was a static instead of a local variable.
2. Preparation for multi-tier servers – To support multiple port numbers the SandBox connector is updated. Tests today to make it work properly have been cancelled. To be picked up later as it is only a priority to test Server to Server communication.
3. Test – Auto connect seems to work fine. Within 20 seconds of a Server becoming available, the separate applications are synchronizing Remote objects.
4. User Authentication – Including a basic “plug and play” scenario for logging into SandBoxes and assuring messages sent from a Client are “signed” by the user Credentials
1. Status: June 19, 2011
1. No further progress
2. Status: July 5, 2011
1. No progress
5. Data manipulation – Using a “plug and play” approach via: “Transparant Data Access Layers” to access, change and store data in databases
1. Status: June 19, 2011
1. Tested first version three weeks ago. To be revisited after finalizing items 1 to 4.
2. Status: July 5, 2011
1. No progress
6. Data Conversion – Including the conversions of Objects to “Comma Separated Values” vice versa, in order to:
1. Send updates – On Remote Objects through the network
2. Reduce code for working with Databases – As we can inject data from databases into objects and database records

Phase 2: Securing the framework – Started per June 16

Phase 2 is about making the Framework more solid and more protected against people who want to hijack data and events.

1. Server to Server updates – Including: “I am currently serving Sandboxes A, B and C with a total of N connections”
1. Status: June 19, 2011
1. Started implementation of Secured SandBoxes to make this possible and simple
2. Implemented Server Sate Manifest as a Remote Object that contains the data per Server and automatically updates all other Servers when the state on that Server changes
2. Status: July 5, 2011
1. Bug fix: Secret sandboxes not registered properly – Due to a bug in the refactored code, allowing for the creation of three different sandboxes: Open/Public, Trusted/Public  and Hidden, the Internal Server Manafest was not sent and received properly. This bug has been fixed yesterday.
2. Other progress – Briefly tested, except from the bug fix, no progress made
2. Load Balancing – Including a smart distribution of Sandboxes so that the load to serve Clients is shared by many
1. Status: June 19, 2011
1. Implemented, not tested yet. When a Client requests an IP number, the Client Socket will provide the IP of the Server that is least loaded.
2. Status: July 5, 2011
1. No progress made
3. Server to Server security – Using handshakes and shared keys. Assuring that when a user plugs in a “Server” to intercept messages and sandboxes, this “Server” is refused by the network
1. Status: June 19, 2011
1. Work in progress. Servers now build up a Secured SandBox using the exact same mechanism as Clients, with the difference that they use an internal Shared Keyword to test if they can Trust the other Server
2. Status: July 5, 2011
1. Trust relationships seem to work – The basic principle has been tested and trust-relationships seem to work. Connections are refused when a Client Server can not be trusted and three different type of SandBox Connections have been defined. See “Server to Server updates” under the update of July 5, 2011.
2. More testing required – To be sure that the code is solid, more testing is required.
4. SSL encryption – Currently all data sent is in Clear Text, making it extremely easy to hijack and abuse the system
1. Status: June 19, 2011
1. Not started yet
2. Status: July 5, 2011
1. Not started yet

When these items are done, the Framework can withstand most basic attacks, can be scaled up by hot-plugging new Servers in the network and will continue to work even when multiple servers drop out.

Phase 3: Opening up the framework – Not started yet

While the basis is for specific use in a controlled environment, some things are missing to make the framework even more useful. These include:

1. The use of Web-sockets – For Socket-communication over port 80 in environments behind firewalls
2. Porting the code from C# to Java and ActionScript – To provide a broader use base, allowing you to run the framework on:
1.  Android, Apple, Linux and in Flash
2. Intel and ARM-based devices

Phase 4: Testing and cleaning up – Not started yet

In Phase 4 all crap and remaining leaks will be filtered out of the system. During phase 2, most issues have been tested and addressed, but as a part of the process.

The work includes:

1. Testing specific use-cases for performance - By running as much connections and messages as possible, until the system collapses
2. Testing possible ways to hijack users and data – By scrutinizing the security model and implementing code that should break it
3. Improving performance – By taking away data-hogs and