Remote Objects within HotForestGreen are:
- Shared – Over all clients who are connected to the same Application Cluster and stated their interest in them
- Automatically updated – When one or more Remote Objects on some other Remote Client send their update to the Application Cluster
The Remote Objects are created and sent with the following starting points and concepts to take into mind:
- Remote Clients -Each Application connected to the Framework is called a Remote Client
- Distributed Applicatons – Instead of one application performing all tasks and containing all code, you can create many small applications, each performing specific tasks and each running on multiple systems. The collection of all these local, small and independent applications is called a “Distributed Application” where “distributed” is used in the sense of “distributed over multiple places”. Within these Distributed Applications, all independent applications work together as a team: where each team member performs a specific task, does not specifically needs to know who else is there (limited awareness) and the team as a whole delivers the total end result
- Application Clusters – Each of your Applications or Remote Clients run in an Application Cluster. The Application Cluster
- Remote Data Manager – All Remote Objects are managed using the Remote Data Manager. This manager is explained later in this document
- Bi-directional Socket Connection – Remote Objects are sent over bi-directional Socket Connections
- Message based – Each update on a Remote Object is sent in a Message over the bi-directional Socket Connection. This message has a Message Header and a Message Body.
- Message Header – The Message Header is a serialized String Array with several parameters, put in a specific and fixed order of appearance. The header in this setup has a relatively low overhead for de-serialization
- Message Body – The Message Body can contain any content in any structure of your liking. By default the Framework will use its internal Serialization Format which is optimized to reduce overhead on serialization, de-serialization and the number of bytes sent over the network. More about this later
- Local Data Repositories – Each Remote Client will keep a Local Data Repository with all Data Objects
- Data Hooks – To define which local Object Class of objects will receive what data, you register that physical Object Class to a Data Hook to the Server. Each event on that Data Hook will be pushed to objects in that Object Class
- Distributed Event Model – Each Remote Client in the Application Cluster registers liseners to specific Data Hooks to specific Application Sandboxes in specific Application Clusters. When some Remote Client sends an update on a specific Remote Object over a Socket Connection to the Application Cluster, the Application Server will dispatch this update to all other Remote Clients registered against that Data Hook. Each registered Remote Client will then automatically receive a Data Push
- Serialization and de-serialization – To send data over the Socket connection, we serialize and de-serialize that data using a common and shared format. Internally the Framework uses “Solid Comma Separated Values” compareable to CSV but with a more solid separator for items/fields/columns and rows. Apart from the serialization format used in the Framework you can use your own via your own custom Data Serializers
- Data Serializers – As stated HotForestGreen uses an internal format to Serialize and de-serialize data. This process is done via a Data Serializer which is pluggable: meaning that you can create your own Serializer/De-serializer. For the Framework, data is data and as long as all your Remote Objects of that specific class use the same Serialization / De-serialization format it does not matter what fomat you use.
- Binary data – Bottom line: all data sent over whatever type of Socket connecton is binary data. For most Socket connections and for the Framework this is without any problem
- WebSockets – HotForestGreen supports WebSockets. WebSockets from version XX support binary data when sent in a separate package.
The Remote Data Manager
- Application Clusters – Application Clusters are the sum of all Remote Clients running in a specific Application SandBox
- Application SandBoxes – All Remote Data and Remote Objects run in specific SandBoxes. Each SandBox is a closed environment in which objects and data runs independantly from other Application SandBoxes in your Application and Remote Client. Assumed is that (Remote) Objects in “SandBox 1” are not the the same as (Remote) Objects in “SandBox 2”. This means that when “Object 1” in “SandBox 2” receives an update, “Object 1” in “SandBox 2” will remain unaffected
- Multiton Instances – The Remote Data Manager is a Multiton, where each Instance of a Remote Data Manager manager the data of the specific SandBox represented by that Remote Data Manager
- Persistancy of Remote Objects – Each Remote Object is by default a Persistent Object. This means that when “Person 1” in “Sandbox 1” is sent to all Remote Clients — instead of creating a new “Person 1” each time it is received– all local Objects on all Remote Clients in “Sandbox 1” in your Application Cluster representing “Person 1” will automatically be updated by the Remote Data Manager. To do this, each Remote Data Manager holds a Local DataMap
- Local Data Repository and Simple Object Database – Each instance of the Remote Data Manager holds a Local Data Repository of all Remote Objects registered against a specific SandBox. This Local Data Repository forms the bridge between your local objects and the Remote Data that is received over your Socket Connections. So when the Remote OBject “Person 1” is received, it will first be resolved against the Local Data Repository. If it already exists, that specific object will be updated. If it does not exists, a new Local Object will be created and added to the Local Data Repository for all future updates
- Messages, Observers and Remote Event Dispatches – As stated before, the Framework uses Messages and Remote Event Dispatches. Each Remote Client and each Local Data Repository is in the end an Observer on specific Events dispatched against the HotForestGreen Servers
- Local Events and Local Event Dispatches – To create a simple basis for updates, the Remote Data Manager offers several hooks to Observe Events on Remote Objects. These hooks are:
- Added – A Remote Object is added to the collection
- Updated – The Remote Object is updated somewhere in the Application Cluster: either locally or on another Remote Client
- Removed – The Remote Object is removed from the stack
Serialization and de-serialization of Object Data
This topic is also covered in other posts:
- Formats to serialize to – Here. About considerating and dropping the use of CSV, JSON and XML and moving towards the SCSV (Solid CSV) format used in HotForestGreen
Basis of serialization and de-serialization
- Sending “objects” – When you send “objects” over any connection, what is really sent is information within and about that object. This information includes:
- A reference to the object/object class – In order to de-serialize, you need to know what your basis is.
- The content of that object – Being values in variables and other objects nested within that object
- Serialization – When you serialize an object and object structure, you basically iterate through that object (and any sub-object) and read and store all found values into some kind of format. XML could be one. JSON another, your own format yet another. In the end, all objects store basic primitives like Strings, Numbers and Boolean values and your object structure is nothing more than that: a structure to hold your data
- De-serialization – When you de-serialize data, you translate your “flattened” data structure into a (new) object structure that resembles the original you serialized. To do so you need:
- Object Classes – That mirror the Classes (containing the structure) of the objects you serialized. For instance: the class “Person”
- Shared Class References – To resolve the right Object Class to instantiate when re-creating the object structure that was serialized. In most cases this happens by mapping: MyClassMap.addClass(“PersonClass”, my.classes.Person.class() ). When we serialize the data, we can include info on the object Class we serialized. For instance: <object classreference=”PersonClass” >. Where “PersonClass” can be resolved into the Class: Person
- References to map the variables – To inject the values from the serialized dataset into your object, you need to know where to put that data. For instance: <firstName>John</firstName> will do that trick, or: <object classreference=”PersonClass” firstName=”John”>
- The code to do the de-serialization – In most cases this is done by iteration and reflection, using the structure of your serialized data as starting point: person[ dataItem.name ] = dataItem.value
Data Maps and Persistancy of Remote Objects
- One single object for one Date Item – When you send updates “person 1” to all Remote Clients, you want those updates to happen in the one single Data Object you created for “person 1” on each Client
- Using Data Maps and Object IDs – To find objects already created, HFG uses Data Maps to store the objects we create and receive on each type (“Person”, “Car”). This Data Map has the following layers:
- SandBox – The SandBox in which the data is running. “Person 1” in “SandBox A” is not the same “Person 1” as we find it in “SandBox B”
- HFG Class Reference – When we store the Object Reference, we do that based on the HotForestGreen ClassReference. For instance: the class “Person” can be stored under “PersonClass” for “SandBox A”. “Car” can be referred to as “CarClass”
- Object ID – This is the unique ID per object, either created by HotForestGreen or by the Data Provider (a database or your code)
- Mapping Remote Objects – A Remote Object is mapped on the following parameters:
- Class Reference – We need to know to which Class it has to be de-serialized
- Object Identifier – The name of the variable that contains the Object ID Record ID
- Resolving Remote Objects – When a Remote Object is received, we use the SandBox ID to access the datamap containing separate Data Maps for each HFG Class Map within that SandBox. Once we have resolved that, we have the final Data Map with references to all existing Data Objects, using the Object ID as the Key to resolve that object from the Data Map
- No object found – When no object with those keys are found, a new instance will be created by the Remote Data Manager and injected into the Data Map. That new object will then be offered for de-serialization of the data
- De-serializing the data – When the DataMap Manager returns an Object corresponding the given keys, the de-serialization process can start and the values from the Remote Object can be injected into our local representative
Sending any (registered) object
- No specific Extentions needed – In an older version of the Framework you had to extend the Remote Object Class to make any object a Remote Object. This process has been removed.
- Mapping Local Classes against Remote Object Class References – As your Class Structure can differ between projects using and Sharing the same Remote Objects you do need some kind of common name to make the mapping / injection possible. This is done by mapping your local Class agains the Remote Object Class Reference. For instance: “your.project.classes.Person” is mapped against “PersonClass” in “SandBox A”. When you receive Remote Object Data with Class Reference “PersonClass”, your mapping will resolve that into your Class: “your.project.classes.Person”
- Knowing your SandBox – One thing you need to do and continue to do is keep tabs on your SandBoxes. There are two ways to do that:
- Manually – You code it into your application. This is the most effecient way for your resources, as you do not need extra processing time to resolve the SandBox
- From a Map – Each Object you receive and create can be used as a key to map the SandBox it was created for. The advantage is that your application will keep track of where what Object belongs without your further help. The disadvantage is that this costs resources in memory and time to resolve the SandBox from the Map you use
- Registering persistant objects against a Object/Sandbox Map – When you want to use maps to register where your object belongs, you use:
- RemoteDataManager.mapObject( yourObject, yourSandBox )
- Sending your object to your SandBox – Your object can be sent in the following ways:
- Using all parameters – This is the fastest way as the Framework does not have to waste extra time resolving all implicit data:
- RemoteDataManager.sendUpdate( yourObject, theSandBox_ObjectClass_ReferenceName, yourSandBox).
- Using the SandBox name – When you only use the SandBox name the Framework will try and resolve the SandBox ObjectClass Reference Name from the Class/Class Reference Name Map you made for that Object, for that SandBox.
- RemoteDataManager.sendUpdate( yourObject, yourSandBox).
- Using the Object – When you built an Object/SandBox Map, the call is even shorter in code:
- RemoteDataManager.sendUpdate( yourObject )
- Using an Extention on the RemoteObject Class – Using / extending this Class, you can tell each Object explicitly to which SandBox it belongs and with which Remote Object Class Reference Name it is associated. The call will be moved to the Remote Object itself, to keep coding short and simple:
Sending lists of Remote Objects
- Using an Array or Collection – To indicate which Remote Objects you want to send in a List, you place them inside a Array or Collection
- Sending the list via the RemoteDataManager – As the Array itself does not hold info on where or how to send the objects you use the RemoteDataManager directly, even when your Remote Objects are Extended form the RemoteObject Class. Like this:
- RemoteDataManager.sendList( yourCollection, theSandBoxClassReferenceID, yourSandBox)
- All objects of the same Type – You can not send a list of mixed types as the Framework will serialize the objects as if they are from the same Class
Serializing nested objects
- Any type – Nested object can be of any type, including collections of other Objects
- Single Objects – Any object nested in another object is serialized as a sub-set of that parent object.
- Collection of primitives – Primitives like String values and Numeric values are serialized as a sub-set of that Data Object and placed inside the DataRow of the object containing that collection of primitives
- Collections of Objects – Collections of objects inside an object are not serialized by the Framework. If you need to send collections, you can do two things:
- Hide the actual Collection and share references – Store the actual objects into a private Collection and share a pubic reference list using String values or numbers to resolve the actual objects on the Remote Client side.
- Create / download your own Serializer – As the Serializer/de-serializers are pluggable, you can create, inject and use your own versions.
De-serializing nested objects
- Limited implementation – The Framework will offer you a very limited implementation of Serialization/de-serialization of Nested Objects. The main reason is that:
- In most cases more is not needed – In most cases you do not a very elaborate Deep Serialization
- In those cases you need it: it is not good enough – In the cases where you DO need Deep Serialization/Deep de-serialization, your own solutions will be much better than what the Framework might be able to provide
- Plug and play: download or write your own – The Serializer / De-serializer is a pluggable system. You can inject your own Serializers / De-serializers, state in the Message Headers which specific one you used and skip whatever the Framework might have provided you with your own (downloaded) awesomness
- Single Objects – Using a object- and class reference inside the Main Object, the Remote Object / Data Object that is the result of de-serializin the sub-set belonging to that nested object is put into the Main Object. This process can repeat itself for nested objects inside nested objects
- Collections of Objects – Collections are de-serialized as a sub-set of that parent object and placed in the collection object of the Main Object. Any existing data in that Collection is replaced in that process. Object references to Data Objects / Remote Objects like “Person 1” will be pointing to the exact same object “Person 1” as before the purge and replacement of the existing Collection
- Use references to resolve the Objects – If you sent references, the Framework can use the Local Data Map can use to resolve the actual objects. You will need to write your own setters or mechanisms in your Remote Objects to do the actual work, as the Framework will not do that for you
- Use your own process – See “Plug and play: download or write your own” in item 1: “Limited implementation” of this list
- Maps – Maps can not be serialized by the Framework. As they can be built up in any way possible, with any type of key possible, they form a potential danger for overhead in both coding and processing. Short and simple: avoid Maps.
- Collection of primitives – Primitives like String values and Numeric values are de-serialized and will replace the existing Collection. If your programming language treats primitives as Objects, like Java does, the object references to the String with value “String 1” will not be the same as the Object reference of the previous value “String 1”
Serializing arrays of primitives like Dates, numbers and Strings
- End points – Any object that can be considered an “end point” will be translated to a String value representing that value.
- Date/Time – For date/time a standard encoding will be used that can be de-serialized in any coding language. The de-serialization is part of the Framework
- String values – In most languages, Strings are Strings and not Objects. The framework will simply translate Strings to a String value
- Numeric values – Numbers can be translated to String values and will be so. The decimal-separator will be a dot
Receiving and processing Remote Objects
- SandBoxes and Sockets – For security reasons and scalebility, each SandBox runs in its own and explicit Socket Connection.
- Messages – Each update on Remote Objects is sent in a Message. This Message contains:
- A Message header – Which has been discussed earlier and contains all meta-data for the Server and the Remote Clients
- A Message Body – Which contains for SCSV:
- One Header with field names – Used to map the values to the variables on the local instances of the Remote Objects
- One or more DataRows – As a Message can contain data for one or more Remote Objects, we package them in DataRows. Each DataRow contains data for one instances of one specific Remote Object
- Headers and Meta Data – Each message we receive on our Sockets contain Meta-Data. This meta-data includes:
- Remote Object Class Reference Name – This is a generic and shared key over all applications using these Remote Objects, to resolve the Serialized data to the right Class
- SandBox name – Even though we run data within SandBox specific Sockets, we send also the SandBox name with the header.
- Automated process – An Extened Remote Object is received and handled automatically. Using the local SandBox and the Remote Object Class Reference Name, the Framework will (try and) resolve the local instance of the Remote Object and once created or found, it will inject the values as sent in the Message Body into the corresponding Local instance of the Remote Object
Optimizing sending lists by using Challenge/Response
- Challenge: Remote Client makes generic Data Query – For instance: “give me all people with first name ‘John’.”
- Response: Sending IDs only – As some, or even most of all Data Objects covering that Query might already be in possession of the Remote Client, sending all Objects again might be overkill. Instead, the first Response can be to send only the IDs of each Data Object, so the Remote Client can refine its request for those Objects not already in the Local DataMap
- Challenge 2: Requesting specific DataSets – As soon as the Remote Client has resolved which IDs are not yet present in its Local DataMap, it will request the specific dataset by stating the specific Object IDs it requires.
- Response 2: Sending requested data – Based on the given Object IDs, the Receiver of the Request will send the missing Data Objects
- Resolving the Data Objects – Once the data is received, the Remote Client will resolve the Data Objects into its local set of Remote Objects
Optimizing Remote Object Handling
- Explicit coding – The more explicit your coding is, the less the Framework will have to resolve and the faster your processing will be
- Extending the Remote Object Class – This class allows you to state explicitly per Object to which Class and SandBox it belongs, keeping your code short and simple
- Instantiating an Extended Remote Object – A Remote Object is instantiated in the following way:
- myRemoteObject = new ExtendedRemoteObject( sandBoxName, sandBboxClassReferenceName )
- Sending an Extended Remote Object – An Extened Remote Object is sent in the following way:
- myRemoteObject.sendUpdate() ;