Typical Scenario

The Hosting API is designed to be low level enough that middleware systems can be built on top of it, and high level enough to be useful for applications without supporting middleware.

**Figure:** A subsystem running on a hosting service. The subsystem may comprise many netlets. The agency which started the subsystem is not shown.
$\resizebox*{0.33\textwidth}{!}{\includegraphics{images/notation0.eps}}$

**Figure:** A netlet running on a hosting service, under the management of an application controller. The netlet is running on a particular node, and using a resource supplied by that node under the management of the hosting service.
$\resizebox*{0.75\textwidth}{!}{\includegraphics{images/notation1.eps}}$

**Figure:** The same situation as in figure , except that the node isn't important and isn't shown.
$\resizebox*{0.33\textwidth}{!}{\includegraphics{images/notation2.eps}}$

**Figure:** A Netlet using a service. The access point is downloaded and runs under the same hosting service as the consumer netlet. The service is using a resource from the hosting service. It has been passed that resource by the consumer netlet, through the service interface.
$\resizebox*{0.5\textwidth}{!}{\includegraphics{images/notation3.eps}}$

**Figure:** The same situation as in figure , in an abbreviated notation.
$\resizebox*{0.4\textwidth}{!}{\includegraphics{images/notation4.eps}}$

**Figure:** A service being provided from provider to consumer. There may be multiple service connections and access points involved in the connection.
$\resizebox*{0.3\textwidth}{!}{\includegraphics{images/notation6.eps}}$

Figures through show the various ways we depict relationships between hosting services, netlets, and resources.

As an illustration, consider a simple pattern for Jtrix services. Applications following this pattern have the following characteristics:

The use a Helper service which acts as application controller
The Helper service deals with boilerplate details like service advertisement, user interface, codebase download, and master election. It may be an entirely separate service, or be created to run only this application.
They maintain a group of worker netlets each of which maintains a copy of the application state.
There may be different types of worker. Each worker type may be associated with resource instances. When a worker fails, it should be replaced with a worker connected to the same resources, if possible.
Exactly one worker is designated as a master.
Only workers of a particular type are eligibile to be master. The other workers of that type are designated as backup masters
All changes to application state occur on the master and are propagated from the master to the backups. In the event of master failure, a new master is elected from the available backups. The most up to date backup is preferred.
If the application provides an external service, the service may optionally be supplied by any worker, or by the master alone.
Some operations must be performed by the master. These operations require access to up to date state information, for state integrity purposes. These operations are routed from backup controllers to the master over worker-to-worker RPC. Other operations may be performed using the (possibly out of date) backup copy of the state available on any worker. These decisions are application specific.
If the application provides a Jtrix service, the service is provided to clients via access point netlets.
Access point netlets use RPC to communicate with a worker. The criteria for selecting the worker are as above.

We denote the person responsible for maintaining the application as the ``Application Administrator,'' or ``KEN'' for short.

**Figure:** A typical Jtrix service style application
$\resizebox*{0.8\textwidth}{!}{\includegraphics{images/std_pattern.eps}}$

The lifecycle of such an application will involve the following steps:

Contract Negotiation
KEN must first obtain a Warrant for a hosting service. This involves some kind of contract negotiation, which may be explicit or implicit, inside or outside of Jtrix. The Warrant itself is merely an XML document. Some possibilities are:
- KEN runs his own cluster. He can then issue himself a warrant.
- KEN obtains a warrant directly from a Hosting Service Provider.
  He uses a web interface to agree to the contract, make payment provision, and download the warrant. Payment may have to be made when the service is used, via a Wallet service, or the wallet may have to be provided to the web site. Maybe just a credit card is required.
- KEN obtains the warrant from a broker.

KEN is working for some organisation which runs a cluster. He gets a warrant as part of his work.

Application Deployment
KEN now initialises his application. This involves the creation of a Bootstrap Descriptor which describes the application's bootstrap netlet, which is a special netlet written to start the application. Applications will be distributed with a mechanism to create this initial descriptor. This mechanism may require arguments to configure the application, or may be a complicated GUI style app. The bootstrap netlet will need, as a minimum, the warrant for the hosting service.
KEN uses a launcher application to start the application. The launcher application starts the bootstrap netlet, using the Bootstrap Descriptor. A complicated application may include its own launcher as part of the installation mechanism, bypassing the use of an explicit Bootstrap Descriptor.
The launcher application stays around long enough to ensure that the bootstrap has completed successfully, or to report the reason for failure. It can do this by interacting directly with the bootstrap netlet.
The bootstrap netlet uses the hosting warrant to create the first worker netlet. It can use the hosting services status reporting mechanism to retrieve initialisation dependent objects from the worker netlet, or the reason for startup failure.
In order to bind the hosting service warrant, the launcher application needs to provide a node environment. A convenient way of doing this is to embed nodality in the laucher.
Bootstrap
Once the bootstrap netlet is started, it must initialise the application, and provide any required feedback to KEN.
1. Helper service initialisation
  Helper services used by the application may need to be initialised. If a helper service is used to serve the application codebase, then the bootstrap should upload the codebase at this point. The filenames for the codebase and the warrant for the helper service (and any other configuration parameters necessary to the application) are supplied during the creation of the bootstrap descriptor. The interaction mechanism used to control the bootstrap process may be used to read or write files during the bootstrap process.
  A helper service may also be used to handle binding requests, and that service may also be initialised at this point. It may be necessary to initialise hosting resources or helper services in order to generate an initial warrant for the applications own service, for instance, to find binding protocol addresses if these are not known in advance.
  If the helper service is to run on the same hosting service, then it should be bootstrapped separately, before the application is bootstrapped.
Worker initialisation
Application initialisation consists of using the hosting service to run the first application worker, which will become the master worker. Resources may need to be instantiated, connected and mapped for the controller use. Helper services may also be mapped as resources, if required.

Feedback to KEN
Once the worker has initialised, any required feedback may be picked out of the controllers status via the hosting service, and reported back to KEN. Similarly, any failure feedback can be reported back.
Once the controller has sucessfully started, the bootstrap netlet can exit,and the launcher program can terminate.

First Controller startup
1. Service advertisement
  If the application provides a Jtrix service, it must advertise that service. This may be done internally, or more likely, via a helper service. The helper service will likely need configuring to properly service the bind requests for this application.
2. Backup integrity
  The first controller should arrange for at least one backup if the application is to be fault tolerant. RPC links must be established to backups, and initial state propagated if necessary.
  The hosting services also allow for an auto bootstrap mechanism if a control lease expires. This lease may have only one active lessor, and this mechanism may be used in a master election process if necessary. The autobootstrap mechanism may be used to reconnect ``orphaned'' parts of an application by binding an external service. If this mechanism is to be used, it should be initialised, and the lease periodically renewed.
3. External services
  If the application provides an external service, then the listener should also be established at this point.
4. Feedback
  Any necessary information about the deployment of the service may be fed back to KEN via the Hosting services status reporting mechanisms.

Run time events

Backup controller netlet or node failure

If a backup controller fails, due to netlet or node malfunction, it should be replaced. Ideally, it should be replaced with a new netlet bound to the same resources. If this is not possible, the state must be reinitialised from another backup or the master. This is shown in figure .

**Figure:** Reusing an exiting resource if possible.
[Starting point. Two copies of the data exist, in two different resources.] $\resizebox{0.9\textwidth}{!}{\includegraphics{images/failover0.eps}}$ [Basic failover. The resource contents have to be recreated in a different resource.] $\resizebox{0.9\textwidth}{!}{\includegraphics{images/failover1.eps}}$ [Optimal situation. A new netlet can be created with the original resource.] $\resizebox*{0.9\textwidth}{!}{\includegraphics{images/failover2.eps}}$

Reuse of the resource might be worthwhile for a number of reasons, including the preservation of an IP address, or to avoid copying a large amount of data.

Master controller netlet or node failure
If the master node fails, a new master should be elected from the remaining backups. The new master controller should follow a startup sequence similar to that detailed for the first master controller.

Service Binding
Service binding will result in the creation of access point netlets. These netlets will make network requests which must be serviced.
External Services
The master and backups may have to service external network requests.

Application Shutdown
At some point, the application will need to be terminated. The master should arrange for all netlets to terminate.
The hosting warrant can be used to reconnect to the hosting service if desired, for manual status inspection or forced termination. Standard user interfaces can be used for this, much as a traditional operating system provides standard interfaces for process inspection or termination.

This pattern is used by many of the basic applications and services which form the Jtrix core software suite. It seems suitable for fault tolerant, client/server based network applications. Jtrix provides a support framework (see section

) which handles much of the boilerplate involved in such an application.

Other patterns and corresponding frameworks are of course possible.

Jim Chapman 2001-08-16