Apples and oranges: techniques in comparing technologies

Companies and government agencies need to be able to compare how different spectrum dependent technologies fare in meeting expressed requirements when evaluating tender responses. The problem will get more difficult as technology neutrality in spectrum management allows all manner of competing systems to have spectrum access. Previously this was seen in a mild form with TETRA, Tetrapol and MPT1327. All were a form of radio trunking. All provided basically the same user function but all did it in a different way. How does the contracting company choose a winner?

The answer is by simulating the offer against the requirements.

To show this method let us look at a recent example. This analysis was undertaken to compare the utility of the new WiMAX technology in military communications. The method involved considering a typical brigade deployment of around 1,100 military units or force elements in a territory of around 10,000 square kilometres. Each force element needed combat net radios and extensive data transmission emulating a network enabled communications capability.

The scenario was then metricated using an established modern military communications technology: the Enhanced Position Location Reporting System (EPLRS) from Raytheon. This established a benchmark for performance against which WiMAX could be compared.

WiMAX was interesting because it was available, it could operate in an ad hoc mode and it could pass significant data traffic. The only problem seen in WiMAX was that it was intended for use above 2GHz. This would limit the spread of the brigade and perhaps the nature of the terrain in which it could operate but nonetheless it was considered of interest.

The scenario involved a fighting force spread across the terrain. There are three dependent variables that measure the quality of communications between the force elements. The first is that every unit should be able to communicate with others in its company, battalion or whatever (although not necessarily directly). The second is that each should be able to pass the desired traffic. Finally this communications should not be interrupted unduly by interference.

A secondary set of metrics were then developed from the user requirement. One example was the number of elements who at that point in time and for that traffic were completely out of communications (orphan elements). Another was the number of critically connected elements connected only with voice or with only one link to a peer rather than a robust multiple interconnects.

These metrics could then be reported for each technology as the scenario unfolds. Other measures such as spectrum efficiency could also be reflected in the dependent variables by allocating a particular amount of spectrum to both and then progressively constraining the spectrum whilst watching the stress on the variables. The image below shows one such report for one battalion area within the overall scenario reporting connections, connection success rates, orphans and critical connections. This scenario includes bandwidth scaling to reduce the data rate of WiMAX to match one of the EPLRS modes. This makes WiMAX more sensitive thereby approaching EPLRS success in connectivity.

In developing one or more scenarios and then evaluating the technology under test against a benchmark the following comparisons tables can be made. The technical world of decibels, Watts and microvolts disappear within the scenario leaving a series of metrics that would be of interest to the user as a measure of how good the technology is.

Clearly there are issues. One must build the scenario to be representative of the user application. The exact user scenario must be built. If on the sea, give the sea state. If on land, give the speed of movement and say if prone or standing. These are just examples but the detail is critical to fair evaluation. Simulation is one of the only ways of comparing apples and oranges in an objective way based on the experience of the user. Simulation permits comparison on a level battlefield and avoids any claim of unfair treatment from any bidder - particularly if all the bidders are given the scenarios with the tender and are told of the benchmark