Multiple input/multiple output (MIMO) systems have shifted from theory through design to development and are now entering the testing and deployment phases. The MIMO articles in this issue are focused on performance testing in real world conditions, which is the validation of MIMO performance that the industry has long been waiting for.


This first article covers the standardization efforts for MIMO "Over the Air" (OTA) testing. MIMO is used here to refer to any multi-antenna technique including Rx and Tx diversity, beamsteering and spatial multi-plexing. When this work started a couple of years ago, I quickly concluded that MIMO OTA was the biggest test challenge facing the industry that I had seen in 20 years of standards involvement. By the end of this article I hope you will appreciate why. Following this article are four related articles focused on specific aspects of the test process. Dr. Michael Foegelle of ETS-Lindgren explains the classical anechoic chamber method for creating the required spatially diverse test signals, while Derek Skousen of MI Technologies and Charlie Orlenius of Bluetest describe how to use a reverberation chamber for the same purpose. Madhu Gurumurthy of Spirent describes the spatial correlation properties of real radio channels and the importance of realistically modeling them. Finally, Azimuth Systems, as a part of the product feature section, presents a drive test solution for measuring real channels so they can be replayed in the lab.

The MIMO Antenna: Unseen
When mobile phones first appeared on the scene, the antenna was very much in evidence. These early phones typically operated in the 900 MHz band and were often "graced" with a folding or pull-out whip antenna whose length was optimized for the frequency of operation. Over time the whip antenna gave way to much smaller helical designs and then in 1994 we saw the first of the integral "patch" antennas that became widespread by 1998. Thus, in a few short years the antenna has transitioned from its traditional, highly visible form to being nothing short of invisible.

The MIMO Antenna: Unloved
The transition from seen to unseen was clearly a gift to the industrial designer, who was now freed from the need to design around this functional "wart." Seen through the eyes of fashion, the integration of the antenna represents huge progress, but from an antenna design and performance perspective, the opposite is true. In the clash between industrial and antenna designers, suffice it to say that for the sake of a millimeter here or there, fashion often triumphs over function.

The MIMO Antenna: Untested!
And now for the reckoning—Single input/single output (SISO) OTA standards have existed since 2001, but nothing yet exists for MIMO. The factors affecting MIMO antenna performance are much more complex than SISO and today's antenna designers are under huge pressure. They have had to deal with the transition from external to integrated design as well as the change from single to multiple ports. The volume required for one good antenna can easily be divided up to make room for two bad antennas—and without standard test methods, who is to know? A similar challenge comes from the huge increase in the number of supported frequency bands, with hex band devices now being common. Each band imposes unique demands on the optimal receive and transmit antennas resulting in the need for separate antennas for some bands. And although the focus here is cellular, there are further antenna demands for the support of Wi-Fi, Bluetooth, GPS, FM radio, DVB, etc. Before getting into the specific challenges of MIMO OTA testing, it is important to first have a working understanding of SISO.

A Brief History of Radiated Testing
Most handset testing is done using conducted methods; that is, by connecting a cable to what is known as the "temporary antenna connector," which bypasses the DUT antenna to provide direct access to the transceiver. The requirements at this port are based on an assumption that the DUT antenna can be fairly represented by an isotropic antenna with 0 dB gain. In the days when the antenna was a dipole tuned to the single band of interest, this assumption was not unreasonable. However, with the advent of multi-band integrated antennas, the 0 dBi assumption is no longer safe. This being the case it is easy to see how the results from conducted tests for requirements such as reference sensitivity and maximum output power may no longer represent the radiated performance of the device in a real network.

Radiated testing for regulatory purposes started with Electromagnetic Compatibility (EMC) testing for spurious emissions, and more recently both a hearing aid compatibility and a safety test were added for specific absorption ratio (SAR) to assess how much of the DUT radiated power is absorbed by a phantom head. However, these tests do not assess the desired radio performance of the DUT for the purpose of communication. The CTIA published its "Test Plan for Mobile Station Over the Air Performance" in 2001, and beginning in 2006 3GPP published Technical Report (TR) 25.914, Technical Specification (TS) 25.144, and finally in 2008 the associated test specification TS 34.114.

SISO OTA Figures of Merit
SISO OTA is conceptually simple, comprising one figure of merit (FOM) for the DUT transmitter called Total Radiated Power (TRP) and another for the DUT receiver called Total Reference Sensitivity (TRS). For TRS, the CTIA uses the term Total Isotropic Sensitivity (TIS). The TRP is defined as the integral of the power transmitted in different directions over the entire radiation sphere. TRS is a similar measure, but it represents the reference sensitivity of the DUT receiver. With these two FOMs agreed upon, the bulk of the standards work for SISO was in defining the method, the test system uncertainty and finally the performance requirements.

The test method was initially developed using an anechoic chamber. Substantial theoretical analysis of the measurement uncertainty was performed by CTIA and the European COST273 project, resulting in an error model with over 20 terms. The requirement for overall test system uncertainty was calculated to be in the region of ±2 dB; this figure has since been validated using a golden radio by the majority of the nearly 50 CTIA accredited labs. Further work on an alternative test method using a reverberation chamber has also been done and test results indicate a similar level of uncertainty.

SISO OTA Test Requirements
A summary of DUT minimum test requirements for UTRA (W-CDMA) from TS 34.114 is given in Table 1. The figures in Table 1 are the test requirements, which have been relaxed from the minimum requirements in TS 25.144 by an amount known as the test tolerance, which is a function of the maximum allowed test system uncertainty. 3GPP normally relax the minimum requirements by the full test system uncertainty, but since the OTA test uncertainty is quite high (±1.9 dB for TRP and ±2.3 dB for TRS), the relaxation is limited to about half of the allowed test system uncertainty. This choice prevents the requirements from becoming too relaxed and allowing bad DUTs to pass, but it does slightly increase the risk that a good DUT might fail the test.In the pursuit of improved user experience and network performance, a non-mandatory recommended target is also defined.

The figures presented in Table 1 conceal the details of the actual tests, which are quite thorough. Both TRP and TRS are the average of measurements made across a sphere typically using steps of 15 degrees for TRP and 30 degrees for TRS in both azimuth φ (Phi) and elevation θ (Theta). At each point, measurements are made at the low, middle and high channels of all the frequency bands supported by the DUT and at two orthogonal RF polarizations; e.g., vertical and horizontal. The DUT has to be tested in its primary mechanical mode, and may involve sliding or folding open the DUT. Testing in other mechanical modes is not part of the minimum requirement, although some carriers require all modes to be tested.

The final consideration is the physical environment. Tests are carried out in free space or in the proximity of a Specific Anthropomorphic Mannequin phantom head, better known as SAM. Tests are carried out on both the left and right sides of the head, which is filled with different liquids to match the frequency-dependent RF loading effect of the human head. In order to ensure repeatability across labs, detailed guidelines on how to align the DUT to the test environment are provided. The latest CTIA test plan has added a phantom hand, which can take four positions depending on the DUT design: monoblock, fold, narrow data and PDA. The hand may be used on its own for "data" positions or in conjunction with the head to emulate a more realistic speech position than the head-only tests. Currently, the hand tests are limited to the right hand. Testing may be extended to the left hand since the interaction between device and hand can be highly asymmetric.

From this brief overview of the scope of SISO OTA testing it is easy to see that characterizing one multi-band device requires thousands of measurements. Testing can take up to two weeks in an expensive anechoic facility.

Figure 1 TRP vs. TRS for GSM 1800 commercial devices.

Development of the SISO OTA Test Requirements
The normal process in standardization is to simulate the desired performance and provide design targets for developers. Since OTA performance requirements are very much retrospective, simulation was not an option. Also, the sheer complexity of trying to simulate a realistic performance target was considered out of scope. However, with the creation of a repeatable test method, the requirements were instead developed through a series of measurement campaigns using real devices. This is clearly not ideal, but under the circumstances was the only practical solution. A typical set of free space TRP and TRS measurements from a variety of commercial devices is shown in Figure 1. Each point defines the TRP and TRS for one DUT.

Note the variation of performance covering approximately 7 dB for TRS and 4.5 dB for TRP. With the addition of head and hand loading the figures will spread significantly further. All of these devices will have passed the conducted tests, which have a narrower spread, suggesting that the increased variation is due to the previously untested antenna. A carrier company choosing devices for a network will want to see a high TRP and low TRS. The availability of standardized OTA measurements gives the carrier for the first time a deterministic and repeatable way (compared to field measurements) of selecting the best handsets, which in turn will directly improve end-user quality of experience (QoE). In the conducted domain many a battle has been fought over a tenth of a dB here or there, but in the radiated domain there are dBs of performance at stake, which is why the recent introduction of OTA testing is long overdue and so important.

Agreeing to minimum requirements in 3GPP was by no means straightforward. Carriers naturally wanted to set high targets, while vendors had to protect the installed base of handsets and existing designs from being classed as non-compliant and to protect design margins for the ever-decreasing size of future handsets. The end result was a compromise. Figures were agreed on for average and minimum performance that allowed the bulk of legacy devices to remain compliant. Carriers accepted tougher, though non-mandatory, "recommended" average performance—typically 3 dB better—to give the industry something to aim for. Even then the recommended TRP performance is some 6 dB below the nominal power for the conducted test, suggesting that there is still considerable scope for improvement, although with the continual downward pressure on device size further improvement may not be realistic. To date, figures for GSM OTA requirements have not yet been reached and many CTIA requirements are still to be defined.

Moving from SISO to MIMO
MIMO OTA standardization started a couple of years ago in CTIA and 3GPP along with the European COST2100 project that succeeded COST273. We saw that SISO OTA is conceptually simple, with just one primary test method and two figures of merit based on existing conducted measurements in order to provide the missing insight into the antenna performance. With the exception of the phantom head and hand, the SISO measurements are independent of the external radio environment.

The situation for MIMO OTA is very different. The desired FOM for MIMO OTA is end-to-end data throughput in realistic conditions, which provides a direct measure of QoE. MIMO is all about taking advantage of instantaneous spatial diversity in the radio channel, and thus the measured performance is tightly coupled to the radio propagation, noise and interference conditions in which a device is tested. This extends to the closed loop behavior of the end-to-end system; that is, the real-time interaction between how the DUT measures the radio environment and the subsequent behavior of the base station scheduler, which can choose to reconfigure the downlink as frequently as 1000 times per second.

The state of play of MIMO OTA standardization in 3GPP is summarized in TR 37.976, which documents the progress of the study phase prior to formal standardization. Much of the discussion has been on methods for creating spatially diverse signals. A table contrasting the seven different proposals runs to 25 criteria, with most of the objective data still to be provided and agreed upon. Throughput is the main FOM, but six others are also being considered.

MIMO OTA Test Methods
The test methods for MIMO OTA can be broadly grouped into three categories: reverberation chamber (with and without external channel fading), anechoic chamber (using from 2 to 32 antennas) and multi-stage. The articles that follow from ETS-Lindgren and MI Technologies/Bluetest will cover the first two categories and so will not be discussed here other than to draw attention to the robust debate that exists over which method is viable or affordable.

Two methods are classed as multi-stage. The first and simpler method involves antenna pattern measurement2 from which various FOM such as correlation and gain imbalance can be developed, making it possible to compute the antenna impact on theoretical throughput. Since this method measures only part of the DUT and does not measure end-to-end throughput, the impact of secondary factors such as self-blocking (desensitization) are excluded. That said desensitization is largely covered by the SISO OTA tests. To accurately measure the antenna pattern on an unmodified handset requires the definition of a non-intrusive method. The CTIA has developed a standard format for documenting pattern information for GPS devices, but techniques used for pattern measurement are currently device-specific and proprietary. From a test perspective, the ideal solution for handsets would be to standardize a non-intrusive device test mode that allows the measurement of antenna relative gain and phase using the same anechoic chamber as for SISO OTA tests. This approach would require some development work but would be highly useful.

The other multi-stage proposal is called the two-stage method.3,4 The first stage is the same radiated antenna pattern measurement just described, although the calculation of FOM and theoretical throughput is not required. The second stage is a conducted test that combines the measured antenna pattern with any desired radio propagation environment using a channel emulator. The two output signals are then injected into the standard, temporary antenna ports used for conducted testing, and a throughput measurement is made with a signal that consists of the fading channel modified by the antenna pattern. The main advantage of the two-stage method is that it can reuse the existing simple SISO anechoic chamber for the antenna measurement and then, using only a two-port channel emulator, can emulate arbitrarily complex spatial channel conditions without the need for a large anechoic chamber and multiple probe antennas. With this brief overview of the proposed test methods we will now look at the primary FOM, MIMO throughput.

Figure 2 Shannon-bound spectral efficiency for rank 2 spatial multiplexing as a function of antenna correlation and SNR.

Theoretical MIMO Performance Gains
The SISO OTA requirements were defined retrospectively, so there could be no preconceived expectations. However, for MIMO to have any value, it has to demonstrate a gain over SISO. Figure 2 plots the spectral efficiency versus SNR for five different combinations of transmit (α) and receive (β) antenna correlation, showing how the theoretical spatial multi-plexing gain varies with SNR. Low antenna correlation is much easier to achieve at high frequencies e.g., above 1.7 GHz. In reality, the end-to-end correlation is continually varying due to the additional impact of the radio channel. Spirent's paper discusses this in more depth. The correlation can also vary widely within one channel bandwidth; to get the optimum MIMO gain out of the system, frequency-selective scheduling is required to target the best part of the channel. This is something that OFDM systems can do but CDMA systems cannot.

When MIMO performance is discussed, it is common to highlight the potential 100 percent spatial multi-plexing gains over SISO without mentioning that this only occurs with high SNR and low correlation. In real loaded networks, however, the median SNR is in the vicinity of 5 dB. Combining this median SNR with a realistic correlation value (which includes the impact of all antennas and the channel) gives realizable gains over SISO nearer to 20 percent than 100 percent. This performance difference is problematic for testing.

Efficacy of Test
The whole point of testing is to differentiate good performance from bad. If in realistic conditions the theoretical gain is limited, then the figure for acceptable performance needs to be somewhat lower. The challenge is to devise a test that can distinguish between SISO performance and limited MIMO gain. We know that the SISO OTA radio conditions can be controlled within ±2 dB and there are no obvious reasons why the more complex MIMO environment will be more accurate. If we then look at the impact of a possible 4 dB change in conditions on theoretical throughput, we see that the subtle gains we are trying to measure could easily be swamped.

Indeed, calibration of multi-probe test environments is proving to be a significant challenge, and failure to understand the factors that influence uncertainty always results in underestimation of their contribution. If five thermometer manufacturers independently predict the accuracy of their devices as less than 1 degree, but when the thermometers are exposed to the same environment the range of readings spans five degrees, you know there is a problem even if the theory does not yet exist to explain the cause. This is why the CTIA golden radio validation exercise was so important for the new discipline of OTA measurement.

It is reasonable to assume that 3GPP will relax the MIMO OTA minimum requirement by some proportion of the test uncertainty. With the figures suggested here, it would be easy to see how a DUT operating in SISO mode could then pass a MIMO test requirement, completely negating the efficacy or usefulness of the test.

The problem can be alleviated by testing under more extreme conditions (high SNR and low channel correlation) to maximize the expected gain, but there is no guarantee that the results obtained in near ideal conditions would correlate with those in real life conditions. Consider this analogy: If you were trying to differentiate the demodulation accuracy of two optical character recognition systems, would you use large, high contrast, uncorrelated characters such as WOXI or small, low contrast, correlated characters such as OCQD? My hunch is that if the demodulation target is easy, it is not useful as a performance differentiator. If for measurement accuracy reasons MIMO throughput gain turns out to be problematic as an FOM, an alternative option is to consider an antenna-only metric such as correlation or gain imbalance. The timeliness, cost and lower scope of such an approach need to be weighed against the alternatives.

Selecting MIMO OTA Test Methods and Figures of Merit
One practical way to distinguish useful FOM and test methods from the not so useful is to define two reference DUTs, one with known, agreed-upon good performance and the other with some controlled impairment such as a deliberate gain imbalance or highly correlated antenna pattern. The goal of any candidate FOM and supporting test method would then be to demonstrate repeatable and accurate differentiation of the good DUT from the impaired one. It would also be necessary to define more than one type of impairment since many DUT capabilities can impact performance in a closed loop MIMO system; for example, the timing and accuracy of the reporting of channel state information.

We might expect such capabilities to be covered by the conducted receiver tests, but for reasons of simplification these tests are executed primarily open loop, i.e., with fixed channel coding, regardless of radio conditions. Any optimized system relies on unspecified scheduling algorithms in the network. If tests are to be realistic, they need to be closed loop, but then it becomes necessary to precisely define the behavior of network algorithms so that differences in test equipment implementation do not impact the measured DUT performance. With the air interface becoming ever more complex, the gap between the simplified open loop testing we have become used to and real closed loop performance continues to grow.

Take LTE as an example. LTE supports seven different downlink transmission modes ranging from SISO through six forms of MIMO with transmit diversity and beam steering to fully precoded spatial multi-plexing. Each mode is designed to offer the best system performance for particular network conditions. When the radio conditions vary, a fully optimized system has to configure the DUT to measure the channel in the best way; then interpret the results to select the correct transmission mode and subsequent coding gain and rank.

Another teasing issue is the definition of noise. Some test scenarios are done without noise in order to highlight the issue of desensitization, while other tests are performed under realistic noise levels as seen in loaded networks. But what is this noise? Uniform Gaussian white noise is easy to generate and is relevant for testing CDMA systems, but due to frequency dependent scheduling in no way represents reality for OFDM systems. A real UE will be faced with statistically varying narrow band frequency hopping interference coming from different directions to that of the signal. If the test environment uses spatially uniform interference, a good DUT with low correlation will gain no advantage. This is unrealistic and unfair.

Conclusion
The big issue facing MIMO OTA test develpoment is the laudable but hugely challenging goal of measuring realistic end-user performance. Considerable work lies ahead to determine what level of sophistication is required in the test system to differentiate between a good MIMO DUT and one that is undesirable. SISO OTA standardization had the advantage that every DUT on the planet was a potential measurement candidate, but the lack of wide availability of a rich diversity of MIMO devices will limit the rate at which progress can be made towards a final MIMO OTA standard. Everyone agrees that testing needs to be no more complex, time consuming, or expensive than necessary, but it is clear from current proposals that the industry is still some way from agreeing on reference performance in specific conditions and the associated accuracy and method of test.

References

  1. Orange, "OTA TRP and TRS requirements for GSM 900 and 1800," 3GPP R4-091762.
  2. Qualcomm, "Concept for Multi-antenna Radiated Performance Test," 3GPP R4-101311.
  3. Agilent Technologies, "MIMO OTA test methodology proposal," 3GPP R4-091361.
  4. Agilent Technologies, "MIMO OTA experiment and validation for multiple probe antenna based method and two-stage method," 3GPP R4-101180.

Moray Rumney joined Hewlett-Packard/Agilent Technologies in 1984 after receiving a BSc degree in Electronics from Heriot-Watt University in Edinburgh. His career has spanned manufacturing engineering, product development, applications engineering, and most recently technical marketing. His main focus has been the development and system design of base station emulators used in the development and testing of cellular phones. Rumney joined ETSI in 1991 and 3GPP in 1999 where he was a significant contributor to the development of type approval tests for GSM and UMTS. He currently represents Agilent at 3GPP RAN WG4, developing the air interface for HSPA+ and LTE. His current focus is on radiated testing of MIMO devices. Rumney has published many technical articles and is a regular speaker and chairman at industry conferences.