We perform an asymptotic study of the performance of filter bank multicarrier (FBMC) in the context of massive multi-input multi-output (MIMO). We show that the effects of channel distortions, i.e., intersymbol interference and intercarrier interference, do not vanish as the base station (BS) array size increases. As a result, the signal-to-interference-plus-noise ratio (SINR) cannot grow unboundedly by increasing the number of BS antennas, and is upper bounded by a certain deterministic value. We show that this phenomenon is a result of the correlation between the multi-antenna combining tap values and the channel impulse responses between the mobile terminals and the BS antennas. To resolve this problem, we introduce an efficient equalization method that removes this correlation, enabling us to achieve arbitrarily large SINR values by increasing the number of BS antennas. We perform a thorough analysis of the proposed system and find analytical expressions for both equalizer coeffici 查看全文>>