The universal time-domain windowed-orthogonal frequency division multiplexing (UTW-OFDM) has been proposed for the 5G and beyond as one of the new-waveforms with low out-of-band emission (OOBE). Furthermore, the simplified UTW-OFDM without symbol overlapping process has been proposed for short-delay multipath fading channel. By applying a time-domain window with long transition duration, the simplified UTW-OFDM can significantly improve the OOBE reduction with low additional complexity, while communication quality is deteriorated by the inter-carrier interference (ICI) caused by the long transition duration, resulting in the limit of the OOBE reduction performance in the case of 64QAM. In this paper, we propose a frequency-domain ICI cancellation using a weight matrix based on the window shape for the simplified UTW-OFDM to improve the communication quality in the case of 64QAM. By computer simulation, the proposed ICI cancellation performance is evaluated using LTE downlink parameters under the Extended Pedestrian A (EPA) channel model defined by the 3GPP. The proposed ICI cancellation can improve the OOBE reduction performance of the simplified UTW-OFDM by about 25 dB in the case of 64QAM compared to the conventional cyclic prefix-based OFDM (CP-OFDM) without deteriorating the communication quality.