A new approach of an optimal 16-QAM modulation has been introduced to be used for both Fourier and wavelet based OFDM. The comparison between circular and square 16- QAM is included. It is shown that the power consumption of circular is better than the square by 0.5359 W or -2.71 dB for the same minimum distance 2d. Simulation works for Fourier and wavelet based OFDM have been performed. In Fourier base OFDM, the circular has slightly out-performed the square when the number of subcarriers is varied from 32 to 256 at different BER values. When considering wavelet based OFDM, all wavelet families except Biorthogonal and Reverse-Biorthogonal have performed similar result for both modulation schemes. Of all Biorthogonal families, bior5.5 is the most out-performed one when using circular and square 16-QAM respectively.