Coaxial sensors are effective for measurement of dielectric properties of biological tissues. Several measurement methods used to derive dielectric properties have been investigated for the measurement with a coaxial sensor at microwave frequencies. While the measurement accuracy depends on the method used, there has been insufficient intercomparison of these methods and their model approximation errors. On the other hand, we have developed a coaxial sensor for the measurement of complex permittivity at millimeter-wave (MMW) frequencies of up to 100 GHz. However, the scarcity of reference data at MMW frequencies makes the validation of the measurement system difficult. Thus, it is essential to clarify the model approximation error of the method used in the measurement system, particularly at MMW frequencies. This study aims to clarify the model approximation errors of methods for dielectric property measurement using a coaxial sensor at MMW frequencies. The model approximation errors were assessed by comparing results obtained by the methods with those based on the theoretical formula of the full-wave modal expression of Maxwell's equations. The measurement uncertainty for the theoretical formula was estimated for a standard liquid sample to clarify the contribution of the model approximation errors to the uncertainty. Furthermore, the methods were applied to the measurement of porcine tissues at body temperature, and the measurement accuracy and usability for measurement at MMW frequencies are discussed.