Quantization
i
Quantization
Reducing the number of bits used to store model numbers. This saves memory and can improve speed, with some loss of precision.
Reduce weight precision and inspect the memory savings and numerical distortion.
Numerical format
Model memory7.0 GB
Compression2×
RMSE0.002
i
RMSE
Root-mean-square error: one number summarizing how far quantized weights moved from their original values. Lower is better.
Fewer bits mean fewer representable values. The model gets smaller, but each weight moves to a coarser level.
Original FP16-1.00-0.72-0.44-0.180.080.310.570.831.00
INT8 quantized-1.00-0.72-0.44-0.180.080.310.570.831.00
Quantization error by weight