Welcome to Acta Agronomica Sinica，

Acta Agronomica Sinica ›› 2024, Vol. 50 ›› Issue (12): 3083-3095.doi: 10.3724/SP.J.1006.2024.42016

• TILLAGE & CULTIVATION·PHYSIOLOGY & BIOCHEMISTRY • Previous Articles     Next Articles

Growth visualized prediction method of multi-variety rice based on improved Pix2Pix-HD network

DUAN Ling-Feng1(), WANG Xin-Yi1, WANG Zhi-Hao1, GENG Ze-Dong2, LU Yun-Rui2, YANG Wan-Neng1,2,*()   

  1. 1College of Engineering, Huazhong Agricultural University, Wuhan 430070, Hubei, China
    2National Key Laboratory of Crop Genetic Improvement, Huazhong Agricultural University, Wuhan 430070, Hubei, China
  • Received:2024-03-10 Accepted:2024-08-15 Online:2024-12-12 Published:2024-09-02
  • Contact: *E-mail: ywn@mail.hzau.edu.cn
  • Supported by:
    National Natural Science Foundation of China(32170411);Hubei Hongshan Laboratory Open Project(2021hskf005)

RichHTML431

13

PDF

149

Abstract:

With the advancement of information technology and agricultural digitization, the visualized prediction of digital plant growth holds significant development potential. As an important food crop, achieving visualized growth prediction for rice is of great significance for analyzing its growth and development. However, due to the large size and complex morphology of rice, attaining high-precision and high-resolution visualized growth prediction has been challenging. This study proposes a growth prediction method for rice based on an improved Pix2Pix-HD network, achieving high-precision visualized growth prediction from the heading stage to the filling stage of rice. Comparative experiments were designed to verify the effectiveness of the model improvement scheme. The results showed that the FID, PSNR, and SSIM values between the predicted and real filling stage rice images in the test set were 24.75, 13.58, and 0.78, respectively. The average correlation coefficient between the predicted and actual phenotypes was 0.762, demonstrating good accuracy at different scales. The proposed data-driven rice growth prediction method can achieve high-resolution and high visual realism in rice growth visualization predictions, providing new solutions for rice growth forecasting.

Key words: rice, growth visualized prediction, generative adversarial network, Pix2Pix-HD, digital plants

Fig. 1

Overall technical process RGVP: proposed rice growth visualized predicter; Pix2Pix-HD: a generative adversarial network for image conversion based on deep learning."

Fig. 2

Network structure of Pix2Pix-HD G1: global generator; G2a: the first part of the local enhancer; G2b: the second part of the local enhancer; D1: 1024×1024 resolution scale discriminator; D2: 512×512 resolution scale discriminator; D3: 256×256 resolution scale discriminator."

Fig. 3

Improved network structure of global generator and discriminator in Pix2Pix-HD G1: global generator; D1: 1024×1024 resolution scale discriminator; RefP: reflection padding layer; Conv: convolution layer; BN: batch normalization layer; ReLU, LeakyReLU, Tanh, Sigmoid: different activation functions; ConvT: transpose convolution layer; Dropout: a regularization method."

Fig. 4

Examples of prediction results A1, A2, and A3: heading stage rice image; B1, B2, and B3: predicted filling stage rice image; C1, C2, and C3: ground truth filling stage rice image."

Table 1

Correlation coefficient between predicted phenotypes and real image phenotypes in phenotypic traits"

评价指标
Evaluation indicators		 形态特征相关系数
Correlation coefficient of
morphological traits		 纹理特征相关系数
Correlation coefficient of texture traits		 MCCM MCCT MCCA
TPA GPA YPA FD H W 直方图特征
Histogram		 灰度梯度共生矩阵特征
Gray-level co-occurrence matrix
M SE S MU3 U E C GM GDM GMSE GDMSE
CC 0.77 0.69 0.82 0.78 0.58 0.53 0.84 0.86 0.85 0.86 0.77 0.79 0.79 0.62 0.80 0.80 0.81 0.70 0.80 0.76
CCS 0.15 0.21 -0.05 0.17 0.01 0.27 0.06 0.03 0.06 -0.02 0.16 0.16 -0.08 -0.08 -0.05 -0.01 -0.02 0.13 0.02 0.06

Table 2

Comparison between predicted images and real images at different scales"

分辨率
Resolution		 视觉相似性Visualized similarity 表型准确性Phenotypic accuracy
Fréchet距离FID 峰值信噪比PSNR 结构相似度SSIM MCCM MCCT MCCA
1024×1024 24.75 13.58 0.78 0.70 0.80 0.76
512×512 21.51 14.47 0.73 0.69 0.78 0.75
256×256 40.69 15.77 0.68 0.67 0.70 0.69

Table 3

Comparison of model performance with different improvements"

模型 Model 模型改进设计方案
Model improvement		 视觉相似性
Visualized similarity		 表型准确性
Phenotypic accuracy
训练阶段
随机失活
Training dropout		 测试阶段
随机失活
Testing dropout		 假标签损失
Fake label loss		 空间注意力
机制
Spatial attention		 通道注意力
机制
Channel
attention		 Fréchet距离
FID		 峰值信噪比 PSNR 结构相似度 SSIM MCCM MCCT MCCA
模型1
Model 1		 √ √ × × × 30.44 13.27 0.77 0.64 0.72 0.69
模型2
Model 2		 × × × × × 26.10 13.76 0.78 0.62 0.74 0.70
模型3
Model 3		 √ × × × × 37.60 13.46 0.78 0.66 0.78 0.74
模型4
Model 4		 √ × √ × × 29.65 13.58 0.78 0.65 0.78 0.74
模型5
Model 5		 √ × √ √ × 24.75 13.58 0.78 0.70 0.80 0.76
模型6
Model 6		 √ × √ √ √ 32.80 13.72 0.78 0.68 0.81 0.76

Fig. 5

Comparison of prediction results between Pix2Pix model and RGVP model A1, A2, and A3: heading stage rice image; B1, B2, and B3: Pix2Pix predicted filling stage rice image; C1, C2, and C3: RGVP predicted filling stage rice image; D1, D2, and D3: ground truth filling stage rice image; Pix2Pix: a generative adversarial network for image conversion based on deep learning; RGVP: proposed rice growth visualized predicter."

Table 4

Comparison between Pix2Pix model and RGVP model at two scales"

模型及处理
Model and process		 分辨率
Resolution		 视觉相似性Visualized similarity 表型准确性Phenotypic accuracy
Fréchet距离
FID		 峰值信噪比
PSNR		 结构相似度
SSIM		 MCCM MCCT MCCA
Pix2Pix模型
Pix2Pix model		 256×256 45.60 14.53 0.65 0.50 0.63 0.58
RGVP + 1/4下采样
RGVP + 1/4 Down-sampling		 256×256 40.69 15.77 0.68 0.67 0.70 0.69
Pix2Pix + 4倍超分辨
Pix2Pix + 4× super-resolution		 1024×1024 68.91 13.31 0.76 0.49 0.64 0.59
RGVP模型
RGVP model		 1024×1024 24.75 13.58 0.78 0.70 0.80 0.76
[1] 梁伟军. 农业与相关产业融合发展研究. 华中农业大学博士学位论文, 湖北武汉, 2010.
Liang W J. Research on the Integration and Development of Agriculture and Related Industries. PhD Dissertation of Huazhong Agricultural University, Wuhan, Hubei, China, 2010 (in Chinese with English abstract).
[2] 赵春江, 陆声链, 郭新宇, 肖伯祥, 温维亮. 数字植物及其技术体系探讨. 中国农业科学, 2010, 43: 2023-2030.
Zhao C J, Lu S L, Guo X Y, Xiao B X, Wen W L. Exploration of digital plants and their technical system. Sci Agric Sin, 2010, 43: 2023-2030 (in Chinese with English abstract).
[3] 张洪程, 王夫玉. 中国水稻群体研究进展. 中国水稻科学, 2001, 15: 51-56.
Zhang H C, Wang F Y. Research progress on rice population in China. Chin J Rice Sci, 2001, 15: 51-56 (in Chinese with English abstract).
[4] 程式华, 廖西元, 闵绍楷. 中国超级稻研究: 背景、目标和有关问题的思考. 中国稻米, 1998, (1): 3-5.
Cheng S H, Liao X Y, Min S K. Research on Chinese super rice: background, objectives, and reflections on related issues. China Rice, 1998, (1): 3-5 (in Chinese).
[5] 邹应斌, 周上游, 唐起源. 中国超级杂交水稻超高产栽培研究的现状与展望. 中国农业科技导报, 2003, 5(1): 31-35.
Zou Y B, Zhou S Y, Tang Q Y. The current status and prospects of research on super high yield cultivation of Chinese super hybrid rice. J Agric Sci Technol, 2003, 5(1): 31-35 (in Chinese with English abstract).
[6] Jung K H, An G, Ronald P C. Towards a better bowl of rice: assigning function to tens of thousands of rice genes. Nat Rev Genet, 2008, 9: 91-101.
[7] Alhnaity B, Kollias S, Leontidis G, Jiang S Y, Schamp B, Pearson S. An autoencoder wavelet based deep neural network with attention mechanism for multi-step prediction of plant growth. Inform Sci, 2021, 560: 35-50.
[8] Alhnaity B, Pearson S, Leontidis G, Kollias S. Using deep learning to predict plant growth and yield in greenhouse environments. Int Soc Hortic Sci, 2019, 1296: 425-432.
[9] 张慧春, 王国苏, 边黎明, 郑加强, 周宏平. 基于光学相机的植物表型测量系统与时序生长模型研究. 农业机械学报, 2019, 50(10): 197-207.
Zhang H C, Wang G S, Bian L M, Zheng J Q, Zhou H P. Research on plant phenotype measurement system and temporal growth model based on optical cameras. Trans CSAE, 2019, 50(10): 197-207 (in Chinese with English abstract).
[10] 朱新广, 常天根, 宋青峰, 常硕其, 王重荣, 张国庆, 郭亚, 周少川. 数字植物: 科学内涵、瓶颈及发展策略. 合成生物学, 2020, 1: 285-297.
doi: 10.12211/2096-8280.2020-018
Zhu X G, Chang T G, Song Q F, Chang S Q, Wang C R, Zhang G Q, Guo Y, Zhou S C. Plants: scientific connotation, bottlenecks, and development strategies. Syn Biol J, 2020, 1: 285-297 (in Chinese with English abstract).
[11] Lindenmayer A. Mathematical models for cellular interactions in development. J Theor Biol, 1968, 18: 300-315.
pmid: 5659072
[12] Leitner D, Klepsch S, Knie A, Schnepf A. The algorithmic beauty of plant roots: an L-System model for dynamic root growth simulation. Math Comput Model Dyn Sys, 2010, 16: 575-587.
[13] Espana M, Baret F, Aries F, Chelle M, Andrieu B, Prevot L. Modeling maize canopy 3D architecture: application to reflectance simulation. Ecol Model, 1999, 122: 25-43.
[14] Jallas E, Sequeira R, Martin P, Turner S, Papajorgji P. Mechanistic cirtual modeling: coupling a plant simulation model with a three-dimensional plant architecture component. Environ Model Assess, 2009, 14: 29-45.
[15] Qian B, Huang W J, Xie D H, Ye H C, Guo A, Pan Y H, Jin Y, Xie Q Y, Jiao Q J, Zhang B Y, Ruan C, Xu T J, Zhang Y, Nie T G. Coupled maize model: a 4D maize growth model based on growing degree days. Comput Electron Agric, 2023, 212: 108124.
[16] Minorsky P V. Achieving the in silico plant. Systems biology and the future of plant biological research. Plant Physiol, 2003, 132: 404-409.
pmid: 12822566
[17] Prusinkiewicz P. Modeling plant growth and development. Curr Opin Plant Biol, 2004, 7: 79-83.
pmid: 14732445
[18] 肖亚, 李玉强. 农业人工智能综述. 数字技术与应用, 2020, 38(9): 204-205.
Xiao Y, Li Y Q. Overview of agricultural artificial intelligence. Digit Technol Appl, 2020, 38(9): 204-205 (in Chinese with English abstract).
[19] Fahlgren N, Gehan M A, Baxter I. Lights, camera, action: high-throughput plant phenotyping is ready for a close-up. Curr Opin Plant Biol, 2015, 24: 93-99.
doi: 10.1016/j.pbi.2015.02.006 pmid: 25733069
[20] Kim T H, Lee S H, Oh M M, Kim J O. Plant growth prediction based on hierarchical auto-encoder. In: International Conference on Electronics, Information, and Communication. Jeju: IEEE, 2022. pp 1-3.
[21] Kim T H, Lee S H, Kim J O. A novel shape based plant growth prediction algorithm using deep learning and spatial transformation. IEEE Access, 2022, 10: 37731-37742.
[22] Sakurai S, Uchiyama H, Shimada A, Taniguchi R. Plant growth prediction using convolutional LSTM. VISIGRAPP, 2019, 14: 105-113.
[23] 王春颖, 泮玮婷, 李祥, 刘平. 基于STLSTM的植物生长发育预测模型. 农业机械学报, 2022, 53(6): 250-258.
Wang C Y, Pan W T, Li X, Liu P. A plant growth and development prediction model based on STLSTM. Trans CSAM, 2022, 53(6): 250-258 (in Chinese with English abstract).
[24] Wang C, Pan W, Song X, Yu H, Zhu J, Liu P, Li X. Predicting plant growth and development using time-series images. Agronomy, 2022, 12: 2213.
[25] Yasrab R, Zhang J, Smyth P, Pound M P. Predicting plant growth from time-series data using deep learning. Remote Sens, 2021, 13: 331.
[26] Drees L, Junker-Frohn L V, Kierdorf J, Roscher R. Temporal prediction and evaluation of Brassica growth in the field using conditional generative adversarial networks. Comput Electron Agric, 2021, 190: 106415.
[27] Meng Y, Xu M, Yoon S, Jeong Y, Park D S. Flexible and high-quality plant growth prediction with limited data. Front Plant Sci, 2022, 13: 989304.
[28] Yang W, Guo Z, Huang C, Duan L, Chen G, Jiang N, Fang W, Feng H, Xie W, Lian X. Combining high-throughput phenotyping and genome-wide association studies to reveal natural genetic variation in rice. Nat Commun, 2014, 5: 5087.
doi: 10.1038/ncomms6087 pmid: 25295980
[29] Wang T C, Liu M Y, Zhu J Y, Tao A, Kautz J, Catanzaro B. High-resolution image synthesis and semantic manipulation with conditional GANs. Comput Vis Pattern Recognit, 2017. pp 1-14.
[30] Woo S, Park J, Lee J Y, Kweon I S. CBAM:convolutional block attention module. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y, eds. Proceedings of the European Conference on Computer Vision. Munich: Springer Cham, 2018. pp 3-19.
[31] Dowson D C, Landau B V. The fréchet distance between multivariate normal distributions. J Multivariate Anal, 1982, 12: 450-455.
[32] Korhonen J, You J. Peak signal-to-noise ratio revisited: is simple beautiful?. In: Fourth International Workshop on Quality of Multimedia Experience. Melbourne: IEEE, 2012. pp 37-38.
[33] Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Proc, 2004, 13: 600-612.
[34] Wang X, Xie L, Dong C, Shan Y. Real-esrgan:training real-world blind super-resolution with pure synthetic data. In:Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021. pp 1905-1914.
[1] ZHAO Li-Ming, DUAN Shao-Biao, XIANG Hong-Tao, ZHENG Dian-Feng, FENG Nai-Jie, SHEN Xue-Feng. Effects of alternate wetting and drying irrigation and plant growth regulators on photosynthetic characteristics and endogenous hormones of rice [J]. Acta Agronomica Sinica, 2025, 51(1): 174-188.
[2] JIA Shu-Han, HE Can, CHEN Min, LIU Jia-Xin, HU Wei-Min, HU Jin, GUAN Ya-Jing. Study on the quality differences of seeds with different pre-harvest sprouting levels and the grading of pre-harvest sprouting in hybrid rice [J]. Acta Agronomica Sinica, 2024, 50(9): 2310-2322.
[3] HU Li-Qin, XIAO Zheng-Wu, FANG Sheng-Liang, CAO Fang-Bo, CHEN Jia-Na, HUANG Min. Effects of planting season on digestive characteristics of high amylose content rice [J]. Acta Agronomica Sinica, 2024, 50(9): 2347-2357.
[4] LIU Chen, WANG Kun-Kun, LIAO Shi-Peng, YANG Jia-Qun, CONG Ri-Huan, REN Tao, LI Xiao-Kun, LU Jian-Wei. Effects of nitrogen fertilizer application levels on yield and nitrogen absorption and utilization of oilseed rape under maize-oilseed rape and rice-oilseed rape rotation fields [J]. Acta Agronomica Sinica, 2024, 50(8): 2067-2077.
[5] SONG Zhi-Wen, ZHAO Lei, BI Jun-Guo, TANG Qing-Yun, WANG Guo-Dong, LI Yu-Xiang. Effects of nitrogen fertilization levels on matter accumulation and nutrient uptake in rice cultivar with different nitrogen efficiency under drip irrigation [J]. Acta Agronomica Sinica, 2024, 50(8): 2025-2038.
[6] SHAO Mei-Hong, ZHAO Ling-Ling, CHENG Chu, CHENG Si-Ming, ZHU Shuang-Bing, ZHAI Lai-Yuan, CHEN Kai, XU Jian-Long. Screening, evaluation, and utilization of low nitrogen tolerance for the selected introgression lines in rice with Huanghuazhan background [J]. Acta Agronomica Sinica, 2024, 50(8): 1907-1919.
[7] HE Dan-Dan, SHU Ya-Zhou, ZHOU Hai-Lian, WU Song-Guo, WEI Xiao-Shuang, YANG Ming-Chong, LI Bo, WU Zheng-Dan, HAN Shi-Jian, YANG Juan, WANG Ji-Bin, WANG Ling-Qiang. OsRPTA18 participated in the regulation of leaf inclination in rice [J]. Acta Agronomica Sinica, 2024, 50(8): 1934-1947.
[8] GUO Chun-Lin, LIN Man-Hong, CHEN Ting, CHEN Hong-Fei, LIN Wen-Fang, LIN Wen-Xiong. Evolution characteristics of rhizosphere microorganisms in response to ratoon rice senescence and underlying carry-over effect mechanism [J]. Acta Agronomica Sinica, 2024, 50(8): 2039-2052.
[9] FU Jing, MA Meng-Juan, ZHANG Qi-Fei, DUAN Ju-Qi, WANG Yue-Tao, WANG Fu-Hua, WANG Sheng-Xuan, BAI Tao, YIN Hai-Qing, WANG Ya. Effects of alternate wetting and drying irrigation and different nitrogen application levels on photosynthetic characteristics and nitrogen absorption and utilization of japonica rice [J]. Acta Agronomica Sinica, 2024, 50(7): 1787-1804.
[10] CHENG Shuang, XING Zhi-Peng, TIAN Chao, HU Qun, WEI Hai-Yan, ZHANG Hong-Cheng. Effects of an integrated dryland tillage and soaking pattern on the reducing substances in rice field and early growth of machine transplanted rice [J]. Acta Agronomica Sinica, 2024, 50(7): 1762-1775.
[11] PEI Fa-Jing, ZHANG Wen-Xuan, ZHANG Xiao, WANG Xin-Yu, PENG Shao-Bing, MI Jia-Ming. Developing new rice lines with ultrashort-duration, long-grain, and fragrance [J]. Acta Agronomica Sinica, 2024, 50(7): 1684-1698.
[12] TANG Qing-Yun, YANG Jing-Jing, ZHAO Lei, SONG Zhi-Wen, WANG Guo-Dong, LI Yu-Xiang. Effect of nitrogen application on morphological conformation and fractal characteristics of drip irrigated rice roots [J]. Acta Agronomica Sinica, 2024, 50(6): 1540-1553.
[13] ZHANG Xiao-Fang, ZHU Qi, HUA Yun-Yan, JIA Li-Hui-Ying, QIU Shi-You, CHEN Yu-Jie, MA Tao, DING Wo-Na. Screening and validation of OsCYP22 interacting proteins in rice [J]. Acta Agronomica Sinica, 2024, 50(6): 1628-1634.
[14] ZHU Zhong-Lin, WEN Yue, ZHOU Qi, WU Yan-Fei, DU Xue-Zhu, SHENG Feng. Mechanism of loding residence and drought tolerance of OsCNGC10 gene in rice [J]. Acta Agronomica Sinica, 2024, 50(5): 1351-1360.
[15] HU Ming-Ming, DING Feng, PENG Zhi-Yun, XIANG Kai-Hong, LI Yu, ZHANG Yu-Jie, YANG Zhi-Yuan, SUN Yong-Jian, MA Jun. Effects of straw returning to field combined with water and N management on rice yield formation and N uptake and utilization under diversified cropping patterns [J]. Acta Agronomica Sinica, 2024, 50(5): 1236-1252.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] Li Shaoqing, Li Yangsheng, Wu Fushun, Liao Jianglin, Li Damo. Optimum Fertilization and Its Corresponding Mechanism under Complete Submergence at Booting Stage in Rice[J]. Acta Agronomica Sinica, 2002, 28(01): 115 -120 .
[2] Wang Lanzhen;Mi Guohua;Chen Fanjun;Zhang Fusuo. Response to Phosphorus Deficiency of Two Winter Wheat Cultivars with Different Yield Components[J]. Acta Agron Sin, 2003, 29(06): 867 -870 .
[3] WANG Yan;QIU Li-Ming;XIE Wen-Juan;HUANG Wei;YE Feng;ZHANG Fu-Chun;MA Ji. Cold Tolerance of Transgenic Tobacco Carrying Gene Encoding Insect Antifreeze Protein[J]. Acta Agron Sin, 2008, 34(03): 397 -402 .
[4] ZHENG Xi;WU Jian-Guo;LOU Xiang-Yang;XU Hai-Ming;SHI Chun-Hai. Mapping and Analysis of QTLs on Maternal and Endosperm Genomes for Histidine and Arginine in Rice (Oryza sativa L.) across Environments[J]. Acta Agron Sin, 2008, 34(03): 369 -375 .
[5] XING Guang-Nan, ZHOU Bin, ZHAO Tuan-Jie, YU De-Yue, XING Han, HEN Shou-Yi, GAI Jun-Yi. Mapping QTLs of Resistance to Megacota cribraria (Fabricius) in Soybean[J]. Acta Agronomica Sinica, 2008, 34(03): 361 -368 .
[6] ZHENG Yong-Mei;DING Yan-Feng;WANG Qiang-Sheng;LI Gang-Hua;WANG Hui-Zhi;WANG Shao-Hua. Effect of Nitrogen Applied before Transplanting on Tillering and Nitrogen Utilization in Rice[J]. Acta Agron Sin, 2008, 34(03): 513 -519 .
[7] QIN En-Hua;YANG Lan-Fang;. Selenium Content in Seedling and Selenium Forms in Rhizospheric Soil of Nicotiana tabacum L.[J]. Acta Agron Sin, 2008, 34(03): 506 -512 .
[8] LÜ Li-Hua;TAO Hong-Bin;XIA Lai-Kun; HANG Ya-Jie;ZHAO Ming;ZHAO Jiu-Ran;WANG Pu;. Canopy Structure and Photosynthesis Traits of Summer Maize under Different Planting Densities[J]. Acta Agron Sin, 2008, 34(03): 447 -455 .
[9] Zhang Shubiao;Yang Rencui. Some Biological Character of eui-hybrid Rice[J]. Acta Agron Sin, 2003, 29(06): 919 -924 .
[10] SHAO Rui-Xin;SHANG-GUAN Zhou-Ping. Effects of Exogenous Nitric Oxide Donor Sodium Nitroprusside on Photosynthetic Pigment Content and Light Use Capability of PS II in Wheat under Water Stress[J]. Acta Agron Sin, 2008, 34(05): 818 -822 .
Add: No. 80 Xueyuan South Rd, Beijing 100081, P. R. China E-mail: zwxb301@caas.cn
Supported by Beijing Magtech Co.Ltd