The robot device that is being addressed in this research has two arms: one for picking the fruit and the other for chopping it. The arms find and locate pods with the help of a complex vision system that employs cameras. In this human-robot workflow, the operator chooses the tomatoes they want picked, and then the robot does the actual picking. The robot management and communication system use the EtherCAT bus to create a link with the graphical user interface (GUI), enabling human administration and control. The objective of this project is to create and assess a robotic system for harvesting tomatoes, equipped with dual arms. This system incorporates a mobile model equipped with two robotic arms and an end effector to enhance the efficiency of tomato harvesting. The system uses a GUI to enhance interaction between the robot and the human operator. Additionally, it employs a vision model to streamline the process of fruit detection. Findings from this study demonstrate that HMI may significantly improve the accuracy of tomato harvesting robots. Finally, there were some difficulties in developing 3D models because this study included outdoor experiments.
Keywords
Tomato Harvesting, Dual-Arm Robotic System, End-Effectors, Vision System, Communication System, Motion Control, Graphic User Interface.
L. Kumar et al., “Climate change and future of agri-food production,” in Elsevier eBooks, 2022, pp. 49–79. doi: 10.1016/b978-0-323-910019.00009-8.
J. M. Alho, “Population forecasts,” in Elsevier eBooks, 2001, pp. 11789–11793. doi: 10.1016/b0-08-043076-7/02114-8.
Y. Zhao, L. Gong, Y. Huang, and C. Liu, “A review of key techniques of vision-based control for harvesting robot,” Computers and Electronics in Agriculture, vol. 127, pp. 311–323, Sep. 2016, doi: 10.1016/j.compag.2016.06.022.
A. Gongal, S. Amatya, M. Karkee, Q. Zhang, and K. M. Lewis, “Sensors and systems for fruit detection and localization: A review,” Computers and Electronics in Agriculture, vol. 116, pp. 8–19, Aug. 2015, doi: 10.1016/j.compag.2015.05.021.
W. C. Seng and S. H. Mirisaee, “A new method for fruits recognition system,” 2009 International Conference on Electrical Engineering and Informatics, Aug. 2009, doi: 10.1109/iceei.2009.5254804.
P. Kurtser and Y. Edan, “Planning the sequence of tasks for harvesting robots,” Robotics and Autonomous Systems, vol. 131, p. 103591, Sep. 2020, doi: 10.1016/j.robot.2020.103591.
N. Noguchi, J. D. Will, J. F. Reid, and Q. Zhang, “Development of a master–slave robot system for farm operations,” Computers and Electronics in Agriculture, vol. 44, no. 1, pp. 1–19, Jul. 2004, doi: 10.1016/j.compag.2004.01.006.
B. Li, Y. Ling, H. Zhang, and S. Zheng, “The design and realization of cherry tomato harvesting robot based on IOT,” International Journal of Online Engineering, vol. 12, no. 12, p. 23, Dec. 2016, doi: 10.3991/ijoe.v12i12.6450.
J. P. Vásconez, G. Kantor, and F. A. Cheein, “Human–robot interaction in agriculture: A survey and current challenges,” Biosystems Engineering, vol. 179, pp. 35–48, Mar. 2019, doi: 10.1016/j.biosystemseng.2018.12.005.
J. E. Slotine and S. S. Sastry, “Tracking control of non-linear systems using sliding surfaces, with application to robot manipulators†,” International Journal of Control, vol. 38, no. 2, pp. 465–492, Aug. 1983, doi: 10.1080/00207178308933088.
W. Ji, D. Zhao, F. Cheng, B. Xu, Y. Zhang, and J. Wang, “Automatic recognition vision system guided for apple harvesting robot,” Computers & Electrical Engineering, vol. 38, no. 5, pp. 1186–1195, Sep. 2012, doi: 10.1016/j.compeleceng.2011.11.005.
C. J. Hohimer, H. Wang, S. Bhusal, J. H. Miller, C. Mo, and M. Karkee, “Design and Field Evaluation of a Robotic Apple Harvesting System with a 3D-Printed Soft-Robotic End-Effector,” Transactions of the ASABE, vol. 62, no. 2, pp. 405–414, Jan. 2019, doi: 10.13031/trans.12986.
H. Zhou, X. Wang, W. Au, H. Kang, and C. Chen, “Intelligent robots for fruit harvesting: recent developments and future challenges,” Research Square (Research Square), Aug. 2021, doi: 10.21203/rs.3.rs-497056/v1.
J. R. Davidson, A. Silwal, C. J. Hohimer, M. Karkee, C. Mo, and Q. Zhang, “Proof-of-concept of a robotic apple harvester,” 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct. 2016, doi: 10.1109/iros.2016.7759119.
Z. Zhang, C. Igathinathane, J. Li, H. Cen, Y. Lu, and P. Flores, “Technology progress in mechanical harvest of fresh market apples,” Computers and Electronics in Agriculture, vol. 175, p. 105606, Aug. 2020, doi: 10.1016/j.compag.2020.105606.
Y. Hua et al., “Recent advances in intelligent automated fruit harvesting robots,” The Open Agriculture Journal, vol. 13, no. 1, pp. 101–106, Aug. 2019, doi: 10.2174/1874331501913010101.
H. Zhou, X. Wang, W. Au, H. Kang, and C. Chen, “Intelligent robots for fruit harvesting: recent developments and future challenges,” Precision Agriculture, vol. 23, no. 5, pp. 1856–1907, Jun. 2022, doi: 10.1007/s11119-022-09913-3.
T. Li, F. Xie, Z. Zhao, H. Zhao, X. Guo, and Q. Feng, “A multi-arm robot system for efficient apple harvesting: Perception, task plan and control,” Computers and Electronics in Agriculture, vol. 211, p. 107979, Aug. 2023, doi: 10.1016/j.compag.2023.107979.
B. Zion, M. P. Mann, D. Levin, A. Shilo, D. Rubinstein, and I. Shmulevich, “Harvest-order planning for a multiarm robotic harvester,” Computers and Electronics in Agriculture, vol. 103, pp. 75–81, Apr. 2014, doi: 10.1016/j.compag.2014.02.008.
E. Navas, R. Fernández, D. Sepúlveda, M. Armada, and P. G. De Santos, “Modular Dual-Arm robot for precision harvesting,” in Advances in intelligent systems and computing, 2019. doi: 10.1007/978-3-030-36150-1_13.
M. Lio and M.-C. Liu, “Governance and agricultural productivity: A cross-national analysis,” Food Policy, vol. 33, no. 6, pp. 504–512, Dec. 2008, doi: 10.1016/j.foodpol.2008.06.003.
T. Yang, J. E. Altland, and U. C. Samarakoon, “Evaluation of substrates for cucumber production in the Dutch bucket hydroponic system,” Scientia Horticulturae, vol. 308, p. 111578, Jan. 2023, doi: 10.1016/j.scienta.2022.111578.
“SWEEPER, the sweet pepper harvesting robot,” WUR. https://www.wur.nl/en/project/SWEEPER-the-sweet-pepper-harvesting-robot.htm
M. Y. H. Khajepour et al., “REACH: Robotic Equipment for Automated Crystal Harvesting using a six-axis robot arm and a micro-gripper,” Acta Crystallographica Section D: Structural Biology, vol. 69, no. 3, pp. 381–387, Feb. 2013, doi: 10.1107/s0907444912048019.
E. J. Van Henten, D. a. V. Slot, C. W. J. Hol, and L. G. Van Willigenburg, “Optimal manipulator design for a cucumber harvesting robot,” Computers and Electronics in Agriculture, vol. 65, no. 2, pp. 247–257, Mar. 2009, doi: 10.1016/j.compag.2008.11.004.
K. Sahoo, R. Bergman, S. Alanya‐Rosenbaum, H. Gu, and S. Liang, “Life Cycle Assessment of Forest-Based Products: A review,” Sustainability, vol. 11, no. 17, p. 4722, Aug. 2019, doi: 10.3390/su11174722.
C. Lehnert, A. English, C. McCool, A. W. Tow, and T. Pérez, “Autonomous sweet pepper harvesting for protected cropping systems,” IEEE Robotics and Automation Letters, vol. 2, no. 2, pp. 872–879, Apr. 2017, doi: 10.1109/lra.2017.2655622.
I. Sa et al., “PeDuncle detection of sweet pepper for Autonomous Crop Harvesting—Combined color and 3-D information,” IEEE Robotics and Automation Letters, vol. 2, no. 2, pp. 765–772, Apr. 2017, doi: 10.1109/lra.2017.2651952.
B. Arad et al., “Development of a sweet pepper harvesting robot,” Journal of Field Robotics, vol. 37, no. 6, pp. 1027–1039, Jan. 2020, doi: 10.1002/rob.21937.
P. M. Zapotezny-Anderson and C. Lehnert, “Towards active Robotic Vision in agriculture: a deep learning approach to visual servoing in occluded and unstructured protected cropping environments,” IFAC-PapersOnLine, vol. 52, no. 30, pp. 120–125, Jan. 2019, doi: 10.1016/j.ifacol.2019.12.508.
G. Kootstra, X. Wang, P. M. Blok, J. Hemming, and E. Van Henten, “Selective Harvesting Robotics: current research, trends, and future directions,” Current Robotics Reports, vol. 2, no. 1, pp. 95–104, Jan. 2021, doi: 10.1007/s43154-020-00034-1.
S. Birrell, J. Hughes, J. Y. Cai, and F. Iida, “A field‐tested robotic harvesting system for iceberg lettuce,” Journal of Field Robotics, vol. 37, no. 2, pp. 225–245, Jul. 2019, doi: 10.1002/rob.21888.
G. Quaglia, C. Visconte, L. Carbonari, A. Botta, and P. Cavallone, “Agri.Q: a sustainable rover for precision agriculture,” in Springer proceedings in energy, 2020, pp. 81–91. doi: 10.1007/978-3-030-55757-7_6.
Y. Tang et al., “Recognition and Localization Methods for Vision-Based Fruit Picking Robots: A review,” Frontiers in Plant Science, vol. 11, May 2020, doi: 10.3389/fpls.2020.00510.
A. Haldorai, A. Ramu, and S. Murugan, “Smart Sensor Networking and Green Technologies in Urban Areas,” Computing and Communication Systems in Urban Development, pp. 205–224, 2019, doi: 10.1007/978-3-030-26013-2_10.
V. Praba and K. Krishnaveni, “A review on crop disease detection techniques,” in Lecture notes in electrical engineering, 2023, pp. 435–448. doi: 10.1007/978-981-19-8136-4_36.
T. Yoshida, Y. Onishi, T. Kawahara, and T. Fukao, “Automated harvesting by a dual-arm fruit harvesting robot,” ROBOMECH Journal, vol. 9, no. 1, Sep. 2022, doi: 10.1186/s40648-022-00233-9.
G. Brahmanage and H. Leung, “Outdoor RGB-D Mapping Using Intel-RealSense,” 2019 IEEE SENSORS, Oct. 2019, doi: 10.1109/sensors43011.2019.8956916.
A. K. Pothula, Z. Zhang, and R. Lu, “Evaluation of a new apple in-field sorting system for fruit singulation, rotation and imaging,” Computers and Electronics in Agriculture, vol. 208, p. 107789, May 2023, doi: 10.1016/j.compag.2023.107789.
T. Zhu et al., “A calculation method of phenotypic traits based on three-dimensional reconstruction of tomato canopy,” Computers and Electronics in Agriculture, vol. 204, p. 107515, Jan. 2023, doi: 10.1016/j.compag.2022.107515.
S. B. Uzayr, “Introduction to Visual Studio Code,” in Apress eBooks, 2021, pp. 1–46. doi: 10.1007/978-1-4842-7344-9_1.
J. Hughes, F. Lida, and S. Birrell, “Field robotics for harvesting,” in Elsevier eBooks, 2022, pp. 69–94. doi: 10.1016/b978-0-12-817634-4.000094.
J. Liu, P. Li, and Z. Li, “A Multi-Sensory End-Effector for Spherical Fruit Harvesting Robot,” 2007 IEEE International Conference on Automation and Logistics, Aug. 2007, doi: 10.1109/ical.2007.4338567.
T. Yoshikawa, “Manipulability of robotic mechanisms,” The International Journal of Robotics Research, vol. 4, no. 2, pp. 3–9, Jun. 1985, doi: 10.1177/027836498500400201.
M. Pan et al., “Soft actuators and robotic devices for rehabilitation and assistance,” Advanced Intelligent Systems, vol. 4, no. 4, Nov. 2021, doi: 10.1002/aisy.202100140.
J. Gao et al., “Development and evaluation of a pneumatic finger-like end-effector for cherry tomato harvesting robot in greenhouse,” Computers and Electronics in Agriculture, vol. 197, p. 106879, Jun. 2022, doi: 10.1016/j.compag.2022.106879.
Acknowledgements
Author(s) thanks to Dr. John Mackey for this research validation and verification support.
Funding
No funding was received to assist with the preparation of this manuscript.
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Availability of data and materials
No data available for above study.
Author information
Contributions
All authors have equal contribution in the paper and all authors have read and agreed to the published version of the manuscript.
Corresponding author
Stanisław Lem
Stanisław Lem
Data Science and Artificial Intelligence, University of Hamburg, Hamburg, Germany.
Open Access This article is licensed under a Creative Commons Attribution NoDerivs is a more restrictive license. It allows you to redistribute the material commercially or non-commercially but the user cannot make any changes whatsoever to the original, i.e. no derivatives of the original work. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/4.0/
Cite this article
Stanisław Lem and John Mackey, “Field Performance of a Dual Arm Robotic System for Efficient Tomato Harvesting”, Journal of Robotics Spectrum, vol.2, pp. 066-075, 2024. doi: 10.53759/9852/JRS202402007.
