Deep learning accelerators are a specialized sort of hardware architecture designed to enhance the computational efficiency of computers engaged in deep neural networks (DNNs) training. The implementation of DNNs in embedded vision applications might potentially be facilitated by the integration of energy-effective accelerators of deep learning into sensors. The lack of recognition for their significant impact on accuracy is a notable oversight. In previous iterations of deep learning accelerators integrated inside sensors, a common approach was bypassing the image signal processor (ISP). This deviation from the traditional vision pipelines had a detrimental impact on the performance of machine learning models trained on data that had undergone post-ISP processing. In this study, we establish a set of energy-efficient techniques that allow ISP to maximize their advantages while also limiting the covariate shift between the target dataset (RAW images) and the training dataset (ISP-analyzed images). This approach enables the practical use of in-sensor accelerators. To clarify, our results do not minimize the relevance of in-sensor accelerators. Instead, we highlight deficiencies in the methodology used in prior research and propose methodologies that empower in-sensor accelerators to fully exploit their capabilities.
Keywords
Image Signal Processor, Digital Signal Processor, Image Processing Unit, Deep Neural Networks
Z. Zhou et al., “Operation of AIMS in deep learning workflow,” ASVIDE, vol. 10, pp. 137–137, 2023.
Lokesh, Chandana, “Vlsi modeling of high performance digital signal processors for wireless sensor nodes,” I-manag. S J. Digit. Signal Process., vol. 5, no. 2, p. 22, 2017.
H. Zhang, J. Cheng, J. Zhang, H. Liu, and Z. Wei, “A regularization perspective based theoretical analysis for adversarial robustness of deep spiking neural networks,” Neural Netw., vol. 165, pp. 164–174, 2023.
H.-K. Fu, Y.-L. Liu, T.-T. Chen, C.-P. Wang, and P.-T. Chou, “The study of spectral correction algorithm of charge-coupled device array spectrometer,” IEEE Trans. Electron Devices, vol. 61, no. 11, pp. 3796–3802, 2014.
P. Kumar, K. Zhu, X. Gao, S.-D. Wang, M. Lanza, and C. S. Thakur, “Hybrid architecture based on two-dimensional memristor crossbar array and CMOS integrated circuit for edge computing,” Npj 2D Mater. Appl., vol. 6, no. 1, 2022.
P. Lin et al., “Three-dimensional memristor circuits as complex neural networks,” Nat. Electron., vol. 3, no. 4, pp. 225–232, 2020.
A. Kumar, K. Abhishek, X. Liu, and A. Haldorai, “An Efficient Privacy-Preserving ID Centric Authentication in IoT Based Cloud Servers for Sustainable Smart Cities,” Wireless Personal Communications, vol. 117, no. 4, pp. 3229–3253, Nov. 2020, doi: 10.1007/s11277-020-07979-8.
K. Ashok, M. Ashraf, J. Thimmia Raja, M. Z. Hussain, D. K. Singh, and A. Haldorai, “Collaborative analysis of audio-visual speech synthesis with sensor measurements for regulating human–robot interaction,” International Journal of System Assurance Engineering and Management, Aug. 2022, doi: 10.1007/s13198-022-01709-y.
F. Han, Y. Bian, B. Liu, Q. Zeng, and Y. Tian, “Research on calibration of a binocular stereo-vision imaging system based on the artificial neural network,” J. Opt. Soc. Am. A Opt. Image Sci. Vis., vol. 40, no. 2, pp. 337–354, 2023.
S.-G. Hong, K. Siau, and J.-W. Kim, “The impact of ISP, BPR, and customization on ERP performance in manufacturing SMEs of Korea,” Asia Pac. J. Innov. Entrep., vol. 10, no. 1, pp. 39–54, 2016.
S. M. A. Sharif, R. Ali Naqvi, and M. Biswas, “Beyond joint demosaicking and denoising: An image processing pipeline for a pixel-bin image sensor,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021.
V. Ngampruetikorn, D. J. Schwab, and G. J. Stephens, “Energy consumption and cooperation for optimal sensing,” Nat. Commun., vol. 11, no. 1, p. 975, 2020.
A. Salari, A. Djavadifar, X. Liu, and H. Najjaran, “Object recognition datasets and challenges: A review,” Neurocomputing, vol. 495, pp. 129–152, 2022.
S. Kumar, R. Kumar, and Saad, “Real-Time Detection of Road-Based Objects using SSD MobileNet-v2 FPNlite with a new Benchmark Dataset,” in 2023 4th International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), 2023.
Acknowledgements
Authors thank Reviewers for taking the time and effort necessary to review the manuscript.
Funding
No funding was received to assist with the preparation of this manuscript.
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Availability of data and materials
No data available for above study.
Author information
Contributions
All authors have equal contribution in the paper and all authors have read and agreed to the published version of the manuscript.
This article is licensed under a Creative Commons Attribution 4.0 International
License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you
give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and
indicate if changes were made. The images or other third party material in this article are included in the article‟s
Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the
article‟s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the
permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/4.0/
Cite this article
Ali-Кhusein and Urquhart, “Image Signal Processing in the Context of Deep Learning Applications”, Journal of Computing and Natural Science, vol.4, no.1, pp. 011-019, January 2024. doi: 10.53759/181X/JCNS202404002.
