Advanced Explainable AI: Self Attention Deep Neural Network of Text Classification
Prasanna V
Prasanna V
Department of Data Science and Business Systems, School of Computing, Faculty of Engineering and Technology,
SRM Institute of Science and Technology, Kattankulathur, Chennai, India.
Department of Computer Science and Applications, Faculty of Science and Humanities, SRM Institute of Science and Technology, Ramapuram, Chennai, India.
The classification of texts is a crucial component of the data retrieval mechanism. By utilizing semantic details representation, and the text vector sequence is condensed, resulting in a reduction in the temporal and spatial order of the memory pattern. This process helps to clarify the context of the text, extract crucial feature information, and fuse these features to determine the classification outcome. This approach represents the preprocessed text data using character-level vectors. The self-attention mechanism is used to understand the interdependence of words in a text, allowing for the extraction of internal structure-related data. Furthermore, the semantic characteristics of text data have been extracted independently using Deep Convolutional Neural Network (DCNN) and Bi-directional Gated Recurrent Unit (BiGRU) using a Soft-Attention mechanism. These two distinct feature extraction outcomes are then merged. The Softmax layer is employed to categorize the deep-extracted attributes, hence enhancing the accuracy of the classification model. This improvement is achieved by including a uniform distribution component into the cross-entropy loss function. Our results demonstrate that our suggested method for explainability outperforms the model that was suggested in terms of accuracy and computing efficiency. For the purpose of assessing the effectiveness of our suggested approach, we developed many baseline models and performed an evaluation their studies.
Keywords
EAI, Deep Learning, Attention, Text Classification, DCNN.
X. Luo and X. Wang, “Research on multi-feature fusion text classification model based on self-attention mechanism,” Journal of Physics: Conference Series, vol. 1693, no. 1, p. 012071, Dec. 2020, doi: 10.1088/1742-6596/1693/1/012071.
Talukdar and S. Kr. Sarma, “Hybrid Model for Efficient Assamese Text Classification using CNN-LSTM,” International Journal of Computing and Digital Systems, vol. 14, no. 1, pp. 10183–10192, Oct. 2023, doi: 10.12785/ijcds/140191.
J. Zhang, F. Liu, W. Xu, and H. Yu, “Feature Fusion Text Classification Model Combining CNN and BiGRU with Multi-Attention Mechanism,” Future Internet, vol. 11, no. 11, p. 237, Nov. 2019, doi: 10.3390/fi11110237.
S. Liu, F. Le, S. Chakraborty, and T. Abdelzaher, “On Exploring Attention-based Explanation for Transformer Models in Text Classification,” 2021 IEEE International Conference on Big Data (Big Data), Dec. 2021, doi: 10.1109/bigdata52589.2021.9671639.
H. Chefer, S. Gur, and L. Wolf, “Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers,” 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 2021, doi: 10.1109/iccv48922.2021.00045
Hashmi, S. Y. Yayilgan, M. M. Yamin, S. Ali, and M. Abomhara, “Advancing Fake News Detection: Hybrid Deep Learning with FastText and Explainable AI,” IEEE Access, vol. 12, pp. 44462–44480, 2024, doi: 10.1109/access.2024.3381038.
H. Sebbaq and N. El Faddouli, “MTBERT-Attention: An Explainable BERT Model based on Multi-Task Learning for Cognitive Text Classification,” Scientific African, vol. 21, p. e01799, Sep. 2023, doi: 10.1016/j.sciaf. 2023.e01799.
L. H. Baniata and S. Kang, “Switching Self-Attention Text Classification Model with Innovative Reverse Positional Encoding for Right-To-Left Languages: A Focus on Arabic Dialects,” Feb. 2024, doi: 10.20944/preprints202402. 0332.v1.
N. R, N. K, S. R, S. Banu S, S. P, and B. P, “Graph-Based Rumor Detection on Social Media Using Posts and Reactions,” International Journal of Computing and Digital Systems, vol. 15, no. 1, pp. 173–182, Jul. 2024, doi: 10.12785/ijcds/160114.
R. Nareshkumar and K. Nimala, “Interactive Deep Neural Network for Aspect-Level Sentiment Analysis,” 2023 International Conference on Artificial Intelligence and Knowledge Discovery in Concurrent Engineering (ICECONF), Jan. 2023, doi: 10.1109/iceconf57129.2023.10083812.
L. Wang, R. Chen, and L. Li, “Knowledge-Guided Prompt Learning for Few-Shot Text Classification,” Electronics, vol. 12, no. 6, p. 1486, Mar. 2023, doi: 10.3390/electronics12061486.
Z. Wang, “Deep Learning Based Text Classification Methods,” Highlights in Science, Engineering and Technology, vol. 34, pp. 238–243, Feb. 2023, doi: 10.54097/hset. v34i.5478.
M. Y. Hailemariam, S. Lynden, A. Matono, and T. Amagasa, “Self-Attention-based Data Augmentation Method for Text Classification,” Proceedings of the 2023 15th International Conference on Machine Learning and Computing, Feb. 2023, doi: 10.1145/3587716.3587779.
S. Lee, L. Liu, and W. Choi, “Iterative Translation-Based Data Augmentation Method for Text Classification Tasks,” IEEE Access, vol. 9, pp. 160437–160445, 2021, doi: 10.1109/access.2021.3131446.
Y. J. Yu, S. J. Yoon, S. Y. Jun, and J. W. Kim, “TABAS: Text augmentation based on attention score for text classification model,” ICT Express, vol. 8, no. 4, pp. 549–554, Dec. 2022, doi: 10.1016/j.icte.2021.11.002.
T. Cai and X. Zhang, “Imbalanced Text Sentiment Classification Based on Multi-Channel BLTCN-BLSTM Self-Attention,” Sensors, vol. 23, no. 4, p. 2257, Feb. 2023, doi: 10.3390/s23042257.
X. Li and H. Ning, “Deep Pyramid Convolutional Neural Network Integrated with Self-attention Mechanism and Highway Network for Text Classification,” Journal of Physics: Conference Series, vol. 1642, no. 1, p. 012008, Sep. 2020, doi: 10.1088/1742-6596/1642/1/012008.
K. Zahoor, N. Zakaria Bawany, and T. Qamar, “Evaluating text classification with explainable artificial intelligence,” IAES International Journal of Artificial Intelligence (IJ-AI), vol. 13, no. 1, p. 278, Mar. 2024, doi: 10.11591/ijai. v13.i1. pp278-286.
R. Jing, “A Self-attention Based LSTM Network for Text Classification,” Journal of Physics: Conference Series, vol. 1207, p. 012008, Apr. 2019, doi: 10.1088/1742-6596/1207/1/012008.
K. J, A. H, P. V, Y. S, and G. N, “A Study Of Dynamic Thresholds Power Detection Spectrum Sensing Techniques In CRN,” 2023 9th International Conference on Advanced Computing and Communication Systems (ICACCS), Mar. 2023, doi: 10.1109/icaccs57279.2023.10112923.
R. Nareshkumar and K. Nimala, “An Enhanced BERT Model for Depression Detection on Social Media Posts,” Lecture Notes in Networks and Systems, pp. 53–64, 2024, doi: 10.1007/978-981-99-8479-4_5.
R. Nareshkumar and K. Nimala, “An Exploration of Intelligent Deep Learning Models for Fine Grained Aspect-Based Opinion Mining,” 2022 International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES), Jul. 2022, doi: 10.1109/icses55317.2022.9914094.
S. Durairaj and R. Sridhar, “Coherent virtual machine provisioning based on balanced optimization using entropy-based conjectured scheduling in cloud environment,” Engineering Applications of Artificial Intelligence, vol. 132, p. 108423, Jun. 2024, doi: 10.1016/j.engappai.2024.108423.
M. Amanullah, S. Thanga Ramya, M. Sudha, V. P. Gladis Pushparathi, A. Haldorai, and B. Pant, “Data sampling approach using heuristic Learning Vector Quantization (LVQ) classifier for software defect prediction,” Journal of Intelligent Fuzzy Systems, vol. 44, no. 3, pp. 3867–3876, Mar. 2023, doi: 10.3233/jifs-220480.
Acknowledgements
The authors would like to thank to the reviewers for nice comments on the manuscript.
Funding
No funding was received to assist with the preparation of this manuscript.
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Availability of data and materials
Data sharing is not applicable to this article as no new data were created or analysed in this study.
Author information
Contributions
All authors have equal contribution in the paper and all authors have read and agreed to the published version of the manuscript.
Corresponding author
Prasanna V
Prasanna V
Department of Data Science and Business Systems, School of Computing, Faculty of Engineering and Technology,
SRM Institute of Science and Technology, Kattankulathur, Chennai, India.
Open Access This article is licensed under a Creative Commons Attribution NoDerivs is a more restrictive license. It allows you to redistribute the material commercially or non-commercially but the user cannot make any changes whatsoever to the original, i.e. no derivatives of the original work. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/4.0/
Cite this article
Prasanna V, Umarani S, Suganthi B, Ranjani V, Manigandan Thangaraju and Uma Maheswari P, “Advanced Explainable AI: Self Attention Deep Neural Network of Text Classification”, Journal of Machine and Computing, pp. 586-593, July 2024. doi: 10.53759/7669/jmc202404056.
