Simulation of English part of speech classification based on artificial intelligence and additive logistic regression

doi:10.21203/rs.3.rs-2749448/v1

Download PDF

Research Article

Simulation of English part of speech classification based on artificial intelligence and additive logistic regression

https://doi.org/10.21203/rs.3.rs-2749448/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 29 May, 2023

Read the published version in Soft Computing →

You are reading this latest preprint version

English part-of-speech classification technology is a technology that can process text data, can effectively solve the problem of messy data in text information categories, make data structured and organized, and facilitate people to obtain effective information implicit in the text. This article transforms the original polynomial distribution into a generalized linear model and uses logistic regression algorithm for specific implementation. Moreover, the model proposed in this paper inherits the good explanatory characteristics of the decision tree, and it locally uses logistic regression to fit the data, which greatly improves the function space that logistic regression can fit. In addition, due to changes in the decision theory of logistic regression leaf nodes, the corresponding tree branch theory also needs to be changed accordingly. Finally, this paper designs experiments to study the performance of the model constructed in this paper. The research results show that the model constructed in this paper has high accuracy in the extraction and classification of English part of speech features.

Artificial intelligence

additive logistic regression

English part of speech

classification

machine recognition

At present, natural language processing technology can't solve the problem of problem understanding in the machine to solve math problems.

English part-of-speech classification generally includes preprocessing, English text representation, feature selection and compression, and classifier construction. There are two modes of feature compression: feature selection and feature extraction. Feature selection includes filtering method and fusion method. Among them, the filtering method judges the importance of features based on a certain standard, and retains relatively important features, which is not related to classification methods. The fusion method is closely integrated with the classifier and judges the pros and cons of a feature subset based on the classification accuracy. Due to the huge number of features in English part of speech classification, usually up to hundreds of thousands of dimensions, it is very difficult to use the fusion method to calculate. Therefore, the filtration method is usually used. At present, the commonly used filtering methods include feature extraction method based on document frequency, information gain method, mutual information method, etc. These methods treat each word as an independent dimension and assume that the word and word are orthogonal to represent each document.

Latent semantic analysis (LSA) is to approximate the original English text by mapping the vector features of English text to a low-dimensional latent semantic space, which can solve the problem of synonyms and polysemous words. The main disadvantage of LSA is the lack of a statistical theoretical basis, and the singular value decomposition takes a long time. Hofmann proposed a PLSA model based on probability and statistics by improving LSA, which uses an enhanced EM algorithm to train latent classes. Compared with LSA, PLSA has a more solid statistical foundation. LDA is an unsupervised topic model belonging to a fully probabilistic model, and LDA has made great progress in many fields of machine learning and information retrieval.

Words are the smallest language component in English, and the units of English information processing are basically words, and English word segmentation is to split the English sequence into meaningful words. Nowadays, word segmentation is more and more important in the application of professional fields, and it is particularly important to improve the accuracy of automatic word segmentation in professional fields. Since the English automatic word segmentation system is a relatively basic system and has a wide range of applications in many aspects, it has attracted the attention of domestic and international scholars in related fields. The literature [1] proposed a word segmentation method with the longest word count of 6. This method is the maximum matching method that is widely used later. The literature [2] applied MM to the word segmentation system and proposed a modern English word segmentation system CDWS. CDWS is the first automatic word segmentation system implemented independently in my country. It adopts MM word segmentation method, and its word segmentation speed is 11.15 characters/sec, and the error segmentation rate is 1/625. The literature [3] adopted the forward-increasing maximum matching method to design and realize the CASS automatic word segmentation system. The word segmentation speed of this system is 200 characters per second, and the word segmentation error rate is 1/300. The literature [4] fully introduced the expert system method into word segmentation technology. The segmentation error rate of this method for closed corpus is less than 1/1600, the segmentation error rate for open corpus is less than 1/625, and the word segmentation speed is up to 200 words/second. The literature [5] applied syntactic analysis technology to realize automatic segmentation of English words, which can better resolve the divergent meanings. So far, the annotated corpus has 50 million characters, has been manually proofread three times, and its accuracy rate is > 98%. The literature [6] used the full segmentation method. The segmentation accuracy of this method can reach 99%, and the word segmentation speed can reach 30 characters per second. The literature [7] used a combination of word shape matching and string frequency statistics to achieve automatic word segmentation in English, which is suitable for automatic abstraction and document retrieval. The literature [8] proposed English name recognition based on statistics. The recognition speed of pure software is 4 characters/sec, and the recognition speed of software and hardware is 15 characters/sec. The literature [9] developed a toolkit for English natural language text processing based on machine learning. The literature [10] developed a general-purpose multi-language processing platform NLPWin, which adopts the "integrated word segmentation and syntactic analysis" method to improve the efficiency of the automatic word segmentation system. Stanford English word segmentation system is a word segmentation tool based on Condition Random Field (cRF). The literature [11] developed a syntax analyzer Berkeley parser based on PCFG (Probabilistic Context Free Grammar), which can output the syntax analysis results of all sentences in the file. The literature [12] realized the initial model of NLPIR, which improved the recall rate while taking into account the accuracy rate. The segmentation accuracy reached 98.45%, and the word segmentation speed was 500KB/s.

The string-matching algorithm have been developed for decades, but some practical algorithms have only been widely used in recent years. The string matching is divided into single-pattern matching, multi-pattern matching, myopic matching and regular matching. The main algorithms for single pattern matching are BM algorithm, KMP algorithm, BDM algorithm, and Sunday algorithm. The classic algorithms for multi-pattern matching include AC algorithm and SBOM algorithm. The approximate matching mainly uses the concept of edit distance to express the difference between the pattern set and the text to be matched. The regular matching is to match all the strings in the text that meet the rules through rules composed of some wildcards. The literature [13] proposed the KMP algorithm. The algorithm idea is to use the information after the matching failure to minimize the number of matching between the pattern string and the main string to achieve the purpose of fast matching. The literature [14] proposed the BM algorithm. Its core idea is that if the pattern string does not match during the matching process, the algorithm can skip as many characters as possible for the next step of matching, thereby improving the matching efficiency. The process of regular matching is to convert the regular expression into a parse tree before matching the string, and then convert the parse tree into a deterministic or non-deterministic automaton. Among all algorithms, the BF algorithm (violent algorithm) is the most basic string-matching algorithm and the simplest algorithm [15].

Examples of invalidation of regression tree minimization criterion

Simulation experiment. We assume $x=1,1.2,1.4, \cdots 99.8,100$, according to,

$$\begin{gathered} y\left| x \right. \leqslant 25\sim N\left( {\mu =1,\sigma =0.1} \right) \hfill \\ y\left| x \right.>25\sim N\left( {\mu =1.1,\sigma =0.1} \right) \hfill \\ \end{gathered}$$

The response observation value Y is generated. Among them, $N\left( {\mu ,\sigma } \right)$ represents a normal distribution with mean $\mu$ and standard deviation $\sigma$. We try to use the quantile criterion and minimize variance criterion formula designed in this paper to get the quantile of the simulated data. When the conditional distribution $y\left| x \right.>\theta ,y\left| x \right. \leqslant \theta ,$ satisfies the normal distribution, the mean value is not much different, and the standard deviation is 10 times different, the use of the minimized variance criterion is even worse than the result of human eye observation. Therefore, this article believes that it is necessary to improve, and analyzes the reasons for the wrong results of the minimization criteria for quantile selection.

It further gives a comparison of simulation experiments under different mean $\mu$ and standard deviation $\sigma$, among them,

$$\begin{gathered} y\left| x \right. \leqslant 25\sim N\left( {{\mu _0},{\sigma _0}} \right) \hfill \\ y\left| x \right.>25\sim N\left( {{\mu _1},{\sigma _1}} \right) \hfill \\ \end{gathered}$$

Normal information gain criterion

First, the calculation formula of continuous variable information entropy is given. We assume that $y\sim p\left( y \right)$ and $p\left( y \right)$ are the density functions of Y. When y is distributed normally, the density function is:

$$p\left( {y,\mu ,\sigma } \right)=\frac{1}{{\sqrt {2\pi {\sigma ^2}} }}\exp \left\{ { - \frac{1}{{2{\sigma ^2}}}{{\left( {y - \mu } \right)}^2}} \right\}$$

The definition of continuous variable information entropy is:

$$Ent\left( y \right)= - \int {p\left( y \right){{\log }^{p\left( y \right)}}} dy$$

Lemma1

We assume $y\sim N\left( {\mu ,\sigma } \right)$, then the information entropy of y is:

$$\frac{1}{2}{\log ^{2\pi }}+\frac{1}{2}{\log ^{{\sigma ^2}}}+\frac{1}{2}$$

Since the first and last terms are constants, ${\log ^{{\sigma ^2}}}$ can be used to represent their relative size.

prove:

$$\begin{gathered} Ent\left( y \right)= - \int {\frac{1}{{\sqrt {2\pi {\sigma ^2}} }}\exp \left\{ { - \frac{1}{{2{\sigma ^2}}}{{\left( {y - \mu } \right)}^2}} \right\}} \hfill \\ {\log ^{\frac{1}{{\sqrt {2\pi {\sigma ^2}} }}\exp \left\{ { - \frac{1}{{2{\sigma ^2}}}{{\left( {y - \mu } \right)}^2}} \right\}}}dy \hfill \\ = - {\log ^{\frac{1}{{\sqrt {2\pi {\sigma ^2}} }}}}+\frac{1}{2} \hfill \\ \end{gathered}$$

Thus, the information entropy based on the normal distribution can be defined as:

$$Ent\left( D \right)={\log ^{{{\hat {\sigma }}^2}}}$$

Among them, ${\hat {\sigma }^2}$ is the estimated value of the variance of the response variable, generally taken as:

$${\hat {\sigma }^2}=\frac{1}{N}\sum {_{{i=1}}^{N}{{\left( {{y_i} - \bar {y}} \right)}^2}}$$

If it is a weighted sample, we have weights

$$\begin{gathered} W=diag\left( {{w_1}, \cdots ,{w_N}} \right), \hfill \\ \sum {_{{i=1}}^{N}{w_i}=1} \hfill \\ \end{gathered}$$

Then, there is:

$${\hat {\sigma }^2}=\sum {_{{i=1}}^{N}{w_i}{{\left( {{y_i} - \bar {y}} \right)}^2}}$$

Since the total information entropy of the marginal distribution is the same, the information gain brought by the variable: quantile (key: value) can be calculated by the following formula,

$$\begin{gathered} Gain\left( {D,key:value} \right)= - \frac{{\left| {{D_{ \leqslant value}}} \right|}}{{\left| D \right|}}Ent\left( {{D_{ \leqslant value}}} \right) \hfill \\ - \frac{{\left| {{D_{>value}}} \right|}}{{\left| D \right|}}Ent\left( {{D_{>value}}} \right) \hfill \\ \end{gathered}$$

The bias analysis assumes that the covariate: site (key: value) is used to divide the sample set into ${D_{ \leqslant value}},{D_{>value}}$,The proportion of the total sample is ${p_0},{p_1}$ respectively, which satisfies:

$${p_0}+{p_1}=1$$

We assume that the variance of the true conditional distribution $y\left| {key \leqslant value,} \right.y\left| {key>value} \right.$ satisfies:

$$\sigma _{0}^{2}=2,\sigma _{1}^{2}=\sigma _{0}^{6}=8$$

Minimize variance criterion is considered.

$$Va{r_{index}}\left( {D,key:value} \right)={p_0}\sigma _{0}^{2}+{p_1}\sigma _{1}^{2}$$

When the difference between the mean ${\mu _0},{\mu _1}$ of the true conditional distribution is very small, in order to minimize the variance, ${p_1}$ will approach 0, and the proportion ${p_1}$ of ${D_{>value}}$ will be very small. At the same time, since $\sigma _{1}^{2}$ is 4 times that of $\sigma _{0}^{2}$, the strength of the quantile being raised can be approximately regarded as 4. Moreover, it corresponds to the normal information gain criterion.

$$- Gain\left( {D,key:value} \right)={p_0}{\log ^{\sigma _{0}^{2}}}+{p_1}{\log ^{\sigma _{1}^{2}}}$$

The coefficient of ${p_1}$ is only 3 times that of the ${p_0}$ coefficient, that is, the strength of the quantile being raised can be approximately regarded as 3. This difference is more obvious when the variance difference is large. For example, $\sigma _{0}^{2}=2,\sigma _{1}^{2}=\sigma _{0}^{{10}}=32$, for the principle of minimizing variance, the quantile elevation force is 16, and the quantile elevation force selected by the normal information gain is 5. This is also the reason why the quantile 67.586 obtained by the criterion of minimizing variance is much higher than the true value when the variance difference is large in the simulation experiment. When the mean difference is large, the correction effect of the mean is more obvious, so the two quantile selection strategies can approximate the true value.

A typical example is shown in Fig. 1. It is obvious from the figure that based on the linear model criterion, the fitted quantile is more reasonable. Although using 50 as the quantile will maximize the degree of data confusion after branching, the subsequent model can better fit the segmented data. Therefore, it is reasonable to consider 50 as the quantile of the branching strategy. In this paper, the leaf node uses a logistic regression model, so maximum likelihood estimation and least quadratic loss are considered as the quantile estimation strategy of branch nodes.

First, the regression problem is considered, we assume that there is data.

$$D=\left\{ {\left( {{x_i},{y_i}} \right)} \right\}_{{i=1}}^{1}$$

The model is:

$$E\left[ {y\left| x \right.} \right]=\alpha \sigma \left( {{\beta ^T}x} \right)$$

Among them, $\beta$ is the parameter vector, $\alpha$ is the connection coefficient, which satisfies $\alpha \in {{\mathbb{R}}^1},\sigma \left( x \right)={1 \mathord{\left/ {\vphantom {1 {\left( {1+{e^{ - x}}} \right)}}} \right. \kern-0pt} {\left( {1+{e^{ - x}}} \right)}}$. The objective function is:

$$J\left( {\alpha ,\beta } \right)=\sum\limits_{{i=1}}^{N} {{{\left( {{y_i} - \alpha \sigma \left( {{\beta ^T}x} \right)} \right)}^2}}$$

The objective function can be solved by using the error back propagation strategy commonly used in neural networks. Because the extended logistic regression model is used as a regression problem, it is essentially a single hidden layer single hidden node neural network. For any sample set, after training $\alpha ,\beta$ is obtained, the residual sum of squares can be calculated, and the residual sum of squares can be minimized as the decision criterion for tree branches, that is:

$$SSE\left( D \right)=J\left( D \right)=J\left( {{\alpha ^*},{\beta ^*}} \right)$$

Among them, ${\alpha ^*},{\beta ^*}$ is the solution obtained by solving the objective function. All covariates and quantiles (key: value) are traversed, and the sample set is divided into ${D_{little}},{D_{big}}$ according to (key: value), and the residual square sum of the model after the fitting of the sample set ${D_{little}},{D_{big}}$ is calculated. The calculated total residual square and the reciprocal of the SSU is called the (key: value) utility, and the most useful (key: value) is selected as the branch strategy. Since the greedy algorithm is used here, there is Lemma2.

$$SSU=SSE\left( {{D_{little}}} \right)+SSE\left( {{D_{big}}} \right)$$

Lemma2

The selected (key:value) based on the sum of squares of the residuals of the model fitting is used for the number of single branches obtained by branching $tre{e_{key:value}}$, is smaller than the residual sum of squares of the single branch tree $tre{e_{key:value}}$ obtained by any $\left( {ke{y^s}:valu{e^s}} \right)$.

Here, we give the quantile decision rules required for the logistic regression model and use the linear model to approximate it instead. The quantile found by the linear model has a smaller difference from the real quantile. Here, we do not intend to use the linear model as a decision strategy for leaf nodes but use it as a quantile finding strategy. Although Lemma2 proved that using the leaf node decision model to minimize the residual sum of squares as a criterion can minimize the loss of the entire tree stump, it does not mean that the true value can be found. Because the quantile that can achieve the smallest sum of squared residuals is not unique. The figure shows that after finding a quantile with a small enough loss, the loss is not significantly reduced, especially in samples with random errors. Since the gradient of the logistic regression model is too smooth after extending to both sides, it is easy to choose a relatively close value as the quantile. In this case, it is better to find a quantile search strategy that can also achieve its approximate function but has a more concise calculation, that is, the linear model search strategy mentioned here. Therefore, the objective function is modified to:

$$J\left( {\alpha ,\beta } \right)=\sum\limits_{{i=1}}^{N} {{{\left( {{y_i} - {\beta ^T}{x_i}} \right)}^2}}$$

Text classification is a guided learning process. It finds the relationship model (classifier) between text attributes (features) and text categories based on a set of marked training texts, and then uses the learned relationship model to make category judgments on new texts. The general process of text classification can be divided into two parts: training and classification. The training process is mainly the process of establishing a relationship model (classifier) through the association relationship between the text and the category in the training set. The classification process is mainly based on the training classification model (classifier) to classify the unknown text according to the classification of the known text, as shown in Fig. 2.

When processing massive text data, the efficiency is often unsatisfactory due to the limitation of the calculation speed and memory capacity of a single machine. Therefore, this section presents a parallel implementation method for text classification.

As shown in Fig. 3, the algorithm first preprocesses the training corpus. Then, the algorithm learns the relevant parameters of the PSTLDA model and uses the obtained text topic distribution as a feature to train the classifier. For the KNN classifier, it is only necessary to save the text topic distribution of the training set. The steps of training the model and training the classifier are also implemented in parallel. The implementation in the Hadoop environment will be described in detail below. For the text set to be classified, the text topic distribution of each text in the text set to be classified is inferred according to the model, and finally they are classified. The Naive Bayes classification process is shown in Fig. 4.

As can be seen from the figure above, the algorithm includes three stages. The first is the preparatory work stage. Its task is to make preparatory work for classification, determine feature attributes according to the input comment text, and appropriately divide each feature attribute, and then manually classify part of the data set as a training set. This stage is the only work that requires human participation in the entire classification process, and the quality of the artificially generated training set has a great impact on the entire classification algorithm. Among them, the quality of the classifier depends largely on the selection of feature attributes and the quality of the training set.

The experimental data uses manual collection of web pages (480 each) on the four topics of finance, education, sports and games from the English Internet. The number of training sets is 360 and the number of test sets is 120.

This article uses precision, recall and F1 score to evaluate the results of web page classification in the experiment.

In the experiment, three traditional classification algorithms: kNN, NaiveBayes and SVM are used as comparisons. Among them, kNN is implemented by the algorithm in the Python machine learning library scikit-learn, NaiveBayes is implemented by Java programming, and SVM is implemented by LIBSVM toolbox. The model constructed in this paper is named AL algorithm. The webpage part-of-speech classification results of the four algorithms are shown in Table 1.

Table 1

Classification results of web page parts of speech of 4 algorithms
Classification algorithm	Page category	Precision	Recall rate	F1
kNN	Finance	0.786	0.825	0.805
	education	0.940	0.909	0.924
	physical education	0.902	0.909	0.905
	game	0.915	0.892	0.903
	average value	0.886	0.884	0.884
Naive Bayes	Finance	0.750	0.800	0.774
	education	0.907	0.892	0.900
	physical education	0.883	0.875	0.879
	game	0.905	0.867	0.885
	average value	0.861	0.859	0.859
SVM	Finance	0.848	0.884	0.866
	education	0.976	0.960	0.968
	physical education	0.983	0.934	0.958
	game	0.920	0.943	0.931
	average value	0.932	0.930	0.931
AL	Finance	0.854	0.875	0.865
	education	0.992	0.951	0.971
	physical education	0.912	0.943	0.927
	game	0.950	0.934	0.942
	average value	0.927	0.926	0.926

In the process of English part-of-speech classification, logistic regression is used to fit data locally, which greatly improves the function space that logistic regression can fit. Due to changes in the decision theory of logistic regression leaf nodes, the corresponding tree branching theory also needs to be changed accordingly. This paper proposes a model-based branching decision theory. Moreover, this paper verifies the rationality of the theory through numerical simulation and actual data set verification. The tree model design based on quantile decision tree and regression tree usually considers the following three problems: branch strategy design problem; decision strategy design problem; pruning to prevent model overfitting.

Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interests

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Data Availability

Data will be made available on request.

F. Mohsen, M. M. Hadhoud, K. Moustafa, et al., “A new image segmentation method based on particle swarm optimization,” Int. Arab J. Inf. Technol., vol. 9, no. 5, pp. 487-493, 2012.
K. Stärk, E. Kidd, and R. L. Frost, “Word segmentation cues in German child-directed speech: A corpus analysis,” Language and Speech, vol. 65, no. 1, pp. 3-27, 2022.
F. S. Matikolaie, Y. Kheddache, and C. Tadj, “Automated newborn cry diagnostic system using machine learning approach,” Biomedical Signal Processing and Control, vol. 73, p. 103434, 2022.
Y. Zhao, H. Li, S. Yin, et al., “A New Chinese Word Segmentation Method Based on Maximum Matching. J. Inf. Hiding Multim,” Signal Process., vol. 9, no. 6, pp. 1528-1535, 2018.
J. WANG, X. XUE, and W. WENG, “Source code summarization technology based on syntactic analysis,” Journal of Computer Applications, vol. 35, no. 7, p. 1999, 2015.
Y. Wu, X. Peng, K. Ruan, and Z. Hu, “Improved image segmentation method based on morphological reconstruction,” Multimedia Tools and Applications, vol. 76, no. 19, pp. 19781-19793, 2017.
A. Gupta, and D. Yadav, “A novel approach to perform context‐based automatic spoken document retrieval of political speeches based on wavelet tree indexing,” Multimedia Tools and Applications, vol. 80, no. 14, pp. 22209-22229, 2021.
C. Delle Luche, C. Floccia, L. Granjon, et al., “Infants' first words are not phonetically specified: Own name recognition in British English‐learning 5‐month‐olds,” Infancy, vol. 22, no. 3, pp. 362-388, 2017.
A. Névéol, H. Dalianis, S. Velupillai, et al., “Clinical natural language processing in languages other than english: opportunities and challenges,” Journal of biomedical semantics, vol. 9, no. 1, pp. 1-13, 2018.
X. S. L. N. L. Weizhong, and L. I. U. Xiaobai, “Design of integration framework for multi-language applications on logging platform in Java,” Acta Petrolei Sinica, vol. 31, no. 5, p. 810, 2010.
S. Ten Oever, K. Kaushik, and A. E. Martin, “Inferring the nature of linguistic computations in the brain,” PLoS Computational Biology, vol. 18, no. 7, p. e1010269, 2022.
P. Li, and Y. Ye, “Chinese spam filtering based on back-propagation neural networks,” Softw Eng, vol. 4, no. 2, pp. 9-12, 2016.
S. Rajesh, S. Prathima, and L. S. S. Reddy, “Unusual pattern detection in DNA database using KMP algorithm,” International Journal of Computer Applications, vol. 1, no. 22, pp. 1-7, 2010.
F. Yu, “Malicious url detection algorithm based on bm pattern matching,” International Journal of Security and Its Applications, vol. 9, no. 9, pp. 33-44, 2015.
R. A. Hooshmand, and S. Soltani, “Fuzzy optimal phase balancing of radial and meshed distribution networks using BF-PSO algorithm,” IEEE Transactions on Power Systems, vol. 27, no. 1, pp. 47-57, 2011.

Download PDF

Journal Publication

published 29 May, 2023

Read the published version in Soft Computing →

Reviewers agreed at journal
10 Apr, 2023
Editor assigned by journal
30 Mar, 2023
First submitted to journal
28 Mar, 2023

You are reading this latest preprint version

Simulation of English part of speech classification based on artificial intelligence and additive logistic regression

Status:

Journal Publication

Version 1

Abstract

Figures

1. Introduction

2. Related Work

3. Branch Strategy Design

4. Selection Criteria Based On Target Model Quantile

5. Model Construction And Performance Analysis

6. Model Performance Verification

7. Conclusion

Declarations

References

Status:

Journal Publication

Version 1