For natural language processing problems, the short text classification is still a research hot topic, with obviously problem in the features sparse, high-dimensional text data and feature representation. In order to express text directly, a simple but new variation which employs one-hot with low-dimension was proposed. In this paper, a Densenet-based model was proposed to short text classification. Furthermore, the feature diversity and reuse were implemented by the concat and average shuffle operation between Resnet and Densenet for enlarging short text feature selection. Finally, some benchmarks were introduced to evaluate the Falcon. From our experimental results, the Falcon method obtained significant improvements in the state-of-art models on most of them in all respects, especially in the first experiment of error rate. To sum up, the Falcon is an efficient and economical model, whilst requiring less computation to achieve high performance.
Nowadays, short text classification is the task of automatically assigning pre-defined categories to documents written in natural languages. Several types of text categorization have been studied, each of which deals with different types of documents and categories. With the popularity of social media, short text classification becomes an essential component in many applications, such as topic categorization to detect discussed topics, information filtering, and sentiment classification to determine the sentiment typically in product or movie reviews [
Compared with the feature of general Chinese short text, the electric power complaint text has the following special characteristics:
・ The text relates to the field of electric power, which contains a large number of electrical professional vocabularies.
・ The characteristics of the text are not obvious, and more margin information is needed to analyze.
・ The text is mixed with too many symbols and numbers.
However, when dealing with shorter text messages, traditional techniques will not perform as well as they would have performed on larger texts.
Some obstructions are encountered when we plan to use Densenet method to deal with electric power complaint text:
・ Sparse features: The method of machine learning is to classify or predict based on features, and various features are constructed in text analysis to match the corresponding features. The effect of text classification is depending on features.
・ Difficult features representation: The sentence modeling aims at representing sentences as meaningful features of tasks such as text classification. The shorter and simpler for text, the harder feature representation.
In order to solve the above problems, we proposed the Falcon approach, which incorporates Resnet Networks and Densely Connected Convolutional Networks. To the best of our knowledge, no Densenet-based work on short text classification has been proposed to date. Meanwhile, we made several innovations in the data processing to accelerate the model training [
・ Feature extraction: We make innovations on the incorporation model to meet the needs of the classification in short text and make feature extraction become easy.
・ Reduce the dimension: In this paper, we use the one-hot vector, PCA dimensionality reduction, vocabulary dictionary and matching the vector id, to avoid additional calculation. The implementation of concat operation will inevitably increase the time complexity of the model.
・ Marginal feature flow: We first time propose the new architecture that incorporates Densenet with Resnet in terms of text classification, in order to increase the transfer of margin features.
We apply the proposed model on the short text classification task and achieved superior performance on various benchmarks.
The rest of this paper is organized as follows. In Section 2, we discuss some related works about the models of sentence representation and feature flow. Propaedeutics will be review in Section 3. The Falcon is presented in Section 4. Section 5 carries out the relevant experiment and analyzes the performance of Falcon. Finally, we conclude the paper and make an acknowledgement.
In this section we first give an overview of the current learning model in feature representation. Next, we review Densenet and channel shuffle that form the basis for Densenet-based Networks.
Models in feature representation. In many recent works of sentence representation, neural network models were constructed on either input word sequences or transformed syntactic parse tree [
It also has been shown that higher-level modeling on x l can help to increase the variation in the input, which should then to make it efficient to learn more margin features of between different layers [
Another model of mix properties of Text RNN + CNN was put forward by Siwei Lai et al. [
Dense connection. Densely connected networks proposed by Dr. Huang Gao [
Channel Shuffle. Xiangyu Zhang et al. [
In summary, none of them can solve the problems we encountered by analyzing the above model. Hence, we proposed the approach to the next section to address these challenges. We adopted a variant of Densenet to increase the margin feature flow and reducing network complexity. Subsequently, it will achieve great performance on short text classification.
Before the model has been proposed, we will review the three state-of-art application of convolutional neural networks to text data.
Consider a word vector group with vocabulary V that is passed through a convolutional network. The network comprises L layers, each layer implement a non-linear transformation H l ( . ) , where l indexes the layer H l ( . ) can be a composite function of operations, which in our case is the “Rectified Linear Unit” (ReLU). We denote the output of the l t h layer as x l .
Resnet. Traditional convolutional feed-forward networks connect the output the l t h as input to the l + 1 t h layer: x l = H l ( x l − 1 ) . ResNets [
x l = H l ( x l − 1 ) + x l − 1 (1)
On the one hand, Resnet bypass signal from one layer to the next via identity connections.
On the other hand, Resnet [
Resnet aims to solve the problem of long distance transmission of feature combination problems or shallow information at different levels.
Densenet. To further improve the information flow between different layers, the densenets was introduced direct connections to any layer to all subsequent layers, the layer receives the feature-maps from all of the preceding layers:
x l = H l ( [ x 0 , x 1 , ... , x l − 1 ] ) (2)
where x k , k ∈ [ 0 , l − 1 ] , refers to the concatenation of the feature-maps. In order to implement easily, Densenet concatenates all of the front inputs into a single map.
Crucially, in contrast to Resnet, Densenet never combine features through summation before they are passed into a layer, instead, it combine features by concatenating them. Hence, the l t h layer has l inputs, consisting of the feature-maps of all preceding convolutional blocks. Its own feature-maps are passed on to all L − l subsequent layers. This introduces L (L + 1)/2 connections with a L-layer network, instead of just L, as in traditional architectures.
Compressed dimension. Since the dimensionality of region vectors determines the dimensionality of weight vectors, having high-dimensional region vectors means more parameters to learn. If p | V | is too large, the model becomes too complex (w.r.t. the amount of training data available) and training becomes unaffordable expensive even with efficient handling of sparse data; Therefore, one has to lower the dimensionality by lowering the vocabulary size | V | and the region size p, which may or may not be desirable, depending on the nature of the task.
With this representation, we have fewer parameters to learn. Essentially, the expressiveness of our alternative (which loses word order only within small regions) is somewhere between one-hot representation and word2vec.
Concat. One can simply implement a concat operation by adding one more concat layer upon existing networks after the 1 × 1 convolutional layer (as shown in Equation (3)). The concat operation is following:
C k = R k ⊕ D k (3)
x l = H l ( [ x 0 , x 1 , ... , x l − 1 ] ) (4)
where R k and D k denote the extracted information at k-th step from Resnet and Densenet, ⊕ denotes the concat operator.
Average Channel Shuffle. The basic idea of average shuffle algorithm is the index i scans and copies the original data from front to back, between [0, 1] random a index j , the main effect is equivalent to exchang the values of i and j in the copy data, which advantage is that the time consumption is minimal. Which is enabling cross-group information flows for group convolution layers. The pseudocode of Average Shuffle is shown in the following Algorithm 1.
The model schematic architecture, which is shown in
In this section, the effectiveness of the proposed model was validated by THUCNews datasets, and then handle the EPCT datasets. Hardware environment: 4 GB RAM, Nvidia Geforce GTX 970 M, 3 G. Software enviroment: All experiments are conducted on a Windows 7 professional 64 bit OS with a simple integrated experimental environment (anaconda 3 (64 bit) + python (3.6) + spyder) and an experimental framework of tensorflow (1.1.0).
The summary statistics of EPCT datasets are in
We divided the dataset into training sets, validation sets, and test sets by 8:1:1. That is, 80% data were used for training the word2vec and classifier. In constructing the word vector model, the size of word vectors was set to 50 (i.e., each word was represented as a 50-dimensional vector.). In word vector representation, each word is represented as a vector in an arbitrary vector space. Then every word is represented as a numerical vector, we can compute relevancy between words. Continuous word vectors representation techniques have been proposed in [
| Date sets Classes Number Ave Length |
|---|
| THUNews 20 20,000 264 |
| EPCT 7 5000 93 |
The concat operation used in Equation (3) is not valid when the size of feature-maps changes. For convolutional layers with filter size 3 × 3 (group convolution) and 1 × 1, The transition layers used in our experiments consist of a batch normalization layer and an 1 × 1 convolutional layer followed by a 2 × 2 average pooling layer.
Several parameters of our model are summarized in
To look into the details, on this task, while our model outperform all the baseline methods, which indicates that in this setting the merit of having fewer parameters is larger than the benefit of keeping word order in each region.
We first perform a set of experiments to validate the F1-score on Resnet, Densenet-BC and Concat model.
Finally, the results with training sets of various sizes on THUNews and EPCT are shown in
On THUNews and EPCT datasets, the best error rates we obtained by training were 8.6 and 7.5. Without exception, which is both better than other methods. Meanwhile, we also find that our model will achieve good performance in a short training time, which is shown in
| Description | Values |
|---|---|
| embedding dim | 64 |
| seq length | 600 |
| vocab size | 500 |
| hidden dim | 128 |
| batch size | 64 |
| num epochs | 10 |
| Methods | THUNews | EPCT |
|---|---|---|
| one-hot + CNN | 11.47 | 9.50 |
| word2vec + CNN | 8.46 | 8.21 |
| one-hot + Densenet | 8.34 | 7.92 |
| word2vec + Densenet | 8.21 | 7.75 |
| [ours] | 8.06 | 7.63 |
| Categories | Model | ||
|---|---|---|---|
| ― | Densenet | Resnet | Concat model |
| praise | 0.76 | 0.60 | 0.76 |
| fault repair | 0.91 | 0.86 | 0.93 |
| suggestion | 0.83 | 0.75 | 0.89 |
| report | 0.76 | 0.62 | 0.76 |
| complaint | 0.75 | 0.69 | 0.89 |
| business consultation | 0.96 | 0.88 | 0.97 |
| opinion | 0.65 | 0.59 | 0.72 |
In this paper, this is the first time that a novel short text classification method based on Densenet networks was proposed to address the electric power complaint text. Meanwhile the improvements in short text classification model making a comprehensive comparison of the classification efficiency were implemented. The model provides a new approach and a train of thought for the study of the other short Chinese texts.
In addition, the highly consistent classification results were found by comparison of the actual classification results of the state grid; this is due to the improvement of feature learning method in this model, just like reading, the richer the content of the book, the more knowledge you will learn.
Although some improvements have been achieved, one of the future works is to develop a more efficient short text classification model in enlarging short text features.
To acknowledge all of the people that have contributed to this paper, the donors of THUNews dataset and the anonymous reviewers for their helpful feedback and suggestions. This work is also supported by the National Natural Science Foundation of China (No. 61272437 and 71203137).
The authors declare no conflicts of interest regarding the publication of this paper.
Li, H.M., Huang, H.N., Cao, X. and Qian, J.G. (2018) Falcon: A Novel Chinese Short Text Classification Method. Journal of Computer and Communications, 6, 216-226. https://doi.org/10.4236/jcc.2018.611021