[natural language processing and text analysis] two sub models of word2vec (supervised neural network model), skip gram and cbow model.

1. 1. 1. Word2vec Premise

First, explain the operation rules of neural network .

On the far left is the input field （3 Neurons ）, middle weights Is the weight hidden layer ,bias Is the partial weight , In the middle is accumulation

Here is the product .Z=4 Is the neuron multiplied by the weight , add bias obtain , And then through the activation function （activation function） machining , The left part is function processing

Take these parts as basic units , Practice and you will get the following neural network

A neuron binds to another neuron , The connection relationship is the activation function .

1. 1. 1. XOR ask

Linearly indivisible , Multiple linear regression cannot be used to calculate , Because the final result must have some predictions failed .

The input layer has 4 A numerical , The exception layer has two nodes , Output results 1 or 0, Training 500 Time

this 500 Adjust the weight value next time , Then adjust the value in the hidden layer

We find that the error value will continue to follow 500 Number of times error falling .

The final adjustment result is ：

Let's take a look at the result table

We began to doubt , Originally linearly indivisible values , Now there is a hidden layer , It can be classified successfully

So we doubt the function of hidden layer .

We can type in ,0,0. See what the hidden layer outputs , Get the results , Make logic LR The regression model （ Here is the neural network framework process mentioned above .）, We found that the input field passes through the hidden layer , It becomes a linearly separable value .

Observe the output of the hidden layer .

We can find that the input layer （input Layer） To the hidden layer （Hidden Layer） The process of is to carry out goal transformation , Reduce analysis dimension .

Because our input layer only 2 individual , Therefore, this case does not carry out dimensionality reduction . But if you use New_X Conduct neural network , It can also be found , The accuracy is 100% .

Neural networks can help us generate new features

There is a foundation here , We will word2Vec

1. 1. 1. Word2Vec（ There's a surveillance model ）

Word model 1（Skip -gram）

CBOW

The input layer is the keyword , The output layer is the result

The hidden layer has two matrices , matrix 1 It represents the word embedding matrix , matrix 2 It stands for

We put the previous 5 An article , Train to get embedded ,word,embedding

The distance is far because the words used before and after are completely different .

Method CBOW

Because there are many words before and after words , So we can , Every word of one hot encoding As an input result , Then we average the results of each hidden layer , If we get the result, we can get what kind of model the news is

MATRIW The matrix is as follows

Again , Let's enter Article 6 , You can also predict the appropriate results .

Similarly, we can get related words and synonyms by doing distance .