Machine Learning

concrete content

2017 New New New TensorFlow CODE repository

Introduction Artifical Neural Network

[Optimization](https://en.wikipedia.org/wiki/Gradient_descent)

**Brief**: Input -> Process layer 1 2 3 ......-> Results
        result provide feedback (reward) 
        Activation function
        example(Training data) -> Training
        final(testing data) -> testing

**Definition**: Artifical neural network(ANN) is a collection of connected artifician neurons, which inspired by Bio-neural network.
supervised or not: manualley labeld or not

**Method**: primarily focus on oprimization method 
        Gradient Descent
        Cost function
        obtaining Global minimum or local Minimum
        a first order iterative optimization algorithm

        asymptotic rate of convergence is inferior???

Preparation(Tensor Flow):

Installation
'''
    pip3 install --upgrade tensorflow

    import tensorflow
'''
Input(example) -> Output(Results)

usage: prediction

overfitting line 

more
underfitting line

Basis Architecture

tensor flow basis

input -> hide layer -> output
1. input : dataset 1 2 3 4.......
2. hide layer: weights, biases, activation rules
3. output
4. Gradient Descent optimize metrics......

tensor flow basis coding

key idea of TF: error back-propagation algorithm through optimization method
optimization method this time: gradient descent

Full Code

tf Session

FuLL Code

tf Variable

Full Code

tf Placeholder

tf Activation Function

manual add layer

Full code

Basis TensorFlow

1. basis architecture

Full Code
2. visualization
Full code

Speeding up training and Optimizer

1. stochastic Gradient Descent(SGD)
2. Momentum (using push) 
3. AdaGrad (using rule to discpline)
4. RMSProp (basic rule + rule)
5. Adam (fully fule + rule)

more link

Tensorboard

inputs -> layer 1 2 3 4….. ->outputs
overview graph: as well as components graph:
Full code
Full code

Advanced Content

1. classification solution case
   regression problem sovling -> output value
   classification problem solving -> output possiblity

   activation function:

    regression using relu
    classification using softmax

   cost function:

    regression: real - prediction
    classification: cross_entropy

   optimizer: both gradient descent

Full code

2. solving overfitting
   
   For general situation rather than specific training data
   states: underfit just_right overfit
   Tensorflow solve: drop out
   Full code

3. Convoltutional neural network(CNN)
   Purpose: image text vedio (computer vision recognization

   Method: object Components.s.s -> combine -> combine ..->.->…. -> object classfied

   pooling?

   example: image->covolution->max pooling->connected -> classified

great resource
4. CNN buidling 1
code concept:image recognition, gradually compress hight and width in order to increase thickness (processing data in validation below ‘cp’)

CNN filtering quintessence

**Overview picture in example**:

image -> convolutionary -> pooling (cp)-> cp .......-> tensorflow(fully connected tf) -> tf -> classifer  

key attribute in code: patch(compression related with padding) stride(step related pooling)

padding method: valid padding ; same padding
pooling method: max pooling ; average pooling

5. CNN building 2
   TensorFlow Libraries implment CNN
   conv2a function
   pooling module
   padding module
   key features: stride, patch, padding, pooling
   Full code

6. CNN building 3
   feeling: capture data -> reframe(reshape) data -> optimizer data
   practical: padding, pooling, conv2d, adamoptimizer

Full code

7. Saver

   purpose: store the trained parameter like(weights and biases)
   issue: save parameters -> reload parameters
   key: tf.train.saver

Full code

8. Recurrent Neural Network (RNN) 1
   key: order matter issue solutions (sentence)

   common method: long term short term memory (LSTM)

   collecting continum experience from previous experience(state) -> incfluence -> next experience (state)

   using LSTM enable avoid

    gradient vanishing and gradient exploding 

   LSTM structure: Write(pre) - forget (mid) - read (after) ————three gates handle

resource1
resource2

9. Recurrent Neural Network (RNN) 2 Classification
   RNN LSTM

   inputs hiden layer -> Cell -> outputs hiden layer

   every cell has write forget read gates function

    solving for gradient vanishing or gradient exploding
   
    1. initial_states
    2. c_state
    3. m_state (basis only have this state)

   Tensorflow
   ‘’’

    tf.nn.dynamic_rnn()

   ‘’’
   difficulties: time_major, transpose

Full Code

Noremal style or Tensorflow style
Extensive BPTT

10. Recurrent Neural Network (RNN) 3 Regression

    1. data handling — batch

    2. LSTM (input, cell, output)

    3. Optimizer
       ‘’’
       key attribute: input_size, output_size, cell_size

       1. input layer

       2. cell
          Basic LSTM Cell
          Lstm_cell.zero_state
          tf.nn.dynamic_rnn

       3. output leyer
          losses = tf.nn.seq2seq.sequence_loss_by_example()

    ‘’’

    general difficulties:

    datashape => reshape function

RNN key: Truncate Backpropagration Again:
Extensive BPTT

Full Code

11. Recurrent Neural Network (RNN) 4 Visualization

    plt_plot()

Full Code

12. Autoencoder
    Much like PCA
    it is unsupervised learning
    feel like trained compiler
    encoder -> activation function
    decoder -> activation function

    tf.nn.sigmoid

    difficulties: learning rate

Full Code

13. Scope
    corresponded with variable
    ‘’’
    tf.name_scope
    tf.variable_scope

    tf.get_variable() 
    tf.Variable()

    ‘’’

    Understanding scope mechanism can reuse variable
    especially in situation(RNN TrainConfig && TestConfig)
    ‘’’

    scope.reuse_variables()

    ‘’’
    Full Code
    Full Code

14. Batch Normalization (BN)
    significance: if not, neurons lose efficacy, sensor capacity vanish
    procedure:forwarding layer process BN after every time activation function

Full Code

extensive resource

15. Transfer Learning

(continum to learn when start again these bracket should be vanished)

current exsited model to operate

Extensive resource