1. Eric Chen
  2. hencrice.bitbucket.org

Commits

Eric Chen  committed c78fc81

Adding sparsity to make an autoencoder.

  • Participants
  • Parent commits d1e51e3
  • Branches master

Comments (0)

Files changed (6)

File deepArch/_modules/CostFunc.html

View file
  • Ignore whitespace
 <span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="nb">sum</span> <span class="k">as</span> <span class="n">npsum</span><span class="p">,</span> <span class="n">asmatrix</span><span class="p">,</span> <span class="n">log</span><span class="p">,</span> <span class="n">asarray</span><span class="p">,</span> <span class="n">append</span>
 <span class="kn">from</span> <span class="nn">Helper</span> <span class="kn">import</span> <span class="n">transform1Dto2D</span>
 
-<div class="viewcode-block" id="courseraML_CostFunc"><a class="viewcode-back" href="../costFunc.html#CostFunc.courseraML_CostFunc">[docs]</a><span class="k">def</span> <span class="nf">courseraML_CostFunc</span><span class="p">(</span><span class="n">forwardWeightAllLayers</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>
+<div class="viewcode-block" id="courseraML_CostFunc"><a class="viewcode-back" href="../costFunc.html#CostFunc.courseraML_CostFunc">[docs]</a><span class="k">def</span> <span class="nf">courseraML_CostFunc</span><span class="p">(</span><span class="n">weightAllLayers</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>
     <span class="sd">r&#39;&#39;&#39;</span>
 <span class="sd">    Vectorized/regulated cost function (described in the Coursera Stanford Machine Learning course) that computes the total cost over multiple inputs:</span>
 
 
 <span class="sd">    where m=the number of inputs; K=the number of labels in targets; y=a single target array; x=a single input array.</span>
 
-<span class="sd">    :param forwardWeightAllLayers: A flatten array contains all forward weights from input layer to the last hidden layer before output layer.</span>
+<span class="sd">    :param weightAllLayers: A flatten array contains all forward weights from input layer to the last hidden layer before output layer.</span>
 <span class="sd">    :param *args: Must in the following order:</span>
 
 <span class="sd">        **inputArr2D**: 1 training example per row.</span>
 
 <span class="sd">        **nn**: An instance of class FeedforwardNeuNet.</span>
 
-<span class="sd">    :returns: A scalar representing the cost of current input using forwardWeightAllLayers.</span>
+<span class="sd">    :returns: A scalar representing the cost of current input using weightAllLayers.</span>
 <span class="sd">    &#39;&#39;&#39;</span>
     <span class="n">inputArr2D</span><span class="p">,</span> <span class="n">targets</span><span class="p">,</span> <span class="n">weightDecayParam</span><span class="p">,</span> <span class="n">nn</span> <span class="o">=</span> <span class="n">args</span>
-    <span class="n">startIndex</span><span class="p">,</span> <span class="n">weights2D</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="p">[]</span>
+    <span class="n">startIndex</span><span class="p">,</span> <span class="n">weightsExcBias</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span>
     <span class="k">for</span> <span class="n">ly</span> <span class="ow">in</span> <span class="n">nn</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">:</span> <span class="c"># update all forward weights</span>
-        <span class="n">newWeight</span> <span class="o">=</span> <span class="n">transform1Dto2D</span><span class="p">(</span><span class="n">forwardWeightAllLayers</span><span class="p">[</span><span class="n">startIndex</span><span class="p">:</span><span class="n">startIndex</span> <span class="o">+</span> <span class="n">ly</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span><span class="o">.</span><span class="n">size</span><span class="p">],</span> <span class="o">*</span><span class="n">ly</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
-        <span class="n">ly</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span> <span class="o">=</span> <span class="n">asmatrix</span><span class="p">(</span><span class="n">newWeight</span><span class="p">)</span>
-        <span class="n">startIndex</span> <span class="o">+=</span> <span class="n">ly</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span><span class="o">.</span><span class="n">size</span>
-        <span class="n">weights2D</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">newWeight</span><span class="p">)</span>
+        <span class="n">newWeight</span> <span class="o">=</span> <span class="n">transform1Dto2D</span><span class="p">(</span><span class="n">weightAllLayers</span><span class="p">[</span><span class="n">startIndex</span><span class="p">:</span><span class="n">startIndex</span> <span class="o">+</span> <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="o">.</span><span class="n">size</span><span class="p">],</span> <span class="o">*</span><span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
+        <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span> <span class="o">=</span> <span class="n">asmatrix</span><span class="p">(</span><span class="n">newWeight</span><span class="p">)</span>
+        <span class="n">startIndex</span> <span class="o">+=</span> <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="o">.</span><span class="n">size</span>
+        <span class="n">weightsExcBias</span> <span class="o">=</span> <span class="n">append</span><span class="p">(</span><span class="n">weightsExcBias</span><span class="p">,</span> <span class="n">newWeight</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="c"># exclude weights for bias unit with [:-1]</span>
     <span class="n">output</span> <span class="o">=</span> <span class="n">asarray</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">forwardPropogateAllInput</span><span class="p">(</span><span class="n">inputArr2D</span><span class="p">))</span>
-    <span class="k">assert</span> <span class="n">output</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="n">targets</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="s">&#39;dimension mismatch in output layer&#39;</span>
-    <span class="k">return</span> <span class="mf">1.0</span> <span class="o">/</span> <span class="n">targets</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">*</span> <span class="p">(</span><span class="n">npsum</span><span class="p">(</span><span class="o">-</span><span class="n">targets</span> <span class="o">*</span> <span class="n">log</span><span class="p">(</span><span class="n">output</span><span class="p">)</span> <span class="o">-</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">targets</span><span class="p">)</span> <span class="o">*</span> <span class="n">log</span><span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">output</span><span class="p">))</span> <span class="o">+</span> <span class="n">weightDecayParam</span> <span class="o">/</span> <span class="mf">2.0</span> <span class="o">*</span> <span class="n">npsum</span><span class="p">(</span><span class="n">npsum</span><span class="p">(</span><span class="n">w</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="n">weights2D</span><span class="p">))</span> <span class="c"># exclude weights for bias unit with [:-1]</span>
+    <span class="k">assert</span> <span class="n">output</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="n">targets</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="s">&#39;dimension mismatch in next layer&#39;</span>
+    <span class="k">return</span> <span class="mf">1.0</span> <span class="o">/</span> <span class="n">targets</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">*</span> <span class="p">(</span><span class="n">npsum</span><span class="p">(</span><span class="o">-</span><span class="n">targets</span> <span class="o">*</span> <span class="n">log</span><span class="p">(</span><span class="n">output</span><span class="p">)</span> <span class="o">-</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">targets</span><span class="p">)</span> <span class="o">*</span> <span class="n">log</span><span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">output</span><span class="p">))</span> <span class="o">+</span> <span class="n">weightDecayParam</span> <span class="o">/</span> <span class="mf">2.0</span> <span class="o">*</span> <span class="n">npsum</span><span class="p">(</span><span class="n">weightsExcBias</span> <span class="o">**</span> <span class="mi">2</span><span class="p">))</span>
 </div>
-<div class="viewcode-block" id="courseraML_CostFuncGrad"><a class="viewcode-back" href="../costFunc.html#CostFunc.courseraML_CostFuncGrad">[docs]</a><span class="k">def</span> <span class="nf">courseraML_CostFuncGrad</span><span class="p">(</span><span class="n">forwardWeightAllLayers</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>
+<div class="viewcode-block" id="courseraML_CostFuncGrad"><a class="viewcode-back" href="../costFunc.html#CostFunc.courseraML_CostFuncGrad">[docs]</a><span class="k">def</span> <span class="nf">courseraML_CostFuncGrad</span><span class="p">(</span><span class="n">weightAllLayers</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>
     <span class="sd">r&#39;&#39;&#39;</span>
-<span class="sd">    Vectorized/regulated implementation that computes the partial derivatives of the :func:`courseraML_CostFunc` over each weight in forwardWeightAllLayers:</span>
+<span class="sd">    Vectorized/regulated implementation that computes the partial derivatives of the :func:`courseraML_CostFunc` over each weight in weightAllLayers:</span>
 
 <span class="sd">    .. math::</span>
 
 <span class="sd">        \frac{\partial~courseraMLCostFunc()}{\partial~weight_{ij}^{(layer~l)}}</span>
 
-<span class="sd">    :param forwardWeightAllLayers: A flatten array contains all forward weights from input layer to the last hidden layer before output layer.</span>
+<span class="sd">    :param weightAllLayers: A flatten array contains all forward weights from input layer to the last hidden layer before output layer.</span>
 <span class="sd">    :param *args: Must in the following order:</span>
 
 <span class="sd">        **inputArr2D**: 1 training example per row.</span>
 
 <span class="sd">        **nn**: An instance of class FeedforwardNeuNet.</span>
 
-<span class="sd">    :returns: A flatten array represents the partial derivatives of the :func:`courseraML_CostFunc` over each weight in forwardWeightAllLayers.</span>
+<span class="sd">    :returns: A flatten array represents the partial derivatives of the :func:`courseraML_CostFunc` over each weight in weightAllLayers.</span>
 <span class="sd">    &#39;&#39;&#39;</span>
-    <span class="n">_</span><span class="p">,</span> <span class="n">targets</span><span class="p">,</span> <span class="n">weightDecayParam</span><span class="p">,</span> <span class="n">nn</span> <span class="o">=</span> <span class="n">args</span> <span class="c"># no need to use forwardWeightAllLayers cause ly._NnLayer__forwardWeight will be updated in courseraML_CostFunc(), which will be used in FeedforwardNeuNet.train() together with courseraML_CostFuncGrad()</span>
-    <span class="n">costGradAllLyOutToIn</span> <span class="o">=</span> <span class="p">[</span><span class="n">ly</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span> <span class="k">for</span> <span class="n">ly</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">)]</span> <span class="c"># each is a triangle^{l}(a matrix) on Courera ML L9, p8</span>
-    <span class="n">errDeltaArrNxtLv</span> <span class="o">=</span> <span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">outputs</span> <span class="o">-</span> <span class="n">targets</span><span class="p">)</span>
-    <span class="k">for</span> <span class="n">n</span><span class="p">,</span> <span class="n">ly</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="nb">reversed</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="mi">1</span><span class="p">:])):</span>
-        <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="o">.</span><span class="n">T</span> <span class="o">*</span> <span class="n">errDeltaArrNxtLv</span> <span class="c"># Set m=(# of training examples), ly=layer below output layer =&gt; ly.self2D is a (m by (# of units, including bias)) matrix, and errDeltaArrNxtLv is a (m by (# of classes in output layer)) matrix. We originally have: outputLayerMatrix(which shares the same dimensions as errDeltaArrNxtLv)==ly.self2D*weightsMatrixBetweenOutputLyAndLyBelow(which shares the same dimensions as costGradAllLyOutToIn[n]), now we have: self.2D.T*outputLayerMatrix==self.2D.T*self.2D*weightsMatrixBetweenOutputLyAndLyBelow==weightsMatrixBetweenOutputLyAndLyBelow</span>
-        <span class="n">numOfExamples</span> <span class="o">=</span> <span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
-        <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="n">n</span><span class="p">]</span>
-        <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="n">n</span><span class="p">][:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="n">weightDecayParam</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">ly</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="c"># add regularization but exclude weights for bias unit</span>
+    <span class="n">_</span><span class="p">,</span> <span class="n">targets</span><span class="p">,</span> <span class="n">weightDecayParam</span><span class="p">,</span> <span class="n">nn</span> <span class="o">=</span> <span class="n">args</span> <span class="c"># no need to use weightAllLayers cause ly.forwardWeight will be updated in courseraML_CostFunc(), which will be used in FeedforwardNeuNet.train() together with courseraML_CostFuncGrad()</span>
+    <span class="n">costGradAllLyOutToIn</span> <span class="o">=</span> <span class="p">[]</span> <span class="c"># each is a triangle^{l}(a matrix) on Courera ML L9, p8</span>
+    <span class="n">numOfExamples</span> <span class="o">=</span> <span class="n">targets</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">*</span> <span class="mf">1.0</span>
+    <span class="n">errDeltaMatNxtLv</span> <span class="o">=</span> <span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">outputs</span> <span class="o">-</span> <span class="n">targets</span><span class="p">)</span>
+    <span class="k">for</span> <span class="n">ly</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="mi">1</span><span class="p">:]):</span>
+        <span class="n">costGradAllLyOutToIn</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="o">.</span><span class="n">T</span> <span class="o">*</span> <span class="n">errDeltaMatNxtLv</span><span class="p">)</span> <span class="c"># Set m=(# of training examples), ly=layer below output layer =&gt; ly.self2D is a (m by (# of units, including bias)) matrix, and errDeltaMatNxtLv is a (m by (# of classes in output layer)) matrix. We originally have: outputLayerMatrix(which shares the same dimensions as errDeltaMatNxtLv)==ly.self2D*weightsMatrixBetweenOutputLyAndLyBelow(which shares the same dimensions as costGradAllLyOutToIn[n]), now we have: self.2D.T*outputLayerMatrix==self.2D.T*self.2D*weightsMatrixBetweenOutputLyAndLyBelow==weightsMatrixBetweenOutputLyAndLyBelow</span>
+        <span class="c"># (check sparseAutoencoder_2011new.pdf p8) cause all bias unit has value 1, so after the statement above, costGradAllLyOutToIn[-1][-1] == npsum(errDeltaMatNxtLv, 0)==the partial deri of bi at the botm of p7</span>
+        <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
+        <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="n">weightDecayParam</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="c"># add regularization but exclude weights for bias unit</span>
         <span class="n">arr</span> <span class="o">=</span> <span class="n">asarray</span><span class="p">(</span><span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="p">)</span>
-        <span class="n">errDeltaArrNxtLv</span> <span class="o">=</span> <span class="n">asmatrix</span><span class="p">((</span><span class="n">asarray</span><span class="p">(</span><span class="n">errDeltaArrNxtLv</span> <span class="o">*</span> <span class="n">ly</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span><span class="o">.</span><span class="n">T</span><span class="p">)</span> <span class="o">*</span> <span class="n">arr</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">arr</span><span class="p">))[:,</span> <span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="c"># exclude bias unit</span>
-    <span class="n">ly</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="c"># no need to calculate errDeltaArrNxtLv for input layer, so separate it from for loop above</span>
-    <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="o">.</span><span class="n">T</span> <span class="o">*</span> <span class="n">errDeltaArrNxtLv</span> <span class="c"># costGradAllLyOutToIn[-1] is the gradient of cost func over input layer&#39;s weights</span>
-    <span class="n">numOfExamples</span> <span class="o">=</span> <span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
-    <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
-    <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="n">weightDecayParam</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">ly</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
+        <span class="n">errDeltaMatNxtLv</span> <span class="o">=</span> <span class="n">asmatrix</span><span class="p">((</span><span class="n">asarray</span><span class="p">(</span><span class="n">errDeltaMatNxtLv</span> <span class="o">*</span> <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="o">.</span><span class="n">T</span><span class="p">)</span> <span class="o">*</span> <span class="n">arr</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">arr</span><span class="p">))[:,</span> <span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="c"># exclude bias unit</span>
+    <span class="n">ly</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="c"># no need to calculate errDeltaMatNxtLv for input layer, so separate it from for loop above</span>
+    <span class="n">costGradAllLyOutToIn</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="o">.</span><span class="n">T</span> <span class="o">*</span> <span class="n">errDeltaMatNxtLv</span><span class="p">)</span> <span class="c"># costGradAllLyOutToIn[-1] is the gradient of cost func over input layer&#39;s weights</span>
+    <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
+    <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="n">weightDecayParam</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
     <span class="n">flat</span> <span class="o">=</span> <span class="n">asarray</span><span class="p">(</span><span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
     <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="n">costGradAllLyOutToIn</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]):</span>
         <span class="n">flat</span> <span class="o">=</span> <span class="n">append</span><span class="p">(</span><span class="n">flat</span><span class="p">,</span> <span class="n">asarray</span><span class="p">(</span><span class="n">f</span><span class="p">))</span>
-    <span class="k">return</span> <span class="n">flat</span></div>
+    <span class="k">return</span> <span class="n">flat</span>
+</div>
+<span class="k">def</span> <span class="nf">sparse_CostFunc</span><span class="p">(</span><span class="n">weightAllLayers</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>
+    <span class="sd">r&#39;&#39;&#39;</span>
+<span class="sd">    Vectorized/regulated cost function (described in the Coursera Stanford Machine Learning course) that computes the total cost over multiple inputs:</span>
+
+<span class="sd">    .. math::</span>
+
+<span class="sd">        \frac{1}{m}&amp;\sum_{i=1}^{m}\sum_{k=1}^{K}[-y_k^i log(forwardPropThruAllLayers(x^i)_k)-(1-y_k^i) log(1-forwardPropThruAllLayers(x^i)_k)]\\</span>
+<span class="sd">        &amp;+\frac{\lambda}{2m}\sum^{allLayers} (weight^2~if~weight~is~NOT~multiplied~with~any~bias~unit~otherwise~0)\\</span>
+
+<span class="sd">    where m=the number of inputs; K=the number of labels in targets; y=a single target array; x=a single input array.</span>
+
+<span class="sd">    :param weightAllLayers: A flatten array contains all forward weights from input layer to the last hidden layer before output layer.</span>
+<span class="sd">    :param *args: Must in the following order:</span>
+
+<span class="sd">        **inputArr2D**: 1 training example per row.</span>
+
+<span class="sd">        **targets**: The number of labels must match the number of units in output layer.</span>
+
+<span class="sd">        **weightDecayParam**: For model complexity regulation.</span>
+
+<span class="sd">        **nn**: An instance of class FeedforwardNeuNet.</span>
+
+<span class="sd">    :returns: A scalar representing the cost of current input using weightAllLayers.</span>
+<span class="sd">    &#39;&#39;&#39;</span>
+    <span class="n">inputArr2D</span><span class="p">,</span> <span class="n">targets</span><span class="p">,</span> <span class="n">weightDecayParam</span><span class="p">,</span> <span class="n">sparsity</span><span class="p">,</span> <span class="n">sparseParam</span><span class="p">,</span> <span class="n">nn</span> <span class="o">=</span> <span class="n">args</span>
+    <span class="n">startIndex</span><span class="p">,</span> <span class="n">weightsExcBias</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span>
+    <span class="n">avgEx</span> <span class="o">=</span> <span class="mf">1.0</span> <span class="o">/</span> <span class="n">targets</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
+    <span class="k">for</span> <span class="n">ly</span> <span class="ow">in</span> <span class="n">nn</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">:</span> <span class="c"># update all forward weights</span>
+        <span class="n">newWeight</span> <span class="o">=</span> <span class="n">transform1Dto2D</span><span class="p">(</span><span class="n">weightAllLayers</span><span class="p">[</span><span class="n">startIndex</span><span class="p">:</span><span class="n">startIndex</span> <span class="o">+</span> <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="o">.</span><span class="n">size</span><span class="p">],</span> <span class="o">*</span><span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
+        <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span> <span class="o">=</span> <span class="n">asmatrix</span><span class="p">(</span><span class="n">newWeight</span><span class="p">)</span>
+        <span class="n">startIndex</span> <span class="o">+=</span> <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="o">.</span><span class="n">size</span>
+        <span class="n">weightsExcBias</span> <span class="o">=</span> <span class="n">append</span><span class="p">(</span><span class="n">weightsExcBias</span><span class="p">,</span> <span class="n">newWeight</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="c"># exclude weights for bias unit with [:-1]</span>
+    <span class="n">output</span> <span class="o">=</span> <span class="n">asarray</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">forwardPropogateAllInput</span><span class="p">(</span><span class="n">inputArr2D</span><span class="p">))</span>
+    <span class="k">assert</span> <span class="n">output</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="n">targets</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="s">&#39;dimension mismatch in next layer&#39;</span>
+<span class="c">#     avgActvArrAllLyAllEx = 0</span>
+<span class="c">#     for ly in nn.layersExOutputLy[1:]:</span>
+<span class="c">#         ly.avgActvArrAllEx = avgEx * npsum(ly.self2D, 0)</span>
+<span class="c">#         avgActvArrAllLyAllEx = append(avgActvArrAllLyAllEx, ly.avgActvArrAllEx[:-1]) # exclude bias?</span>
+    <span class="k">return</span> <span class="n">avgEx</span> <span class="o">*</span> <span class="p">(</span><span class="n">npsum</span><span class="p">(</span><span class="o">-</span><span class="n">targets</span> <span class="o">*</span> <span class="n">log</span><span class="p">(</span><span class="n">output</span><span class="p">)</span> <span class="o">-</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">targets</span><span class="p">)</span> <span class="o">*</span> <span class="n">log</span><span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">output</span><span class="p">))</span> <span class="o">+</span> <span class="n">weightDecayParam</span> <span class="o">/</span> <span class="mf">2.0</span> <span class="o">*</span> <span class="n">npsum</span><span class="p">(</span><span class="n">weightsExcBias</span> <span class="o">**</span> <span class="mi">2</span><span class="p">))</span> <span class="o">+</span> <span class="n">sparseParam</span> <span class="o">*</span> <span class="n">npsum</span><span class="p">(</span><span class="n">sparsity</span> <span class="o">*</span> <span class="n">log</span><span class="p">(</span><span class="n">sparsity</span> <span class="o">/</span> <span class="n">avgActvArrAllLyAllEx</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">sparsity</span><span class="p">)</span> <span class="o">*</span> <span class="n">log</span><span class="p">((</span><span class="mi">1</span> <span class="o">-</span> <span class="n">sparsity</span><span class="p">)</span> <span class="o">/</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">avgActvArrAllLyAllEx</span><span class="p">)))</span>
+
+<span class="k">def</span> <span class="nf">sparse_CostFuncGrad</span><span class="p">(</span><span class="n">weightAllLayers</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>
+    <span class="sd">r&#39;&#39;&#39;</span>
+<span class="sd">    Vectorized/regulated implementation that computes the partial derivatives of the :func:`courseraML_CostFunc` over each weight in weightAllLayers:</span>
+
+<span class="sd">    .. math::</span>
+
+<span class="sd">        \frac{\partial~courseraMLCostFunc()}{\partial~weight_{ij}^{(layer~l)}}</span>
+
+<span class="sd">    :param weightAllLayers: A flatten array contains all forward weights from input layer to the last hidden layer before output layer.</span>
+<span class="sd">    :param *args: Must in the following order:</span>
+
+<span class="sd">        **inputArr2D**: 1 training example per row.</span>
+
+<span class="sd">        **targets**: The number of labels must match the number of units in output layer.</span>
+
+<span class="sd">        **weightDecayParam**: For model complexity regulation.</span>
+
+<span class="sd">        **nn**: An instance of class FeedforwardNeuNet.</span>
+
+<span class="sd">    :returns: A flatten array represents the partial derivatives of the :func:`courseraML_CostFunc` over each weight in weightAllLayers.</span>
+<span class="sd">    &#39;&#39;&#39;</span>
+    <span class="n">_</span><span class="p">,</span> <span class="n">targets</span><span class="p">,</span> <span class="n">weightDecayParam</span><span class="p">,</span> <span class="n">nn</span> <span class="o">=</span> <span class="n">args</span> <span class="c"># no need to use weightAllLayers cause ly.forwardWeight will be updated in courseraML_CostFunc(), which will be used in FeedforwardNeuNet.train() together with courseraML_CostFuncGrad()</span>
+    <span class="n">costGradAllLyOutToIn</span> <span class="o">=</span> <span class="p">[]</span> <span class="c"># each is a triangle^{l}(a matrix) on Courera ML L9, p8</span>
+    <span class="n">numOfExamples</span> <span class="o">=</span> <span class="n">targets</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">*</span> <span class="mf">1.0</span>
+    <span class="n">errDeltaMatNxtLv</span> <span class="o">=</span> <span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">outputs</span> <span class="o">-</span> <span class="n">targets</span><span class="p">)</span>
+    <span class="k">for</span> <span class="n">ly</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="mi">1</span><span class="p">:]):</span>
+        <span class="n">costGradAllLyOutToIn</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="o">.</span><span class="n">T</span> <span class="o">*</span> <span class="n">errDeltaMatNxtLv</span><span class="p">)</span> <span class="c"># Set m=(# of training examples), ly=layer below output layer =&gt; ly.self2D is a (m by (# of units, including bias)) matrix, and errDeltaMatNxtLv is a (m by (# of classes in output layer)) matrix. We originally have: outputLayerMatrix(which shares the same dimensions as errDeltaMatNxtLv)==ly.self2D*weightsMatrixBetweenOutputLyAndLyBelow(which shares the same dimensions as costGradAllLyOutToIn[n]), now we have: self.2D.T*outputLayerMatrix==self.2D.T*self.2D*weightsMatrixBetweenOutputLyAndLyBelow==weightsMatrixBetweenOutputLyAndLyBelow</span>
+        <span class="c"># (check sparseAutoencoder_2011new.pdf p8) cause all bias unit has value 1, so after the statement above, costGradAllLyOutToIn[-1][-1] == npsum(errDeltaMatNxtLv, 0)==the partial deri of bi at the botm of p7</span>
+        <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
+        <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="n">weightDecayParam</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="c"># add regularization but exclude weights for bias unit</span>
+        <span class="n">arr</span> <span class="o">=</span> <span class="n">asarray</span><span class="p">(</span><span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="p">)</span>
+        <span class="n">errDeltaMatNxtLv</span> <span class="o">=</span> <span class="n">asmatrix</span><span class="p">((</span><span class="n">asarray</span><span class="p">(</span><span class="n">errDeltaMatNxtLv</span> <span class="o">*</span> <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="o">.</span><span class="n">T</span><span class="p">)</span> <span class="o">*</span> <span class="n">arr</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">arr</span><span class="p">))[:,</span> <span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span> <span class="c"># exclude bias unit</span>
+    <span class="n">ly</span> <span class="o">=</span> <span class="n">nn</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="c"># no need to calculate errDeltaMatNxtLv for input layer, so separate it from for loop above</span>
+    <span class="n">costGradAllLyOutToIn</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="o">.</span><span class="n">T</span> <span class="o">*</span> <span class="n">errDeltaMatNxtLv</span><span class="p">)</span> <span class="c"># costGradAllLyOutToIn[-1] is the gradient of cost func over input layer&#39;s weights</span>
+    <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
+    <span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">][:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">+=</span> <span class="n">weightDecayParam</span> <span class="o">/</span> <span class="n">numOfExamples</span> <span class="o">*</span> <span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
+    <span class="n">flat</span> <span class="o">=</span> <span class="n">asarray</span><span class="p">(</span><span class="n">costGradAllLyOutToIn</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
+    <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="n">costGradAllLyOutToIn</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]):</span>
+        <span class="n">flat</span> <span class="o">=</span> <span class="n">append</span><span class="p">(</span><span class="n">flat</span><span class="p">,</span> <span class="n">asarray</span><span class="p">(</span><span class="n">f</span><span class="p">))</span>
+    <span class="k">return</span> <span class="n">flat</span>
 </pre></div>
 
           </div>

File deepArch/_modules/FeedforwardNeuNet.html

View file
  • Ignore whitespace
 <span class="sd">This module contains 2 classes inherited from ndarray, NnLayer and NeuNet, which are the building</span>
 <span class="sd">blocks of neural network. It also implements relating activation function such as sigmoid.</span>
 <span class="sd">&#39;&#39;&#39;</span>
-<span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="n">exp</span><span class="p">,</span> <span class="n">ndarray</span><span class="p">,</span> <span class="n">append</span><span class="p">,</span> <span class="n">empty</span><span class="p">,</span> <span class="n">ones</span><span class="p">,</span> <span class="n">ravel</span><span class="p">,</span> <span class="n">asmatrix</span><span class="p">,</span> <span class="n">asarray</span>
+<span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="n">exp</span><span class="p">,</span> <span class="n">ndarray</span><span class="p">,</span> <span class="n">append</span><span class="p">,</span> <span class="n">empty</span><span class="p">,</span> <span class="n">ones</span><span class="p">,</span> <span class="n">ravel</span><span class="p">,</span> <span class="n">asmatrix</span><span class="p">,</span> <span class="n">asarray</span><span class="p">,</span> <span class="nb">sum</span> <span class="k">as</span> <span class="n">npsum</span>
 <span class="kn">from</span> <span class="nn">scipy.optimize</span> <span class="kn">import</span> <span class="n">fmin_cg</span>
 
 <div class="viewcode-block" id="sigmoid"><a class="viewcode-back" href="../deepArchitecture.html#FeedforwardNeuNet.sigmoid">[docs]</a><span class="k">def</span> <span class="nf">sigmoid</span><span class="p">(</span><span class="n">arr</span><span class="p">):</span>
 <span class="sd">        &#39;&#39;&#39;</span>
         <span class="n">obj</span> <span class="o">=</span> <span class="n">append</span><span class="p">(</span><span class="n">empty</span><span class="p">(</span><span class="n">numOfUnit</span><span class="p">),</span> <span class="n">biasUnitValue</span><span class="p">)</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="n">cls</span><span class="p">)</span>
         <span class="n">obj</span><span class="o">.</span><span class="n">__activFunc</span> <span class="o">=</span> <span class="n">activFunc</span>
-        <span class="n">obj</span><span class="o">.</span><span class="n">__forwardWeight</span> <span class="o">=</span> <span class="n">asmatrix</span><span class="p">(</span><span class="n">empty</span><span class="p">((</span><span class="n">obj</span><span class="o">.</span><span class="n">size</span><span class="p">,</span> <span class="n">numOfUnitNextLv</span><span class="p">)))</span>
+        <span class="n">obj</span><span class="o">.</span><span class="n">forwardWeight</span> <span class="o">=</span> <span class="n">asmatrix</span><span class="p">(</span><span class="n">empty</span><span class="p">((</span><span class="n">obj</span><span class="o">.</span><span class="n">size</span><span class="p">,</span> <span class="n">numOfUnitNextLv</span><span class="p">)))</span>
         <span class="k">return</span> <span class="n">obj</span>
 
 <span class="c">#    __array_finalize__ is the only method that always sees new instances being created, it is the sensible place to fill in instance defaults for new object attributes, among other tasks.</span>
 <span class="sd">        NnLayer([0.1, 0.2, 0.3, 1.])</span>
 <span class="sd">        &#39;&#39;&#39;</span>
         <span class="bp">self</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">inputArr</span>
-        <span class="k">return</span> <span class="n">ravel</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">__activFunc</span><span class="p">(</span><span class="n">asarray</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">__forwardWeight</span><span class="p">))</span>
+        <span class="k">return</span> <span class="n">ravel</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">__activFunc</span><span class="p">(</span><span class="n">asarray</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">forwardWeight</span><span class="p">))</span>
 </div>
 <div class="viewcode-block" id="NnLayer.actvByAllInput"><a class="viewcode-back" href="../deepArchitecture.html#FeedforwardNeuNet.NnLayer.actvByAllInput">[docs]</a>    <span class="k">def</span> <span class="nf">actvByAllInput</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">inputArr2D</span><span class="p">):</span>
         <span class="sd">&#39;&#39;&#39;</span>
 <span class="sd">        NnLayer([0.23, 0.37, 0.28, 1.])</span>
 <span class="sd">        &#39;&#39;&#39;</span>
         <span class="bp">self</span><span class="o">.</span><span class="n">self2D</span> <span class="o">=</span> <span class="n">append</span><span class="p">(</span><span class="n">inputArr2D</span><span class="p">,</span> <span class="n">ones</span><span class="p">((</span><span class="n">inputArr2D</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="mi">1</span><span class="p">))</span> <span class="o">*</span> <span class="bp">self</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">],</span> <span class="mi">1</span><span class="p">)</span> <span class="c"># self2D[-1] == bias</span>
-        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__activFunc</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">self2D</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">__forwardWeight</span><span class="p">)</span>
+        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">__activFunc</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">self2D</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">forwardWeight</span><span class="p">)</span>
 </div>
 <div class="viewcode-block" id="NnLayer.updateForwardWeight"><a class="viewcode-back" href="../deepArchitecture.html#FeedforwardNeuNet.NnLayer.updateForwardWeight">[docs]</a>    <span class="k">def</span> <span class="nf">updateForwardWeight</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">newWeight</span><span class="p">):</span>
         <span class="sd">&#39;&#39;&#39;</span>
 <span class="sd">        :param newWeight: Can be either nested tuple/list or numpy matrix.</span>
 <span class="sd">        &#39;&#39;&#39;</span>
         <span class="n">newWeight</span> <span class="o">=</span> <span class="n">asmatrix</span><span class="p">(</span><span class="n">newWeight</span><span class="p">)</span><span class="o">.</span><span class="n">T</span>
-        <span class="k">assert</span> <span class="bp">self</span><span class="o">.</span><span class="n">__forwardWeight</span><span class="o">.</span><span class="n">shape</span> <span class="o">==</span> <span class="n">newWeight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="s">&#39;weight matrices dimension mismatch&#39;</span>
-        <span class="bp">self</span><span class="o">.</span><span class="n">__forwardWeight</span> <span class="o">=</span> <span class="n">newWeight</span>
+        <span class="k">assert</span> <span class="bp">self</span><span class="o">.</span><span class="n">forwardWeight</span><span class="o">.</span><span class="n">shape</span> <span class="o">==</span> <span class="n">newWeight</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="s">&#39;weight matrices dimension mismatch&#39;</span>
+        <span class="bp">self</span><span class="o">.</span><span class="n">forwardWeight</span> <span class="o">=</span> <span class="n">newWeight</span>
 </div></div>
 <div class="viewcode-block" id="FeedforwardNeuNet"><a class="viewcode-back" href="../deepArchitecture.html#FeedforwardNeuNet.FeedforwardNeuNet">[docs]</a><span class="k">class</span> <span class="nc">FeedforwardNeuNet</span><span class="p">(</span><span class="n">ndarray</span><span class="p">):</span>
     <span class="sd">&#39;&#39;&#39;</span>
 <span class="sd">    An instance of this class represents a single neural network. The instance itself is an 1D numpy array served as the output layer.</span>
 <span class="sd">    &#39;&#39;&#39;</span>
-<div class="viewcode-block" id="FeedforwardNeuNet.__new__"><a class="viewcode-back" href="../deepArchitecture.html#FeedforwardNeuNet.FeedforwardNeuNet.__new__">[docs]</a>    <span class="k">def</span> <span class="nf">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">layersExOutputLy</span><span class="p">,</span> <span class="n">weightDecayParam</span><span class="p">):</span>
+<div class="viewcode-block" id="FeedforwardNeuNet.__new__"><a class="viewcode-back" href="../deepArchitecture.html#FeedforwardNeuNet.FeedforwardNeuNet.__new__">[docs]</a>    <span class="k">def</span> <span class="nf">__new__</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">layersExOutputLy</span><span class="p">,</span> <span class="n">weightDecayParam</span><span class="p">,</span> <span class="n">sparsity</span><span class="p">,</span> <span class="n">sparseParam</span><span class="p">):</span>
         <span class="sd">&#39;&#39;&#39;</span>
 <span class="sd">        :param layersExOutputLy: A tuple that contains all layers except the output layer.</span>
 <span class="sd">        :param weightDecayParam: Parameter that regulates network model complexity.</span>
+<span class="sd">        :param sparsity: Parameter that set the threshold of the sparsity of the neural network.</span>
+<span class="sd">        :param sparseParam: Parameter that regulates the sparsity.</span>
 
 <span class="sd">        &gt;&gt;&gt; layer0, layer1 = NnLayer(sigmoid, 3, 1, 2), NnLayer(sigmoid, 2, 1, 6)</span>
 <span class="sd">        &gt;&gt;&gt; layer0.updateForwardWeight(((0.11, 0.12, 0.13, 0.14), (0.15, 0.16, 0.17, 0.18)))</span>
 <span class="sd">        &gt;&gt;&gt; layer1.updateForwardWeight(((0.201, 0.202, 0.203), (0.204, 0.205, 0.206), (0.207, 0.208, 0.209), (0.21, 0.211, 0.212), (0.213, 0.214, 0.215), (0.216, 0.217, 0.218)))</span>
-<span class="sd">        &gt;&gt;&gt; nn = NeuNet((layers0, layer1), 1, costFunc)</span>
+<span class="sd">        &gt;&gt;&gt; nn = NeuNet((layers0, layer1), 1, 0.05, 1)</span>
 <span class="sd">        &#39;&#39;&#39;</span>
-        <span class="n">obj</span> <span class="o">=</span> <span class="n">empty</span><span class="p">(</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="n">cls</span><span class="p">)</span>
+        <span class="n">obj</span> <span class="o">=</span> <span class="n">empty</span><span class="p">(</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">forwardWeight</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span><span class="o">.</span><span class="n">view</span><span class="p">(</span><span class="n">cls</span><span class="p">)</span>
         <span class="n">obj</span><span class="o">.</span><span class="n">layersExOutputLy</span> <span class="o">=</span> <span class="n">layersExOutputLy</span>
+        <span class="n">obj</span><span class="o">.</span><span class="n">avgActvAllLyExOutputLyOverAllEx</span> <span class="o">=</span> <span class="bp">None</span>
         <span class="n">obj</span><span class="o">.</span><span class="n">__weightDecayParam</span> <span class="o">=</span> <span class="n">weightDecayParam</span>
+        <span class="n">obj</span><span class="o">.</span><span class="n">__sparsity</span> <span class="o">=</span> <span class="n">sparsity</span>
+        <span class="n">obj</span><span class="o">.</span><span class="n">__sparseParam</span> <span class="o">=</span> <span class="n">sparseParam</span>
         <span class="k">return</span> <span class="n">obj</span>
 </div>
 <div class="viewcode-block" id="FeedforwardNeuNet.train"><a class="viewcode-back" href="../deepArchitecture.html#FeedforwardNeuNet.FeedforwardNeuNet.train">[docs]</a>    <span class="k">def</span> <span class="nf">train</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">inputArr2D</span><span class="p">,</span> <span class="n">targets</span><span class="p">,</span> <span class="n">costFunc</span><span class="p">,</span> <span class="n">costFuncGrad</span><span class="p">,</span> <span class="n">maxIter</span><span class="o">=</span><span class="mi">100</span><span class="p">):</span>
 <span class="sd">        :param costFuncGrad: callable *f&#39;(paramToOptimize, \*arg)* that will be used to compute partial derivative of cost function over each parameter in paramToOptimize.</span>
 <span class="sd">        &#39;&#39;&#39;</span>
         <span class="bp">self</span><span class="o">.</span><span class="n">forwardPropogateAllInput</span><span class="p">(</span><span class="n">inputArr2D</span><span class="p">)</span> <span class="c"># perform forward propagation to set self.outputs</span>
-        <span class="n">flatWeights</span> <span class="o">=</span> <span class="n">asarray</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span><span class="p">)</span>
+        <span class="n">avgEx</span> <span class="o">=</span> <span class="mf">1.0</span> <span class="o">/</span> <span class="n">targets</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
+        <span class="n">flatWeights</span> <span class="o">=</span> <span class="n">asarray</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">forwardWeight</span><span class="p">)</span>
         <span class="k">for</span> <span class="n">ly</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">layersExOutputLy</span><span class="p">[</span><span class="mi">1</span><span class="p">:]:</span>
-            <span class="n">flatWeights</span> <span class="o">=</span> <span class="n">append</span><span class="p">(</span><span class="n">flatWeights</span><span class="p">,</span> <span class="n">asarray</span><span class="p">(</span><span class="n">ly</span><span class="o">.</span><span class="n">_NnLayer__forwardWeight</span><span class="p">))</span>
-        <span class="n">fmin_cg</span><span class="p">(</span><span class="n">costFunc</span><span class="p">,</span> <span class="n">flatWeights</span><span class="p">,</span> <span class="n">costFuncGrad</span><span class="p">,</span> <span class="p">(</span><span class="n">inputArr2D</span><span class="p">,</span> <span class="n">targets</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">__weightDecayParam</span><span class="p">,</span> <span class="bp">self</span><span class="p">),</span> <span class="n">maxiter</span><span class="o">=</span><span class="n">maxIter</span><span class="p">,</span> <span class="n">full_output</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="c"># fmin_cg calls grad before cost func</span>
+            <span class="n">ly</span><span class="o">.</span><span class="n">avgActvArrAllEx</span> <span class="o">=</span> <span class="n">avgEx</span> <span class="o">*</span> <span class="n">npsum</span><span class="p">(</span><span class="n">ly</span><span class="o">.</span><span class="n">self2D</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
+            <span class="n">flatWeights</span> <span class="o">=</span> <span class="n">append</span><span class="p">(</span><span class="n">flatWeights</span><span class="p">,</span> <span class="n">asarray</span><span class="p">(</span><span class="n">ly</span><span class="o">.</span><span class="n">forwardWeight</span><span class="p">))</span>
+        <span class="n">fmin_cg</span><span class="p">(</span><span class="n">costFunc</span><span class="p">,</span> <span class="n">flatWeights</span><span class="p">,</span> <span class="n">costFuncGrad</span><span class="p">,</span> <span class="p">(</span><span class="n">inputArr2D</span><span class="p">,</span> <span class="n">targets</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">__weightDecayParam</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">__sparsity</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">__sparseParam</span><span class="p">,</span> <span class="bp">self</span><span class="p">),</span> <span class="n">maxiter</span><span class="o">=</span><span class="n">maxIter</span><span class="p">,</span> <span class="n">full_output</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="c"># fmin_cg calls grad before cost func</span>
 <span class="c">#         startIndex = 0 # update all forward weights(not really sure this is necessary cause I already done this in cost func)</span>
 <span class="c">#         for ly in self.layersExOutputLy:</span>
-<span class="c">#             newWeight = transform1Dto2D(optForwardWeights[startIndex:startIndex + ly._NnLayer__forwardWeight.size], *ly._NnLayer__forwardWeight.shape)</span>
-<span class="c">#             ly._NnLayer__forwardWeight = asmatrix(newWeight)</span>
-<span class="c">#             startIndex += ly._NnLayer__forwardWeight.size</span>
+<span class="c">#             newWeight = transform1Dto2D(optForwardWeights[startIndex:startIndex + ly.forwardWeight.size], *ly.forwardWeight.shape)</span>
+<span class="c">#             ly.forwardWeight = asmatrix(newWeight)</span>
+<span class="c">#             startIndex += ly.forwardWeight.size</span>
 </div>
 <div class="viewcode-block" id="FeedforwardNeuNet.forwardPropogateOneInput"><a class="viewcode-back" href="../deepArchitecture.html#FeedforwardNeuNet.FeedforwardNeuNet.forwardPropogateOneInput">[docs]</a>    <span class="k">def</span> <span class="nf">forwardPropogateOneInput</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">inputArr</span><span class="p">):</span>
         <span class="sd">&#39;&#39;&#39;</span>

File deepArch/_sources/deepArchitecture.txt

View file
  • Ignore whitespace
 
 Feedforward Neural Network Structure
 ------------------------------------
-.. image:: https://lh3.googleusercontent.com/-jxMPh0i6q5M/UeZFRAWHtrI/AAAAAAAAGpM/Y888-Vz2ri8/w987-h549-no/FeedforwardNNStruct.jpg
+.. image:: https://lh6.googleusercontent.com/-h9JB8KqOcK8/UfI_lmuSTwI/AAAAAAAAGxg/DjrkXas5u5c/w926-h522-no/FeedForwardNNStruct.png
 
 .. |addBias| image:: https://lh3.googleusercontent.com/-ZoJswguVmxQ/UeZHe6Y8zeI/AAAAAAAAGpk/RDza9WEoHzk/w48-h44-no/addBias.png
 	:width: 40px

File deepArch/costFunc.html

View file
  • Ignore whitespace
 <span id="stanford-coursera-machine-learning-course"></span><h2>3.1. Stanford Coursera Machine Learning course<a class="headerlink" href="#module-CostFunc" title="Permalink to this headline">¶</a></h2>
 <dl class="function">
 <dt id="CostFunc.courseraML_CostFunc">
-<tt class="descclassname">CostFunc.</tt><tt class="descname">courseraML_CostFunc</tt><big>(</big><em>forwardWeightAllLayers</em>, <em>*args</em><big>)</big><a class="reference internal" href="_modules/CostFunc.html#courseraML_CostFunc"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#CostFunc.courseraML_CostFunc" title="Permalink to this definition">¶</a></dt>
+<tt class="descclassname">CostFunc.</tt><tt class="descname">courseraML_CostFunc</tt><big>(</big><em>weightAllLayers</em>, <em>*args</em><big>)</big><a class="reference internal" href="_modules/CostFunc.html#courseraML_CostFunc"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#CostFunc.courseraML_CostFunc" title="Permalink to this definition">¶</a></dt>
 <dd><p>Vectorized/regulated cost function (described in the Coursera Stanford Machine Learning course) that computes the total cost over multiple inputs:</p>
 <div class="math">
 <p><img src="_images/math/6e00b17315c0ef793e7fea24014ae1f68a0387f8.png" alt="\frac{1}{m}&amp;\sum_{i=1}^{m}\sum_{k=1}^{K}[-y_k^i log(forwardPropThruAllLayers(x^i)_k)-(1-y_k^i) log(1-forwardPropThruAllLayers(x^i)_k)]\\
 <col class="field-body" />
 <tbody valign="top">
 <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
-<li><strong>forwardWeightAllLayers</strong> &#8211; A flatten array contains all forward weights from input layer to the last hidden layer before output layer.</li>
+<li><strong>weightAllLayers</strong> &#8211; A flatten array contains all forward weights from input layer to the last hidden layer before output layer.</li>
 <li><strong>*args</strong> &#8211; <p>Must in the following order:</p>
 <p><strong>inputArr2D</strong>: 1 training example per row.</p>
 <p><strong>targets</strong>: The number of labels must match the number of units in output layer.</p>
 </ul>
 </td>
 </tr>
-<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">A scalar representing the cost of current input using forwardWeightAllLayers.</p>
+<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">A scalar representing the cost of current input using weightAllLayers.</p>
 </td>
 </tr>
 </tbody>
 
 <dl class="function">
 <dt id="CostFunc.courseraML_CostFuncGrad">
-<tt class="descclassname">CostFunc.</tt><tt class="descname">courseraML_CostFuncGrad</tt><big>(</big><em>forwardWeightAllLayers</em>, <em>*args</em><big>)</big><a class="reference internal" href="_modules/CostFunc.html#courseraML_CostFuncGrad"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#CostFunc.courseraML_CostFuncGrad" title="Permalink to this definition">¶</a></dt>
-<dd><p>Vectorized/regulated implementation that computes the partial derivatives of the <a class="reference internal" href="#CostFunc.courseraML_CostFunc" title="CostFunc.courseraML_CostFunc"><tt class="xref py py-func docutils literal"><span class="pre">courseraML_CostFunc()</span></tt></a> over each weight in forwardWeightAllLayers:</p>
+<tt class="descclassname">CostFunc.</tt><tt class="descname">courseraML_CostFuncGrad</tt><big>(</big><em>weightAllLayers</em>, <em>*args</em><big>)</big><a class="reference internal" href="_modules/CostFunc.html#courseraML_CostFuncGrad"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#CostFunc.courseraML_CostFuncGrad" title="Permalink to this definition">¶</a></dt>
+<dd><p>Vectorized/regulated implementation that computes the partial derivatives of the <a class="reference internal" href="#CostFunc.courseraML_CostFunc" title="CostFunc.courseraML_CostFunc"><tt class="xref py py-func docutils literal"><span class="pre">courseraML_CostFunc()</span></tt></a> over each weight in weightAllLayers:</p>
 <div class="math">
 <p><img src="_images/math/25c8a77c4942de9aa1a7a923c9a77e2dde90f076.png" alt="\frac{\partial~courseraMLCostFunc()}{\partial~weight_{ij}^{(layer~l)}}"/></p>
 </div><table class="docutils field-list" frame="void" rules="none">
 <col class="field-body" />
 <tbody valign="top">
 <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first simple">
-<li><strong>forwardWeightAllLayers</strong> &#8211; A flatten array contains all forward weights from input layer to the last hidden layer before output layer.</li>
+<li><strong>weightAllLayers</strong> &#8211; A flatten array contains all forward weights from input layer to the last hidden layer before output layer.</li>
 <li><strong>*args</strong> &#8211; <p>Must in the following order:</p>
 <p><strong>inputArr2D</strong>: 1 training example per row.</p>
 <p><strong>targets</strong>: The number of labels must match the number of units in output layer.</p>
 </ul>
 </td>
 </tr>
-<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">A flatten array represents the partial derivatives of the <a class="reference internal" href="#CostFunc.courseraML_CostFunc" title="CostFunc.courseraML_CostFunc"><tt class="xref py py-func docutils literal"><span class="pre">courseraML_CostFunc()</span></tt></a> over each weight in forwardWeightAllLayers.</p>
+<tr class="field-even field"><th class="field-name">Returns:</th><td class="field-body"><p class="first last">A flatten array represents the partial derivatives of the <a class="reference internal" href="#CostFunc.courseraML_CostFunc" title="CostFunc.courseraML_CostFunc"><tt class="xref py py-func docutils literal"><span class="pre">courseraML_CostFunc()</span></tt></a> over each weight in weightAllLayers.</p>
 </td>
 </tr>
 </tbody>

File deepArch/deepArchitecture.html

View file
  • Ignore whitespace
 <p>Various types of deep architectures.</p>
 <div class="section" id="feedforward-neural-network-structure">
 <h2>2.1. Feedforward Neural Network Structure<a class="headerlink" href="#feedforward-neural-network-structure" title="Permalink to this headline">¶</a></h2>
-<img alt="https://lh3.googleusercontent.com/-jxMPh0i6q5M/UeZFRAWHtrI/AAAAAAAAGpM/Y888-Vz2ri8/w987-h549-no/FeedforwardNNStruct.jpg" src="https://lh3.googleusercontent.com/-jxMPh0i6q5M/UeZFRAWHtrI/AAAAAAAAGpM/Y888-Vz2ri8/w987-h549-no/FeedforwardNNStruct.jpg" />
+<img alt="https://lh6.googleusercontent.com/-h9JB8KqOcK8/UfI_lmuSTwI/AAAAAAAAGxg/DjrkXas5u5c/w926-h522-no/FeedForwardNNStruct.png" src="https://lh6.googleusercontent.com/-h9JB8KqOcK8/UfI_lmuSTwI/AAAAAAAAGxg/DjrkXas5u5c/w926-h522-no/FeedForwardNNStruct.png" />
 <p><img alt="addBias" src="https://lh3.googleusercontent.com/-ZoJswguVmxQ/UeZHe6Y8zeI/AAAAAAAAGpk/RDza9WEoHzk/w48-h44-no/addBias.png" style="width: 40px; height: 40px;" /> : add bias to the input. <img alt="weights" src="https://lh6.googleusercontent.com/-4RBDIb7_VRY/UeZHiRa8nrI/AAAAAAAAGps/XCWAa7E8vGA/w89-h51-no/weights.png" style="width: 60px; height: 30px;" /> : multiple the input by the corresponding weights of that layer. <img alt="save_icon" src="https://lh6.googleusercontent.com/-dmBSipdmYWY/UeZHnDw6L7I/AAAAAAAAGp0/iNiHZykQ2Lc/s14-no/save_icon.gif" style="width: 20px; height: 20px;" /> : save the result as the values of this layer. <img alt="sigmoid" src="https://lh4.googleusercontent.com/-MC8-zpYp6aU/UeZHpFFwCkI/AAAAAAAAGp8/sGCCrSxa9hI/w73-h40-no/sigmoid.png" style="width: 60px; height: 30px;" /> : execute sigmoid function on the input element-wise.</p>
 </div>
 <div class="section" id="feedforward-neural-network-implementation">
 <dd><p>An instance of this class represents a single neural network. The instance itself is an 1D numpy array served as the output layer.</p>
 <dl class="staticmethod">
 <dt id="FeedforwardNeuNet.FeedforwardNeuNet.__new__">
-<em class="property">static </em><tt class="descname">__new__</tt><big>(</big><em>layersExOutputLy</em>, <em>weightDecayParam</em><big>)</big><a class="reference internal" href="_modules/FeedforwardNeuNet.html#FeedforwardNeuNet.__new__"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#FeedforwardNeuNet.FeedforwardNeuNet.__new__" title="Permalink to this definition">¶</a></dt>
+<em class="property">static </em><tt class="descname">__new__</tt><big>(</big><em>layersExOutputLy</em>, <em>weightDecayParam</em>, <em>sparsity</em>, <em>sparseParam</em><big>)</big><a class="reference internal" href="_modules/FeedforwardNeuNet.html#FeedforwardNeuNet.__new__"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#FeedforwardNeuNet.FeedforwardNeuNet.__new__" title="Permalink to this definition">¶</a></dt>
 <dd><table class="docutils field-list" frame="void" rules="none">
 <col class="field-name" />
 <col class="field-body" />
 <tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
 <li><strong>layersExOutputLy</strong> &#8211; A tuple that contains all layers except the output layer.</li>
 <li><strong>weightDecayParam</strong> &#8211; Parameter that regulates network model complexity.</li>
+<li><strong>sparsity</strong> &#8211; Parameter that set the threshold of the sparsity of the neural network.</li>
+<li><strong>sparseParam</strong> &#8211; Parameter that regulates the sparsity.</li>
 </ul>
 </td>
 </tr>
 <div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">layer0</span><span class="p">,</span> <span class="n">layer1</span> <span class="o">=</span> <span class="n">NnLayer</span><span class="p">(</span><span class="n">sigmoid</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">),</span> <span class="n">NnLayer</span><span class="p">(</span><span class="n">sigmoid</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">6</span><span class="p">)</span>
 <span class="gp">&gt;&gt;&gt; </span><span class="n">layer0</span><span class="o">.</span><span class="n">updateForwardWeight</span><span class="p">(((</span><span class="mf">0.11</span><span class="p">,</span> <span class="mf">0.12</span><span class="p">,</span> <span class="mf">0.13</span><span class="p">,</span> <span class="mf">0.14</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.15</span><span class="p">,</span> <span class="mf">0.16</span><span class="p">,</span> <span class="mf">0.17</span><span class="p">,</span> <span class="mf">0.18</span><span class="p">)))</span>
 <span class="gp">&gt;&gt;&gt; </span><span class="n">layer1</span><span class="o">.</span><span class="n">updateForwardWeight</span><span class="p">(((</span><span class="mf">0.201</span><span class="p">,</span> <span class="mf">0.202</span><span class="p">,</span> <span class="mf">0.203</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.204</span><span class="p">,</span> <span class="mf">0.205</span><span class="p">,</span> <span class="mf">0.206</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.207</span><span class="p">,</span> <span class="mf">0.208</span><span class="p">,</span> <span class="mf">0.209</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.21</span><span class="p">,</span> <span class="mf">0.211</span><span class="p">,</span> <span class="mf">0.212</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.213</span><span class="p">,</span> <span class="mf">0.214</span><span class="p">,</span> <span class="mf">0.215</span><span class="p">),</span> <span class="p">(</span><span class="mf">0.216</span><span class="p">,</span> <span class="mf">0.217</span><span class="p">,</span> <span class="mf">0.218</span><span class="p">)))</span>
-<span class="gp">&gt;&gt;&gt; </span><span class="n">nn</span> <span class="o">=</span> <span class="n">NeuNet</span><span class="p">((</span><span class="n">layers0</span><span class="p">,</span> <span class="n">layer1</span><span class="p">),</span> <span class="mi">1</span><span class="p">,</span> <span class="n">costFunc</span><span class="p">)</span>
+<span class="gp">&gt;&gt;&gt; </span><span class="n">nn</span> <span class="o">=</span> <span class="n">NeuNet</span><span class="p">((</span><span class="n">layers0</span><span class="p">,</span> <span class="n">layer1</span><span class="p">),</span> <span class="mi">1</span><span class="p">,</span> <span class="mf">0.05</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
 </pre></div>
 </div>
 </dd></dl>

File deepArch/searchindex.js

View file
  • Ignore whitespace
-Search.setIndex({objects:{"":{CostFunc:[2,0,1,""]},"FeedforwardNeuNet.FeedforwardNeuNet":{forwardPropogateOneInput:[5,1,1,""],train:[5,1,1,""],forwardPropogateAllInput:[5,1,1,""],"__new__":[5,4,1,""]},CostFunc:{courseraML_CostFuncGrad:[2,3,1,""],courseraML_CostFunc:[2,3,1,""]},"FeedforwardNeuNet.NnLayer":{updateForwardWeight:[5,1,1,""],recvAndActvByOneInput:[5,1,1,""],actvByAllInput:[5,1,1,""],"__new__":[5,4,1,""]},FeedforwardNeuNet:{sigmoid:[5,3,1,""],NnLayer:[5,2,1,""],FeedforwardNeuNet:[5,2,1,""]}},terms:{all:[5,2],numofunitnextlv:5,partial:[5,2],execut:5,sigmoid:5,over:[5,2],layersexoutputli:5,paramtooptim:5,scalar:2,code:1,through:4,edu:2,follow:2,actvbyallinput:5,newweight:5,paramet:[5,2],feedforward:[0,5,4],fit:5,weightdecayparam:[5,2],activ:5,except:5,param:5,matrix:5,add:5,input:[5,1,2],hidden:2,feedforwardneunet:[5,2],modul:[0,2],match:2,"case":5,contrain:5,sourc:[5,1,2],"return":[5,2],costfunc:[5,2],variou:[4,5],ufldl:2,self2d:5,initi:5,bia:5,numofunit:5,zmnoatzigik:3,courseraml_costfunc:2,"class":[5,2],introduct:[3,0,2],layer1:5,layer0:5,like:5,level:5,edit:2,list:5,"public":4,achiev:4,vector:[5,2],either:[5,1],truth:[5,1],each:[5,2],output:[5,1,2],kaggl:4,where:2,page:0,mean:1,wiki:2,arr:5,set:4,repres:[5,1,2],weight:[5,2],inputarr2d:[5,2],neunet:5,some:1,intern:5,cost:[0,5,2],sec:2,flatten:2,arg:[5,2],http:[3,2],deriv:[5,2],index:[0,2],what:3,network:[0,5,2,4],per:[5,2],label:[5,1,2],current:2,forwardpropogateallinput:5,biasunitvalu:5,row:[5,2],method:5,refer:[0,1,2],machin:[0,2],linear:5,contain:[5,2],costfuncgrad:5,here:4,multilay:2,ipython:4,modifi:5,valu:5,search:0,last:[5,2],architectur:[3,0,5,1,4],befor:2,column:5,howev:5,instanc:[5,2],implement:[0,5,2,4],whole:5,produc:5,comment:1,updateforwardweight:5,ethem:2,ground:[5,1],appli:4,arrai:[5,2],number:[5,2],terminolog:[0,1],"2nd":2,comput:[5,2],total:2,perceptron:2,unit:[5,2],given:5,from:2,describ:2,numpi:5,transform:5,next:5,nnlayer:5,treat:5,which:5,forward:[5,2],type:5,store:5,"function":[0,5,2],regul:[5,2],tupl:5,ufldl_tutori:2,coursera:[0,2],forwardweightalllay:2,ani:5,train:[5,2],deep:[3,0,5,1,4],actual:[5,1],educ:3,than:5,must:2,serv:5,forwardpropogateoneinput:5,save:5,wise:5,target:[5,1,2],stanford:[0,2],provid:5,recvandactvbyoneinput:5,youtu:3,structur:[0,5],project:3,defin:1,activfunc:5,can:[4,5],learn:[3,0,1,2],"static":5,purpos:3,result:5,propag:5,layer:[5,1,2],constant:5,creat:3,viewer:4,certain:5,deeparch:0,sometim:1,digit:4,indic:0,lectur:2,maxit:5,itself:5,out:[3,4],want:5,tabl:0,need:5,courseraml_costfuncgrad:2,element:5,check:[3,4],cours:[0,2],multipl:[5,2],welcom:0,perform:5,self:5,when:5,detail:[0,5],accuraci:4,note:5,also:5,complex:[5,2],pure:3,test:4,instead:5,you:[4,5],document:[0,1],singl:[5,2],updat:5,competit:4,recogn:4,layers0:5,supervis:1,multipli:5,nest:5,ndarrai:5,capabl:3,notebook:4,php:2,data:[4,1],inputarr:5,demonstr:[0,4],"__new__":5,algorithm:2,neural:[0,5,2,4],backpropag:2,correspond:[5,2],alpaydin:2,exclud:5,rather:5,exampl:[5,2],combin:5,thi:[3,5,1,2],callabl:5,model:[5,2],order:2},objtypes:{"0":"py:module","1":"py:method","2":"py:class","3":"py:function","4":"py:staticmethod"},titles:["Welcome to deepArch&#8217;s documentation!","5. Terminologies","3. Cost Functions","1. Introduction","4. Demonstrations","2. Deep Architectures"],objnames:{"0":["py","module","Python module"],"1":["py","method","Python method"],"2":["py","class","Python class"],"3":["py","function","Python function"],"4":["py","staticmethod","Python static method"]},filenames:["index","jargon","costFunc","introduction","demo","deepArchitecture"]})
+Search.setIndex({objects:{"":{CostFunc:[2,0,1,""]},"FeedforwardNeuNet.FeedforwardNeuNet":{forwardPropogateOneInput:[5,1,1,""],train:[5,1,1,""],forwardPropogateAllInput:[5,1,1,""],"__new__":[5,4,1,""]},CostFunc:{courseraML_CostFuncGrad:[2,3,1,""],courseraML_CostFunc:[2,3,1,""]},"FeedforwardNeuNet.NnLayer":{updateForwardWeight:[5,1,1,""],recvAndActvByOneInput:[5,1,1,""],actvByAllInput:[5,1,1,""],"__new__":[5,4,1,""]},FeedforwardNeuNet:{sigmoid:[5,3,1,""],NnLayer:[5,2,1,""],FeedforwardNeuNet:[5,2,1,""]}},terms:{all:[5,2],numofunitnextlv:5,partial:[5,2],execut:5,sigmoid:5,over:[5,2],layersexoutputli:5,paramtooptim:5,scalar:2,code:1,through:4,edu:2,threshold:5,follow:2,actvbyallinput:5,newweight:5,paramet:[5,2],feedforward:[0,5,4],fit:5,weightdecayparam:[5,2],activ:5,except:5,param:5,matrix:5,add:5,input:[5,1,2],hidden:2,feedforwardneunet:[5,2],modul:[0,2],match:2,"case":5,contrain:5,sourc:[5,1,2],"return":[5,2],costfunc:[5,2],variou:[4,5],ufldl:2,self2d:5,initi:5,bia:5,numofunit:5,zmnoatzigik:3,courseraml_costfunc:2,"class":[5,2],introduct:[3,0,2],sparsiti:5,layer1:5,layer0:5,like:5,level:5,edit:2,list:5,"public":4,achiev:4,vector:[5,2],either:[5,1],truth:[5,1],each:[5,2],output:[5,1,2],kaggl:4,where:2,page:0,mean:1,wiki:2,arr:5,set:[4,5],repres:[5,1,2],weight:[5,2],inputarr2d:[5,2],neunet:5,some:1,intern:5,cost:[0,5,2],sec:2,flatten:2,arg:[5,2],http:[3,2],deriv:[5,2],index:[0,2],what:3,network:[0,5,2,4],per:[5,2],label:[5,1,2],current:2,forwardpropogateallinput:5,biasunitvalu:5,row:[5,2],method:5,refer:[0,1,2],machin:[0,2],linear:5,contain:[5,2],costfuncgrad:5,here:4,multilay:2,ipython:4,modifi:5,valu:5,search:0,last:[5,2],architectur:[3,0,5,1,4],befor:2,column:5,howev:5,instanc:[5,2],implement:[0,5,2,4],whole:5,produc:5,comment:1,updateforwardweight:5,ethem:2,ground:[5,1],appli:4,arrai:[5,2],number:[5,2],terminolog:[0,1],"2nd":2,comput:[5,2],total:2,perceptron:2,unit:[5,2],given:5,from:2,describ:2,numpi:5,transform:5,next:5,nnlayer:5,treat:5,which:5,forward:[5,2],type:5,store:5,"function":[0,5,2],regul:[5,2],tupl:5,ufldl_tutori:2,coursera:[0,2],ani:5,train:[5,2],deep:[3,0,5,1,4],actual:[5,1],educ:3,than:5,must:2,serv:5,forwardpropogateoneinput:5,save:5,wise:5,target:[5,1,2],stanford:[0,2],provid:5,recvandactvbyoneinput:5,youtu:3,structur:[0,5],project:3,defin:1,activfunc:5,can:[4,5],learn:[3,0,1,2],"static":5,purpos:3,result:5,propag:5,layer:[5,1,2],constant:5,creat:3,viewer:4,certain:5,deeparch:0,sometim:1,digit:4,indic:0,lectur:2,maxit:5,itself:5,out:[3,4],want:5,tabl:0,need:5,courseraml_costfuncgrad:2,element:5,check:[3,4],cours:[0,2],multipl:[5,2],welcom:0,weightalllay:2,rather:5,self:5,when:5,detail:[0,5],accuraci:4,note:5,also:5,complex:[5,2],pure:3,test:4,instead:5,you:[4,5],document:[0,1],singl:[5,2],updat:5,competit:4,recogn:4,layers0:5,supervis:1,multipli:5,nest:5,ndarrai:5,capabl:3,notebook:4,php:2,data:[4,1],inputarr:5,demonstr:[0,4],"__new__":5,algorithm:2,neural:[0,5,2,4],backpropag:2,correspond:[5,2],alpaydin:2,exclud:5,sparseparam:5,perform:5,exampl:[5,2],combin:5,thi:[3,5,1,2],callabl:5,model:[5,2],order:2},objtypes:{"0":"py:module","1":"py:method","2":"py:class","3":"py:function","4":"py:staticmethod"},titles:["Welcome to deepArch&#8217;s documentation!","5. Terminologies","3. Cost Functions","1. Introduction","4. Demonstrations","2. Deep Architectures"],objnames:{"0":["py","module","Python module"],"1":["py","method","Python method"],"2":["py","class","Python class"],"3":["py","function","Python function"],"4":["py","staticmethod","Python static method"]},filenames:["index","jargon","costFunc","introduction","demo","deepArchitecture"]})