Question Simple Nerual network not swapping outputs during execution after being trained

[Keldecknight]

Greetings Everyone, I am working on learning a bit more about neural networks and came across this tutorial and attempted to recreate the code. I believe I have the correct sigmoid function, however, when sending my inputs of 0,1 and training the output to be 1,0 like the tutorial talks about, I also do another 30k loop of training in the reverse where the input is 1,0 and the output is 0,1.

The issue I am seeing is when I passed a test through the execute command and it seems no matter what order I send the input in the output always comes out to be 0.0001xxxxxxx and the output is always 0.9998xxxxxx where I would expect these to swap from my training. Anyone know what I may be doing incorrect by any chance?

Once the form load, the training completes and then I click button 1 and 2 to swap the inputs, however, I get this as the output no what which button I press

0.503787955138265
0.503787955138265
100000-16.6666666666667%
200000-33.3333333333333%
300000-50%
The thread 0x3044 has exited with code 0 (0x0).
400000-66.6666666666667%
500000-83.3333333333333%
600000-100%
100000-16.6666666666667%
200000-33.3333333333333%
300000-50%
400000-66.6666666666667%
500000-83.3333333333333%
600000-100%
0.000132203712251734
0.999867688588757
0.000132203712251734
0.999867688588757
0.000132203712251734
0.999867688588757
0.000132203712251734
0.999867688588757

Thank you for any feedback

https://social.technet.microsoft.com/wiki/contents/articles/32140.visual-basic-net-neural-networks-basics.aspx#Running_a_network

Public Class Form1
    Public Shared r As New Random()
    Dim network As NeuralNetwork

    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
        Dim layerList As New List(Of Integer)
        With layerList
            .Add(2)
            .Add(4)
            .Add(4)
            .Add(4)
            .Add(4)
            .Add(4)
            .Add(4)
            .Add(2)
        End With

        network = New NeuralNetwork(21.5, layerList)

        Dim inputs As New List(Of Double)
        inputs.Add(0)
        inputs.Add(1)

        Dim ots As List(Of Double) = network.Execute(inputs)
        Debug.WriteLine(ots(0))
        Debug.WriteLine(ots(1))

        Dim Trainedoutputs As New List(Of Double)
        Trainedoutputs.Add(1)
        Trainedoutputs.Add(0)

        Dim TotalCycles As Integer = 600000
        For i = 0 To TotalCycles
            If i <> 0 AndAlso i Mod 100000 = 0 Then
                Debug.WriteLine(i & "-" & CDbl(100 / (TotalCycles / i)) & "%")
            End If
            network.Train(inputs, Trainedoutputs)
        Next

        Dim inputs2 As New List(Of Double)
        inputs2.Add(1)
        inputs2.Add(0)

        Dim Trainedoutputs2 As New List(Of Double)
        Trainedoutputs2.Add(0)
        Trainedoutputs2.Add(1)

        For i = 0 To TotalCycles
            If i <> 0 AndAlso i Mod 100000 = 0 Then
                Debug.WriteLine(i & "-" & CDbl(100 / (TotalCycles / i)) & "%")
            End If
            network.Train(inputs2, Trainedoutputs2)
        Next

    End Sub

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        Dim inputs As New List(Of Double)
        inputs.Add(0)
        inputs.Add(1)
        Dim ots As List(Of Double) = network.Execute(inputs)
        Debug.WriteLine(ots(0))
        Debug.WriteLine(ots(1))
    End Sub

    Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
        Dim inputs As New List(Of Double)
        inputs.Add(1)
        inputs.Add(0)
        Dim ots As List(Of Double) = network.Execute(inputs)
        Debug.WriteLine(ots(0))
        Debug.WriteLine(ots(1))
    End Sub
End Class

Public Class Dendrite
    Dim r As New Random
    Dim _weight As Double

    Property Weight As Double
        Get
            Return _weight
        End Get
        Set(value As Double)
            _weight = value
        End Set
    End Property

    Public Sub New()
        Me.Weight = r.NextDouble()
    End Sub
End Class


Public Class Neuron
    Dim r As New Random
    Dim _dendrites As New List(Of Dendrite)
    Dim _dendriteCount As Integer
    Dim _bias As Double
    Dim _value As Double
    Dim _delta As Double

    Public Property Dendrites As List(Of Dendrite)
        Get
            Return _dendrites
        End Get
        Set(value As List(Of Dendrite))
            _dendrites = value
        End Set
    End Property

    Public Property Bias As Double
        Get
            Return _bias
        End Get
        Set(value As Double)
            _bias = value
        End Set
    End Property

    Public Property Value As Double
        Get
            Return _value
        End Get
        Set(value As Double)
            _value = value
        End Set
    End Property

    Public Property Delta As Double
        Get
            Return _delta
        End Get
        Set(value As Double)
            _delta = value
        End Set
    End Property

    Public ReadOnly Property DendriteCount As Integer
        Get
            Return _dendrites.Count
        End Get
    End Property

    Public Sub New()
        Me.Bias = r.NextDouble()
    End Sub
End Class

Public Class Layer
    Dim _neurons As New List(Of Neuron)
    Dim _neuronCount As Integer

    Public Property Neurons As List(Of Neuron)
        Get
            Return _neurons
        End Get
        Set(value As List(Of Neuron))
            _neurons = value
        End Set
    End Property

    Public ReadOnly Property NeuronCount As Integer
        Get
            Return _neurons.Count
        End Get
    End Property

    Public Sub New(neuronNum As Integer)
        _neuronCount = neuronNum
    End Sub
End Class

Public Class NeuralNetwork
    Dim _layers As New List(Of Layer)
    Dim _learningRate As Double

    Public Property Layers As List(Of Layer)
        Get
            Return _layers
        End Get
        Set(value As List(Of Layer))
            _layers = value
        End Set
    End Property

    Public Property LearningRate As Double
        Get
            Return _learningRate
        End Get
        Set(value As Double)
            _learningRate = value
        End Set
    End Property

    Public ReadOnly Property LayerCount As Integer
        Get
            Return _layers.Count
        End Get
    End Property

    Sub New(LearningRate As Double, nLayers As List(Of Integer))
        If nLayers.Count < 2 Then Exit Sub

        Me.LearningRate = LearningRate

        For ii As Integer = 0 To nLayers.Count - 1

            Dim l As Layer = New Layer(nLayers(ii) - 1)
            Me.Layers.Add(l)

            For jj As Integer = 0 To nLayers(ii) - 1
                l.Neurons.Add(New Neuron())
            Next

            For Each n As Neuron In l.Neurons
                If ii = 0 Then n.Bias = 0

                If ii > 0 Then
                    For k As Integer = 0 To nLayers(ii - 1) - 1
                        n.Dendrites.Add(New Dendrite)
                    Next
                End If

            Next

        Next
    End Sub
    Function sigmoid(x As Double) As Double
        Return 1 / (1 + Math.Exp(-x))
    End Function

    Function Execute(inputs As List(Of Double)) As List(Of Double)
        If inputs.Count <> Me.Layers(0).NeuronCount Then
            Return Nothing
        End If

        For ii As Integer = 0 To Me.LayerCount - 1
            Dim curLayer As Layer = Me.Layers(ii)

            For jj As Integer = 0 To curLayer.NeuronCount - 1
                Dim curNeuron As Neuron = curLayer.Neurons(jj)

                If ii = 0 Then
                    curNeuron.Value = inputs(jj)
                Else
                    curNeuron.Value = 0
                    For k = 0 To Me.Layers(ii - 1).NeuronCount - 1
                        curNeuron.Value = curNeuron.Value + Me.Layers(ii - 1).Neurons(k).Value * curNeuron.Dendrites(k).Weight
                    Next k

                    curNeuron.Value = Sigmoid(curNeuron.Value + curNeuron.Bias)
                End If

            Next
        Next

        Dim outputs As New List(Of Double)
        Dim la As Layer = Me.Layers(Me.LayerCount - 1)
        For ii As Integer = 0 To la.NeuronCount - 1
            outputs.Add(la.Neurons(ii).Value)
        Next

        Return outputs
    End Function

    Public Function Train(inputs As List(Of Double), outputs As List(Of Double)) As Boolean
        If inputs.Count <> Me.Layers(0).NeuronCount Or outputs.Count <> Me.Layers(Me.LayerCount - 1).NeuronCount Then
            Debug.WriteLine("output.count <> Layers " & Me.Layers(Me.LayerCount - 1).NeuronCount)
            Return False
        End If

        Execute(inputs)

        For ii = 0 To Me.Layers(Me.LayerCount - 1).NeuronCount - 1
            Dim curNeuron As Neuron = Me.Layers(Me.LayerCount - 1).Neurons(ii)

            curNeuron.Delta = curNeuron.Value * (1 - curNeuron.Value) * (outputs(ii) - curNeuron.Value)

            For jj = Me.LayerCount - 2 To 1 Step -1
                For kk = 0 To Me.Layers(jj).NeuronCount - 1
                    Dim iNeuron As Neuron = Me.Layers(jj).Neurons(kk)
                    iNeuron.Delta = iNeuron.Value * (1 - iNeuron.Value) * Me.Layers(jj + 1).Neurons(ii).Dendrites(kk).Weight * Me.Layers(jj + 1).Neurons(ii).Delta
                Next kk
            Next jj
        Next ii


        For ii = Me.LayerCount - 1 To 0 Step -1
            For jj = 0 To Me.Layers(ii).NeuronCount - 1
                Dim iNeuron As Neuron = Me.Layers(ii).Neurons(jj)
                iNeuron.Bias = iNeuron.Bias + (Me.LearningRate * iNeuron.Delta)

                For kk = 0 To iNeuron.DendriteCount - 1
                    iNeuron.Dendrites(kk).Weight = iNeuron.Dendrites(kk).Weight + (Me.LearningRate * Me.Layers(ii - 1).Neurons(kk).Value * iNeuron.Delta)
                Next kk
            Next jj
        Next ii

        Return True
    End Function

End Class

 

[jmcilhinney]

Please don't post duplicate threads. New members need to have their posts approved by a moderator, which you would have been told when you posted. That's why your post didn't show initially. I have deleted your original thread, as the second one seemed slightly more informative.

Also, please don't post unformatted code snippets. They are too hard to read. The post editor works just like every text editor you've ever used, with a toolbar at the top for formatting. Format code snippets as code for easy reading. The easier you make it for us to understand your post, the more chance you have of getting the help you want.

 

[jmcilhinney]

As for the issue, have you actually debugged your code properly, i.e. by setting breakpoints and stepping through the code and examining the state at each step? If not, you need to do that now. If you don't know how, stop what you're doing and learn, because debugging is a critical part of software development. If you have debugged then you should be able to explain exactly where and how the behaviour of the code - not just the application from the user's perspective but the actual code - differs from your expectations. Once you understand that, you may well be able to fix the issue on your own. If you can't, at least you can provide us with all the relevant information. If your code behaves exactly as expected but produces the wrong result then it's probably your expectations that are wrong, so you need to reexamine them. You haven't actually explained the logic that your code is supposed to implement, so we can't really tell you whether it actually is implementing it. This is why you need to break the whole thing down into smaller parts. As it is, there's really too much code there to expect us to go over it all, try to work out what it's supposed to be doing, whether that makes sense and whether it's actually doing it.