Question

我试图用两个约束来实现随机梯度下降，因此不能使用scikit-learn。不幸的是，我已经在没有这两个限制的情况下与常规SGD挣扎。训练集上的损失（平方损失）在一些迭代中下降，但在一段时间后开始增加，如图中所示。这些是我使用的功能：

def loss_prime_simple(w,node,feature,data):
   x = data[3]
   y = data[2]
   x_f = x[node][feature]
   y_node = y[node]
   ret = (y_node - w[feature] * x_f) * (-x_f)
   return ret

def update_weights(w,data,predecs,children,node, learning_rate):
   len_features = len(data[3][0])
   w_new = np.zeros(len_features)
   for feature_ in range(len_features):
      w_new[feature_] = loss_prime_simple(w,node,feature_,data)
   return w - learning_rate * w_new

def loss_simple(w,data):
   y_p = data[2]
   x = data[3]
   return ((y_p - np.dot(w,np.array(x).T)) ** 2).sum()

这显示了训练集的损失，具有两种不同的学习率（0.001,0.0001）http://postimg.org/image/43nbmh8x5/

任何人都可以发现错误或有建议如何调试此问题？感谢

编辑：

正如lejlot指出的那样，拥有数据会很好。以下是我用于x（单个样本）的数据：http://textuploader.com/5x0f1

Y = 2

这会导致丢失：http://postimg.org/image/o9d97kt9v/

更新的代码：

def loss_prime_simple(w,node,feature,data):
   x = data[3]
   y = data[2]
   x_f = x[node][feature]
   y_node = y[node]
   return -(y_node - w[feature] * x_f) * x_f

def update_weights(w,data,predecs,children,node, learning_rate):
   len_features = len(data[3][0])
   w_new = np.zeros(len_features)
   for feature_ in range(len_features):
      w_new[feature_] = loss_prime_simple(w,node,feature_,data)
   return w - learning_rate * w_new

def loss_simple2(w,data):
   y_p = data[2]
   x = data[3]
   return ((y_p - np.dot(w,np.array(x).T)) ** 2).sum()

import numpy as np
X = [#put array from http://textuploader.com/5x0f1 here]
y = [2]

data = None, None, y, X

w = np.random.rand(4096)

a = [ loss_simple2(w, data) ]

for _ in range(200):
    for j in range(X.shape[0]):
        w = update_weights(w,data,None,None,j, 0.0001)
        a.append( loss_simple2(w, data) )

from matplotlib import pyplot as plt
plt.figure()
plt.plot(a)
plt.show()

Answer 1

您可以注意到的主要错误是reshape而非transpose，比较：

>>> import numpy as np
>>> X = np.array(range(10)).reshape(2,-1)
>>> X
array([[0, 1, 2, 3, 4],
       [5, 6, 7, 8, 9]])
>>> X.reshape(-1, 2)
array([[0, 1],
       [2, 3],
       [4, 5],
       [6, 7],
       [8, 9]])
>>> X.T
array([[0, 5],
       [1, 6],
       [2, 7],
       [3, 8],
       [4, 9]])
>>> X.reshape(-1, 2) == X.T
array([[ True, False],
       [False, False],
       [False, False],
       [False, False],
       [False,  True]], dtype=bool)

接下来看起来很糟糕的是调用sum（数组），你应该调用array.sum（）

>>> import numpy as np
>>> x = np.array(range(10)).reshape(2, 5)
>>> x
array([[0, 1, 2, 3, 4],
       [5, 6, 7, 8, 9]])
>>> sum(x)
array([ 5,  7,  9, 11, 13])
>>> x.sum()
45

在此之后，它运作得很好

def loss_prime_simple(w,node,feature,data):
   x = data[3]
   y = data[2]
   x_f = x[node][feature]
   y_node = y[node]
   ret = w[feature]
   return -(y_node - w[feature] * x_f) * x_f

def update_weights(w,data,predecs,children,node, learning_rate):
   len_features = len(data[3][0])
   w_new = np.zeros(len_features)
   for feature_ in range(len_features):
      w_new[feature_] = loss_prime_simple(w,node,feature_,data)
   return w - learning_rate * w_new

def loss_simple(w,data):
   y_p = data[2]
   x = data[3]
   return ((y_p - np.dot(w,np.array(x).T)) ** 2).sum()

import numpy as np

X = np.random.randn(1000, 3)
y = np.random.randn(1000)

data = None, None, y, X

w = np.array([1,3,3])

loss = [loss_simple(w, data)]

for _ in range(20):
    for j in range(X.shape[0]):
        w = update_weights(w, data, None, None, j, 0.001)
        loss.append(loss_simple(w, data))

from matplotlib import pyplot as plt
plt.figure()
plt.plot(loss)
plt.show()

Answer 2

问题是我使用代替

更新了权重

这样可行：

def update_weights(w,x,y, learning_rate):
    inner_product = 0.0    
    for f_ in range(len(x)):
        inner_product += (w[f_] * x[f_])
    dloss = inner_product - y
    for f_ in range(len(x)):
        w[f_] += (learning_rate * (-x[f_] * dloss))
    return w

新元 - 经过一些迭代后损失开始增加

2 个答案: