|
2018 區(qū)塊鏈技術(shù)及應(yīng)用峰會(huì)(BTA)·中國 倒計(jì)時(shí) 1 天
2018，想要follow最火的區(qū)塊鏈技術(shù)？你還差一場嚴(yán)謹(jǐn)純粹的技術(shù)交流會(huì)——2018區(qū)塊鏈技術(shù)及應(yīng)用峰會(huì)(BTA)·中國將于2018年3月30-31日登陸北京喜來登長城飯店。追求專業(yè)性？你要的這里全都有：當(dāng)超強(qiáng)嘉賓陣容遇上業(yè)界同好的腦洞大聯(lián)歡，1 1=無限可能，目前門票預(yù)購火熱進(jìn)行中。
活動(dòng)詳情： http:///7FI1Ch
編譯 | 林椿眄 出品 | 人工智能頭條（公眾號ID：AI_Thinker）
【AI科技大本營導(dǎo)讀】Python 被稱為是最接近 AI 的語言。最近一位名叫Anna-Lena Popkes的小姐姐在GitHub上分享了自己如何使用Python（3.6及以上版本）實(shí)現(xiàn)7種機(jī)器學(xué)習(xí)算法的筆記，并附有完整代碼。所有這些算法的實(shí)現(xiàn)都沒有使用其他機(jī)器學(xué)習(xí)庫。這份筆記可以幫大家對算法以及其底層結(jié)構(gòu)有個(gè)基本的了解，但并不是提供最有效的實(shí)現(xiàn)。
小姐姐她是德國波恩大學(xué)計(jì)算機(jī)科學(xué)專業(yè)的研究生，主要關(guān)注機(jī)器學(xué)習(xí)和神經(jīng)網(wǎng)絡(luò)。
七種算法包括：
▌1. 線性回歸算法
在線性回歸中，我們想要建立一個(gè)模型，來擬合一個(gè)因變量 y 與一個(gè)或多個(gè)獨(dú)立自變量(預(yù)測變量) x 之間的關(guān)系。
給定：
線性回歸模型可以理解為一個(gè)非常簡單的神經(jīng)網(wǎng)絡(luò)：
線性回歸模型可以使用以下方法進(jìn)行訓(xùn)練
a) 梯度下降法
b) 正態(tài)方程(封閉形式解)：
其中 X 是一個(gè)矩陣，其形式為 ，包含所有訓(xùn)練樣本的維度信息。
而正態(tài)方程需要計(jì)算 的轉(zhuǎn)置。這個(gè)操作的計(jì)算復(fù)雜度介于 ）和 之間，而這取決于所選擇的實(shí)現(xiàn)方法。因此，如果訓(xùn)練集中數(shù)據(jù)的特征數(shù)量很大，那么使用正態(tài)方程訓(xùn)練的過程將變得非常緩慢。 線性回歸模型的訓(xùn)練過程有不同的步驟。首先(在步驟 0 中)，模型的參數(shù)將被初始化。在達(dá)到指定訓(xùn)練次數(shù)或參數(shù)收斂前，重復(fù)以下其他步驟。
第 0 步：
用0 (或小的隨機(jī)值)來初始化權(quán)重向量和偏置量，或者直接使用正態(tài)方程計(jì)算模型參數(shù)
第 1 步(只有在使用梯度下降法訓(xùn)練時(shí)需要)：
計(jì)算輸入的特征與權(quán)重值的線性組合，這可以通過矢量化和矢量傳播來對所有訓(xùn)練樣本進(jìn)行處理：
其中 X 是所有訓(xùn)練樣本的維度矩陣，其形式為；· 表示點(diǎn)積。
第 2 步(只有在使用梯度下降法訓(xùn)練時(shí)需要)：
用均方誤差計(jì)算訓(xùn)練集上的損失：
第 3 步(只有在使用梯度下降法訓(xùn)練時(shí)需要):
對每個(gè)參數(shù)，計(jì)算其對損失函數(shù)的偏導(dǎo)數(shù)：
所有偏導(dǎo)數(shù)的梯度計(jì)算如下：
第 4 步(只有在使用梯度下降法訓(xùn)練時(shí)需要）:
更新權(quán)重向量和偏置量：
其中，表示學(xué)習(xí)率。
In [4]:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
np.random.seed(123)
數(shù)據(jù)集
In [5]:
# We will use a simple training set
X = 2 * np.random.rand(500, 1)
y = 5 3 * X np.random.randn(500, 1)
fig = plt.figure(figsize=(8,6))
plt.scatter(X, y)
plt.title('Dataset')
plt.xlabel('First feature')
plt.ylabel('Second feature')
plt.show()
In [6]:
# Split the data into a training and test set
X_train, X_test, y_train, y_test = train_test_split(X, y)
print(f'Shape X_train: {X_train.shape}')
print(f'Shape y_train: {y_train.shape}')
print(f'Shape X_test: {X_test.shape}')
print(f'Shape y_test: {y_test.shape}')
Shape X_train: (375, 1)
Shape y_train: (375, 1)
Shape X_test: (125, 1)
Shape y_test: (125, 1)
線性回歸分類
In [23]:
class LinearRegression:
def __init__(self):
pass
def train_gradient_descent(self, X, y, learning_rate=0.01, n_iters=100):
'''
Trains a linear regression model using gradient descent
'''
# Step 0: Initialize the parameters
n_samples, n_features = X.shape
self.weights = np.zeros(shape=(n_features,1))
self.bias = 0
costs = []
for i in range(n_iters):
# Step 1: Compute a linear combination of the input features and weights
y_predict = np.dot(X, self.weights) self.bias
# Step 2: Compute cost over training set
cost = (1 / n_samples) * np.sum((y_predict - y)**2)
costs.append(cost)
if i % 100 == 0:
print(f'Cost at iteration {i}: {cost}')
# Step 3: Compute the gradients
dJ_dw = (2 / n_samples) * np.dot(X.T, (y_predict - y))
dJ_db = (2 / n_samples) * np.sum((y_predict - y))
# Step 4: Update the parameters
self.weights = self.weights - learning_rate * dJ_dw
self.bias = self.bias - learning_rate * dJ_db
return self.weights, self.bias, costs
def train_normal_equation(self, X, y):
'''
Trains a linear regression model using the normal equation
'''
self.weights = np.dot(np.dot(np.linalg.inv(np.dot(X.T, X)), X.T), y)
self.bias = 0
return self.weights, self.bias
def predict(self, X):
return np.dot(X, self.weights) self.bias
使用梯度下降進(jìn)行訓(xùn)練
In [24]:
regressor = LinearRegression()
w_trained, b_trained, costs = regressor.train_gradient_descent(X_train, y_train, learning_rate=0.005, n_iters=600)
fig = plt.figure(figsize=(8,6))
plt.plot(np.arange(n_iters), costs)
plt.title('Development of cost during training')
plt.xlabel('Number of iterations')
plt.ylabel('Cost')
plt.show()
Cost at iteration 0: 66.45256981003433
Cost at iteration 100: 2.2084346146095934
Cost at iteration 200: 1.2797812854182806
Cost at iteration 300: 1.2042189195356685
Cost at iteration 400: 1.1564867816573
Cost at iteration 500: 1.121391041394467
測試（梯度下降模型）
In [28]:
n_samples, _ = X_train.shape
n_samples_test, _ = X_test.shape
y_p_train = regressor.predict(X_train)
y_p_test = regressor.predict(X_test)
error_train = (1 / n_samples) * np.sum((y_p_train - y_train) ** 2)
error_test = (1 / n_samples_test) * np.sum((y_p_test - y_test) ** 2)
print(f'Error on training set: {np.round(error_train, 4)}')
print(f'Error on test set: {np.round(error_test)}')
Error on training set: 1.0955 Error on test set: 1.0
使用正規(guī)方程（normal equation）訓(xùn)練
# To compute the parameters using the normal equation, we add a bias value of 1 to each input example
X_b_train = np.c_[np.ones((n_samples)), X_train]
X_b_test = np.c_[np.ones((n_samples_test)), X_test]
reg_normal = LinearRegression()
w_trained = reg_normal.train_normal_equation(X_b_train, y_train)
測試（正規(guī)方程模型）
y_p_train = reg_normal.predict(X_b_train)
y_p_test = reg_normal.predict(X_b_test)
error_train = (1 / n_samples) * np.sum((y_p_train - y_train) ** 2)
error_test = (1 / n_samples_test) * np.sum((y_p_test - y_test) ** 2)
print(f'Error on training set: {np.round(error_train, 4)}')
print(f'Error on test set: {np.round(error_test, 4)}')
Error on training set: 1.0228 Error on test set: 1.0432
可視化測試預(yù)測
# Plot the test predictions
fig = plt.figure(figsize=(8,6))
plt.scatter(X_train, y_train)
plt.scatter(X_test, y_p_test)
plt.xlabel('First feature')
plt.ylabel('Second feature')
plt.show()
▌2. Logistic 回歸算法
在 Logistic 回歸中，我們試圖對給定輸入特征的線性組合進(jìn)行建模，來得到其二元變量的輸出結(jié)果。例如，我們可以嘗試使用競選候選人花費(fèi)的金錢和時(shí)間信息來預(yù)測選舉的結(jié)果(勝或負(fù))。Logistic 回歸算法的工作原理如下。
給定：
數(shù)據(jù)集 是d-維向量 是一個(gè)二元的目標(biāo)變量
Logistic 回歸模型可以理解為一個(gè)非常簡單的神經(jīng)網(wǎng)絡(luò)：
與線性回歸不同，Logistic 回歸沒有封閉解。但由于損失函數(shù)是凸函數(shù)，因此我們可以使用梯度下降法來訓(xùn)練模型。事實(shí)上，在保證學(xué)習(xí)速率足夠小且使用足夠的訓(xùn)練迭代步數(shù)的前提下，梯度下降法(或任何其他優(yōu)化算法)可以是能夠找到全局最小值。
訓(xùn)練 Logistic 回歸模型有不同的步驟。首先(在步驟 0 中)，模型的參數(shù)將被初始化。在達(dá)到指定訓(xùn)練次數(shù)或參數(shù)收斂前，重復(fù)以下其他步驟。
第 0 步：用 0 (或小的隨機(jī)值)來初始化權(quán)重向量和偏置值
第 1 步：計(jì)算輸入的特征與權(quán)重值的線性組合，這可以通過矢量化和矢量傳播來對所有訓(xùn)練樣本進(jìn)行處理：
其中 X 是所有訓(xùn)練樣本的維度矩陣，其形式為；·表示點(diǎn)積。
第 2 步：用 sigmoid 函數(shù)作為激活函數(shù)，其返回值介于0到1之間：
第 3 步：計(jì)算整個(gè)訓(xùn)練集的損失值。
我們希望模型得到的目標(biāo)值概率落在 0 到 1 之間。因此在訓(xùn)練期間，我們希望調(diào)整參數(shù)，使得模型較大的輸出值對應(yīng)正標(biāo)簽(真實(shí)標(biāo)簽為 1)，較小的輸出值對應(yīng)負(fù)標(biāo)簽(真實(shí)標(biāo)簽為 0 )。這在損失函數(shù)中表現(xiàn)為如下形式：
第 4 步：對權(quán)重向量和偏置量，計(jì)算其對損失函數(shù)的梯度。
關(guān)于這個(gè)導(dǎo)數(shù)實(shí)現(xiàn)的詳細(xì)解釋，可以參見這里（https://stats./questions/278771/how-is-the-cost-function-from-logistic-regression-derivated）。
一般形式如下：
對于偏置量的導(dǎo)數(shù)計(jì)算，此時(shí)為 1。
第 5 步：更新權(quán)重和偏置值。
其中，表示學(xué)習(xí)率。
In [24]:
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt
np.random.seed(123)
% matplotlib inline
數(shù)據(jù)集
In [25]:
# We will perform logistic regression using a simple toy dataset of two classes
X, y_true = make_blobs(n_samples= 1000, centers=2)
fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y_true)
plt.title('Dataset')
plt.xlabel('First feature')
plt.ylabel('Second feature')
plt.show()
In [26]:
# Reshape targets to get column vector with shape (n_samples, 1)
y_true = y_true[:, np.newaxis]
# Split the data into a training and test set
X_train, X_test, y_train, y_test = train_test_split(X, y_true)
print(f'Shape X_train: {X_train.shape}')
print(f'Shape y_train: {y_train.shape}')
print(f'Shape X_test: {X_test.shape}')
print(f'Shape y_test: {y_test.shape}')
Shape X_train: (750, 2) Shape y_train: (750, 1) Shape X_test: (250, 2) Shape y_test: (250, 1)
Logistic回歸分類
In [27]:
class LogisticRegression:
def __init__(self):
pass
def sigmoid(self, a):
return 1 / (1 np.exp(-a))
def train(self, X, y_true, n_iters, learning_rate):
'''
Trains the logistic regression model on given data X and targets y
'''
# Step 0: Initialize the parameters
n_samples, n_features = X.shape
self.weights = np.zeros((n_features, 1))
self.bias = 0
costs = []
for i in range(n_iters):
# Step 1 and 2: Compute a linear combination of the input features and weights,
# apply the sigmoid activation function
y_predict = self.sigmoid(np.dot(X, self.weights) self.bias)
# Step 3: Compute the cost over the whole training set.
cost = (- 1 / n_samples) * np.sum(y_true * np.log(y_predict) (1 - y_true) * (np.log(1 - y_predict)))
# Step 4: Compute the gradients
dw = (1 / n_samples) * np.dot(X.T, (y_predict - y_true))
db = (1 / n_samples) * np.sum(y_predict - y_true)
# Step 5: Update the parameters
self.weights = self.weights - learning_rate * dw
self.bias = self.bias - learning_rate * db
costs.append(cost)
if i % 100 == 0:
print(f'Cost after iteration {i}: {cost}')
return self.weights, self.bias, costs
def predict(self, X):
'''
Predicts binary labels for a set of examples X.
'''
y_predict = self.sigmoid(np.dot(X, self.weights) self.bias)
y_predict_labels = [1 if elem > 0.5 else 0 for elem in y_predict]
return np.array(y_predict_labels)[:, np.newaxis]
初始化并訓(xùn)練模型
In [29]:
regressor = LogisticRegression()
w_trained, b_trained, costs = regressor.train(X_train, y_train, n_iters=600, learning_rate=0.009)
fig = plt.figure(figsize=(8,6))
plt.plot(np.arange(600), costs)
plt.title('Development of cost over training')
plt.xlabel('Number of iterations')
plt.ylabel('Cost')
plt.show()
Cost after iteration 0: 0.6931471805599453 Cost after iteration 100: 0.046514002935609956 Cost after iteration 200: 0.02405337743999163 Cost after iteration 300: 0.016354408151412207 Cost after iteration 400: 0.012445770521974634 Cost after iteration 500: 0.010073981792906512
測試模型
In [31]:
y_p_train = regressor.predict(X_train)
y_p_test = regressor.predict(X_test)
print(f'train accuracy: {100 - np.mean(np.abs(y_p_train - y_train)) * 100}%')
print(f'test accuracy: {100 - np.mean(np.abs(y_p_test - y_test))}%')
train accuracy: 100.0% test accuracy: 100.0%
▌3. 感知器算法
感知器是一種簡單的監(jiān)督式的機(jī)器學(xué)習(xí)算法，也是最早的神經(jīng)網(wǎng)絡(luò)體系結(jié)構(gòu)之一。它由 Rosenblatt 在 20 世紀(jì) 50 年代末提出。感知器是一種二元的線性分類器，其使用 d- 維超平面來將一組訓(xùn)練樣本( d- 維輸入向量)映射成二進(jìn)制輸出值。它的原理如下：
給定：
感知器可以理解為一個(gè)非常簡單的神經(jīng)網(wǎng)絡(luò)：
感知器的訓(xùn)練可以使用梯度下降法，訓(xùn)練算法有不同的步驟。首先(在步驟0中)，模型的參數(shù)將被初始化。在達(dá)到指定訓(xùn)練次數(shù)或參數(shù)收斂前，重復(fù)以下其他步驟。
第 0 步：用 0 (或小的隨機(jī)值)來初始化權(quán)重向量和偏置值
第 1 步：計(jì)算輸入的特征與權(quán)重值的線性組合，這可以通過矢量化和矢量傳播法則來對所有訓(xùn)練樣本進(jìn)行處理：
其中 X 是所有訓(xùn)練示例的維度矩陣，其形式為；·表示點(diǎn)積。
第 2 步：用 Heaviside step 函數(shù)作為激活函數(shù)，其返回一個(gè)二進(jìn)制值：
第 3 步：使用感知器的學(xué)習(xí)規(guī)則來計(jì)算權(quán)重向量和偏置量的更新值。
其中，表示學(xué)習(xí)率。
第 4 步：更新權(quán)重向量和偏置量。
In [1]:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split
np.random.seed(123)
% matplotlib inline
數(shù)據(jù)集
In [2]:
X, y = make_blobs(n_samples=1000, centers=2)
fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y)
plt.title('Dataset')
plt.xlabel('First feature')
plt.ylabel('Second feature')
plt.show()
In [3]:
y_true = y[:, np.newaxis]
X_train, X_test, y_train, y_test = train_test_split(X, y_true)
print(f'Shape X_train: {X_train.shape}')
print(f'Shape y_train: {y_train.shape})')
print(f'Shape X_test: {X_test.shape}')
print(f'Shape y_test: {y_test.shape}')
Shape X_train: (750, 2) Shape y_train: (750, 1)) Shape X_test: (250, 2) Shape y_test: (250, 1)
感知器分類
In [6]:
class Perceptron():
def __init__(self):
pass
def train(self, X, y, learning_rate=0.05, n_iters=100):
n_samples, n_features = X.shape
# Step 0: Initialize the parameters
self.weights = np.zeros((n_features,1))
self.bias = 0
for i in range(n_iters):
# Step 1: Compute the activation
a = np.dot(X, self.weights) self.bias
# Step 2: Compute the output
y_predict = self.step_function(a)
# Step 3: Compute weight updates
delta_w = learning_rate * np.dot(X.T, (y - y_predict))
delta_b = learning_rate * np.sum(y - y_predict)
# Step 4: Update the parameters
self.weights = delta_w
self.bias = delta_b
return self.weights, self.bias
def step_function(self, x):
return np.array([1 if elem >= 0 else 0 for elem in x])[:, np.newaxis]
def predict(self, X):
a = np.dot(X, self.weights) self.bias
return self.step_function(a)
初始化并訓(xùn)練模型
In [7]:
p = Perceptron()
w_trained, b_trained = p.train(X_train, y_train,learning_rate=0.05, n_iters=500)
測試
In [10]:
y_p_train = p.predict(X_train)
y_p_test = p.predict(X_test)
print(f'training accuracy: {100 - np.mean(np.abs(y_p_train - y_train)) * 100}%')
print(f'test accuracy: {100 - np.mean(np.abs(y_p_test - y_test)) * 100}%')
training accuracy: 100.0% test accuracy: 100.0%
可視化決策邊界
In [13]:
def plot_hyperplane(X, y, weights, bias):
'''
Plots the dataset and the estimated decision hyperplane
'''
slope = - weights[0]/weights[1]
intercept = - bias/weights[1]
x_hyperplane = np.linspace(-10,10,10)
y_hyperplane = slope * x_hyperplane intercept
fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y)
plt.plot(x_hyperplane, y_hyperplane, '-')
plt.title('Dataset and fitted decision hyperplane')
plt.xlabel('First feature')
plt.ylabel('Second feature')
plt.show()
In [14]:
plot_hyperplane(X, y, w_trained, b_trained)
▌4. K 最近鄰算法
k-nn 算法是一種簡單的監(jiān)督式的機(jī)器學(xué)習(xí)算法，可以用于解決分類和回歸問題。這是一個(gè)基于實(shí)例的算法，并不是估算模型，而是將所有訓(xùn)練樣本存儲在內(nèi)存中，并使用相似性度量進(jìn)行預(yù)測。
給定一個(gè)輸入示例，k-nn 算法將從內(nèi)存中檢索 k 個(gè)最相似的實(shí)例。相似性是根據(jù)距離來定義的，也就是說，與輸入示例之間距離最小(歐幾里得距離)的訓(xùn)練樣本被認(rèn)為是最相似的樣本。
輸入示例的目標(biāo)值計(jì)算如下：
分類問題：
a) 不加權(quán)：輸出 k 個(gè)最近鄰中最常見的分類 b) 加權(quán)：將每個(gè)分類值的k個(gè)最近鄰的權(quán)重相加，輸出權(quán)重最高的分類
回歸問題：
a) 不加權(quán)：輸出k個(gè)最近鄰值的平均值 b) 加權(quán)：對于所有分類值，將分類值加權(quán)求和并將結(jié)果除以所有權(quán)重的總和
加權(quán)版本的 k-nn 算法是改進(jìn)版本，其中每個(gè)近鄰的貢獻(xiàn)值根據(jù)其與查詢點(diǎn)之間的距離進(jìn)行加權(quán)。下面，我們在 sklearn 用 k-nn 算法的原始版本實(shí)現(xiàn)數(shù)字?jǐn)?shù)據(jù)集的分類。
In [1]:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
np.random.seed(123)
% matplotlib inline
數(shù)據(jù)集
In [2]:
# We will use the digits dataset as an example. It consists of the 1797 images of hand-written digits. Each digit is
# represented by a 64-dimensional vector of pixel values.
digits = load_digits()
X, y = digits.data, digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y)
print(f'X_train shape: {X_train.shape}')
print(f'y_train shape: {y_train.shape}')
print(f'X_test shape: {X_test.shape}')
print(f'y_test shape: {y_test.shape}')
# Example digits
fig = plt.figure(figsize=(10,8))
for i in range(10):
ax = fig.add_subplot(2, 5, i 1)
plt.imshow(X[i].reshape((8,8)), cmap='gray')
X_train shape: (1347, 64) y_train shape: (1347,) X_test shape: (450, 64) y_test shape: (450,)
K 最鄰近類別
In [3]:
class kNN():
def __init__(self):
pass
def fit(self, X, y):
self.data = X
self.targets = y
def euclidean_distance(self, X):
'''
Computes the euclidean distance between the training data and
a new input example or matrix of input examples X
'''
# input: single data point
if X.ndim == 1:
l2 = np.sqrt(np.sum((self.data - X)**2, axis=1))
# input: matrix of data points
if X.ndim == 2:
n_samples, _ = X.shape
l2 = [np.sqrt(np.sum((self.data - X[i])**2, axis=1)) for i in range(n_samples)]
return np.array(l2)
def predict(self, X, k=1):
'''
Predicts the classification for an input example or matrix of input examples X
'''
# step 1: compute distance between input and training data
dists = self.euclidean_distance(X)
# step 2: find the k nearest neighbors and their classifications
if X.ndim == 1:
if k == 1:
nn = np.argmin(dists)
return self.targets[nn]
else:
knn = np.argsort(dists)[:k]
y_knn = self.targets[knn]
max_vote = max(y_knn, key=list(y_knn).count)
return max_vote
if X.ndim == 2:
knn = np.argsort(dists)[:, :k]
y_knn = self.targets[knn]
if k == 1:
return y_knn.T
else:
n_samples, _ = X.shape
max_votes = [max(y_knn[i], key=list(y_knn[i]).count) for i in range(n_samples)]
return max_votes
初始化并訓(xùn)練模型
In [11]:
knn = kNN()
knn.fit(X_train, y_train)
print('Testing one datapoint, k=1')
print(f'Predicted label: {knn.predict(X_test[0], k=1)}')
print(f'True label: {y_test[0]}')
print()
print('Testing one datapoint, k=5')
print(f'Predicted label: {knn.predict(X_test[20], k=5)}')
print(f'True label: {y_test[20]}')
print()
print('Testing 10 datapoint, k=1')
print(f'Predicted labels: {knn.predict(X_test[5:15], k=1)}')
print(f'True labels: {y_test[5:15]}')
print()
print('Testing 10 datapoint, k=4')
print(f'Predicted labels: {knn.predict(X_test[5:15], k=4)}')
print(f'True labels: {y_test[5:15]}')
print()
Testing one datapoint, k=1 Predicted label: 3 True label: 3
Testing one datapoint, k=5 Predicted label: 9 True label: 9
Testing 10 datapoint, k=1 Predicted labels: [[3 1 0 7 4 0 0 5 1 6]] True labels: [3 1 0 7 4 0 0 5 1 6]
Testing 10 datapoint, k=4 Predicted labels: [3, 1, 0, 7, 4, 0, 0, 5, 1, 6] True labels: [3 1 0 7 4 0 0 5 1 6]
測試集精度
In [12]:
# Compute accuracy on test set
y_p_test1 = knn.predict(X_test, k=1)
test_acc1= np.sum(y_p_test1[0] == y_test)/len(y_p_test1[0]) * 100
print(f'Test accuracy with k = 1: {format(test_acc1)}')
y_p_test8 = knn.predict(X_test, k=5)
test_acc8= np.sum(y_p_test8 == y_test)/len(y_p_test8) * 100
print(f'Test accuracy with k = 8: {format(test_acc8)}')
Test accuracy with k = 1: 97.77777777777777 Test accuracy with k = 8: 97.55555555555556
▌5. K均值聚類算法
K-Means 是一種非常簡單的聚類算法(聚類算法都屬于無監(jiān)督學(xué)習(xí))。給定固定數(shù)量的聚類和輸入數(shù)據(jù)集，該算法試圖將數(shù)據(jù)劃分為聚類，使得聚類內(nèi)部具有較高的相似性，聚類與聚類之間具有較低的相似性。
算法原理
1. 初始化聚類中心，或者在輸入數(shù)據(jù)范圍內(nèi)隨機(jī)選擇，或者使用一些現(xiàn)有的訓(xùn)練樣本(推薦) 2. 直到收斂
目標(biāo)函數(shù)
聚類算法的目標(biāo)函數(shù)試圖找到聚類中心，以便數(shù)據(jù)將劃分到相應(yīng)的聚類中，并使得數(shù)據(jù)與其最接近的聚類中心之間的距離盡可能小。
給定一組數(shù)據(jù)X1，...，Xn和一個(gè)正數(shù)k，找到k個(gè)聚類中心C1，...，Ck并最小化目標(biāo)函數(shù)：
這里：
K-Means 算法的缺點(diǎn)：
聚類的個(gè)數(shù)在開始就要設(shè)定 聚類的結(jié)果取決于初始設(shè)定的聚類中心 對異常值很敏感 不適合用于發(fā)現(xiàn)非凸聚類問題 該算法不能保證能夠找到全局最優(yōu)解，因此它往往會(huì)陷入一個(gè)局部最優(yōu)解
In [21]:
import numpy as np
import matplotlib.pyplot as plt
import random
from sklearn.datasets import make_blobs
np.random.seed(123)
% matplotlib inline
數(shù)據(jù)集
In [22]:
X, y = make_blobs(centers=4, n_samples=1000)
print(f'Shape of dataset: {X.shape}')
fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y)
plt.title('Dataset with 4 clusters')
plt.xlabel('First feature')
plt.ylabel('Second feature')
plt.show()
Shape of dataset: (1000, 2)
K均值分類
In [23]:
class KMeans():
def __init__(self, n_clusters=4):
self.k = n_clusters
def fit(self, data):
'''
Fits the k-means model to the given dataset
'''
n_samples, _ = data.shape
# initialize cluster centers
self.centers = np.array(random.sample(list(data), self.k))
self.initial_centers = np.copy(self.centers)
# We will keep track of whether the assignment of data points
# to the clusters has changed. If it stops changing, we are
# done fitting the model
old_assigns = None
n_iters = 0
while True:
new_assigns = [self.classify(datapoint) for datapoint in data]
if new_assigns == old_assigns:
print(f'Training finished after {n_iters} iterations!')
return
old_assigns = new_assigns
n_iters = 1
# recalculate centers
for id_ in range(self.k):
points_idx = np.where(np.array(new_assigns) == id_)
datapoints = data[points_idx]
self.centers[id_] = datapoints.mean(axis=0)
def l2_distance(self, datapoint):
dists = np.sqrt(np.sum((self.centers - datapoint)**2, axis=1))
return dists
def classify(self, datapoint):
'''
Given a datapoint, compute the cluster closest to the
datapoint. Return the cluster ID of that cluster.
'''
dists = self.l2_distance(datapoint)
return np.argmin(dists)
def plot_clusters(self, data):
plt.figure(figsize=(12,10))
plt.title('Initial centers in black, final centers in red')
plt.scatter(data[:, 0], data[:, 1], marker='.', c=y)
plt.scatter(self.centers[:, 0], self.centers[:,1], c='r')
plt.scatter(self.initial_centers[:, 0], self.initial_centers[:,1], c='k')
plt.show()
初始化并調(diào)整模型
kmeans = KMeans(n_clusters=4)
kmeans.fit(X)
Training finished after 4 iterations!
描繪初始和最終的聚類中心
▌6. 簡單的神經(jīng)網(wǎng)絡(luò)
在這一章節(jié)里，我們將實(shí)現(xiàn)一個(gè)簡單的神經(jīng)網(wǎng)絡(luò)架構(gòu)，將 2 維的輸入向量映射成二進(jìn)制輸出值。我們的神經(jīng)網(wǎng)絡(luò)有 2 個(gè)輸入神經(jīng)元，含 6 個(gè)隱藏神經(jīng)元隱藏層及 1 個(gè)輸出神經(jīng)元。
我們將通過層之間的權(quán)重矩陣來表示神經(jīng)網(wǎng)絡(luò)結(jié)構(gòu)。在下面的例子中，輸入層和隱藏層之間的權(quán)重矩陣將被表示為，隱藏層和輸出層之間的權(quán)重矩陣為。除了連接神經(jīng)元的權(quán)重向量外，每個(gè)隱藏和輸出的神經(jīng)元都會(huì)有一個(gè)大小為 1 的偏置量。
我們的訓(xùn)練集由 m = 750 個(gè)樣本組成。因此，我們的矩陣維度如下：
訓(xùn)練集維度： X = (750，2) 目標(biāo)維度： Y = (750，1) 維度：(m，nhidden) = (2,6)
維度：(bias vector)：(1，nhidden) = (1,6) 維度： (nhidden，noutput)= (6,1) 維度：(bias vector)：(1，noutput) = (1,1)
損失函數(shù)
我們使用與 Logistic 回歸算法相同的損失函數(shù)：
對于多類別的分類任務(wù)，我們將使用這個(gè)函數(shù)的通用形式作為損失函數(shù)，稱之為分類交叉熵函數(shù)。
訓(xùn)練 我們將用梯度下降法來訓(xùn)練我們的神經(jīng)網(wǎng)絡(luò)，并通過反向傳播法來計(jì)算所需的偏導(dǎo)數(shù)。訓(xùn)練過程主要有以下幾個(gè)步驟：
1. 初始化參數(shù)(即權(quán)重量和偏差量) 2. 重復(fù)以下過程，直到收斂：
前向傳播過程
首先，我們計(jì)算網(wǎng)絡(luò)中每個(gè)單元的激活值和輸出值。為了加速這個(gè)過程的實(shí)現(xiàn)，我們不會(huì)單獨(dú)為每個(gè)輸入樣本執(zhí)行此操作，而是通過矢量化對所有樣本一次性進(jìn)行處理。其中：
表示對所有訓(xùn)練樣本激活隱層單元的矩陣 表示對所有訓(xùn)練樣本輸出隱層單位的矩陣
隱層神經(jīng)元將使用 tanh 函數(shù)作為其激活函數(shù)：
輸出層神經(jīng)元將使用 sigmoid 函數(shù)作為激活函數(shù)：
激活值和輸出值計(jì)算如下(·表示點(diǎn)乘)：
反向傳播過程
為了計(jì)算權(quán)重向量的更新值，我們需要計(jì)算每個(gè)神經(jīng)元對損失函數(shù)的偏導(dǎo)數(shù)。這里不會(huì)給出這些公式的推導(dǎo)，你會(huì)在其他網(wǎng)站上找到很多更好的解釋(https:///2015/03/17/a-step-by-step-backpropagation-example/)。
對于輸出神經(jīng)元，梯度計(jì)算如下(矩陣符號)：
對于輸入和隱層的權(quán)重矩陣，梯度計(jì)算如下：
權(quán)重更新
In [3]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split
np.random.seed(123)
% matplotlib inline
數(shù)據(jù)集
In [4]:
X, y = make_circles(n_samples=1000, factor=0.5, noise=.1)
fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y)
plt.xlim([-1.5, 1.5])
plt.ylim([-1.5, 1.5])
plt.title('Dataset')
plt.xlabel('First feature')
plt.ylabel('Second feature')
plt.show()
In [5]:
# reshape targets to get column vector with shape (n_samples, 1)
y_true = y[:, np.newaxis]
# Split the data into a training and test set
X_train, X_test, y_train, y_test = train_test_split(X, y_true)
print(f'Shape X_train: {X_train.shape}')
print(f'Shape y_train: {y_train.shape}')
print(f'Shape X_test: {X_test.shape}')
print(f'Shape y_test: {y_test.shape}')
Shape X_train: (750, 2) Shape y_train: (750, 1) Shape X_test: (250, 2) Shape y_test: (250, 1)
Neural Network Class
以下部分實(shí)現(xiàn)受益于吳恩達(dá)的課程 https://www./learn/neural-networks-deep-learning
class NeuralNet():
def __init__(self, n_inputs, n_outputs, n_hidden):
self.n_inputs = n_inputs
self.n_outputs = n_outputs
self.hidden = n_hidden
# Initialize weight matrices and bias vectors
self.W_h = np.random.randn(self.n_inputs, self.hidden)
self.b_h = np.zeros((1, self.hidden))
self.W_o = np.random.randn(self.hidden, self.n_outputs)
self.b_o = np.zeros((1, self.n_outputs))
def sigmoid(self, a):
return 1 / (1 np.exp(-a))
def forward_pass(self, X):
'''
Propagates the given input X forward through the net.
Returns:
A_h: matrix with activations of all hidden neurons for all input examples
O_h: matrix with outputs of all hidden neurons for all input examples
A_o: matrix with activations of all output neurons for all input examples
O_o: matrix with outputs of all output neurons for all input examples
'''
# Compute activations and outputs of hidden units
A_h = np.dot(X, self.W_h) self.b_h
O_h = np.tanh(A_h)
# Compute activations and outputs of output units
A_o = np.dot(O_h, self.W_o) self.b_o
O_o = self.sigmoid(A_o)
outputs = {
'A_h': A_h,
'A_o': A_o,
'O_h': O_h,
'O_o': O_o,
}
return outputs
def cost(self, y_true, y_predict, n_samples):
'''
Computes and returns the cost over all examples
'''
# same cost function as in logistic regression
cost = (- 1 / n_samples) * np.sum(y_true * np.log(y_predict) (1 - y_true) * (np.log(1 - y_predict)))
cost = np.squeeze(cost)
assert isinstance(cost, float)
return cost
def backward_pass(self, X, Y, n_samples, outputs):
'''
Propagates the errors backward through the net.
Returns:
dW_h: partial derivatives of loss function w.r.t hidden weights
db_h: partial derivatives of loss function w.r.t hidden bias
dW_o: partial derivatives of loss function w.r.t output weights
db_o: partial derivatives of loss function w.r.t output bias
'''
dA_o = (outputs['O_o'] - Y)
dW_o = (1 / n_samples) * np.dot(outputs['O_h'].T, dA_o)
db_o = (1 / n_samples) * np.sum(dA_o)
dA_h = (np.dot(dA_o, self.W_o.T)) * (1 - np.power(outputs['O_h'], 2))
dW_h = (1 / n_samples) * np.dot(X.T, dA_h)
db_h = (1 / n_samples) * np.sum(dA_h)
gradients = {
'dW_o': dW_o,
'db_o': db_o,
'dW_h': dW_h,
'db_h': db_h,
}
return gradients
def update_weights(self, gradients, eta):
'''
Updates the model parameters using a fixed learning rate
'''
self.W_o = self.W_o - eta * gradients['dW_o']
self.W_h = self.W_h - eta * gradients['dW_h']
self.b_o = self.b_o - eta * gradients['db_o']
self.b_h = self.b_h - eta * gradients['db_h']
def train(self, X, y, n_iters=500, eta=0.3):
'''
Trains the neural net on the given input data
'''
n_samples, _ = X.shape
for i in range(n_iters):
outputs = self.forward_pass(X)
cost = self.cost(y, outputs['O_o'], n_samples=n_samples)
gradients = self.backward_pass(X, y, n_samples, outputs)
if i % 100 == 0:
print(f'Cost at iteration {i}: {np.round(cost, 4)}')
self.update_weights(gradients, eta)
def predict(self, X):
'''
Computes and returns network predictions for given dataset
'''
outputs = self.forward_pass(X)
y_pred = [1 if elem >= 0.5 else 0 for elem in outputs['O_o']]
return np.array(y_pred)[:, np.newaxis]
初始化并訓(xùn)練神經(jīng)網(wǎng)絡(luò)
nn = NeuralNet(n_inputs=2, n_hidden=6, n_outputs=1)
print('Shape of weight matrices and bias vectors:')
print(f'W_h shape: {nn.W_h.shape}')
print(f'b_h shape: {nn.b_h.shape}')
print(f'W_o shape: {nn.W_o.shape}')
print(f'b_o shape: {nn.b_o.shape}')
print()
print('Training:')
nn.train(X_train, y_train, n_iters=2000, eta=0.7)
Shape of weight matrices and bias vectors: W_h shape: (2, 6) b_h shape: (1, 6) W_o shape: (6, 1) b_o shape: (1, 1)
Training: Cost at iteration 0: 1.0872 Cost at iteration 100: 0.2723 Cost at iteration 200: 0.1712 Cost at iteration 300: 0.1386 Cost at iteration 400: 0.1208 Cost at iteration 500: 0.1084 Cost at iteration 600: 0.0986 Cost at iteration 700: 0.0907 Cost at iteration 800: 0.0841 Cost at iteration 900: 0.0785 Cost at iteration 1000: 0.0739 Cost at iteration 1100: 0.0699 Cost at iteration 1200: 0.0665 Cost at iteration 1300: 0.0635 Cost at iteration 1400: 0.061 Cost at iteration 1500: 0.0587 Cost at iteration 1600: 0.0566 Cost at iteration 1700: 0.0547 Cost at iteration 1800: 0.0531 Cost at iteration 1900: 0.0515
測試神經(jīng)網(wǎng)絡(luò)
n_test_samples, _ = X_test.shape
y_predict = nn.predict(X_test)
print(f'Classification accuracy on test set: {(np.sum(y_predict == y_test)/n_test_samples)*100} %')
Classification accuracy on test set: 98.4 %
可視化決策邊界
X_temp, y_temp = make_circles(n_samples=60000, noise=.5)
y_predict_temp = nn.predict(X_temp)
y_predict_temp = np.ravel(y_predict_temp)
fig = plt.figure(figsize=(8,12))
ax = fig.add_subplot(2,1,1)
plt.scatter(X[:,0], X[:,1], c=y)
plt.xlim([-1.5, 1.5])
plt.ylim([-1.5, 1.5])
plt.xlabel('First feature')
plt.ylabel('Second feature')
plt.title('Training and test set')
ax = fig.add_subplot(2,1,2)
plt.scatter(X_temp[:,0], X_temp[:,1], c=y_predict_temp)
plt.xlim([-1.5, 1.5])
plt.ylim([-1.5, 1.5])
plt.xlabel('First feature')
plt.ylabel('Second feature')
plt.title('Decision boundary')
Out[11]:Text(0.5,1,'Decision boundary')
▌7. Softmax 回歸算法
Softmax 回歸算法，又稱為多項(xiàng)式或多類別的 Logistic 回歸算法。
給定：
數(shù)據(jù)集
Softmax 回歸模型有以下幾個(gè)特點(diǎn)：
對于每個(gè)類別，都存在一個(gè)獨(dú)立的、實(shí)值加權(quán)向量 這個(gè)權(quán)重向量通常作為權(quán)重矩陣中的行。 對于每個(gè)類別，都存在一個(gè)獨(dú)立的、實(shí)值偏置量b 它使用 softmax 函數(shù)作為其激活函數(shù) 它使用交叉熵( cross-entropy )作為損失函數(shù)
訓(xùn)練 Softmax 回歸模型有不同步驟。首先(在步驟0中)，模型的參數(shù)將被初始化。在達(dá)到指定訓(xùn)練次數(shù)或參數(shù)收斂前，重復(fù)以下其他步驟。
第 0 步：用 0 (或小的隨機(jī)值)來初始化權(quán)重向量和偏置值
第 1 步：對于每個(gè)類別k，計(jì)算其輸入的特征與權(quán)重值的線性組合，也就是說為每個(gè)類別的訓(xùn)練樣本計(jì)算一個(gè)得分值。對于類別k，輸入向量為,則得分值的計(jì)算如下：
其中表示類別k的權(quán)重矩陣，·表示點(diǎn)積。
我們可以通過矢量化和矢量傳播法則計(jì)算所有類別及其訓(xùn)練樣本的得分值：
其中 X 是所有訓(xùn)練樣本的維度矩陣，W 表示每個(gè)類別的權(quán)重矩陣維度，其形式為；
第 2 步：用 softmax 函數(shù)作為激活函數(shù)，將得分值轉(zhuǎn)化為概率值形式。屬于類別 k 的輸入向量的概率值為：
同樣地，我們可以通過矢量化來對所有類別同時(shí)處理，得到其概率輸出。模型預(yù)測出的表示的是該類別的最高概率。
第 3 步：計(jì)算整個(gè)訓(xùn)練集的損失值。
我們希望模型預(yù)測出的高概率值是目標(biāo)類別，而低概率值表示其他類別。這可以通過以下的交叉熵?fù)p失函數(shù)來實(shí)現(xiàn)：
在上面公式中，目標(biāo)類別標(biāo)簽表示成獨(dú)熱編碼形式( one-hot )。因此為1時(shí)表示的目標(biāo)類別是 k，反之則為 0。
第 4 步：對權(quán)重向量和偏置量，計(jì)算其對損失函數(shù)的梯度。
關(guān)于這個(gè)導(dǎo)數(shù)實(shí)現(xiàn)的詳細(xì)解釋，可以參見這里（http://ufldl./tutorial/supervised/SoftmaxRegression/）。
一般形式如下：
對于偏置量的導(dǎo)數(shù)計(jì)算，此時(shí)為1。
第 5 步：對每個(gè)類別k，更新其權(quán)重和偏置值。
其中，表示學(xué)習(xí)率。
In [1]:
from sklearn.datasets import load_iris
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt
np.random.seed(13)
數(shù)據(jù)集
In [2]:
X, y_true = make_blobs(centers=4, n_samples = 5000)
fig = plt.figure(figsize=(8,6))
plt.scatter(X[:,0], X[:,1], c=y_true)
plt.title('Dataset')
plt.xlabel('First feature')
plt.ylabel('Second feature')
plt.show()
In [3]:
# reshape targets to get column vector with shape (n_samples, 1)
y_true = y_true[:, np.newaxis]
# Split the data into a training and test set
X_train, X_test, y_train, y_test = train_test_split(X, y_true)
print(f'Shape X_train: {X_train.shape}')
print(f'Shape y_train: {y_train.shape}')
print(f'Shape X_test: {X_test.shape}')
print(f'Shape y_test: {y_test.shape}')
Shape X_train: (3750, 2) Shape y_train: (3750, 1) Shape X_test: (1250, 2) Shape y_test: (1250, 1)
Softmax回歸分類
class SoftmaxRegressor:
def __init__(self):
pass
def train(self, X, y_true, n_classes, n_iters=10, learning_rate=0.1):
'''
Trains a multinomial logistic regression model on given set of training data
'''
self.n_samples, n_features = X.shape
self.n_classes = n_classes
self.weights = np.random.rand(self.n_classes, n_features)
self.bias = np.zeros((1, self.n_classes))
all_losses = []
for i in range(n_iters):
scores = self.compute_scores(X)
probs = self.softmax(scores)
y_predict = np.argmax(probs, axis=1)[:, np.newaxis]
y_one_hot = self.one_hot(y_true)
loss = self.cross_entropy(y_one_hot, probs)
all_losses.append(loss)
dw = (1 / self.n_samples) * np.dot(X.T, (probs - y_one_hot))
db = (1 / self.n_samples) * np.sum(probs - y_one_hot, axis=0)
self.weights = self.weights - learning_rate * dw.T
self.bias = self.bias - learning_rate * db
if i % 100 == 0:
print(f'Iteration number: {i}, loss: {np.round(loss, 4)}')
return self.weights, self.bias, all_losses
def predict(self, X):
'''
Predict class labels for samples in X.
Args:
X: numpy array of shape (n_samples, n_features)
Returns:
numpy array of shape (n_samples, 1) with predicted classes
'''
scores = self.compute_scores(X)
probs = self.softmax(scores)
return np.argmax(probs, axis=1)[:, np.newaxis]
def softmax(self, scores):
'''
Tranforms matrix of predicted scores to matrix of probabilities
Args:
scores: numpy array of shape (n_samples, n_classes)
with unnormalized scores
Returns:
softmax: numpy array of shape (n_samples, n_classes)
with probabilities
'''
exp = np.exp(scores)
sum_exp = np.sum(np.exp(scores), axis=1, keepdims=True)
softmax = exp / sum_exp
return softmax
def compute_scores(self, X):
'''
Computes class-scores for samples in X
Args:
X: numpy array of shape (n_samples, n_features)
Returns:
scores: numpy array of shape (n_samples, n_classes)
'''
return np.dot(X, self.weights.T) self.bias
def cross_entropy(self, y_true, scores):
loss = - (1 / self.n_samples) * np.sum(y_true * np.log(scores))
return loss
def one_hot(self, y):
'''
Tranforms vector y of labels to one-hot encoded matrix
'''
one_hot = np.zeros((self.n_samples, self.n_classes))
one_hot[np.arange(self.n_samples), y.T] = 1
return one_hot
初始化并訓(xùn)練模型
regressor = SoftmaxRegressor()
w_trained, b_trained, loss = regressor.train(X_train, y_train, learning_rate=0.1, n_iters=800, n_classes=4)
fig = plt.figure(figsize=(8,6))
plt.plot(np.arange(800), loss)
plt.title('Development of loss during training')
plt.xlabel('Number of iterations')
plt.ylabel('Loss')
plt.show()Iteration number: 0, loss: 1.393 Iteration number: 100, loss: 0.2051 Iteration number: 200, loss: 0.1605 Iteration number: 300, loss: 0.1371 Iteration number: 400, loss: 0.121 Iteration number: 500, loss: 0.1087 Iteration number: 600, loss: 0.0989 Iteration number: 700, loss: 0.0909
測試模型
n_test_samples, _ = X_test.shape
y_predict = regressor.predict(X_test)
print(f'Classification accuracy on test set: {(np.sum(y_predict == y_test)/n_test_samples) * 100}%')
測試集分類準(zhǔn)確率：99.03999999999999%
|
