Cost Function of Support Vector Machine

Logistic Regression 中的函数 f,g $f,g$

f(x)=ln(1+ex),x∈R,g(x)=f(−x) $f(x)=\ln (1+e^x),x\in \mathbb{R},g(x)=f(-x)$

f,g $f,g$ 的性质

f′(x)=ex1+ex>0,x∈R $f^{\prime }(x)={\displaystyle \frac{e^x}{1+e^x}}>0,x\in \mathbb{R}$
f′′(x)=(ex1+ex)′=ex(1+ex)2>0,x∈R $f^{″}(x)={\left({\displaystyle \frac{e^x}{1+e^x}}\right)}^{\prime }={\displaystyle \frac{e^x}{(1+e^x{)}^2}}>0,x\in \mathbb{R}$
limx→+∞f(x)=+∞,limx→−∞f(x)=0 $\underset{x\to +\mathrm{\infty }}{lim}f(x)=+\mathrm{\infty },\underset{x\to -\mathrm{\infty }}{lim}f(x)=0$
limx→−∞f(x)x=limx→−∞f′(x)=1 $\underset{x\to -\mathrm{\infty }}{lim}{\displaystyle \frac{f(x)}{x}}=\underset{x\to -\mathrm{\infty }}{lim}{f}^{\prime }(x)=1$
limx→+∞[f(x)−x]=limx→+∞ln(1+e−x)=0 $\underset{x\to +\mathrm{\infty }}{lim}\left[f(x)-x\right]=\underset{x\to +\mathrm{\infty }}{lim}\ln (1+e^{-x})=0$

g(x)=f(−x)=ln(1+e−x),x∈R $g(x)=f(-x)=\ln (1+e^{-x}),x\in \mathbb{R}$
g′(x)=11+e−xe−x(−1)=−11+ex<0,x∈R $g^{\prime }(x)={\displaystyle \frac{1}{1+e^{-x}}}{e}^{-x}(-1)=-{\displaystyle \frac{1}{1+e^x}}<0,x\in \mathbb{R}$
g′′(x)=−(11+ex)′=ex(1+ex)2>0,x∈R $g^{″}(x)=-{\left({\displaystyle \frac{1}{1+e^x}}\right)}^{\prime }={\displaystyle \frac{e^x}{(1+e^x{)}^2}}>0,x\in \mathbb{R}$

SVM 的函数 cost0,cost1 ${\mathrm{cost}}_0,{\mathrm{cost}}_1$

cost0(x)=max(0,x−1) ${cost}_0(x)=max(0,x-1)$
cost1(x)=max(0,1−x) ${cost}_1(x)=max(0,1-x)$

Cost function of Logistic Regression

hθ(Xi)=11+e−θ⊺Xi $h_{\theta }\left(X_i\right)={\displaystyle \frac{1}{1+e^{-{\theta }^{⊺}{X}_i}}}$
J(θ)=−1m∑i=1m{yiln(hθ(Xi))+(1−yi)ln(1−hθ(Xi))}+λ2m∑j=1nθ2j $J\left(\theta \right)=-{\displaystyle \frac{1}{m}}\underset{i=1}{\overset{m}{\sum }}\left\{y_i\ln \left(h_{\theta }\left(X_i\right)\right)+\left(1-y_i\right)\ln \left(1-h_{\theta }\left(X_i\right)\right)\right\}+{\displaystyle \frac{\lambda }{2m}}\underset{j=1}{\overset{n}{\sum }}{\theta }_j^2$
则 −lnhθ(Xi)=g(θ⊺Xi) $-\ln h_{\theta }\left(X_i\right)=g({\theta }^{⊺}{X}_i)$
−ln(1−hθ(Xi))=f(θ⊺Xi) $-\ln \left(1-h_{\theta }\left(X_i\right)\right)=f({\theta }^{⊺}{X}_i)$
于是 J(θ)=∑i=1m[yig(θ⊺Xi)+(1−yi)f(θ⊺Xi)]+λ2m∑j=1nθ2j $J\left(\theta \right)=\underset{i=1}{\overset{m}{\sum }}\left[y_{i}g({\theta }^{⊺}{X}_i)+\left(1-y_i\right)f({\theta }^{⊺}{X}_i)\right]+{\displaystyle \frac{\lambda }{2m}}\underset{j=1}{\overset{n}{\sum }}{\theta }_j^2$

Cost function of Support Vector Machine

hθ(Xi)={1,0,θ⊺Xi≥0,otherwise, $h_{\theta }\left(X_i\right)=\{\begin{array}{ll}1,& {\theta }^{⊺}{X}_i\ge 0,\\ 0,& \text{otherwise},\end{array}$
J(θ)=C∑i=1m[yicost1(θ⊺Xi)+(1−yi)cost0(θ⊺Xi)]+∑j=1nλ2θ2j $J\left(\theta \right)=C\underset{i=1}{\overset{m}{\sum }}\left[y_i{\mathrm{cost}}_1({\theta }^{⊺}{X}_i)+\left(1-y_i\right){\mathrm{cost}}_0({\theta }^{⊺}{X}_i)\right]+\underset{j=1}{\overset{n}{\sum }}{\displaystyle \frac{\lambda }{2}}{\theta }_j^2$