像Excel一样使用SQL进行数据分析

时间:2022-06-01 17:43:25

Excel是数据分析中最常用的工具 ,利用Excel可以完成数据清洗,预处理,以及最常见的数据分类,数据筛选,分类汇总,以及数据透视等操作,而这些操作用SQL一样可以实现。SQL不仅可以从数据库中读取数据,还能通过不同的SQL函数语句直接返回所需要的结果,从而大大提高了自己在客户端应用程序中计算的效率。

像Excel一样使用SQL进行数据分析

1 重复数据处理

查找重复记录

  1. SELECT * FROM user
  2. Where (nick_name,password) in
  3. (
  4. SELECT nick_name,password
  5. FROM user
  6. group by nick_name,password
  7. having count(nick_name)>1
  8. );

查找去重记录

查找id最大的记录

  1. SELECT * FROM user
  2. WHERE id in
  3. (SELECT max(id) FROM user
  4. group by nick_name,password
  5. having count(nick_name)>1
  6. );

删除重复记录

只保留id值最小的记录

  1. DELETE c1
  2. FROM customer c1,customer c2
  3. WHERE c1.cust_email=c2.cust_email
  4. AND c1.id>c2.id;
  5. DELETE FROM user Where (nick_name,password) in
  6. (SELECT nick_name,password FROM
  7. (SELECT nick_name,password FROM user
  8. group by nick_name,password
  9. having count(nick_name)>1) as tmp1
  10. )
  11. and id not in
  12. (SELECT id FROM
  13. (SELECT min(id) id FROM user
  14. group by nick_name,password
  15. having count(nick_name)>1) as tmp2
  16. );

2 缺失值处理

查找缺失值记录

  1. SELECT * FROM customer
  2. WHERE cust_email IS NULL;

更新列填充空值

  1. UPDATE sale set city = "未知"
  2. WHERE city IS NULL;
  3.  
  4. UPDATE orderitems set
  5. price_new=IFNULL(price_new,5.74);

查询并填充空值列

  1. SELECT AVG(price_new) FROM orderitems;
  2.  
  3. SELECT IFNULL(price_new,5.74) AS bus_ifnull
  4. FROM orderitems;

3 计算列

更新表添加计算列

  1. ALTER TABLE orderitems ADD price_new DECIMAL(8,2) NOT NULL;
  2.  
  3. UPDATE orderitems set price_new= item_price*count;

查询计算列

  1. SELECT item_price*count as sales FROM orderitems;

4 排序

多列排序

  1. SELECT * FROM orderitems
  2. ORDER BY price_new DESC,quantity;

查询排名前几的记录

  1. SELECT * FROM orderitems
  2. ORDER BY price_new DESC LIMIT 5;

查询第10大的值

  1. SELECT DISTINCT price_new
  2. FROM orderitems
  3. ORDER BY price_new DESC LIMIT 9,1;

排名

数值相同的排名相同且排名连续

  1. SELECT prod_price,
  2. (SELECT COUNT(DISTINCT prod_price)
  3. FROM products
  4. WHERE prod_price>=a.prod_price
  5. ) AS rank
  6. FROM products AS a
  7. ORDER BY rank ;

5 字符串处理

字符串替换

  1. UPDATE data1 SET city=REPLACE(city,'SH','shanghai');
  2.  
  3. SELECT city FROM data1;

按位置字符串截取

字符串截取可用于数据分列

MySQL 字符串截取函数:left(), right(), substring(), substring_index()

  1. SELECT left('example.com', 3);

从字符串的第 4 个字符位置开始取,直到结束

  1. SELECT substring('example.com', 4);

从字符串的第 4 个字符位置开始取,只取 2 个字符

  1. SELECT substring('example.com', 4, 2);

按关键字截取字符串

取第一个分隔符之前的所有字符,结果是www

  1. SELECT substring_index('www.google.com','.',1);

取倒数第二个分隔符之后的所有字符,结果是google.com;

  1. SELECT substring_index('www.google.com','.',-2);

6 筛选

通过操作符实现高级筛选

使用 AND OR IN NOT 等操作符实现高级筛选过滤

  1. SELECT prod_name,prod_price FROM Products
  2. WHERE vend_id IN('DLL01','BRS01');
  3. SELECT prod_name FROM Products WHERE NOT vend_id='DLL01';

通配符筛选

常用通配符有% _ [] ^

  1. SELECT * from customers WHERE country LIKE "CH%";

7 表联结

SQL表连接可以实现类似于Excel中的Vlookup函数的功能

  1. SELECT vend_id,prod_name,prod_price
  2. FROM Vendors INNER JOIN Products
  3. ON Vendors.vend_id=Products.vend_id;
  4.  
  5. SELECT prod_name,vend_name,prod_price,quantity
  6. FROM OderItems,Products,Vendors
  7. WHERE Products.vend_id=Vendors.vend_id
  8. AND OrderItems.prod_id=Products.prod_id
  9. AND order_num=20007;

自联结 在一条SELECT语句中多次使用相同的表

  1. SELECT c1.cust_od,c1.cust_name,c1.cust_contact
  2. FROM Customers as c1,Customers as c2
  3. WHERE c1.cust_name=c2.cust_name
  4. AND c2.cust_contact='Jim Jones';

8 数据透视

数据分组可以实现Excel中数据透视表的功能

数据分组

group by 用于数据分组 having 用于分组后数据的过滤

  1. SELECT order_num,COUNT(*) as items
  2. FROM OrderItems
  3. GROUP BY order_num HAVING COUNT(*)>=3;

交叉表

通过CASE WHEN函数实现

  1. SELECT data1.city,
  2. CASE WHEN colour = "A" THEN price END AS A,
  3. CASE WHEN colour = "B" THEN price END AS B,
  4. CASE WHEN colour = "C" THEN price END AS C,
  5. CASE WHEN colour = "F" THEN price END AS F
  6. FROM data1

原文链接:https://mp.weixin.qq.com/s?__biz=MzA3MTg4NjY4Mw==&mid=2457324446&idx=2&sn=9cbf3a9184e66e115c867a3326ca88e9&chksm=88a5d9aabfd250bc3f12aca480686547379692d25efc1e3b223670f27c1a8473f2329729dc9f&mpshare=1&