基于Python的大数据的分页模型代码

时间：2024-07-06 21:04:14

最近在写一个cmdb系统的分页，尽管Django本身有分页的模块儿，但是还是想实现一下自己心中想的分页的一种逻辑

因为，在我们工作当中，当我们的数据量超级大的时候，其实我们每次分页查询都不必将所有的数据查询出来，而是可以按阶段的查询，举个例子

每次查询5页数据，当需要第六页的时候，再次进行加载,为了更加明了，做了举例

首先我们规定好每次查询5页，每页4条数据　　

 db_data = list(range(1,100))

 final_page = len(db_data)

 one_page_num = 5 #每页4条

 one_time_num = 5 #每次查5页

1、查询索引的逻辑 设定findIndex查询的索引

     # page = 1  int(1/5) + 1  findIndex = 1

     # page = 2  int(2/5) + 1  findIndex = 1

     # page = 3  int(3/5) + 1  findIndex = 1

     # page = 4  int(4/5) + 1  findIndex = 1

     # page = 5  int(5/5)  findIndex = 1

     # page = 6  int(6/5) + 1  findIndex = 2

     # page = 7  int(7/5) + 1  findIndex = 2

     # page = 8  int(8/5) + 1  findIndex = 2

     # page = 9  int(9/5) + 1  findIndex = 2

     # page = 10  int(10/5)  findIndex = 2

     # page = 11  int(11/5) + 1  findIndex = 3

     # page = 12  int(12/5) + 1  findIndex = 3

     # page = 13  int(13/5) + 1  findIndex = 3

     # page = 14  int(14/5) + 1  findIndex = 3

     # page = 15  int(15/5)  findIndex = 3

所以我们可以写如下代码

 if page/one_time_num > int(page/one_time_num):

     findIndex = int(page / one_time_num) + 1

 else:

     findIndex = int(page / one_time_num)

 #进行第一次查询，查询每次5页，每页4条的数据

 select_num = one_time_num * one_page_num #这里决一次定查询20条

 select_start = (findIndex - 1) * select_num  # 开始查询的索引

 select_down =  findIndex * select_num  # 结束查询的索引

 select_data = db_data[select_start: select_down]  #这里查询出了一次20条数据

这个时候是查询出20条数据了，但是，我们想要的具体的数据和页码对不上，所以做了以下的数据

 #下面对20条数据进行截取

     #findIndex 1

         # page = 1  1-(0*5)  gnPage = 1

         # page = 2  2-(0*5)  gnPage = 2

         # page = 3  3-(0*5)  gnPage = 3

         # page = 4  4-(0*5)  gnPage = 4

         # page = 5  5-(0*5)  gnPage = 5

     #findIndex 2

         # page = 6  6-(1*5)  gnPage = 1

         # page = 7  7-(1*5)  gnPage = 2

         # page = 8  8-(1*5)  gnPage = 3

         # page = 9  9-(1*5)  gnPage = 4

         # page = 10  10-(1*5)  gnPage = 5

     #findIndex 3

         # page = 11  11-(2*5)  gnPage = 1

         # page = 12  12-(2*5)  gnPage = 2

         # page = 13  13-(2*5)  gnPage = 3

         # page = 14  14-(2*5)  gnPage = 4

         # page = 15  15-(2*5)  gnPage = 5

所以我们可以得到以下代码来获取具体的一页数据

#设定 截取的索引位now_index

now_index = page - (findIndex-1)*one_time_num

#设定单页的截取起始

page_start = (now_index - 1) * one_page_num

page_end = now_index * one_page_num

#开始截取

page_data = select_data[page_start:page_end]

最后进行代码结合

 #coding:utf-8

 db_data = list(range(1,100))

 one_page_num = 5 #每页4条

 one_time_num = 5 #每次查5页

 while True:

     page = int(input("page >>>"))  #输入页码

     #查询索引的逻辑 设定findIndex查询的索引

     # page = 1  int(1/5) + 1  findIndex = 1

     # page = 2  int(2/5) + 1  findIndex = 1

     # page = 3  int(3/5) + 1  findIndex = 1

     # page = 4  int(4/5) + 1  findIndex = 1

     # page = 5  int(5/5)  findIndex = 1

     # page = 6  int(6/5) + 1  findIndex = 2

     # page = 7  int(7/5) + 1  findIndex = 2

     # page = 8  int(8/5) + 1  findIndex = 2

     # page = 9  int(9/5) + 1  findIndex = 2

     # page = 10  int(10/5)  findIndex = 2

     # page = 11  int(11/5) + 1  findIndex = 3

     # page = 12  int(12/5) + 1  findIndex = 3

     # page = 13  int(13/5) + 1  findIndex = 3

     # page = 14  int(14/5) + 1  findIndex = 3

     # page = 15  int(15/5)  findIndex = 3

     if page/one_time_num > int(page/one_time_num):

         findIndex = int(page / one_time_num) + 1

     else:

         findIndex = int(page / one_time_num)

     #进行第一次查询，查询每次5页，每页4条的数据

     select_num = one_time_num * one_page_num #这里决一次定查询20条

     select_start = (findIndex - 1) * select_num  # 开始查询的索引

     select_down =  findIndex * select_num  # 结束查询的索引

     select_data = db_data[select_start: select_down]  #这里查询出了一次20条数据

     #下面对20条数据进行截取

     #findIndex 1

         # page = 1  1-(0*5)  gnPage = 1

         # page = 2  2-(0*5)  gnPage = 2

         # page = 3  3-(0*5)  gnPage = 3

         # page = 4  4-(0*5)  gnPage = 4

         # page = 5  5-(0*5)  gnPage = 5

     #findIndex 2

         # page = 6  6-(1*5)  gnPage = 1

         # page = 7  7-(1*5)  gnPage = 2

         # page = 8  8-(1*5)  gnPage = 3

         # page = 9  9-(1*5)  gnPage = 4

         # page = 10  10-(1*5)  gnPage = 5

     #findIndex 3

         # page = 11  11-(2*5)  gnPage = 1

         # page = 12  12-(2*5)  gnPage = 2

         # page = 13  13-(2*5)  gnPage = 3

         # page = 14  14-(2*5)  gnPage = 4

         # page = 15  15-(2*5)  gnPage = 5

     #设定 截取的索引位now_index

     now_index = page - (findIndex-1)*one_time_num

     #设定单页的截取起始

     page_start = (now_index - 1) * one_page_num

     page_end = now_index * one_page_num

     #开始截取

     page_data = select_data[page_start:page_end]

     #输出效果

     print("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

     print("当前是第%s次查询"%findIndex)

     print("当前查询到的所有数据是 \n %s" % select_data)

     print("当前是 %s 页" % page)

     print("当前的数据是 \n %s " % page_data)

     print("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")

大家试一下吧。

相关文章