I'm working with a somewhat large set (~30000 records) of data that my Django app needs to retrieve on a regular basis. This data doesn't really change often (maybe once a month or so), and the changes that are made are done in a batch, so the DB solution I'm trying to arrive at is pretty much read-only.
我正在处理我的Django应用程序需要定期检索的大量数据(大约30000条记录)。这个数据不会经常改变(可能一个月左右)，并且所做的更改都是成批完成的，所以我试图得到的DB解决方案基本上是只读的。
The total size of this dataset is about 20mb, and my first thought is that I can load it into memory (possibly as a singleton on an object) and access it very fast that way, though I'm wondering if there are other, more efficient ways of decreasing the fetch time by avoiding disk I/O. Would memcached be the best solution here? Or would loading it into an in-memory SQLite DB be better? Or loading it on app startup simply as an in-memory variable?
这个数据集的总大小是大约20 mb,我首先想到的是我可以加载到内存中(可能是一个单例对象)和访问它非常快,但是我想知道如果有其他更有效的方法减少避免磁盘I / O的获取时间。memcached是这里的最佳解决方案吗?或者将它加载到内存中的SQLite DB中会更好吗?还是仅仅将它作为内存变量加载到app startup中?
2 个解决方案
#1
2
The simplest solution I think it's to load all the objects into memory with
我认为最简单的解决办法是将所有的对象加载到内存中
cached_records = Record.objects.all()
list(cached_records) # by using list() we force Django load all data into memory
Then you are free to use this cached_records in your app, and you also can use QuerySet methods like filter, etc. But filter on the cached records will trigger DB query.
然后您可以在应用程序中使用这个cached_records，还可以使用QuerySet方法，比如filter等。但是对缓存的记录进行筛选将触发DB查询。
If you will query these records based on conditions, using cache would be a good idea.
如果您将根据条件查询这些记录，使用缓存将是一个好主意。
#2
0
Does the disk IO really become the bottleneck of your application's performance and affect your user experience? If not, I don't think this kind of optimization is necessary.
磁盘IO真的成为应用程序性能的瓶颈并影响用户体验吗?如果没有，我不认为这种优化是必要的。
Operating system and RDBMS (e.g MySQL , PostgresQL) are really smart nowdays. The data in the disk will be cached in memory by RDBMS and OS automatically.
操作系统和RDBMS。g MySQL, PostgresQL)现在真的很聪明。磁盘中的数据将被RDBMS和OS自动缓存在内存中。
#1
2
The simplest solution I think it's to load all the objects into memory with
我认为最简单的解决办法是将所有的对象加载到内存中
cached_records = Record.objects.all()
list(cached_records) # by using list() we force Django load all data into memory
Then you are free to use this cached_records in your app, and you also can use QuerySet methods like filter, etc. But filter on the cached records will trigger DB query.
然后您可以在应用程序中使用这个cached_records，还可以使用QuerySet方法，比如filter等。但是对缓存的记录进行筛选将触发DB查询。
If you will query these records based on conditions, using cache would be a good idea.
如果您将根据条件查询这些记录，使用缓存将是一个好主意。
#2
0
Does the disk IO really become the bottleneck of your application's performance and affect your user experience? If not, I don't think this kind of optimization is necessary.
磁盘IO真的成为应用程序性能的瓶颈并影响用户体验吗?如果没有，我不认为这种优化是必要的。
Operating system and RDBMS (e.g MySQL , PostgresQL) are really smart nowdays. The data in the disk will be cached in memory by RDBMS and OS automatically.
操作系统和RDBMS。g MySQL, PostgresQL)现在真的很聪明。磁盘中的数据将被RDBMS和OS自动缓存在内存中。