I have an application where I am storing about 1kbyte of data every 0.1 second. That's 36MByte/hour, or roughly 600MByte/day.
我有一个应用程序，我每0.1秒存储大约1kbyte的数据。这是36MByte /小时，或大约600MByte /天。
The data is highly compressible, so it should compress between 10:1 and 100:1. All the data is referenced by timestamp.
数据是高度可压缩的，因此它应该在10：1和100：1之间压缩。所有数据都由时间戳引用。
My question is this: what technique can I use to store this data?
我的问题是：我可以用什么技术来存储这些数据？
Constraints:
约束：
- The time to insert the data into the database cannot increase as the database size gets larger. This constraint rules out Microsoft SQL Server (we tried it, and after 5 days it ground to a halt as each "insert" was taking a minute).
- 将数据插入数据库的时间不会随着数据库大小的增加而增加。这个约束排除了Microsoft SQL Server（我们尝试了它，并且在5天之后它停止了，因为每个“插入”花了一分钟）。
- We can effectively pause the data recording for 4 hours per day, which would give us time to do compression, etc.
- 我们可以有效地暂停每天4小时的数据记录，这将给我们时间进行压缩等。
- We would like to compatible with LINQ for .NET, which means that we would probably need a database that a LINQ adapter (a MySQL style interface would be ok).
- 我们想与LINQ for .NET兼容，这意味着我们可能需要一个LINQ适配器（MySQL风格的接口可以）的数据库。
3 个解决方案
#1
3
One approach is to simply append your incoming data to a file on disk. After a day, switch to a new file, and then spawn a process to compress and store the previous day's file.
一种方法是简单地将传入的数据附加到磁盘上的文件中。一天之后，切换到新文件，然后生成一个进程来压缩和存储前一天的文件。
You seem to assume that you need to store your data in a database, without stating a reason why. Do you?
您似乎假设您需要将数据存储在数据库中，而不说明原因。你做？
#2
1
If you can't get SQL Server to handle that small of a load in a timely fashion, then I wonder if any RDBMS will be effective for you if you don't explore how you are inserting the data.
如果你不能让SQL Server及时处理那么小的负载，那么我想知道如果你没有探索如何插入数据，是否有任何RDBMS对你有效。
Are you doing just a very simple insert into a single table (with a primary key) that has no other indexes/functions/process blocking/reads ? Or is this process actually a little more involved than this simple/small insert you are talking about?
您是否只是将一个非常简单的插入到一个没有其他索引/函数/进程阻塞/读取的表中（使用主键）？或者这个过程实际上比你正在讨论的这个简单/小插入更复杂？
If you are dead set on using Linq, are you profiling your linq statements to make sure you are not telling the ORM to do something stupid?
如果您已经开始使用Linq了，您是否正在分析您的linq语句以确保您没有告诉ORM做一些愚蠢的事情？
#3
1
Perhaps you can store all the contents onto a binary file and the meta data to the DB.
也许您可以将所有内容存储到二进制文件中，将元数据存储到数据库中。
#1
3
One approach is to simply append your incoming data to a file on disk. After a day, switch to a new file, and then spawn a process to compress and store the previous day's file.
一种方法是简单地将传入的数据附加到磁盘上的文件中。一天之后，切换到新文件，然后生成一个进程来压缩和存储前一天的文件。
You seem to assume that you need to store your data in a database, without stating a reason why. Do you?
您似乎假设您需要将数据存储在数据库中，而不说明原因。你做？
#2
1
If you can't get SQL Server to handle that small of a load in a timely fashion, then I wonder if any RDBMS will be effective for you if you don't explore how you are inserting the data.
如果你不能让SQL Server及时处理那么小的负载，那么我想知道如果你没有探索如何插入数据，是否有任何RDBMS对你有效。
Are you doing just a very simple insert into a single table (with a primary key) that has no other indexes/functions/process blocking/reads ? Or is this process actually a little more involved than this simple/small insert you are talking about?
您是否只是将一个非常简单的插入到一个没有其他索引/函数/进程阻塞/读取的表中（使用主键）？或者这个过程实际上比你正在讨论的这个简单/小插入更复杂？
If you are dead set on using Linq, are you profiling your linq statements to make sure you are not telling the ORM to do something stupid?
如果您已经开始使用Linq了，您是否正在分析您的linq语句以确保您没有告诉ORM做一些愚蠢的事情？
#3
1
Perhaps you can store all the contents onto a binary file and the meta data to the DB.
也许您可以将所有内容存储到二进制文件中，将元数据存储到数据库中。