My node.js app is generating a lot of text logs (100 per user per day). I have 10k active users daily, about 100k logs each day for my app to store. Format something like this:
我的节点。js应用程序生成大量的文本日志(每天100个用户)。我每天有10k的活跃用户，每天大约有100k的日志用来存储我的应用。格式是这样的:
time | log_type_id | action_id | message | user_ip
Cannot archive. Most of this logging data must be available for display to it's user.
不能存档。大多数日志数据必须是可用的，以便显示给用户。
Question is what's the fastest, most efficient disk space and performance wise way to store logs?
问题是，存储日志的最快、最有效的磁盘空间和性能的方法是什么?
Bonus questions what's the best way to sort, analyze all of this data?
额外的问题，整理和分析所有这些数据的最好方法是什么?
1 个解决方案
#1
3
Question is what's the fastest, most efficient disk space and performance wise way to store logs?
问题是，存储日志的最快、最有效的磁盘空间和性能的方法是什么?
Fastest, most efficient way to store logs? well, in sequentially written files of course...
最快、最高效的日志存储方式?当然，在顺序写的文件中……
Bonus questions what's the best way to sort, analyze all of this data?
额外的问题，整理和分析所有这些数据的最好方法是什么?
Best to use a relational database for this.
最好为此使用关系数据库。
This highlights the tradeoff. If all you are doing is saving the logs for a human to review, you will use less space, and write faster, to log files. On the other hand, while log files are great for writing information they are not so great for digesting and analyzing it. In general relational databases will win out even though they are not as efficient storage-wise or performance-wise on writing.
这凸显了权衡。如果您所做的只是保存日志供人审阅，那么您将使用更少的空间和更快的编写日志文件。另一方面，虽然日志文件对于编写信息非常有用，但是对于理解和分析信息却不是那么有用。一般来说，关系数据库会胜出，即使它们在编写时没有那么高效的存储或性能。
#1
3
Question is what's the fastest, most efficient disk space and performance wise way to store logs?
问题是，存储日志的最快、最有效的磁盘空间和性能的方法是什么?
Fastest, most efficient way to store logs? well, in sequentially written files of course...
最快、最高效的日志存储方式?当然，在顺序写的文件中……
Bonus questions what's the best way to sort, analyze all of this data?
额外的问题，整理和分析所有这些数据的最好方法是什么?
Best to use a relational database for this.
最好为此使用关系数据库。
This highlights the tradeoff. If all you are doing is saving the logs for a human to review, you will use less space, and write faster, to log files. On the other hand, while log files are great for writing information they are not so great for digesting and analyzing it. In general relational databases will win out even though they are not as efficient storage-wise or performance-wise on writing.
这凸显了权衡。如果您所做的只是保存日志供人审阅，那么您将使用更少的空间和更快的编写日志文件。另一方面，虽然日志文件对于编写信息非常有用，但是对于理解和分析信息却不是那么有用。一般来说，关系数据库会胜出，即使它们在编写时没有那么高效的存储或性能。