如何从许多日志文件中创建数据库并执行一些SQL查询？

Consider you have many log files that you need to process all of them and to create a database out of it which will be able to be queried with SQL queries.
Which technology should I use for it?
Is LINQ to SQL appropriate?

假设您有许多日志文件需要处理所有日志文件并从中创建数据库,这些数据库将能够使用SQL查询进行查询。我应该使用哪种技术? LINQ to SQL是否合适?

2 个解决方案

#1

SQL is used with structured data, if your log files are simply raw text files, then you are out of luck. What I would suggest is to parse the log files, create local data structures that encapsulate the important data from the file
You can have something like

SQL与结构化数据一起使用,如果您的日志文件只是原始文本文件,那么您就不走运了。我建议解析日志文件,创建本地数据结构,封装文件中的重要数据你可以有类似的东西

class LogFileRow
{
  DateTime timestamp{get; set;}
  int ActivityId{get; set;}
  string Message{get; set;}
....
}

Then fire LINQ queries on collections of LogFileRow.

然后在LogFileRow集合上触发LINQ查询。

#2

My suggestion would be to use something like the Filehelpers library http://filehelpers.sourceforge.net/ to get the logfiles in your database.

我的建议是使用Filehelpers库http://filehelpers.sourceforge.net/之类的东西来获取数据库中的日志文件。

A simple foreach based on the DirectoryInfo class (http://msdn.microsoft.com/en-us/library/system.io.directoryinfo.aspx) where your logfiles are will do.

一个简单的foreach基于DirectoryInfo类(http://msdn.microsoft.com/en-us/library/system.io.directoryinfo.aspx),您的日志文件将在其中执行。

Wether you use Linq-2-sql or another technology for the insertion depends on the number of records. Linq-2-sql is not very suitbable for large batch insertions because it generates a single insert statement per record. I would combine SqlBulkCopy with FileHelpers. It will be easy and high performane.

您使用Linq-2-sql或其他技术进行插入取决于记录的数量。 Linq-2-sql不适合大批量插入,因为它为每个记录生成一个插入语句。我会将SqlBulkCopy与FileHelpers结合起来。它将很容易和高性能。

Afterwards, Linq-2-sql would be fine to query against the table(s) in the database.

之后,Linq-2-sql可以查询数据库中的表。

#1

class LogFileRow
{
  DateTime timestamp{get; set;}
  int ActivityId{get; set;}
  string Message{get; set;}
....
}