I have Multiple CSV files in Folder
我在文件夹中有多个CSV文件
Example :
示例：
Member.CSv
Leader.CSv
I need to load them in to Data base tables .
我需要将它们加载到数据库表中。
I have worked on it using ForEachLoop Container ,Data FlowTask, Excel Source and OLEDB Destination
我使用ForEachLoop Container，Data FlowTask，Excel Source和OLEDB Destination进行了研究
we can do if by using Expressions and Precedence Constraints but how can I do using Script task if I have more than 10 files ..I got Stuck with this one
如果使用表达式和优先级约束我们可以做，但如果我有超过10个文件，我怎么能使用脚本任务..我得到了这个
1 个解决方案
#1
0
We have a similar issue, our solution is a mixture of the suggestions above.
我们有类似的问题，我们的解决方案是上述建议的混合。
- We have a number of files types sent from our client on a daily basis.
- 我们每天都会从客户端发送许多文件类型。
- These have a specific filename pattern (e.g. SalesTransaction20160218.csv, Product20160218.csv)
- 它们具有特定的文件名模式（例如SalesTransaction20160218.csv，Product20160218.csv）
- Each of these file types have a staging "landing" table of the structure you expect
- 这些文件类型中的每一种都具有您期望的结构的登台“登陆”表
- We then have a .net script task that takes the filename pattern and loads that data into a landing table.
- 然后我们有一个.net脚本任务，它接受文件名模式并将该数据加载到登陆表中。
- There are also various checks that are done within the csv parser - matching number of columns, some basic data validation, before loading into the landing table
- 在加载到登陆表之前，还会在csv解析器中完成各种检查 - 匹配列数，一些基本数据验证
We are not good enough .net programmers to be able to dynamically parse an unknown file structure, create SQL table and then load the data in. I expect it is feasible, after all, that is what the SSIS Import/Export Wizard does (with some manual intervention)
我们不够好.net程序员能够动态解析未知的文件结构，创建SQL表然后加载数据。我希望它是可行的，毕竟这是SSIS导入/导出向导所做的（与一些人工干预）
As an alternative to this (the process is quite delicate), we are experimenting with a HDFS data landing area, then it allows us to use analytic tools like R to parse the data within HDFS. After that utilising PIG to load the data into SQL.
作为替代方案（该过程非常精细），我们正在试验HDFS数据着陆区域，然后它允许我们使用像R这样的分析工具来解析HDFS中的数据。之后利用PIG将数据加载到SQL中。
#1
0
We have a similar issue, our solution is a mixture of the suggestions above.
我们有类似的问题，我们的解决方案是上述建议的混合。
- We have a number of files types sent from our client on a daily basis.
- 我们每天都会从客户端发送许多文件类型。
- These have a specific filename pattern (e.g. SalesTransaction20160218.csv, Product20160218.csv)
- 它们具有特定的文件名模式（例如SalesTransaction20160218.csv，Product20160218.csv）
- Each of these file types have a staging "landing" table of the structure you expect
- 这些文件类型中的每一种都具有您期望的结构的登台“登陆”表
- We then have a .net script task that takes the filename pattern and loads that data into a landing table.
- 然后我们有一个.net脚本任务，它接受文件名模式并将该数据加载到登陆表中。
- There are also various checks that are done within the csv parser - matching number of columns, some basic data validation, before loading into the landing table
- 在加载到登陆表之前，还会在csv解析器中完成各种检查 - 匹配列数，一些基本数据验证
We are not good enough .net programmers to be able to dynamically parse an unknown file structure, create SQL table and then load the data in. I expect it is feasible, after all, that is what the SSIS Import/Export Wizard does (with some manual intervention)
我们不够好.net程序员能够动态解析未知的文件结构，创建SQL表然后加载数据。我希望它是可行的，毕竟这是SSIS导入/导出向导所做的（与一些人工干预）
As an alternative to this (the process is quite delicate), we are experimenting with a HDFS data landing area, then it allows us to use analytic tools like R to parse the data within HDFS. After that utilising PIG to load the data into SQL.
作为替代方案（该过程非常精细），我们正在试验HDFS数据着陆区域，然后它允许我们使用像R这样的分析工具来解析HDFS中的数据。之后利用PIG将数据加载到SQL中。