ynamic SSIS Package to Import Excel files into SQL Server Database

Problem/Scenario
Need to import Excel files to a SQL table. Everyday one file is created at specified location in
the source system. We need to copy that file from Source System to Local system and then load
to SQL Table.
Conditions:
1. Each file should be loaded only once. Everynight ob should be executed to load data into
reporting !ata mart.
". Source system #ill maintain all the history files so files at souce should not be deleted.
$. %f ob failed due to some reason &schema changes' server do#n' connection issues etc.(' it
should load all the files from last run date to current date in next successul run. )or example' *ob
didn+t run last one #ee, then #henever *ob runs next time successfully' it should load current file
as #ell as all the files of last #ee, #hich #ere missing.
-. .ll the source files #ill have the same structure &Schema(
/. Nomenclature 0 Each file #ill have name Transaction follo#ed by current date in 11110
220!! format. )or example' if a file #as created on 310.ug0"313 then its name should be
Transaction"313034031.
Soltion
% #ill ta,e advantage of the )orEach Loop 5ontainer. %+ll create a ne# SS%S pac,age to solve the
above problem.
STE6 17
5reate follo#ing tables in your local database &destination database(.
CREATE TABLE [TransactionHistory](
[TransactionID] [int] NOT NULL,
[ProductID] [int] NOT NULL,
[ReerenceOrderID] [int] NOT NULL,
[TransactionDate] [dateti!e] NOT NULL,
[TransactionTy"e] [nc#ar]($% NOT NULL,
[&uantity] [int] NOT NULL,
[Actua'Cost] [!oney] NOT NULL,
CON(TRAINT [P)*TransactionHistory*TransactionID]
PRI+AR, )E, CLU(TERED ([TransactionID] A(C%
% ON [PRI+AR,]
-O
CREATE TABLE [E.ce'/i'esLo0](
[/i'eNa!e] [1arc#ar]($22%,
[/i'eCreatedDate] [dateti!e],
[/i'eLoadDate] [dateti!e],
[RecordCount] [int],
% ON [PRI+AR,]
-O
STE6 "7 5reate a ne# SS%S pac,age and rename it to Load!ynamicExcel)iles.dtsx
STE6 $7 .dd follo#ing pac,age variables7
!ariable"ame Description Examle
Last8un!ate To store last run date "3130390$3
5urrent!ate To hold running date "313034013
!irectory
Source
To store source
directory path
!7:SS%S:;ari:Source)iles
!irectory
!estination
To store local
directory path
!7:SS%S:;ari:!estination)iles
5urrent)ile
Name
To store current
file name
!7:SS%S:;ari:Source)iles:
Transaction"313034031
5reate one #LE D$ connection &% #ill use %local&'(est)" connection manager( for local
database #here you #ant to load excel files data. 5reate one Excel Connection *anager for
excel files located in !7:SS%S:;ari:Destination+iles. .t least one file should be there to create
excel connection manager.
5lic, on Excel 5onnection 2anager 00< go to 6roperties #indo# 00< Select Expression and set
Excel+ilePat, #ith pac,age variable -ser::Crrent+ile"ame as sho#n belo#7
)or this article' %+ll use t#o directories 0 one for sources files and one for destination files.
Location of these files are given belo#7
Sorce +iles: !7:SS%S:;ari:Source)iles
Destination +iles: !7:SS%S:;ari:!estination)iles
% have created fe# excel files from Prodction'(ransaction)istory table of
.dventre/orks011230 database as sho#n belo#7
% have created files from "3130390"3 to "313034031. .fter executing the pac,age first time' % #ill
create files from "31303403" to "31303403= to test the pac,age.
STE6 -7
!rag and drop +ile System (ask and double clic, to open File System Task Editor. Enter +S( 4
Delete destination directory content in name and select Delete directory content as
>peration. Set %sSource6ath?ariable to (re and select Source?ariable as
-ser::DirectoryDestination. )inally clic, on >@ and save changes.
!rag and drop Execte SQL (ask and double clic, to open Execute SQL Task Editor. Enter
Execute SQL Tas, 0 Aet Last8un!ate in Name' select Single ro# as 8esult Set' 5onection Type
as >LE !B and 5onnection as &local(.Test;N and SQLSourceType as !irect input. Enter belo#
Cuery in SQLStatement7
(ELECT I(NULL(+A3([/i'eCreatedDate]%,452$262$62$4% A( LastRunDate
/RO+ [d7o]8[E.ce'/i'esLo0] (NOLOC)%
%n 3eslt Set tab' set 8esult Name 3 to variable -ser::Last3nDate.
)inally clic, >@ and save changes.
!rag and drop Script (ask and double clic, to open Script Task Editor. Select
Dser77!irectory!estination' Dser77!irectorySource' Dser77Last8un!ate in 3ead#nly!ariables.
5lic, on Edit Script''' and paste belo# code7
usin0 (yste!8IO9
"u7'ic 1oid +ain(%
:
try
:
strin0 Directory(ource ;
Dts8<aria7'es[=User>>Directory(ource=]8<a'ue8To(trin0(%9
strin0 DirectoryDestination ;
Dts8<aria7'es[=User>>DirectoryDestination=]8<a'ue8To(trin0(%9
DateTi!e LastRunDate ;
(DateTi!e%Dts8<aria7'es[=User>>LastRunDate=]8<a'ue9
strin0 i'eNa!e,i'eE.tension9
strin0[] i'es ; Directory8-et/i'es(Directory(ource%9
??Co"y source i'es to destination
oreac# (strin0 in i'es%
:
i'eNa!e ; Pat#8-et/i'eNa!e(%9
i'eE.tension ; Pat#8-etE.tension(%9
DateTi!e CurrentDate ;
DateTi!e8Parse(i'eNa!e8(u7strin0(=Transaction=8Len0t#, $2%%9
i ((DateTi!e8Co!"are(CurrentDate,LastRunDate%@2% AA (i'eE.tension
;; =8.'s.=%%
:
??+essa0eBo.8(#oB(i'eNa!e8To(trin0(%%9
/i'e8Co"y(, Pat#8Co!7ine(DirectoryDestination, i'eNa!e%, true%9
C
C
Dts8TasDResu't ; (int%(cri"tResu'ts8(uccess9
C
catc# (E.ce"tion e.%
:
Dts8Lo0(e.8+essa0e, 2, nu''%9
Dts8TasDResu't ; (int%(cri"tResu'ts8/ai'ure9
C
C
!rag and drop )oreach Loop container. Select +oreac, +ile Enmerator as Enumerator' enter
D:5SSIS5)ari5Destination+iles in )older path and 6'xlsx in files textbox. Select +lly 7alified
as 8etrieve file name. %n ?ariable 2appings' Select -ser::Crrent+ile"ame for %ndex 3 to
store current file name for each iteration.
No# drag and drop !ata )lo# Tas, inside )oreach loop container. Dse Excel Sorce reader to
read excel files from destination directory. Dse Excel Connection *anager as connection
manager for excel files. %n connection properties' Select 89-ser::Crrent+ile"ame: as
Excel)ile6ath. Dse Data Conversion' if reCuired. Dse 3o; Cont (ask to count number of
ro#s in data flo# and store it in -ser::3ecordCont variable. Dse #LE D$ Destination to
load data into SQL table.
!rag and drop Execte SQL (ask inside )oreach loop container to log information about
current file.
!ouble clic, on Execute SQL Tas, to open Execute SQL Tas, Editor' enter Execte SQL (ask
4 Insert info into Log table as Name' "one as 8esult Set' #LE D$ as connection type'
%local&'(est)" as connection' Direct inpt as SQLSourceType and belo# Cuery as
SQLStatement.
DECLARE
E/i'eNa!e 1arc#ar(F22%
,E/i'ePat# 1arc#ar(F22%
,EDestinationPat# 1arc#ar(F22%
(ET EDestinationPat# ; G
(ET E/i'ePat# ; G
(ET E/i'eNa!e ; REPLACE(E/i'ePat#,EDestinationPat# H 4I4,44%
IN(ERT INTO [E.ce'/i'esLo0]
(
[/i'eNa!e]
,[/i'eCreatedDate]
,[/i'eLoadDate]
,[RecordCount]
%
(ELECT
E/i'eNa!e [/i'eNa!e]
,CA(T((UB(TRIN-(E/i'eNa!e,$5,$2% as dateti!e% [/i'eCreatedDate]
,-ETDATE(% [/i'eLoadDate]
,G
%n 6arameter mapping' map Dser77!irectory!estination' Dser775urrent)ileName' and
Dser778ecord5ount #ith parameter 3'1'" respectively as sho#n belo#.
)inally clic, >@ and save changes.
$elo; is t,e final layot of or package:
STE6 E7
We are done #ith the pac,age development. To execute the pac,age' go to SS%S pac,age
location' &in this example' it is D:\SSIS\Hari\Sample\SSIS-Sample1(' right clic, on
LoadDynamicExcel+iles'dtsx 00< #pen ;it, 00< SQL Server 0112 Integration Services
Package Exection -tility. This #ill open Execute Packae !tility. 5lic, on Execte button to
run ssis pac,age. No# you can see the progress of pac,age execution in Packae Executio"
Proress #indo#.
No# you can chec, Excel+ileLog table to cross chec, the result of pac,age.
No# %+ll add fe# more excel files in source location 0 from
Transaction"31303403" to Transaction"31303403= as sho#n belo#7
When % execute this pac,age next time' it #ill load only ne# files. 1on can chec, Excel)ileLog
for each iteration7