When creating an SSIS package it is always a best practice to use variables, and parameters in 2012,  to avoid hard coding values into any part of your package. But there are some best practices involved with creating those variables/parameters also. The rest of this article will refer to variables. But each one could be a parameter except for the variables with expressions. This is because an expression will over write any value passed in.

Let’s look at a common situation in an SSIS package. A common need is to save a file with a date appended to it. This can be done with an expression and a variable. The first example is going to be using one variable without the best practices applied.

Of course we are going to need a file system task to save the file for us. One of the nice features of the file system task is the fact that the rename function can rename and move a file. You simply set the source and destination to a different location. You should never need two file system tasks back to back, with one doing the move and the other doing the rename.

The first thing we are going to do is create a file connection in the connection manager of a package. The file location will be c:\test\test.txt. We are going to back this file up in the folder c:\test\backup and rename the file to test_YYYYMMDD.txt. This will give us today’s date at the end of each file we back up.

This could be a flat file connection also.

The source in the File system task will be set to this connection. We could use a variable to pass this name in if we needed. In this example the destination is the important part. The destination will use a variable so we set the property IsDestinationPathVariable to True.

We will need a variable on the package also. I will create a variable called strFileName. It is always a best practice to create variables with the first few letters of the name describing the data type and camel case the rest of the variable name. Here is a list of some data types and the extensions to use.

Now we set the properties of the variable. Set the property EvaluateAsExpression to true. Then click on the expression ellipse and set the expression to:

“c:\\test\\backup\\test”+ (DT_WSTR, 10) (DT_DBDATE) GETDATE() +”.txt”

Notice the double slashes. These are necessary due to escape characters. For example “\n” is new line. Also notie the double quotes around the literals. Then we simple use three functions. Getdate gives us the date, DT_DBDate trims the time from the date, and DT_WSTR converts the date to a string with 10 characters.

This is an easy way to save a file with the date. But this is not using best practices in SSIS. What if the folder you want to back the file up into changes? What if the file extension changes? Using configuration files or tables you can update these values from outside the package. But if the variable contains an expression then configuration file value would be over written when the expression is evaluated. Therefore it is necessary and a best practice to use multiple variables.

Now I will show the same example using variable best practices in SSIS. We are going to keep the same connection in the connection manager and the source in the file system task will still be set to this connection. But instead of creating one variable for the backup file location we will create several variables. Here they are:

strBackupFolder – The backup location

strFileName – The name of the file without the folder or extension

strFileExtension – The extension of the file

strFullFileName – All the other variables combine in this one

strDate – The date represented as a string

strBackupFolder will be set to “c:\test\backup\”

strFileName will be set to “test”

strFileExtension will be set to “.txt”

strFullFileName will be set to an expression

strDate will be set to an expression

Now for the two expressions, the strDate will be just like the expression from the previous example. We use the same three functions to get a date string: (DT_WSTR, 10) (DT_DBDATE) GETDATE(). If you want the date to be in a different format then you will need to use the function Year, Day, and Month to control the format of the date.

The strFullFileName will simply combine all the other variables together to form a complete file name.

Now each of the variables that do not contain expressions can be updated from outside the package with configuration file or configuration tables and they still play nicely with the expressions in the other variables. In any packages you create try to ask the questions like “What if something changes?” This will save you tons of headaches in the future from maintenance. Always try to combine multiple variables with expressions, do not code everything into one variable.

The Web Service task in SSIS can be used to call a web service command to perform a needed operation in your package. The results of most web services will be in XML Format. You can save these results into a variable or into a file. Either way, you can then parse the XML to find the node values you need. This can be done with the XML task in SSIS. I have a blog on the XML task also.

The Web Service I am using is a free demo service. It allows you to enter a zip code and returns the city and state associated with that zip code. This first thing you need on the package is an HTTP connection. The Server URL for this connection is http://www.webservicex.net/uszip.asmx?op=GetInfoByZIP. The rest of the settings for the connection are default.

Then drag a Web Service Task into the control flow. Set the connection to the HTTP connection you just created. You will need the WSDL file for the web service. This can be downloaded from the website where the web service is hosted. This file will need to be saved locally. Set the WSDL File to the location where you saved the WSDL file in the Web Service task in SSIS.

Click on the input node on the left window pane and set the input properties as shown below. These are drop down menus that are populated automatically by the WSDL file. Create a package variable with a data type of string to hold the zip code. Map that in the fields below as shown.

Under the output node set the Output type to Variable and create a results variable with a datatype of string.

If you enter the zip code of 32065 you will receive back this xml list.

This shows us the proper city and state associated with the zip code we entered. Now we can parse through the XML with an XML task and use this data. I show you how to do this in the next blog here.

The XML Task in SSIS allows you to parse through an XML file and read the nodes in the XML. In a previous blog I showed how to use a web service task to get the city and state when entering a zip code.

The results we got back from the web service were in XML format and we saved them into a variable named Results. This image shows the value in the variable

We need to get the city and the state out of this XML and save them each into a variable. You will need to create an XML task in the Control Flow of a package. Set the operation type to XPATH. The source is going to be the results variable. The destination will be the city variable. The second operand will be the node you want to be read, “//CITY” in this case. Last we set the XPATH Operation to values because you want the value of the city node.

The XML Task that retrieves the State value will be identical except for the Second Operand and the Destination variable.

After all of this is complete the package with the web service task will come before the XML Tasks. I placed a break point on the last XML Task and I am showing the results of the variables in the watch window. You can see that the results contain the XML, the City, and State variables contain the values from their respective nodes.

Packages can be scheduled to run at a time when you expect data to be in a database. Instead of guessing the time when the data will be in the database we can have the package look for data in a SQL table. When the table has data then the package will begin.

First we will create two variables on the package, intDelayTime and intSQLCount, both are integers.Then we drag out a For Loop. Set the InitExpression to @intSQLCOunt = 0 and the EvalExpression to @intSQLCount == 0. This will cause the loop to run until the intSQLCount is not zero.

Drag and drop a Script Task inside the For Loop. Set the intDelayTime as a read only variable. Enter the following code.

Public Sub Main()

‘

Dim sec As Double = Convert.ToDouble(Dts.Variables(“intDelayTime”).Value)

Dim ms As Int32 = Convert.ToInt32(sec * 1000)

System.Threading.Thread.Sleep(ms)

‘

Dts.TaskResult = ScriptResults.Success

End Sub

This saves the time delay variable as an integer and changes it to milliseconds. This will cause the task to pause for the number of second saved in the variable.

Drag in an execute SQL task and connect the delay script task to it with a success constraint. The execute SQL task will look at a the table and do something like a Select Count (*). We will set the result set to single row and save the count in the intSQLCount variable.

Select Count(*) from WatchMe

This will save the row count in the variable and the For Loop will continue until this number is not  zero. Once it is not zero the loop will complete and the rest of the package can run.

If you have two or more records on the same row, and need to write each record on its own row in a destination, you have two choices. You can do this in series or parallel in a single data flow in SSIS. Here is the input table I am using for my example.

Notice I have three names on one row. I need these to be inserted into a table with a first name and last name column only. So all three first name fields need to be mapped to the only first name columns on the destination and the same is true for the last name column.

The first method I will show is using a multicast and a union all as seen below.

The multicast clones the data into three data flow line. In the union all we will now select the first name and last name columns to union together. We are going to stack them as shown and delete the unused columns. This gives us only two columns out of the union all.

This makes the mapping in the destination easy. It is simply two columns to two columns.

The second method is to spilt the data up and write it to the database in parallel. Here is that data flow.

This will write the data to the data base for each customer in a separate destination.

Name 1 Destination mapping.

Name 2 Destination mapping

Name 3 Destination mapping

Here is the data on the destination table after the load. Notice the names are all on individual rows.

In terms of performance, the parallel load works about twice as fast. This is in part due to the union all being a partially blocking transform and the parallel is writing three fields at once. This is a huge performance hit. If the parallel load time is 5 minutes, then the series load time would be about 10 minutes. This may seem small, but scalability should always be considered.

The pivot transform in SSIS is already a pain to use. When you have bad data it makes it even worse. In this blog I am going to show how to handle bad data when using the pivot transform. You should understand how to use the pivot transform already and Devin Knight has a great blog on how to do that here.

Here is the input table:

The output table should look like so:

This is a situation where the users have entered data and they have left off the types on the input table and therefore we do not know where the value should go. These values will be dropped in this example. When loading the output table we need to pivot the data. Another issue is the IDs of the incoming data are sequential and not matching. So IDs 1-5 are the first row, 6-10 are the second row and so on.

Here is the data flow used to perform all of this work.

All of these issues can be handled in SSIS with native task. We will use an aggregate transform in this example. Remember an aggregate transform is an Asynchronous transform and does not perform well if you have a lot of rows. This aggregation could be done with a staging table if that is the case.

Here is the query used to pull the information from the input table:

SELECT ID, isnull(Type,’X’) as Type, [Value]

FROM dbo.PivotInput

This will give us the following table:

In the pivot transform you can create a column to catch all of the X columns. These are the rows missing the type.

After the pivot transform the data will look like the below image in a data viewer:

Here you can see that the data has been pivoted but the ID issue still needs to be resolved. You need to place ID 1-5 on the same row and 6-10 on the same row and make this work for all numbers. You will do this with a derived column and the aggregate transform.

The next transform is the derived column. Here you will create a new ID column with the following expression:

Now after the derived column the data will look like the below image:

Notice now you have a New ID that can be grouped together. The aggregate transform will do this.

Here is how the aggregate transform is set up:

Notice you are dropping the X column. You could do a multi cast before this to map those bad rows to another output like a flat file for someone to examine manually.

After all of this we map it to the output and the table looks like so:

Let me know if you have any weird situations like this. I always love a good challenge.

If you get the error in SSIS that says:

…failed with the following error: “No disconnected record set is available for the specified SQL statement.”. Possible failure reasons: Problems with the query, “ResultSet” property not set correctly, parameters not set correctly, or connection not established correctly.

This is can be due to the record set in an Execute SQL task being set to the wrong result set and the task not returning a dataset. For example, if the Execute SQL task is executing an insert statement, there is no returned dataset. But keep in mind that the insert statement does run and will write the data to the table. In this case the record set should be none.

When running a For Each Loop through a set of files, sometimes you will have specific files that you do not want to load.

For example, I have a set of files named:

Abc.txt
Mno.txt
Rts.txt
Wln.txt
Xyz.txt

If I want to skip the file that starts with “W” then I will need an expression in my For Each Loop to detect this file.

Inside the For Each loop I am going to place a sequence container. This will give me a place to anchor my expression which I will place on the precedence constraint coming from the sequence container. There are no tasks in the sequence container.

On the precedence constraint line I am going to set it to constraint and expression. The expression will be:

substring(Upper(@strFileName),1,1) != “W”

This is looking at the first letter in the filename and comparing it to the letter “W”. I would place the “W” in a variable and use that instead, I am just showing this way for simplicity. Notice I convert the file name variable to upper case and compare it to an uppercase “W”. That way the case will not matter.

The dreaded table lock can occur and cause your SSIS packages to fail. A popular request I receive asks “How can I get rid of these table locks?” This blog will show you how to build a package that will kill any SPID’s that are running on your system that could be locking a table.

Note: Be careful using this technique, you could kill a critical process.

In this package you will have five variables.

objSpids = Holds the data from sp_Who2

strDatabase = Name of the database to look in Spids

strSpid = Current Spid in the for each loop

strSQLKill = Expression: “Kill ” + @[User::strSpid]

strSQLSPWho = Expression

“CREATE TABLE #sp_who2

(SPID INT,

Status VARCHAR(1000) NULL,

Login SYSNAME NULL,

HostName SYSNAME NULL,

BlkBy SYSNAME NULL,

DBName SYSNAME NULL,

Command VARCHAR(1000) NULL,

CPUTime INT NULL,

DiskIO INT NULL,

LastBatch VARCHAR(1000) NULL,

ProgramName VARCHAR(1000) NULL,

SPID2 INT,

REQUESTID int

)

INSERT INTO #sp_who2

EXEC sp_who2

SELECT cast(spid as varchar(10)) as spid

FROM #sp_who2

WHERE DBName = ‘”+ @[User::strDataBase] +”‘

and HostName is not null

and Status <> ‘BACKGROUND’

group by spid

DROP TABLE #sp_who2″

Notice the strSQLSPWho variable holds the query to create the table and put all of the SP_Who data into it. The database name comes from the strDatabase variable.

The first thing you will need to do is get the information from SP_who. This is done with an Execute SQL task. Set the SQL source type to variable and choose the strSQLSPWho variable as the source variable. Set the Results set to Full Results Set. In the Results set pane add a result set and set the name to 0 and the variable to objSpids.

Now you will need to loop through each row in the object variable with the Spids. The For Each Loop will do this. The Enumerator needs to be set to For Each ADO. Select the objSpids variable. Under variable mappings set the variable to strSpid and the index to 0.

Now drop an Execute SQL task in the For Each Loop. Set the SQL source type to variable and choose the strSQLKill variable as the source variable. Leave the Results set to None.

That is it for building the package. The next step is to test the package. Place a breakpoint on the For Each loop. Set this breakpoint to “Break at the beginning of every iteration of the loop”.

Start debugging the package and check the watch window or the locals window to for the value of the variables. To get these windows click on debug >Windows> Locals or Watch1.

Here is the watch window:

If the package is picking up Spids you don’t want you will need to adjust the where clause in strSPWho variable.

Now in SQL 2012 we have parameters that make it easy, but configuration files are still an option and I still see a lot of my clients using them even on 2012 due to several reasons, but mostly because of the work to convert over.

SSIS packages are great ETL tools and can do just about anything you need in terms of ETL. Most organizations start out creating SSIS package one by one until they have dozens, hundreds, or even thousands of packages. I have worked with one client that ran over 4,000 packages. This can be a nightmare to maintain. You can save yourself a lot of work by deciding upfront how to configure your packages using configuration files or tables. We are going to discuss configuration files in this article.

We are going to look at a simple example of passing information to a package with a configuration file. Then we will go over using configuration files on multiple packages. Imagine running dozens of packages that point to a server and the server name changes. If you have a configuration file that is feeding this server name to every package you can make a single change to the configuration file and all the packages are updated. This can reduce your maintenance time significantly.

Here is a simple package example:

1. Drag in a script task into a blank SSIS package.

2. Create a string variable on the package named strData

3. Set the value of the variable to “Package”

4. Double click on the script task.

5. Add the strData variable to the read only variables.

6. Click Edit Script

7. Under the main function add the code MsgBox(Dts.Variables(“strData”).Value)

8. Click save and then close the window

9. Close the script editor by clicking ok

10. Run the package

When you run the package a popup box appears show the work Package. This is the value of the variable saved in the package. Now we will set up a configuration file on the package to give us the ability to change the value of the variable from outside the package.

1. Close the popup box and stop the package.

2. Right click in the control flow and select Package Configurations.

3. Place a check in Enable Package Configurations.

4. Click Add.

5. Click Next in the in the welcome screen if it appears.

6. Click on browse and select a location you have rights to write to.

7. Name the file FirstConfig

8. Click save and then click next

9. Click the plus next to variables >strData > Properties

10. Place a check next to value, notice the value on the right

11. Click Next > Finish > Close

We now have a configuration file on the package but the value is still the value from the package. Now we will open the configuration file and change the value. The configuration file is an XML file and I like to use XML notepad (Free from Microsoft) to open it. You will look for the configured value in this file. This configured value is the value passed to the package.

1. Open the folder containing the configuration file

2. Open the configuration file by right clicking and select open with

3. Select a program you can use to edit the file (Example: Notepad, Wordpad, XML notepad)

4. Check the configured value from Package to Config

5. Save and Close the File

6. Return to the package and run it

You should see a popup showing Config. This is the value from the configuration file. The value saved in the package is overwritten at run time.

This is just one small example of using configuration files. A popular way to use configuration files is on connections. When you have a connection on a package the properties of this connection show in the configuration manager. You can place a check next to the connection string property or you can place a check next to the individual elements that make up the connection string, initial catalog, server name, user name, and password. The user name and password are not needed when using windows authentication.

The password is not stored in the configuration file automatically even if you select it in the configuration manager. This is done by design for security. Microsoft did not want you saving your configuration file in plain text without knowing it. So you will have to open the configuration file and add the password. If you selected the connection string the password will go right after the user name. You must type in “Password =####;”, (#### Represents your password). Don’t forget the semicolon after the password. Now this configuration file can be used in any other package using this connection.

There is an issue when using a configuration file in multiple packages. The package that is using the configuration files will try to load every connection in the configuration file. It the package does not contain that connection it will fail validation and the package will not run. This causes an issue when trying to share a configuration file with many packages. There are three methods for handling these issues. You can create a configuration file for each package or create a configuration file for each connection or a combination of both.

The first method of a configuration file for each package works well if you do not have a lot of packages. If you have a thousand connections and fifty packages, a per-package solution is the obvious choice. If every package has a different set of connections this is almost necessary.

The second method of a configuration file for each connection works well if you have a lot of packages and fewer connections. If you have fifty connection and a thousand packages it will be much easier to maintain a per connection solution. In this situation a package with ten connections would have ten configuration files, each with one connection.

The third option is to combine the first two options in some form. For example, if you have one connection that is used by every package use this configuration file in every package. The other connections can have a package level configuration file. This is harder to maintain and you need document which packages are using which configuration files.

With all the options of configuration files it is important to plan out how you will use them in your environment before you create thousands of packages and create a maintenance nightmare. Planning your SSIS package configuration architecture is important and should not be over looked. It is easy to put it off when you only have a couple of packages. Most environments have their packages grow in number faster than anticipated. Planning your configuration files will save you a huge retro fit project in the future.

When creating an SSIS package it is always a best practice to use variables, and parameters in 2012,  to avoid hard coding values into any part of your package. But there are some best practices involved with creating those variables/parameters also. The rest of this article will refer to variables. But each one could be a parameter except for the variables with expressions. This is because an expression will over write any value passed in.

Let’s look at a common situation in an SSIS package. A common need is to save a file with a date appended to it. This can be done with an expression and a variable. The first example is going to be using one variable without the best practices applied.

Of course we are going to need a file system task to save the file for us. One of the nice features of the file system task is the fact that the rename function can rename and move a file. You simply set the source and destination to a different location. You should never need two file system tasks back to back, with one doing the move and the other doing the rename.

The first thing we are going to do is create a file connection in the connection manager of a package. The file location will be c:\test\test.txt. We are going to back this file up in the folder c:\test\backup and rename the file to test_YYYYMMDD.txt. This will give us today’s date at the end of each file we back up.

This could be a flat file connection also.

The source in the File system task will be set to this connection. We could use a variable to pass this name in if we needed. In this example the destination is the important part. The destination will use a variable so we set the property IsDestinationPathVariable to True.

We will need a variable on the package also. I will create a variable called strFileName. It is always a best practice to create variables with the first few letters of the name describing the data type and camel case the rest of the variable name. Here is a list of some data types and the extensions to use.

Now we set the properties of the variable. Set the property EvaluateAsExpression to true. Then click on the expression ellipse and set the expression to:

“c:\\test\\backup\\test”+ (DT_WSTR, 10) (DT_DBDATE) GETDATE() +”.txt”

Notice the double slashes. These are necessary due to escape characters. For example “\n” is new line. Also notie the double quotes around the literals. Then we simple use three functions. Getdate gives us the date, DT_DBDate trims the time from the date, and DT_WSTR converts the date to a string with 10 characters.

This is an easy way to save a file with the date. But this is not using best practices in SSIS. What if the folder you want to back the file up into changes? What if the file extension changes? Using configuration files or tables you can update these values from outside the package. But if the variable contains an expression then configuration file value would be over written when the expression is evaluated. Therefore it is necessary and a best practice to use multiple variables.

Now I will show the same example using variable best practices in SSIS. We are going to keep the same connection in the connection manager and the source in the file system task will still be set to this connection. But instead of creating one variable for the backup file location we will create several variables. Here they are:

strBackupFolder – The backup location

strFileName – The name of the file without the folder or extension

strFileExtension – The extension of the file

strFullFileName – All the other variables combine in this one

strDate – The date represented as a string

strBackupFolder will be set to “c:\test\backup\”

strFileName will be set to “test”

strFileExtension will be set to “.txt”

strFullFileName will be set to an expression

strDate will be set to an expression

Now for the two expressions, the strDate will be just like the expression from the previous example. We use the same three functions to get a date string: (DT_WSTR, 10) (DT_DBDATE) GETDATE(). If you want the date to be in a different format then you will need to use the function Year, Day, and Month to control the format of the date.

The strFullFileName will simply combine all the other variables together to form a complete file name.

Now each of the variables that do not contain expressions can be updated from outside the package with configuration file or configuration tables and they still play nicely with the expressions in the other variables. In any packages you create try to ask the questions like “What if something changes?” This will save you tons of headaches in the future from maintenance. Always try to combine multiple variables with expressions, do not code everything into one variable.

The Web Service task in SSIS can be used to call a web service command to perform a needed operation in your package. The results of most web services will be in XML Format. You can save these results into a variable or into a file. Either way, you can then parse the XML to find the node values you need. This can be done with the XML task in SSIS. I have a blog on the XML task also.

The Web Service I am using is a free demo service. It allows you to enter a zip code and returns the city and state associated with that zip code. This first thing you need on the package is an HTTP connection. The Server URL for this connection is http://www.webservicex.net/uszip.asmx?op=GetInfoByZIP. The rest of the settings for the connection are default.

Then drag a Web Service Task into the control flow. Set the connection to the HTTP connection you just created. You will need the WSDL file for the web service. This can be downloaded from the website where the web service is hosted. This file will need to be saved locally. Set the WSDL File to the location where you saved the WSDL file in the Web Service task in SSIS.

Click on the input node on the left window pane and set the input properties as shown below. These are drop down menus that are populated automatically by the WSDL file. Create a package variable with a data type of string to hold the zip code. Map that in the fields below as shown.

Under the output node set the Output type to Variable and create a results variable with a datatype of string.

If you enter the zip code of 32065 you will receive back this xml list.

This shows us the proper city and state associated with the zip code we entered. Now we can parse through the XML with an XML task and use this data. I show you how to do this in the next blog here.

The XML Task in SSIS allows you to parse through an XML file and read the nodes in the XML. In a previous blog I showed how to use a web service task to get the city and state when entering a zip code.

The results we got back from the web service were in XML format and we saved them into a variable named Results. This image shows the value in the variable

We need to get the city and the state out of this XML and save them each into a variable. You will need to create an XML task in the Control Flow of a package. Set the operation type to XPATH. The source is going to be the results variable. The destination will be the city variable. The second operand will be the node you want to be read, “//CITY” in this case. Last we set the XPATH Operation to values because you want the value of the city node.

The XML Task that retrieves the State value will be identical except for the Second Operand and the Destination variable.

After all of this is complete the package with the web service task will come before the XML Tasks. I placed a break point on the last XML Task and I am showing the results of the variables in the watch window. You can see that the results contain the XML, the City, and State variables contain the values from their respective nodes.