| As I explained earlier about For Each Loop Container, which process each file from selected folder. There is a requirement to save File name along with record, so later on we can identify which record comes from which file. Let me explain it how to achieve with FileNameColumnName property of Flat File Connection to get it easily. 1. Setup Flat File Connection with CSV file, as mentioned Basic Example of Data Flow Task. 2. Right click on Flat File Source, and click on Show Advanced Editor: 3. Click on "Component Properties" and go to FileNameColumnName, Custom properties: 4. Setup FileNameColumnName value with desired column name. Let's say "File Name". Congratulations, This column is added to output list with actual filename of that connection. |
Learn SQL and database management at SQLYoga for articles, tutorials, and tips to improve your skills and streamline data operations. Join our community!
April 6, 2010
SQL SERVER SSIS: Get File Name with Flat File Source, Data Flow Component
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
March 7, 2010
SQL SERVER: Execute Stored Procedure when SQL SERVER is started
We have a requirements to execute Stored Procedure when SQL SERVER is started/restarted and we need to start some processes. I found that SQL SERVER provides a way to call Stored Procedure when SQL services are restarted.
SQL SERVER provides this SP: "sp_procoption", which is auto executed every time when SQL SERVER service has been started. I found this SP and it helps me to figure it out the solution for the request as following way.
Let me show you how to use it. Syntax to use SP:
EXEC SP_PROCOPTION
@ProcName = 'SPNAME',
@OptionName = 'startup',
@OptionValue = 'true/false OR on/off'
- @ProcName, should be Stored procedure name which should be executed when SQL SERVER is started. This stored procedure must be in "master" database.
- @OptionName, should be "startup" always.
- @OptionValue, this should be set up to execute this given sp or not. If it is "true/on", given sp will be execute every time when SQL SERVER is started. If it is "false/off", it will not.
That's it, I hope this is very clear to use this feature.
Reference : Tejas Shah (http://www.SQLYoga.com)
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
January 10, 2010
SQL SERVER: SSIS - Derived Column Data Flow Transformation
| As I explained earlier about Foreach Loop Container. One of regular reader of blog send me an email about one issue. Let me share that problem with all readers. With this example, Foreach Loop Container, What to do if we want to save file name along with each row, so we can come to know that which row is from which file ? This is very practical problem that we need to fix. To solve this, I come up with following solution. 1. I used "Derived Column", one of Data Flow Transformations in Data Flow Operations. 2. Configure Derived Column: As we have variable, FileName, as defined in, SQL SERVER: SSIS - Foreach Loop Container. Here I used that variable as a new column. By dragging that User variable to Expression. By default it assign UNICODE STRING DataType to this new column. We need to change it by: A. Right click on "Derived Column", Go to Show Advanced Editor B. Set DataType to String as: 3. That's it. Now just add it to Destination Column Mapping with your Database column. Let me know your suggestions. |
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
December 7, 2009
SQL SERVER: SSIS - Transfer Jobs Task
The Transfer Jobs task can be configured to transfer all jobs, or only specified jobs. You can also indicate whether the transferred jobs are enabled at the destination.
The jobs to be transferred may already exist on the destination. The Transfer Jobs task can be configured to handle existing jobs in the following ways:
- Overwrite existing jobs.
- Fail the task when duplicate jobs exist.
- Skip duplicate jobs.
1. Select and Drag, Transfer Jobs Task, from Container Flow Items to designer surface.
2. To configure a task, Right click on Transfer Jobs Task, which we dragged to Design surface. Click on "Edit.", you will get page as:
3. SSIS Transfer Jobs Task - General : Here we need to assign unique name to this task and also we can specify brief description, so we will get idea why we need to design this task.
4. SSIS Transfer Jobs Task - Jobs : Jobs page of the Transfer Jobs Task Editor dialog box is required to specify properties for copying one or more SQL Server Agent jobs from one instance of SQL Server to another.
Let's take a view how each properties are used.
SourceConnection: Select a SMO connection manager in the list, or click <New connection...> to create a new connection to the source server
DestinationConnection: Select a SMO connection manager in the list, or click <New connection...> to create a new connection to the destination server.
JobsList: Click the browse button (.) to select the jobs to copy. At least one job must be selected.
FailTask: If job of the same name already exists on the Destination Server then task will fail.
Overwrite: If job of the same name already exists on the Destination Server then task will overwrite the job.
Skip: If job of the same name already exists on the Destination Server then task will skip that job.
TRUE: Enable jobs on destination server.
FALSE: Disable jobs on destination server.
5. SSIS Transfer Jobs Task - Expressions: Click the ellipsis to open the Property Expressions Editor dialog box.
Property expressions update the values of properties when the package is run. The expressions are evaluated and their results are used instead of the values to which you set the properties when you configured the package and package objects. The expressions can include variables and the functions and operators that the expression language provides.
Now let's run task, by right click on task and click on Execute Task, as shown in following figure. You can either Execute Package by right click on Package name, from Solution Explorer.
Once you run this then all/selected jobs will be transferred to destination server as per given criteria.
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
December 3, 2009
SQL SERVER: How to Read Excel file by TSQL
| Many times developers asked that, they want to import data from Excel file. We can do this by many ways with SQL SERVER. 1. We can use SSIS package 2. Import/Export Wizard 3. T-SQL Today, I am going to explain, How to import data from Excel file by TSQL. To import Excel file by TSQL, we need to do following: 1. Put Excel file on server, means we need to put files on server, if we are accessing it from local. 2. Write following TSQL, to read data from excel file SELECT Name, Email, PhoneFROM OPENROWSET('Microsoft.Jet.OLEDB.4.0','Excel 8.0;Database=C:\SQLYoga.xls', [SQL$]) NOTE: Here, Excel file is on "C:\" named "SQLYoga.xls", and I am reading sheet "SQL" from this excel file If you want to insert excel data into table, INSERT INTO [Info]SELECT Name, Email, PhoneFROM OPENROWSET('Microsoft.Jet.OLEDB.4.0','Excel 8.0;Database=C:\test\SQLYoga.xls', [SQL$]) That's it. |
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
December 2, 2009
SQL SERVER: How to remove cursor
| Many times developer ask me that How can they remove Cursor? They need to increase Query Performance, that's why they need to remove SQL SERVER Cursor and find the alternate way to accomplish the same. Please find this code to remove cursor with Table variable: --declare table to keep records to be processedDECLARE @Table AS TABLE(AutoID INT IDENTITY, Column1 VARCHAR(100), Column2 VARCHAR(100))--populate table variable with data that we want to processINSERT INTO @Table(Column1, Column2)SELECT Column1, Column2FROM <Table>WHERE <Conditions>--declare variables to process each recordDECLARE @inc INT, @cnt INT--Assign increment counterSELECT @inc = 1--Get Number of records to be processedSELECT @cnt = COUNT(*)FROM @TableWHILE @inc <= @cnt BEGIN--As we have AutoID declared as IDENTITY, it always get only one record.--Get values in Variable and process it as you want.SELECT @Column1 = Column1,@Column2 = Column2FROM @TableWHERE AutoID = @inc--do your calculation here........--Select next recordSET @inc = @inc = 1END By this way, we can remove CURSOR by Table variable. It is quite easy to implement. One more benefit is: It will process one record at a time, so it locks only that record at a time. Let me know if you have any questions. |
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
November 28, 2009
SQL SERVER SSIS: How to assign Connection from variable
| Last Article, SSIS - Foreach Loop Container, We need to assign dynamic connection to file connection, so SSIS For each loop Task can take each file from folder. Lets configure File connection from variable for SSIS - Foreach Loop Container. What we need to do is, we need to process each file from folder, so we need to assign value from variable to File connection, so SSIS Task will read that file and process that file. To assign FileConnection dynamicaly we need to do following. 1. Right click on File Connection, click Properties. 2. Set DelayValidation = "False", as we need to assign connection dynamically. 3. Click on "Expression", and enter variable name, which we used in SSIS - Foreach Loop Container. That's it. It will assign connection from variable and process that file. Let me know if you have any question for the same. |
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
November 2, 2009
SQL SERVER SSIS - For Loop Container
Today, I am going to explain SQL SERVER SSIS, For Loop Container.
The For Loop container defines a repeating control flow in a package. The loop implementation is the same concept of the For looping structure in programming languages. In each repeat of the loop, the For Loop container evaluates an expression and repeats its workflow until the expression evaluates to False.
The For Loop container uses the following elements to define the loop:
- An optional initialization expression that assigns values to the loop counters.
- An evaluation expression that contains the expression used to test whether the loop should stop or continue.
- An optional iteration expression that increments or decrements the loop counter.
Let's take an example to easily understand how to use For Loop Container with SSIS. Here I take example to iterate ActiveX Task.
1. Select and Drag, For Loop Container, from Container Flow Items to designer surface and add ActiveX Script Task to run inside the loop.
2. To configure this container, right click on this and click on 'edit'.
First we need to create variables to run this package based on variables. We can create variables by: View -> Other Windows -> Variables. Please find variable screen as below:
Here, I specified both variables. Count and Increment, that I am going to use for this example. I specified value Count = 20. So loop will be executed 20 times.
Let's take a view how each properties are used:
For Loop Properties:
1. InitExpression: Type an Initialization Expression in the given textbox. Initialization ensures that we are starting by setting out increment counter to 1. Here I specified variable to 1.
2.EvalExpression: Type an Evaluation Expression in the given textbox. For each iteration the evaluation expression checks to see if we have reached our maximum iteration count as set above when we defined @Counter. Here I specified that @Increment <= @Count, code inside the for loop will execute @Count times.
3. AssignExpression: Type an Assignment Expression in the given textbox. This is used to increment the counter by one for each iteration of the loop, otherwise the loop would never finish. Here I specified to increment variable by 1.
That's it. We have configured SSIS For Loop container.
Now when we execute this package, it will execute ActiveX Task, 20 times (as specified in variable count).
Let me know if you have any questions.
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
October 24, 2009
SQL SERVER SSIS: Basic Example Of Data Flow Task
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
October 22, 2009
SQL SERVER: What is SSIS?
| SSIS (SQL Server Integration Services) is an upgrade of DTS (Data Transformation Services), which is a feature of the previous version of SQL Server SSIS (SQL Server Integration Services) is introduced in SQL SERVER 2005. It is totally new transformation of data. Previously we have DTS package in SQL SERVER 2000. Now they introduced SSIS with many features. SSIS is an extensive data integration platform which is used to transfer and merge the information from various sources and load to multiple systems. It contains wide range of capabilities. Business Intelligence is used for decision making, future trends based on collected data. I can find very good article written by Microsoft SQL SERVER MVP, Pinalkumar Dave. He write about BI on his one of the best article: http://blog.sqlauthority.com/2009/10/21/sql-server-introduction-to-business-intelligence-important-terms-definitions/. SSIS packages can be created in BIDS (Business Intelligence Development Studio). I am going to explain all features of SSIS for easy and better understanding by following articles. Reference: Tejas Shah (http://www.SQLYoga.com ) |
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
August 12, 2009
SQL SERVER: Check if Node exists in XML or not
| Today, I have one requirement to check dynamically if a node exists in my xml or NOT. I have a stored procedure that receives XML and I need to check if the message information xml contains one Node or NOT. If that node exists then I need to execute that Stored Procedure by different logic and if not it should run with different logic. I figure it out by using EXISTS. This is my XML, that I got as parameter. Now I need to check if "BulkData" node exists in XML, then I need to write different logic to get the result.DECLARE @ExportData XMLSELECT @ExportData ='<Data Number="A123"><BulkData><EachData Parts="Test1" /><EachData Parts="Test2" /><EachData Parts="Test3" /></BulkData></Data>' So, I used this This will return "1" if node is exists else return "0".SELECT @ExportData.exist('(//BulkData)') That's it. I can write based on the return result by this statement. Let me know if it helps you. Reference : Tejas Shah(http://www.SQLYoga.com) |
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
July 31, 2009
SQL SERVER: Use CONTEXT_INFO
CREATE TABLE tblA( ID INT IDENTITY, ColVal VARCHAR(100) )
CREATE PROC TestA
AS
BEGIN
INSERT INTO tblA(ColVal)
SELECT 'Allow To insert'
END
CREATE PROC TestB
AS
BEGIN
DECLARE @UID VARBINARY(128)
SELECT @UID = CAST('TestB' AS VARBINARY(128))
SET CONTEXT_INFO 0x5465737442
INSERT INTO tblA(ColVal)
SELECT 'Not Allow To insert'
END
CREATE TRIGGER trg_TblA
ON tblA
AFTER INSERT,DELETE,UPDATE
AS
BEGIN
SET NOCOUNT ON;
DECLARE @Message varbinary(128)
SELECT @Message = cast('TestB' as varbinary(128))
IF @Message = CONTEXT_INFO() BEGIN
RAISERROR('Not Allowed to Insert/Update/Delete from SP: TestB',15,1)
ROLLBACK TRAN
END
END
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
July 15, 2009
SQL SERVER: Reset Setup Values, when SQL SERVER is started/restarted
| We have a requirements to clear all setup values when SQL SERVER is started/restarted and we need to setup default values to setup table. I found one Stored Procedure provided by MS SQL SERVER. Let me share it with all of you. SQL SERVER provides this SP: “sp_procoption”, which is auto executed every time when SQL SERVER service has been started. I found this SP and it helps me to figure it out the solution for the request as following way. Let me show you how to use it Syntax use this SP: EXEC SP_PROCOPTION @ProcName, should be Stored procedure name which should be executed when SQL SERVER is started. This stored procedure must be in “master” database. @OptionName, should be “startup” always. @OptionValue, this should be set up to execute this given sp or not. If it is “true/on”, given sp will be execute every time when SQL SERVER is started. If it is “false/off”, it will not. That’s it, lets take an example. I have one Database called Test, I have created setup table: CREATE TABLE SetupTable( Lets insert some default values to this table: INSERT INTO SetupTable VALUES('A') What I need to do is, I need to wipe out this values when SQL SERVER is started and fill it with the same default values, because these values might be updated by application.So, I created one stored procedure in master database, named, CREATE PROC ClearAllData and set up this stored procedure as auto executed every time when SQL SERVER is started as: EXEC SP_PROCOPTION Now, restart SQL SERVICES, and you find that old values will be deleted and new values with ‘X’, ‘Y’, and ‘Z’ will be inserted automatically. If now you want to stop it to execute automatically, we just need to execute this with “false” as: EXEC SP_PROCOPTION I hope this is very clear to use this feature. Reference : Tejas Shah (http://www.SQLYoga.com) |
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
May 12, 2009
SQL SERVER: Configure Database Mail with SQL SERVER 2005
We used Database mail to send mail to client on each updates.
This is a very simple process to configure. Let me share how to configure Database mail with sql server 2005 with all of you.
After setting up Profile and Account properly, you just need to write following code to send a mail to client:
Step 1:
Step 2:
Step 3:
Step 4: You might get this message:
Step 5: Create Profile
Step 6 : Create Account
That's it.
exec msdb.dbo.sp_send_dbmail
@profile_name = 'ProfileName', IN our CASE, 'Tejas'
@recipients = 'Client Email Address' ,
@blind_copy_recipients = 'BCC Address',
@subject = 'Subject',
@BODY = 'Message Body',
@body_format = 'Message Type', it could be text OR html
Let me know if you have any complexity or comments in setting up Database mail.
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
April 20, 2009
SQL SERVER: Get Result ORDER BY Time regardless Date on SQL DateTime column
Usually we save Time with Dates in DATETIME column.
Today, I came across situation, where I need to sort my result set by Time, regardless the Date.
I have some sample data like this:
DECLARE @Data TABLE(dt DATETIME)
INSERT INTO @Data(dt)
SELECT '2008-12-05 04:00:00.000'
UNION ALL
SELECT '2008-12-10 10:00:00.000'
UNION ALL
SELECT '2009-03-01 08:00:00.000'
UNION ALL
SELECT '2009-03-02 07:15:00.000'
UNION ALL
SELECT '2009-03-10 08:50:00.000'
UNION ALL
SELECT '2008-12-31 23:00:00.000'
UNION ALL
SELECT '2009-05-01 21:10:00.000'
SELECT * FROM @Data
I found very quick solution for this. You can create query as follows:
SELECT *
FROM @Data
ORDER BY Convert(VARCHAR, dt,108)
Let me know if it helps you in any way.
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
April 15, 2009
SQL SERVER: Difference between OpenQuery and OpenRowSet
Today, one developer asked me what is the difference between OpenQuery and OpenRowSet.
Let me share this thing with all of you.
Syntax for both the command:
OPENQUERY ( linked_server ,'query' )
OPENROWSET
( 'provider_name' , 'datasource' ; 'user_id' ; 'password'
, { [ catalog. ] [ schema. ] object | 'query' }
)
Difference is:
OpenQuery uses a predefined linked server,
While OpenRowSet has to specify all the connection options. So with OpenRowSet you can query to your remote SQL server from local.
Else it's the same.
Let me know if it helps you in any way.
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
April 14, 2009
SQL SERVER: SQL Query To Find Most used Tables
| We have very large database and today we want to search the tables which are used mostly. Means tables which are used in Procedures, Constraints, Views, Triggers etc. I know this is very strange requirement, but we need to do this. So, I tried to make an query which will help me to find out the top most tables used in other objects as I mentioned Let me share that sp with all of you: SELECT TableName, COUNT(*) So, I made my life easy, by using this. I can get the list if Tables which are used most. Let me know if it helps you in any way. |
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
April 2, 2009
SQL SEVER: How to make an Entry in Event Viewer using SQL SERVER
| Today I came across the situation where I need to following scenario. I need to make entry in Event Viewer when there is an error in Stored Procedure. I wondered to do this, but finally I come up with the solution. SQL is much powerful. SQL provides us to make an entry in Event Viewer by two ways: 1. using XP_LogEvent 2. By Raiserror WITH LOG Lets first see the way using XP_LogEvent:Here I have created one SP which will raise an error “Divide by zero error encountered.” as I tried to do “10 / 0”. CREATE PROCEDURE TestEventViewer EXEC xp_logevent 60000, @msg, informational END CATCH Lets Execute this SP: EXEC TestEventViewer This will write entry in Event Viewer. Now open Event Viewer. You can find Event Viewer at Control Panel –> Administrative Tools –> Event Viewer. You will get en entry of Error there. So we can do this by Extended Stored Procedure: “xp_logevent”. Let see the parameters of this Procedure. First Parameter: “60000” is the Error Number Second Parameter: “@msg” is the message to be displayed in Event Viewer. Third Parameter: “informational” is the Error Level. It could be “informational”, “Error”, “Warning”. Now, Lets see by another way By Raiserror WITH LOG: It is the same way as we used Raiserror to Raise an Error. CREATE PROCEDURE TestEventViewer RAISERROR(@msg, 11, 1) WITH LOG END CATCH Lets Execute this SP: EXEC TestEventViewer. So By these ways we can make an entry to Event Viewer. Let me know if it helps you in any way |
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
March 26, 2009
SQL SERVER: How much space occupied by Each Table with sp_MSforeachtable procedure
For example, How much space occupied by each table.
I found undocumented Procedure: sp_MSforEachTable in the master database.
The following script reports the space used and allocated for every table in the database.
USE AdventureWorks;
EXECUTE sp_MSforeachtable 'sp_spaceused [?]'
So, We can use sp_MSforeachtable procedure when we need to loop through each table.
Let me know if it helps you in any way.
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS
March 22, 2009
SQL SERVER: Unable to Launch Activity Monitor. How to add VIEW SERVER STATE Permissions to launch Activity Monitor
| Activity Monitor, is used view to current executing queries, Query status, Start Time, Host, Blocking or not, Database etc. Last week, one of my developer has one issue. When he try to open Activity Monitor, System gives the following Error: I found that this user need to get VIEW STATE Permissions, Lets see how to give this permission to the user. We can give permission to user by this two ways: 1. T-SQL: GRANT VIEW SERVER STATE TO YourLogin 2. GUI: Step 1: Right mouse click the SQL Server and go to Properties. You need to give GRANT to “View server State’ for that user. |
18+ years of Hands-on Experience
MICROSOFT CERTIFIED PROFESSIONAL (Microsoft SQL Server)
Proficient in .NET C#
Hands on working experience on MS SQL, DBA, Performance Tuning, Power BI, SSIS, and SSRS