Background

In our Enterprise Data Warehouse (EDW) within the Northwestern University Biomedical Informatics Center (NUBIC), exact copies of many source systems exist. From that source system data we create several data marts to organize clinical data in meaningful ways to assist with reporting. Our current process to populate such data marts is to do the following.

1. Stage the data in a staging table
2. Delete rows from the destination/production table that exist in the staging table
3. Insert the remaining data from the staging table into the production table

Many times in our SSIS ETL packages we will overlap slightly with the last load date/time to make sure all new data is collected. This practice causes numerous delete transactions on the production table on a daily basis before the new rows are inserted from staging. For the most part, these deletes are unnecessary since many of the deleted rows are re-inserted with the same data (because of the deliberate overlap mentioned earlier). In the long-term these unnecessary deletes can cause issues with fragmentation.

Problem

To avoid these excessive deletes we investigated the option of a MERGE statement in place of the 3-step process listed previously.  This functionality initially looked very promising since it would allow us to stage whatever data we needed and then that data could simply be merged into the production data.  The MERGE would update rows that already existed in production and would insert the rows that did not exist.  This would eliminate the unnecessary deletes that occur in our current process.

Our evaluation process uncovered some issues with the MERGE statement and SSIS.

1. There is currently no simple MERGE task available in SSIS 2008. The easiest way to do this is to create an SSIS Execute SQL task and manually type out the MERGE statement.
2. SSIS Execute SQL Task MERGE statement maintenance. If we were to use the Execute SQL Task to execute a MERGE statement, the statements would be rather large. Many of our data mart tables are very wide and are prone to having new columns added to track new clinical information. Manually entering the MERGE statement in an Execute SQL Task creates maintenance issues if a data mart were ever to be modified. If changes to the data mart occur, the MERGE statement would need to be manually updated as well to reflect such changes.
3. SSIS Execute SQL Task character limit. Manual entry of the merge statement into the query editor window of the SSIS Execute SQL task is subject to a 32,767 character limit. Again, many of our data mart tables are very wide and contain several columns where this could potentially be an issue.
4. SSIS Execute SQL Task manual entry errors. Even if the 32,767 character limit is not met, the MERGE statement itself can still be very long. To type all of the statement manually creates several opportunities for typographical errors, including omission of specific columns.

If we were going to use the MERGE statement via SSIS a solution needed to be created that would be easy to use but also would resolve the issues we uncovered in our investigation.

Proposed Solution

Our solution was to create a stored procedure that could dynamically generate the MERGE statement outside of the SSIS Execute SQL Task and execute it.  This article will walk through step-by-step how we built our stored procedure.  An example MERGE statement is provided below.  As we review each step we will see how the stored procedure would generate this example MERGE statement.

To begin, let’s first review the MERGE statement and its construction (Here is a link to a very useful SQLServerCentral technical article which provides a thorough overview of the MERGE statement). A MERGE is comprised of the following parts.

• A source table  (this would be our staging table)
• A target table (this would be our production table)
• A predicate (how to join the source and target tables)
• An Update command to be executed when the source and target rows match on the predicate
• An Insert command to be executed when the source row does not match a target row
• A Delete command to be executed when the target rows do not exist in the source

For our purposes there was no need for the delete command because we want to keep the rows on the target table that do not exist in the source table. Here is the example MERGE statement that we will attempt to generate and execute via this stored procedure.  Within this example we have a staging/source table called edw.adventure_hospital_dm.visits and a production/target table called edw.staging.stg_visits.

MERGE INTO
--1)  Target table
edw.adventure_hospital_dm.visits tgt
USING
--2)  Source table
edw.staging.stg_visits src
ON
--3)  Predicate
src.financial_nbr = tgt.financial_nbr
--4)  Update into Source table when matched
WHEN matched THEN
UPDATE
SET tgt.name                 = src.name,
    tgt.financial_nbr        = src.fininancial_nbr,
    tgt.medical_number       = src.medical_number,
    tgt.registration_date    = src.registration_date,
    tgt.meta_orignl_load_dts = src.meta_orignl_load_dts,
    tgt.meta_update_dts      = src.meta_update_dts
--5)  Insert into Source table when not matched
WHEN NOT matched THEN
INSERT (
    name,
    financial_nbr,
    medical_number,
    registration_date,
    meta_orignl_load_dts,
    meta_update_dts
    )
VALUES (
    src.name,
    src.fininancial_nbr,
    src.medical_number,
    src.registration_date,
    src.meta_orignl_load_dts,
    src.meta_update_dts);

Before the stored procedure was built we determined that a useful MERGE stored procedure would need to satisfy the following requirements.

1. Accept parameters to enter the Source database/schema/table and the Target database/schema/table.
2. Automatically determine the predicate between the Source and Target tables
3. Automatically determine whether a Source column data type matches a Target column data type and store the matching columns in a temp table
4. Is smart enough to NOT update primary key column(s) on the Target table
5. Generate a dynamic SQL MERGE statement based on the matched columns stored in the temp table
6. Execute the dynamic SQL MERGE statement

Step 1: Accept parameters to enter the Source database, schema, and table along with the Target database, schema, and table.

Our goal for this solution was for it to be easy to use.  We wanted to be able to execute a stored procedure and simply tell it what two tables to merge.  To do this we chose to create a stored procedure with 8 parameters.  The first 3 parameters (database, schema, and table name) are used to pass in the Source table.  The second 3 parameters (database, schema, and table name) are used to pass in the Target table.  Even though the stored procedure will automatically generate the matching predicate (which is discussed in Step #2), we created the 7th parameter as an option to manually pass in a comma-separated list of predicate items to match on if the tables being merged either do not have a primary key or the user would like to match on something other than the primary key.

Here is the section of code that was used to create the stored procedure along with its arguments and necessary variables.

CREATE PROCEDURE [adventure_hospital].[generate_merge]
                @SrcDB          SYSNAME,         --Name of the Source database
                @SrcSchema      SYSNAME,         --Name of the Source schema
                @SrcTable       SYSNAME,         --Name of the Source table
                @TgtDB          SYSNAME,         --Name of the Target database
                @TgtSchema      SYSNAME,         --Name of the Target schema
                @TgtTable       SYSNAME,         --Name of the Target table
                @predicate      SYSNAME  = NULL, --(optional)Override to automatic predicate generation.  A comma-separated list of predicate match items
                @debug          SMALLINT = NULL  --(optional)Pass in 1 to kick out just the MERGE statement text without executing it
AS
BEGIN
	DECLARE @merge_sql      NVARCHAR(MAX);  --overall dynamic sql statement for the merge
	DECLARE @columns_sql    NVARCHAR(MAX);  --the dynamic sql to generate the list of columns used in the update, insert, and insert-values portion of the merge dynamic sql
	DECLARE @pred_sql       NVARCHAR(MAX);	--the dynamic sql to generate the predicate/matching-statement of the merge dynamic sql (populates @pred)
	DECLARE @pk_sql         NVARCHAR(MAX);  --the dynamic sql to populate the @pk table variable that holds the primary keys of the target table
	DECLARE @updt           NVARCHAR(MAX);  --contains the comma-seperated columns used in the UPDATE portion of the merge dynamic sql (populated by @columns_sql)
	DECLARE @INSERT         NVARCHAR(MAX);  --contains the comma-seperated columns used in the INSERT portion of the merge dynamic sql (populated by @insert_sql)
	DECLARE @vals           NVARCHAR(MAX);  --contains the comma-seperated columns used in the VALUES portion of the merge dynamic sql (populated by @vals_sql)
	DECLARE @pred           NVARCHAR(MAX);  --contains the predicate/matching-statement of the merge dynamic sql (populated by @pred_sql)
	DECLARE @pred_param     NVARCHAR(MAX) = @predicate;  --populated by @predicate.  used in the dynamic generation of the predicate statment of the merge
	DECLARE @pred_item      NVARCHAR(MAX);  --used as a placeholder of each individual item contained within the explicitley passed in predicate
	DECLARE @done_ind       SMALLINT = 0;   --used in the dynamic generation of the predicate statment of the merge
	DECLARE @dsql_param     NVARCHAR(500);  --contains the necessary parameters for the dynamic sql execution

Step 2: Automatically determine the predicate used between the Source and Target tables
Before the predicate is automatically generated there must be a check to see whether or not a predicate was passed in as a parameter. If the predicate parameter is populated then the comma-separated list that was passed in is broken down into its individual items and the predicate matching statement is generated and assigned to @pred.

If the predicate was not passed in as a parameter a dynamic SQL (@pred_sql) is generated to query the INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE table using the Target database, schema, and table parameters. The INFORMATION_SCHEMA table is automatically available in SQL Server 2000 and above. Since the output of this the dynamic SQL (@pred_sql) needs to be collected and assigned to a variable (@pred) a parameter (@dsql_param) needs to be passed in so that the output can be returned and assigned appropriately. Once the dynamic SQL (@pred_sql) and the dynamic SQL parameter (@dsql_param) have been created, they can be executed. The output can then be assigned to @pred. We now have our predicate statement.

/****************************************************************************************
* This generates the matching statement (aka Predicate) statement of the Merge.        *
* If a predicate is explicitly passed in, use that to generate the matching statement. *
* Else execute the @pred_sql statement to decide what to match on and generate the     *
* matching statement automatically.                                                    *
****************************************************************************************/
 
IF @pred_param IS NOT NULL
  -- If a comma-separated list of predicate match items were passed in via @predicate
  BEGIN
  -- These next two SET statements do basic clean-up on the comma-separated list of predicate items (@pred_param)
  -- if the user passed in a predicate that begins with a comma, strip it out
  SET @pred_param = CASE WHEN SUBSTRING(ltrim(@pred_param),1,1) = ',' THEN SUBSTRING(@pred_param,(charindex(',',@pred_param)+1),LEN(@pred_param)) ELSE @pred_param END
  --if the user passed in a predicate that ends with a comma, strip it out
  SET @pred_param = CASE WHEN SUBSTRING(rtrim(@pred_param),LEN(@pred_param),1) = ',' THEN SUBSTRING(@pred_param,1,LEN(@pred_param)-1) ELSE @pred_param END
  -- End clean-up of(@pred_param) *
  -- loop through the comma-seperated predicate that was passed in via the paramater and construct the predicate statement
  WHILE (@done_ind = 0)
    BEGIN
    SET @pred_item = CASE WHEN charindex(',',@pred_param) > 0 THEN SUBSTRING(@pred_param,1,(charindex(',',@pred_param)-1)) ELSE @pred_param END
    SET @pred_param = SUBSTRING(@pred_param,(charindex(',',@pred_param)+1),LEN(@pred_param))
    SET @pred = CASE WHEN @pred IS NULL THEN (COALESCE(@pred,'') + 'src.[' + @pred_item + '] = ' + 'tgt.[' + @pred_item + ']') ELSE (COALESCE(@pred,'') + ' and ' + 'src.[' + @pred_item + '] = ' + 'tgt.[' + @pred_item + ']') END
    SET @done_ind = CASE WHEN @pred_param = @pred_item THEN 1 ELSE 0 END
    END
  END
ELSE
  -- If an explicite list of predicate match items was NOT passed in then automatically construct the predicate
  -- match statement based on the primary keys of the Source and Target tables
  BEGIN
  SET @pred_sql = ' SELECT @predsqlout = COALESCE(@predsqlout+'' and '','''')+' +
                  '(''''+''src.''+column_name+'' = tgt.''+ccu.column_name)' +
                  ' FROM ' +
                  @TgtDB + '.INFORMATION_SCHEMA.TABLE_CONSTRAINTS tc_tgt' +
                  ' INNER JOIN ' + @TgtDB +'.INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE ccu' +
                  ' ON tc_tgt.CONSTRAINT_NAME = ccu.Constraint_name' +
                  ' AND tc_tgt.table_schema = ccu.table_schema' +
                  ' AND tc_tgt.table_name = ccu.table_name' +
                  ' WHERE' +
                  ' tc_tgt.CONSTRAINT_TYPE = ''Primary Key''' +
                  ' and tc_tgt.table_catalog = ''' + @TgtDB + '''' +
                  ' and tc_tgt.table_name = ''' + @TgtTable + '''' +
                  ' and tc_tgt.table_schema = ''' + @TgtSchema + ''''
  SET @dsql_param = '	@predsqlout nvarchar(max) OUTPUT'
 
  EXEC sp_executesql
  @pred_sql,
  @dsql_param,
  @predsqlout = @pred OUTPUT;
END

The benefit of generating the predicate this way is that this automatically handles multiple primary key constraints on the Target table. All primary key constraints are evaluated and added into the predicate statement. The example output from the @pred variable is listed here.

src.financial_nbr = tgt.financial_nbr

Step 3: Automatically determine if a Source column data type matches the corresponding Target column data type and store the matching columns in a temp table
To do this we first need to create the custom table (@columns) that will be used to hold all the columns that exist between the Source and the Target tables. These columns must have the same name and same data type. By ‘same data type’ I mean both that the data type is the same and that the precision of the Target table column is at the same level or greater than the precision of the Source table column. For example, the medical_number column on the Source table may be varchar(10). As long as the corresponding medical_number column on the Target/Production table is at least a varchar(10) or greater the logic will consider that to be a match and store that column on the temp table.

--Create the temporary table to collect all the columns shared
--between both the Source and Target tables.
 
DECLARE @COLUMNS TABLE (
 table_catalog            VARCHAR(100) NULL,
 table_schema             VARCHAR(100) NULL,
 TABLE_NAME               VARCHAR(100) NULL,
 column_name              VARCHAR(100) NULL,
 data_type                VARCHAR(100) NULL,
 character_maximum_length INT NULL,
 numeric_precision        INT NULL,
 src_column_path          VARCHAR(100) NULL,
 tgt_column_path          VARCHAR(100) NULL
)

When the @columns temp table has been created a dynamic SQL statement is generated (@columns_sql) and executed to populate the temp table. The @columns_sql dynamic SQL statement takes the values passed in via parameters and uses them to query the INFORMATION_SCHEMA.COLUMNS table to find matching columns between the Source and Target tables. Once the @columns_sql dynamic SQL has been constructed it is executed and the values returned are automatically inserted into the @columns temp table.

/************************************************************************************************
* Generate the dynamic sql (@columns_sql) statement that will                                  *
* populate the @columns temp table with the columns that will be used in the merge dynamic sql *
* The @columns table will contain columns that exist in both the source and target             *
* tables that have the same data types.                                                        *
************************************************************************************************/    
 
SET @columns_sql =
'SELECT
tgt.table_catalog,
tgt.table_schema,
tgt.table_name,
tgt.column_name,
tgt.data_type,
tgt.character_maximum_length,
tgt.numeric_precision,
(src.table_catalog+''.''+src.table_schema+''.''+src.table_name+''.''+src.column_name) AS src_column_path,
(tgt.table_catalog+''.''+tgt.table_schema+''.''+tgt.table_name+''.''+tgt.column_name) AS tgt_column_path
FROM
     ' + @TgtDB + '.information_schema.columns tgt
     INNER JOIN ' + @SrcDB + '.information_schema.columns src
       ON tgt.column_name = src.column_name
       AND tgt.data_type = src.data_type
       AND (tgt.character_maximum_length IS NULL OR tgt.character_maximum_length >= src.character_maximum_length)
       AND (tgt.numeric_precision IS NULL OR tgt.numeric_precision >= src.numeric_precision)
     WHERE tgt.table_catalog     = ''' + @TgtDB + '''
     AND tgt.table_schema        = ''' + @TgtSchema + '''
     AND tgt.table_name          = ''' + @TgtTable + '''
     AND src.table_catalog       = ''' + @SrcDB + '''
     AND src.table_schema        = ''' + @SrcSchema + '''
     AND src.table_name          = ''' + @SrcTable + '''
     ORDER BY tgt.ordinal_position'
 
     --execute the @columns_sql dynamic sql and populate @columns table with the data
     INSERT INTO @COLUMNS
     EXEC sp_executesql @columns_sql

Now that the @columns temp table has been populated we have a list of all the columns that will be referenced in the overall MERGE statement.

Step 4:  Has logic to NOT update the primary key column(s) on the target table

While investigating the efficiency gains of using the merge statement we found that we were not getting the gains we were expecting.  We already knew that we were updating all columns within the merge statement when the predicate match was satisfied.  The problem is that when you update a row’s primary key (even if you are updating the primary key to its original value) that the update is essentially handled as an insert and therefore you end up losing some efficiency, actually a lot of efficiency. I wanted to thank Paul White for his blog post regarding “The Impact of Non-Updating Updates“.

To prevent the primary key from being updated within the MERGE update we created a temp table to record the primary key column(s) from the target table.  This temp table is later referenced to filter out the primary key column(s) when the update portion of the MERGE statement (@updt) is generated in the next step.

/**************************************************************************************
* Create the temporary table to collect all the primary key columns                  *
* These primary key columns will be filtered out of the update portion of the merge  *
* We do not want to update any portion of clustered index for performance            *
**************************************************************************************/
 
DECLARE @pk TABLE (
  column_name              VARCHAR(100) NULL
);
 
SET @pk_sql = 'SELECT ' +
              'ccu.column_name ' +
              'FROM ' +
              @TgtDB + '.INFORMATION_SCHEMA.TABLE_CONSTRAINTS tc_tgt ' +
              'INNER JOIN ' + @TgtDB +'.INFORMATION_SCHEMA.CONSTRAINT_COLUMN_USAGE ccu ' +
              'ON tc_tgt.CONSTRAINT_NAME = ccu.Constraint_name ' +
              'AND tc_tgt.table_schema = ccu.table_schema ' +
              'AND tc_tgt.table_name = ccu.table_name ' +
              'WHERE ' +
              'tc_tgt.CONSTRAINT_TYPE = ''Primary Key'' ' +
              'and tc_tgt.table_catalog = ''' + @TgtDB + ''' ' +
              'and tc_tgt.table_name = ''' + @TgtTable + ''' ' +
              'and tc_tgt.table_schema = ''' + @TgtSchema + ''' ' 
 
INSERT INTO @pk
EXEC sp_executesql @pk_sql

Step 5:  Generate a dynamic SQL MERGE statement based on the matched columns stored in the temp table

For the overall MERGE statement there are 3 sets of comma-separated columns that need to be generated from the data collected in the @columns temp table. These will be the update columns, insert columns, and the insert-values columns. In our example MERGE statement the 3 sets of columns are outlined below in red.

For each one of the 3 sets of columns a query needs to be executed against the @columns temp table. We needed a way to query the columns on the @columns temp table and loop through them creating a comma separated list. Since such a query will return more than row, if you try to assign the output of this query to a variable you will receive the error “Subquery returned more than 1 value.”

To get around this we used SQLServer’s FOR XML functionality. For anyone not familiar with FOR XML, simply put FOR XML allows you to run a query and format all of the output as XML. It then assigns that output to a variable of type XML. We essentially ran the same query but used FOR XML PATH(”) which allowed us to loop through the data results by generating an XML version of the data with absolutely no XML formatting. Since FOR XML automatically returns a data type of XML, we then cast the results to convert them from XML to NVARCHAR.

What you are left with is a string that contains all the comma-separated columns that can be used for the MERGE statement. Obviously, this is not the intended use of FOR XML, but this suited our needs nicely.

Here is the section of the stored procedure used to generate the comma-separated columns used for the update portion of the overall MERGE statement.  You can see below that the @pk temp table we created in the previous step is now referenced to filter out primary key column(s) from the update.

--1) List of columns used for Update Statement
--Populate @updt with the list of columns that will be used to construct the Update Statment portion of the Merge
 
SET @updt = CAST((SELECT ',tgt.[' + column_name + '] = src.[' + column_name + ']'
            FROM @COLUMNS c
            WHERE c.column_name != 'meta_orignl_load_dts'                               --we do not want the original time the row was created to be overwritten
            AND NOT EXISTS (SELECT 'x' FROM @pk p WHERE p.column_name = c.column_name)  --we do not want the primary key columns updated for performance
            FOR XML PATH(''))
            AS NVARCHAR(MAX)
            )

**Special Note**   For the columns used in the update portion of the MERGE statement, we filtered out the meta_orginl_load_dts column. The EDW uses this column as meta data to record when a row was originally written out to the data mart. Upon an update, we do not want this column to be updated with the meta_orignl_load_dts from the staging/source table. We still want to preserve the original load date time of that row. The date and time the row was updated will be represented in the meta column meta_update_dts. Second, the comma-separated list of columns used for the insert portion of the overall MERGE statement need to be generated. Here is section of the stored procedure used to generate the comma-separated columns used for the insert portion of the overall MERGE statement.

--2) List of columns used for Insert Statement
--Populate @insert with the list of columns that will be used to construct the Insert Statment portion of the Merge
 
SET @INSERT = CAST((SELECT ',' + '[' + column_name + ']'
              FROM @COLUMNS
              FOR XML PATH(''))
              AS NVARCHAR(MAX)
              )

Here is the output of the @insert variable for our example.

,name
,financial_nbr
,medical_number
,registration_date
,meta_orignl_load_dts
,meta_update_dts

Here is section of the stored procedure used to create the comma-separated list of columns used for the insert-values portion of the overall MERGE statement.

--3) List of columns used for Insert-Values Statement
--Populate @vals with the list of columns that will be used to construct the Insert-Values Statment portion of the Merge	
 
SET @vals = CAST((SELECT ',src.' + '[' + column_name + ']'
            FROM @COLUMNS
            FOR XML PATH(''))
            AS NVARCHAR(MAX)
            )

Here is the output of the @vals variable for our example.

,src.name
,src.fininancial_nbr
,src.medical_number
,src.registration_date
,src.meta_orignl_load_dts
,src.meta_update_dts

Step 6: Execute the dynamic SQL MERGE statement
Now that we have the Source database/schema/table, the Target database/schema/table, the predicate matching statement, and the update/insert/values comma-separated lists of columns, we have everything we need to generate the entire dynamic SQL MERGE statement. Once the MERGE statement is generated it then gets executed automatically.

Here is the section of the stored procedure used to pull everything together into the overall MERGE statement (@merge_sql) and then execute it.

/*************************************************************************************
*  Generate the final Merge statement using the following...                        *
*    -The parameters (@TgtDB, @TgtSchema, @TgtTable, @SrcDB, @SrcSchema, @SrcTable) *
*    -The predicate matching statement (@pred)                                      *
*    -The update column list (@updt)                                                *
*    -The insert column list (@insert)                                              *
*    -The insert-value column list (@vals)                                          *
*    -Filter out Primary Key from the update (updating primary key essentially      *
*     turns the update into an insert and you lose all efficiency benefits)         *
*************************************************************************************/
 
SET @merge_sql = (' MERGE into ' + @TgtDB + '.' + @TgtSchema + '.' + @TgtTable + ' tgt ' +
                  ' using ' + @SrcDB + '.' + @SrcSchema + '.' + @SrcTable + ' src ' +
                  ' on ' + @pred +
                  ' when matched then update ' +
                  ' set ' + SUBSTRING(@updt, 2, LEN(@updt)) +
                  ' when not matched then insert (' + SUBSTRING(@INSERT, 2, LEN(@INSERT)) + ')' +
                  ' values ( ' + SUBSTRING(@vals, 2, LEN(@vals)) + ');'
                 );
 
--Either execute the final Merge statement to merge the staging table into production
--Or kick out the actual merge statement text if debug is turned on (@debug=1)
IF @debug = 1
  BEGIN
  -- If debug is turned on simply select the text of merge statement and return that
  SELECT @merge_sql;
  END
ELSE
  BEGIN
  -- If debug is not turned on then execute the merge statement
  EXEC sp_executesql @merge_sql;
END

The substring() function is used on @updt, @insert, and @vals to remove the preceding comma from each set of columns. This could have been done earlier in the stored procedure but we decided to take care of it here.

Here is the output of the @merge_sql variable from our example.

MERGE INTO edw.adventure_hospital_dm.visits tgt
USING edw.staging.stg_visits src
ON src.financial_nbr = tgt.financial_nbr
WHEN matched THEN UPDATE
SET tgt.name = src.name,
tgt.medical_number = src.medcial_nubmer,
tgt.registration_date = src.registration_date,
tgt.meta_orignl_load_dts = src.meta_orignl_load_dts,
tgt.meta_update_dts = src.meta_update_dts
WHEN NOT matched THEN INSERT (
name,
financial_nbr,
medical_number,
registration_date,
meta_orignl_load_dts,
meta_update_dts
)
VALUES (
src.name,
src.financial_nbr,
src.medical_number,
src.registration_date,
src.meta_orignl_load_dts,
src.meta_update_dts);

Conclusion

We are currently working towards utilizing this stored procedure within our SSIS packages for populating datamarts. Using our current process, a simplified version of one of our SSIS packages would look like the example below.

By using the stored procedure to merge the data from staging into the production table we can replace the last 2 tasks with one Execute SQL task. That Execute SQL task simply executes the stored procedure. Using this method the example SSIS package would now look like this.

Here is the SQL code that is executed by the Execute SQL task.

Although this stored procedure as a bit tedious to build, it is worth it. We now have a stored procedure to run from SSIS that suits our data warehousing needs for merging staging data into production.

There are many benefits to merging data via a stored procedure.

• Easily executed through an SSIS SQL Task
• Easily reusable
• Automatically builds and executes the merge statement so there is no need for manual creation or maintenance of the merge SQL statement.
• Reduces fragmentation since there are less deletes performed on the production table
• Since rows are updated instead of deleted and reinserted, the ETL package runs faster
• Overcomes SSIS SQL Task 32,676 character limit
• Preserves original row load date and time (since the row is updated instead of deleted and re-inserted)

In Practice
The largest table the EDW imports from the primary inpatient Electronic Health Record (EHR) source system stores clinical results and contains about 2.4 billion rows. Currently we merge between 600,000 – 900,000 rows of data from that source system table into the corresponding table within our EDW. With our old 2-step process of deleting rows from the production table that exist in the staging table and then inserting new rows, the whole process took around 2 hours to complete each night. Once we implemented the merge stored procedure into the ETL package that time was reduced to about 6 minutes!A bulk of the processing time in the old process was due to the deletes that were performed on a table that contained over 2 billion rows.

Limitations
This stored procedure is meant for data warehousing type functionality.  It assumes you are using a Staging-To-Production type of model where the staging table and production table are almost identically built with the same column names and very similar data types. Also, this stored procedure does not utilize the delete functionality of the MERGE statement.

Resources
Dhaneenja, T. (2008). Understanding the MERGE DML Statement in SQL Server 2008.

  SQLServerCentral.com. Retrieved from

  http://www.sqlservercentral.com/articles/SQL+Server+2008/64365/

Here’s a rundown of what was built in 24 hours (in the order in which they were presented):

Project: Bundle Recorder
People: Rhett Sutphin

Bundle Recorder is a Jenkins plugin for tracking the gem dependencies used in each build of a bundler-using Ruby project. It stores bundler’s Gemfile.lock at the end of each build and provides a way to view changes between adjacent builds. While it can be used with any Ruby project, it will be most useful for gems, since you don’t usually commit the Gemfile.lock for gems.

I built this plugin to address an issue I ran into more than a few times: NUBIC uses Jenkins for continuous integration. One of the things we use it for is to perform nightly tests of our various gems to ensure that they continue to work as new versions of their own dependencies are released. Sometimes a new dependency does cause a failure — that’s good as far as it goes; it means the builds are doing their jobs. However, when a nightly fails, it’s often hard to tell what changed — by default all we have is the console output from `bundle update`. It lists the dependencies and versions, but — because we also use these builds to verify that none of the dependencies have been yanked from `rubygems.org` and so reinstall everything — not which ones just changed. And they are in no particular order. By parsing the bundler lockfile, Bundle Recorder can give a summary of just what’s changed.

Project: Natural Language Processing (NLP) Abstraction Tool
People: Luke Rasmussen, Tuan Nguyen, Thomas Elbert, Daniel Scheider

Our project was to create a web application to allow someone to go in and make annotations on text documents.  Why is this useful?  When you’re working with text documents in any type of automated fashion, being able to have a “gold standard” to validate against is really important.  For NLP, this gold standard can help with machine learning efforts to improve the NLP engine.  It can also help verify if documents have had all of the patient information removed, and annotate where anything was missed.  The main goals were to create a tool that is:

• lightweight
• easy to learn and use quickly
• built on top of the UIMA framework (http://uima.apache.org/).

Making this all happen from design to development in 24 hours was quite a challenge, but a fun one.  We split into a “data team” and a “UI team”.  The data team worked on getting documents from the EDW so the annotation tool could access them.  The UI team developed the annotation system, and the web services to connect everything together.  Probably the biggest challenge was trying to tone down what we wanted the tool to be able to do (the infamous scope creep – yes, even programmers do it) to the point we could get something working in the time allowed.  Getting free time to work on a pet project is probably the greatest gift a programmer can receive, and we had a lot of fun making the system work and talking about what we’d like to do with it.  We’re definitely hoping for another FedEx Day in the future!

Project: tardis – your friend in time
People: Jeff Lunt

tardis is a grapher for events against a timeline. I wanted a simple way to graph arbitrary events against a timeline using the MIT Simile timeline widget. The goal was to provide a visual representation of events in web apps, for example graphing the incident of application errors vs. time, vs. other events such as server maintenance, server load, etc. The hopeful outcome is that you have a timeline that covers every level of your stack (host, VM usage, process load, web app framework and errors, user load, etc.) all in one view, so that it’s easier to correlate problems and changes in application performance with events at other levels in the stack.

It’s important to realize that typical graphing libraries, that plot numerical data across two axis, don’t really handle momentary or duration events, or if they do, they don’t do it well. That’s what makes the MIT Simile timeline widget especially useful for this purpose.

tardis can also be used to post any series of events, even if it’s not software related, via a simple RESTful API, to be documented and published in the near future.

Project: tahoe file system implementation
People: Dong Fu

Tahoe-LAFS is an open-source distributed file system project that implements the Principle of Least Authority (POLA) and Reliable Array of Independent Nodes (RAIN).  By leveraging an independent pool of SAN storage devices through WAN or LAN links, the software presents end-users with a fault-tolerant and secure resource for backup and online storage.  For the FedEx Day, I was able to set up a virtual machine with Tahoe-LAFS installed and configured.  I was also able to register the test VM onto the public storage Grid to demonstrate its potential.  Other aspects of the product, such as performance, fault-tolerance, and cross-platform compatibility will be explored at the next available opportunity.

A one-page description of the design of Tahoe-LAFS can be found here.

Project: nubic boostrap – a virtual machine build automation using Puppet
People: William Dix, John Dzak, Dong Fu

From Will: On boarding of new developers is a slow and difficult process.  Installing dependencies, particularly Oracle, is time consuming to say the least.  To improve this process, I wanted to build a VM with Oracle installed and set it up as a Vagrant base box (Vagrant is a Ruby gem which provides easy management of Virtualbox VMs).  Using Vagrant, a developer can quickly set up a new Oracle VM on their local machine whenever there is a need.

My project was not as successful as I would have liked.  Primarily because of the time required to move large VMs around, conver them to the proper formats, etc. Because of the time it took just to get the VMs in the right place and in the right format, I was not able to do other desired tasks like automated schema loading.

From John: So, my FedEx Day project was the nubic bootstrap project that I worked on with Dong and William.  Initially I was thinking of working on a set of scripts that would make setting up an Oracle VM easier, but after talking with William and Dong we decided to take an approach similar to how bundler works where you specify your dependencies in a file (shown below), and run the gem command `nubic_bootstrap install`, and all those dependencies are installed.  Here’s an example config file:

example nubic_bootstrap.yml

vms:
  oracle:
    schemas: [cc_pers, cc_notis]
    database: //localhost:15210/XE

I was also hoping to build on some of the functionality that bcdatabase/Oracle offered while building the gem.  After a couple of hours of work I realized that invoking the commands I need on the guest VM was trickier than I initially thought since I needed to know information about the guest VM to invoke certain commands (ex. oracle home).  As time started running out, I retreated back to the initial, more basic idea of commands to import/export databases in the VM along with hard coding some of the settings.  The gem ended up being very hard coded, but it could download the Oracle VM and import a schema into it.  Also, as we neared the end of FedEx Day William found a way to invoke commands inside a VM using Vagrant which could be useful in the future.  I think once I have some more freetime on my hands I will continue with this project since it is something our group could use when migrating to full application environments inside the VM for projects that use Oracle.

Project: The cost of changing health insurance, to insurance companies
People: Justin Starren

————————————-

My technical project for the year, I’ve decided, is to build a continuous delivery system inside the NUBIC dev team.

Here’s a quick reading list of source materials that I’m using to learn how to do it (blog posts to follow as I document the process of building the system internally):

• The Continuous Delivery book
• The Continuous Delivery blog
• Eric Ries’ 5-step primer to Continuous Deployment
• Case Study: Continuous deployment makes releases non-events
• IMVU’s Approach to Integrating Quality Assurance with Continuous Deployment
• A series on the principle of “The 5 whys”
• The Pareto principle, a.k.a. the 80/20 rule

These things couple very well with additional practices that NUBIC embraces as part of its software development process, including:

• Test Driven Development
• Behavior Driven Development
• Continuous Integration
• Releasing much of our code as open source
• Kaizen – continuous improvement philosophy
• Doing well on the “Joel Test”

Finally, ThoughtWorks Studios has a commercial product called Go for automated release management. A couple of people from ThoughtWorks also happen to be the authors of the book on Continuous Delivery.

This post discusses just one way that sensitive information is transparently taken out of your hands, and put into places you didn’t expect, or weren’t obvious from the use of a given technology.

My example is this: putting username/password combinations in a URL, as outlined in this question on StackOverflow (you’ll probably want to read through this, for some background). While the implementation question posed is a simple one, and may make certain programming tasks simpler, it’s turned out to be a bad idea, because of the potential of that information leaking into unintended places.

As a follow up to that StackOverflow question, I received an email from Anthony Castillo (you can follow his blog at inquirious.com), that delved deeper into this problem, specifically in an iOS context. The discussion is germane to anyone thinking about the unintended consequences of transferring data over networks:

Hi Jeff,

[...]

I asked a question similar to one of yours on SO (regarding https://username:password@service.com type urls).

I’m trying to decide if it is safe *enough* for me to use that format for an iOS project I’m working on. Just wondering if you went with it or if you ended up doing something else.

 

I replied…

Hey Anthony, [...]

The short answer to your follow up is “no – it’s not considered good enough”. Security, to some extent, is a matter of managing risk. Username+password in HTTPS links potentially increases your risk with basically no reward. Any logs on systems between you and the destination system that capture that information (or even non-secure logs on your own machine), effectively makes using HTTPS useless for this purpose, since you wind up putting your credentials in the clear.

No bueno.

I originally wrote this question because I was trying to automate a connection between a script on my local machine, and my source repository, thereby preventing a password prompt form coming up at all, so the script could run unattended. The problem I was trying to solve was automating a web app deployment. However, I found that this wasn’t the best thing to do, for the following reasons:

1. It’s better for security by far to use key-based authentication (rather than username+password), and connect over SSL for deployments and other automated processes, rather than HTTPS. The reasons for this are many, but the benefits are basically that automation can happen pretty easily after you get this setup, and because you’re not using passwords, the encryption is generally considered less likely to be broken, since your “secret key” is a high-security key, rather than a human-chosen password.
2. Having fully automated deployments was troublesome without also having an automatic rollback process. In my case, if a developer wasn’t involved in the deployment, that was a bigger problem than the inconvenience of having to be physically present to enter one’s password. [...]

Back to your question, HTTPS connections are, as I understand them, tunnels directly between the client and the server, so theoretically security should be okay. However, you never know how many machines that request goes through before that tunnel is established (your ISP/wireless provider, for example, is one place that might get a hold of the URL with the username+password in it). Ideally, whatever code library you’re using to make the connection should drop the username+password from the URL while establishing the tunnel, however, there’s no guarantee, and it’s just better never to store your username+password in plain text anywhere that you don’t absolutely have to do so, especially since there are much better, and more secure, alternatives that are just as easy to setup. In any cases where you’ve determined that it is absolutely necessary to store credentials in plain text (I can’t think of any in the modern world in which we live), it’s critical that the file system on which it is stored is a system completely within your control, and not accessible via the Internet, lest someone get access to it. This includes not only the file system itself, but any locations/systems, onsite or off, to which that file system is backed up. In the real world, guaranteeing such a situation is difficult at best, and in practical terms, is probably closer to impossible.

As for your iOS app, that’s a bit trickier, depending on your situation, because you’re not talking about one key, and one server, you’re talking about a bunch of users, and a server they connect to. However, I would think that you could achieve key-based authentication for your users by giving them an API/application key that they could copy/paste into your iOS app that would essentially accomplish the same thing. If you need to go over HTTPS, you can pass the key in as a POST parameter to the session, and allow secure data transfer between your app and the server, in addition to the key acting as authentication (this user is who they say they are). I’m not an iOS expert, but one of my colleagues mentioned that iOS offers some encrypted storage on the device to store sensitive information such as credentials, and that it’s been available to iOS devices at least since iOS v4 (give or take). [..]

Anthony replies…

Hi Jeff, [...]

 

After reading your email, I started digging around the web a bit more. Please let me know what you think of the following:

Basic Access Authentication - this is the “username:password@service.com” format that we have been discussing. This is as opposed to Digest Access Authentication. (There may be others as well.)

• Basic authentication can be used with either HTTP or HTTPS.
• With HTTP, it is definitely insecure.
• When used with HTTPS, that implies SSL encryption over the whole connection.

So…I think it’s okay to use basic access authentication, as long as it’s over HTTPS. (1) (2)

However, the point you make about not knowing how many machines a request goes through before reaching the server leaves me with slight doubts.

Perhaps we have slightly different choices here because you have control over both the client and server sides of the connection. Hence your ability to choose a protocol involving an API key.

As for me, I am building an app against a third-party web service. This means I have to go along with whatever protocol they have in place. In this case, it’s username/password authentication over HTTPS.

Maybe it isn’t completely secure, but it might still be the best I can do given my choices. (I think that’s my conclusion.) I don’t believe there is a way for me to force a different authentication method on the service to which I’m connecting.

Regarding encrypted storage on iOS, that is indeed a standard feature offered by Apple. They have what is known as a “Keychain” feature that allows you to store and encrypt data on an app by app basis. I basically let the OS handle the details of that for me. (Pretty nice.)

And finally…

Anthony,4

In your situation, it sounds like you’re right – if this is the only authentication method provided (and they don’t publish per-user API keys) you’re kind of up the creek on that one. You should still avoid putting the username/password in the URL itself if at all possible, depending on the limitations of the API requests. I would, if possible, contact the vendor and ask that they add it [key-based authentication]. In the mean time, I think you’re right that you’re stuck, according to what you’ve outlined.

The digest authentication looks like your best best, but isn’t considered as secure as key-based authentication (as noted in the wikipedia article you linked to), but it looks like you already know that. [...]

So, it’s always a good idea to think about where your information is not only stored, but where it’s transferred to, and what that might mean in terms of that information getting out of your control inadvertently. Thanks to Anthony Castillo for the deeper discussion, and a specific example to which we could apply the principle.

It’s not the users’ fault. Honestly, it’s not.

When answering a technical support question, have you ever asked someone, “Did you read the manual?” Well, put away your superiority complex for a moment, and realize that your users are wondering why they need a manual in the first place.

Manuals stink, plain and simple, so stop using them whenever possible. If you’ve got a complex application, website (or really any training process whatsoever), and you feel that you aren’t receiving the respect you deserve for writing that 900-page, 100% comprehensive training manual, stop spending time on trying to improve the manual, and instead change the system.

• Ask why – first and foremost you need to understand why the user is having a problem with your application, then you need to correct the flaw that is causing the problem in the first place, thereby eliminating the need for the user to ask the question at all. A great method accomplishing this is 5 Whys. During the process of asking “why” it’s important to always be gracious about honest feedback, and curious about how people arrive at their state of confusion. Once you’ve figured out what’s at the root of the problem, it’s usually a trivial thing to change it.
• Show, don’t tell – create a short training video that shows people how to use it, rather than trying to explain it via text and pictures. If your training video can’t correctly explain it in less than three minutes, your app is either too complex, or your video is trying to do too much. Either fix your app, or sharpen the focus of your video. Great examples of awesome instructional videos are the videos that introduce SquareSpace. They are short, focused on a single topic each, and (in the case of SquareSpace) linked directly from the pages in which the related question might be raised in the user’s mind. A user is editing a webpage and wants to know how to add an image? The video for editing pages is linked from the page editing screen. Simple. It’s true that they still maintain a searchable collection of videos that any user can simply watch, but the fact of the matter is that pretty much no one is going to go through this library and watch all the videos first. Users will typically try something, and only when they fail, will they ask for help.
• Protect users from accidents - There are many times that users will do things that they don’t know are dangerous until it’s too late, and they can’t go back! Whenever possible, provide an “undo” function that allows users to fix mistakes with a simple click or keystroke. This method is often far superior then shifting all responsibility to the user, and presenting them with, “Are you sure?! You cannot undo this!” sorts of messages. Those messages make users fearful, cause them to stop and call you for help making a decision about what to do, and ultimately shift blame to the user when simply providing an “undo” function largely avoids the problem from happening the first place. Even the most seasoned users will occasionally make mistakes. These people aren’t “dumb,” and they’re just human after all. Do you really want to have to recover lost data, or blame them for the mistake, when your system could simply protect users from such accidents in the first place?
• Automate it – sometimes people make mistakes when doing repetitive tasks, because humans aren’t as good at doing highly repetitive things accurately 100% of the time, as compared to computers. This problem is exacerbated by processes that have multiple steps, where a mistake in any one of the steps can cause the whole process to break down. Try helping the users of your site or application by pre-filling in values for forms, automatically inserting reasonable default values, or better yet, just completely automate the process whenever possible. If there’s no reason that a human really needs to be involved in a process, take them out of the loop and save everyone some time and energy.
• Language is imprecise – step-by-step instructions, no matter how detailed and precise, no matter how carefully worded, are difficult to follow. Users gets lost in lengthy instructions, misunderstand or misinterpret technical terms, and people simply don’t want to read instructions anyway. Providing users with a glossary of terms (thinking that the manual should explain itself) isn’t really the answer either. So, use pictures instead of words when possible, and video instead of pictures when possible. The complications of interpreting language is part of why IKEA’s assembly instructions contain no words, only pictures.

The introductory page that explains how to avoid damaging your new furniture during assembly, and what to do if you need help or are confused. Pretty clear, yes? (1) put a carpet or rug under the pieces while assembling them, (2) if you're confused, look in the manual for a picture that shows what to do, and (3) call IKEA. Note that the last picture isn't a person on a phone calling IKEA - it's literally a handset connected to IKEA. When I see this, I think only two words: "phone IKEA". The implication is uncomplicated, and clear. Also note this caption of those four pictures took an entire paragraph. Not very efficient, friendly, or helpful, is it?

TLDR:

• Download the sample package (Dynamic DFT Error Handler Sample Project)
• Run the SQL script
• Open up the package and make sure your connections are set up appropriately

Management of Bad Rows in SSIS

For ETL, SSIS does a fine job at letting you manage the basics of copying one column of data in some source table to another column of data in destination table.  Assuming all goes well, you wind up extracting/transforming/loading that data.

If things don’t go well, however…

Exception handling is a central part of any development task and usually winds up representing a significant chunk of your time and code. You wind up covering any number of “what ifs” like:

• What if I failed to connect to a system?
• What if I expected data and didn’t get any?
• What if my expected data type overflowed?
• What if something totally unanticipated happened?

If you’ve used SSIS for ETL you’re accustomed to the idea of data flow paths inside of a transformation.  You connect a source component to a destination component via either a green line (“good output”) or a red line (“bad / error output”).  This is great stuff.  Say you query some rows from a source database table and want to send the rows to a destination database table – you simply wire up the green line from the source to the destination and map the columns.  Done.  Walk away.

But what about the implied red line for bad rows?  What if you actually have an issue with the transformation?  Two immediate reasons come to mind:

• The data was truncated in some way (cast my Oracle number(20,0) to a SQL int)
• Some other unanticipated error occurred (for the sake of explanation, let’s say a primary key violation on insert)

Usually what you’d do with a static transformation is simply use row redirection to handle the exception.  A common solution is to log your error information to a shared error log table for later review.  By attaching the appropriate error output to your destination you “channel” the row information to that destination so you have a hope of figuring out what happened and what you can do about it.

SSIS usually works really well for these situations, with the exception of two nagging challenges you’ll see come up a lot in discussion forums:

• “My row failed – how do I get the error description?”
• “My row failed – how do I tell which row failed?”

Error description is fairly straight forward and I’m not going to get into it too much – there’s a great step-by-step example at (http://consultingblogs.emc.com/jamiethomson/archive/2005/08/08/1969.aspx) which is very instructive.

Error row identifier, though, is a bit more complex because of the way SSIS works.

Error Columns and Lineage IDs

Let’s say you have a row with an integer column “employee_id” which is the primary key on a table.  What you see is a single presentation of that column “employee_id” – it’s labeled that way throughout your data transformation flow, so to you it’s “the same” throughout the flow.  What SSIS sees internally, however, is something completely different.  If you dig a bit you’ll find you have a unique representation of this column at each point throughout the flow of your SSIS package.  That single “column” (“employee_id”) has to be treated uniquely at each input, output, and error output for each step.  Beyond needing to understand how to treat flow direction (ex: input column vs output column), the column itself may change data types, names, or even value as it flows through your package.  SSIS needs to keep track of that “column” at each point throughout the flow and treat it as though it’s unique.  So how does it do that?  LineageID.

There’s a great article on SQL Server Central (http://www.sqlservercentral.com/articles/Integration+Services+(SSIS)/65730/ ) that touches on some of this.  The article describes lineageid as

It’s an integer and it’s unique throughout the data flow. When buffers are reused, the Lineage ID doesn’t change – it’s the same column at the input and output. When buffers are copied, a new column is created – which gets a new (unique) Lineage ID.

That means that as the column “employee_id” flows through the DFT, it gets a unique Lineage ID - for each input and output copy of itself.  And, typically, you have…

• An input column
• An output column for “good” data
• An output column for errors

Taking the “employee_id” example from the “OLE DB Source” step in our DFT we’d have:

• Input (ID = 33)
• Source Error Output (Lineage ID 35)
• Good Output (Lineage ID = 34)

Great!  No problem.  As long as we know the LineageIDs related to our steps we can back track to determine the mapping to “column name” and voila – we know which row failed.  We can simply look up the column by LineageID using “FindColumnByLineageID” in a script task (http://msdn.microsoft.com/en-us/library/microsoft.sqlserver.dts.pipeline.wrapper.idtsbuffermanager100.findcolumnbylineageid.aspx).  Magic.

Not so fast.  One small, but critical catch.  Metadata about a task step is only available within the scope of that task step. Meaning – once we get past “OLE DB Source” I can see “Lineage ID,” but I can’t easily track back to determine the mapping of Lineage ID to column name.  So – if you want to write out error row information (specifically “column name”) in a second DFT (to your error log, for example) there’s no way to look up that name – because the metadata about LineageID is no longer in scope – it’s only available to the prior step.  Incredibly frustrating.

Getting Error Column Information With Static DFTs

For static packages this can be addressed a few ways. The general strategy is to map the Lineage IDs / IDs to column information at design time and then use that information to look up the information you need.

Couple of quick links you may find handy.

• How to Find Out Which Column Caused SSIS to Fail? (http://blogs.msdn.com/b/helloworld/archive/2008/08/01/how-to-find-out-which-column-caused-ssis-to-fail.aspx)
• Error Output’s Description (Component on CodePlex) (http://eod.codeplex.com/ )
• eLog (http://ssisctc.codeplex.com/wikipage?title=eLog&referringTitle=Home)

Again – for static packages, these can mostly if not completely solve the issue and leave you in a far better position to determine which rows failed.  I’m not going to go into these since you can read up online.

So What About a Dynamic DFT?

Note that the links I provided above address design time gathering / mapping of column information.   What do you do about a runtime situation?  We started digging into the CozyRoc dynamic DFT about a year ago.  Basic dynamic mappings worked great.  You can easily remap columns at runtime and, assuming all goes well, you’re done.  But if things don’t go well – what then?

We need to catch and log those bad rows.  But – we can’t map columns / Lineage ID information at design time because that negates the entire point of using a dynamic DFT – you won’t know any of the required information. It’s just not there.  Now that issue with the resolution of metadata from prior steps comes into play.  We can’t generate column information at design time and we can’t inspect metadata from ancestor steps within a DFT.  They’re out of scope.

I’ll admit that when I first looked at this I was stumped.  And incredibly frustrated.  There was this great opportunity to really let SSIS rock using CozyRoc’s dynamic DFT, but the inability to handle bad rows in a data warehousing solution is a showstopper (keep in mind the issue here is an SSIS design constraint, not a CozyRoc fault).  Following the examples for handling static mappings online (thank you very much, above-linked article authors), we had the notion that we should be able to pull some of the DFT information out at runtime and approach the problem somewhat similarly.

• Upon startup, obtain a list of all columns, their IDs, and their Lineage IDs
• Store that list in a collection
• Using the IDs / Lineage IDs from the errors to look up the corresponding record in our collection
• Profit

I rang up CozyRoc and discussed the situation with their engineers.  They immediately understood my intentions and mailed me back a quick sample of some code that exploited a fantastic capability of their dynamic DFT – the ability to add script to the DFT itself. (Thanks, CozyRoc!)  Not code via a script task within the DFT, but on the DFT directly.

CozyRoc DFT+ (http://www.cozyroc.com/ssis/data-flow-task) notes that you can apply script on the DFT by accessing…

• Advanced tab – specifies advanced task options.
• Script page – specifies data flow task script, which is used for Setup tab customization.

Aha.  And the magic snippet they supplied me…

public void OnColumnAdded(IDTSComponentMetaData100 component, bool isInput, string colName)
//do stuff

Great!  They provided event hooks for the dynamic column mapping!  So now I can detect when a column is added to the DFT flow, add it to my reference collection of column information, and then access that collection within the DFT to derive column information critical to error logging.

This will let me take “Lineage ID” 12345 at any point throughout the flow and figure out that it was column “employee_name_concat” or whatever and log that.  We’re in business.

Something to note here.  Handling row truncation behavior is trickier when you’re doing this dynamically.  You can now longer manually address the need to “redirect on truncation” on a column by column basis, so you just extend the magic DFT+ column binding event to do it for you.

if (!isInput)
{
      IDTSOutputColumn100 column = component.OutputCollection[0].OutputColumnCollection[colName];
      column.TruncationRowDisposition = DTSRowDisposition.RD_RedirectRow;
      column.ErrorRowDisposition = DTSRowDisposition.RD_RedirectRow;
}

Done. Setting row disposition behavior accomplished.

From there we wrote up the nastier parts of the whole exercise – the entire collection lookup mechanism to derive column information.  We did that as a script task within the body of the DFT.

The script task pulls rows out of the buffer and evaluates row position to resolve the Lineage ID / ID and determine…

• Source column name (EX: “first_name”)
• Source primary key name (EX: “employee_id”)
• Source primary key value (EX: “12345″)
• Error description (using ComponentMetaData.GetErrorDescription)
• Error data (so we can quickly eyeball the offending column)

You’ll note I said “primary key name” – we felt it was “good enough” for the moment to avoid dealing with compound keys.  That’s definitely a shortcoming, but for the time being we felt that was acceptable since it matched our existing static ETL error handling process.  It’s definitely something that needs to be addressed, though.  We also cheat by explicitly passing in the primary key as an element of the process (we derive it at an earlier step) – again, in consulting speak, an “opportunity for improvement.”

Putting it All Together

Now that we’ve touched on the ideas, let’s see it work.  Rather than walk you through the entire step-by-step process of building a package I’m going to suggest you Dynamic DFT Error Handler Sample Project.  I’ll quickly touch on the major points on how the sample works.

The download includes some SQL scripts to set up…

• [etl_proxy_table, etl_proxy_table_stg, etl_proxy_table_src] We use some fake placeholder “proxy” tables  so you can set up data bindings in the DFT+.  CozyRoc also suggests you use THUNK_COLUMNs to do this, but I’ve found using these placeholder tables to be very helpful.  The reason we use these is that the magic OnColumnAdded method only fires when a column is actually added to the DFT. If you statically map any of the columns the entire error handling approach will fail because we won’t have those “static” columns added to our column collection.  Huge thank-you to CozyRoc for clueing me in on that.
• [etl_errors] our error logging table. YMMV, but remember if you change this you also need to adjust the scripts in the DFT.
• [demo_source_table, demo_dest_table] our source and destination tables.  We’re big Simpsons fans over here, so I’ve provided appropriate sample data.

The overall package has a few steps:

• ["Set Table Information"] - A cheater Script Task to mimic pulling table configuration information.  In a production scenario you’d likely want to provide configuration elements from either a config file or, better yet, a configuration table.
• ["SQL Get First table_keycol name"] - An Execute SQL task which we’ll use to pull out primary key information from our destination table.  This just uses INFORMATION_SCHEMA to look up your target table and pull back the first column for the primary key.  If you use unique constraints or something else, just tweak the SQL or overwrite the destination variable.
• ["Truncate Destination Table"] - A second Execute SQL task to truncate our destination table (for a full load)
• ["Data Flow Task Plus"] - A CozyRoc DFT+ task for our dynamic loading.  The brains of the operation.

We also have variables.  In our production deployment we have lots and lots of variables.

The major points here are:

• table_colmap is a System.Object that is our collection of column names, IDs, and Lineage IDs for all columns in our DFT.  I scoped this to our DFT+ task because it’s specific to that task, but you could get away with scoping it to the package.
• Everything else.  We’re more or less mimicking the variables we used in previous articles.

Let’s move on to the DFT.  Open up the DFT+.  You’re going to see two main paths:

• We had an issue obtaining the source data.  (right side) Yes.  This does happen.  Case in point – you have a date of “-4444 AD” in Oracle.  The OLEDB driver we use for Oracle really doesn’t like that.  Or even a 44 digit numeric.
• We had an issue writing to the destination table. (left side)

Let’s give it a whirl and see what happens.

I’ve intentionally created opportunities for problems.

The destination columns will have conversion issues with

• email_addr length
• first_nm length
• last_nm length
• awesomeness (rating) size

If we run the package and review our error log we’ll see failures related to the highlighted columns.  (Note that I’ve removed some elements of the exception log here solely for formatting)

“You’re failing, Seymour! What is it about you and failure?”

There you go – row exceptions being logged for various issues with data from the dynamic DFT.

How Denali Should Fix This

We’re eagerly anticipating Denali for several reasons, but one fantastic piece of news is that SSIS in Denali should let us bypass most if not all of the issues with LinageID.  As Jorg Klein notes in one of his blog posts (http://sqlblog.com/blogs/jorg_klein/archive/2011/07/22/ssis-denali-ctp3-what-s-new.aspx):

SSIS always mapped columns from source to transformations or destinations with the help of lineage ids. Every column had a unique metadata ID that was known by all components in the data flow. If something changed in the source this would break the lineage ids and raised error messages like: The external metadata column collection is out of synchronization with the data source columns.
To fix this error you would re-map all broken lineage ids with the “Restore Invalid Column References Editor”.
In Denali lineage-ids are no longer used. Mappings are done on column names, which is great because you can now use auto map on column names and even copy/paste pieces of another data flow and connect them by mapping the corresponding column names.

Fan.  Tastic.  Couldn’t come soon enough.  Granted, you’ll have to upgrade to Denali to make use of this, but there are so many other compelling reasons to migrate (http://www.brentozar.com/sql/sql-server-denali-2011-2012/) that this is just icing on the cake.

Appendix – Code

This code is provided in the download, but for quick access / reference I’m also including it here.

DFT+ Column Collection Script

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
 
/*
 
 Add references to ...
     CozyRoc.SSISPlus.2008
     Microsoft.SqlServer.DTSPipelineWrap
     Microsoft.SQLServer.DTSRuntimeWrap 
 
 */
 
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using CozyRoc.SqlServer.SSIS.Attributes;
using CozyRoc.SqlServer.SSIS;
 
using System.Collections;
using System.Collections.Generic;
 
namespace ST_44af5cee356540e294c47d0aa17d41ed.csproj
{
    [System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]
    [DataFlowColumnAdded("OnColumnAdded")]//CozyRoc annotation
    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
    {
 
        #region VSTA generated code
        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion
 
        public void OnColumnAdded(IDTSComponentMetaData100 component, bool isInput, string colName)
        {
 
            try
            {
                if (!isInput)
                {
                    IDTSOutputColumn100 column = component.OutputCollection[0].OutputColumnCollection[colName];
                    column.TruncationRowDisposition = DTSRowDisposition.RD_RedirectRow;
                    column.ErrorRowDisposition = DTSRowDisposition.RD_RedirectRow;
                }
                if (isInput)
                {
 
                    IDTSInputColumn100 column = component.InputCollection[0].InputColumnCollection[colName];
                    Dictionary colmap = new Dictionary();
                    Variables variables = null;
 
                    try
                    {
                        Dts.VariableDispenser.LockOneForWrite("User::table_colmap", ref variables);
                        if (variables["User::table_colmap"].Value.GetType() == colmap.GetType())
                        {
                            colmap = (Dictionary)variables["User::table_colmap"].Value;
                        }
                        else
                        {
                        }
                        colmap.Add(column.ID, column.Name);
                        variables["User::table_colmap"].Value = colmap;//put the column collection back into the variable
                    }
                    catch (Exception exi)
                    {
                    }
                    finally
                    {
                        variables.Unlock();
                    }
                }
            }
            catch
            {
            }
 
        }
 
        public void Main()
        {
 
            Dts.TaskResult = (int)ScriptResults.Success;
        }
    }
}

Error Row Handler Script

/* Microsoft SQL Server Integration Services Script Component
*  This is CozyRoc Script Component Plus Extended Script
*  Write scripts using Microsoft Visual C# 2008.
*  ScriptMain is the entry point class of the script.*/
 
using System;
using System.Text;
 
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
 
using System.Collections;
using System.Collections.Generic; //for our dictionaries / lists
 
/*
 * HOW IT WORKS:
 * ======================================================================
 * Thanks to CozyRoc's great sample code (thanks, CozyRoc! :), we're able to rip through the
 * set of columns and find the error info critical for our logging / fixing.  We get the basic
 * column info on PreExecute() and store the column names, column lineage IDs, and column relative
 * position ("index") in two separate dictionaries for later.  We use LineageID as the key for those
 * and the later on during Input_ProcessInputRow to look up those names and IDs so we can pull back
 * data from the buffer and then also UPDATE the buffer to overwrite our custom error info columns
 *
 * Dictionary 1: Set of column names ("colnames") key: LineageID, value: column.Name
 * Dictionary 2: Set of column relative positions ("colids") key: LineageID, value colIndex
 *
 * PreExecute - set up objects for later.  IDs for columns, dictionaries, variables, etc.
 * Input_ProcessInputRow - the "real work" of adjusting / setting the values in the columns
 *
 * SETUP - READ THIS OR IT WON'T WORK
 * ======================================================================
 * REQUIRED INPUT COLUMNS
 * -------
 * We anticipate the following input columns being present (sent to the script task as inputs)
 *
 * Standard "Error Output" columns from tasks
 * ------
 * ErrorColumn      MSFT - The Lineage ID for the error column
 * ErrorCode        MSFT - The SSIS error code
 *
 * Additional error columns specific to our purposes.  You can reuse these or update the column names
 * ------
 * error_id         CUSTOM - Same as the SSIS error code, but we need them for our table
 * column_nm        CUSTOM - The name of the column where the error occurred
 * record_id_dsc    CUSTOM - the column name for the "primary key" column (EX: employee_id)
 * record_id        CUSTOM - the value/ID for the "primary key" column so you can look up the row later
 *                              EX:"12345" in column "employee_id"
 *
 * error_id         CUSTOM - the SSIS error (same as ErrorCode, but for my purposes we left it here)
 * error_dsc        CUSTOM - the human-readable description of the SSIS error EX: "The data was truncated."
 *
 * REQUIRED VARIABLES
 * -------
 * NOTE: You MUST set these up as a read-only variables within your script task.
 *
 * Package Variable: @colmap (dictionary) - the collection of column names and IDs for our dynamic columns
 *                                        - this is set in the outer DFT+ OnColumnAdded()
 *                                        - we use this to pull out the full list of columns since we can't get ahold
 *                                        - of the prior step's column IDs/LineageIDs when we're in this script task
 *
 * Package Variable: @table_keycol (string) - the name of the column that represents your primary key
 *                                              EX: "employee_id"
 *
 * This is a cheap hack, but for my situation I'm OK with that.  We don't necessarily know what a "key"
 * column is at this point - primary key, I mean here.  So to get around that we set that value in a variable
 * within the overall package.  We then use that variable to say "oh, that's the key column" later and retrieve
 * the column name and the column value so we can write out our primary key reference info.  You'll see the
 * obvious limitation - we don't support compound primary keys.  But neither does my logging table, so...
 * 
 
*/
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
 
    private int[] m_idx;
 
    private string key_col_name;        //"name" of primary key column.  EX: "employee_id".
    private int key_col_id;             //Relative column index / position of our "primary key" column
    //single primary key column.  Does not handle compound primary keys.  Retrieve this from a package variable since we want to handle this
    //dynamically and can't automatically determine it from within the package at runtime
 
    private Dictionary colnames;           //collection to store our colnames for later use within row processing section
    private Dictionary colpositions;
    private Dictionary colidsbyposition;
    private Dictionary colids;           //collection to store our column ids for later use within row processing section
    private Dictionary colmap;
 
    //Internal column tracking numbers.
    //You could probably avoid using these as separate variables, but...
    // 1. I'm not that clever
    // 2. I really, really wanted to explicitly watch them as they moved around
    private int i_error_code_id;
    private int i_error_column_id;
    private int i_error_id;
    private int i_column_nm;
    private int i_record_id;
    private int i_record_id_dsc;
    private int i_error_dsc;
    private int i_error_data;
 
    StringBuilder _sbColIDs = new StringBuilder();
    StringBuilder _sbErrorCols = new StringBuilder();
 
    private bool isSourceErrorOutput = false;// = true;
    private string _OLEDBSourceType = "";
 
    public override void PreExecute()
    {
        base.PreExecute();
 
        colnames = new Dictionary();
        colpositions = new Dictionary();
        colids = new Dictionary();
        colidsbyposition = new Dictionary();
 
        colmap = new Dictionary();
 
        try
        {
            if (Variables.tablecolmap.GetType() == colmap.GetType())
            {
                colmap = (Dictionary)Variables.tablecolmap;
            }
        }
        catch (Exception exi)
        {
        }
 
        IDTSInput100 input = base.ComponentMetaData.InputCollection[0];
        IDTSVirtualInput100 virtInput = input.GetVirtualInput();
        int colsCount = virtInput.VirtualInputColumnCollection.Count;
        m_idx = new int[colsCount];
        for (int colIndex = 0; colIndex < colsCount; colIndex++)         {             IDTSVirtualInputColumn100 column = virtInput.VirtualInputColumnCollection[colIndex];             //================================================================             //pull out the error codes and column IDs             if (string.Compare(column.Name, "ErrorColumn", true) == 0)             {                 i_error_column_id = colIndex;             }             if (string.Compare(column.Name, "ErrorCode", true) == 0)             {                 i_error_code_id = colIndex;             }             if (string.Compare(column.Name, "error_id", true) == 0)             {                 i_error_id = colIndex;             }             if (string.Compare(column.Name, "column_nm", true) == 0)             {                 i_column_nm = colIndex;             }             if (string.Compare(column.Name, "record_id", true) == 0)             {                 i_record_id = colIndex;             }             if (string.Compare(column.Name, "record_id_dsc", true) == 0)             {                 i_record_id_dsc = colIndex;             }             if (string.Compare(column.Name, "error_dsc", true) == 0)             {                 i_error_dsc = colIndex;             }             if (string.Compare(column.Name, "error_data", true) == 0)             {                 i_error_data = colIndex;             }             //add our column names to our list for later use             colnames.Add(column.LineageID, column.Name); //column.LineageID used to look up index of error column name in row             colids.Add(column.LineageID, colIndex); //column.LineageID used to look up index of error column index position in row             colidsbyposition.Add(colIndex, column.LineageID);             try             {                 colpositions.Add(column.Name, colIndex);             }             catch { }             try             {                 //is this column the "key" column we're using to identify the key values for the row? EX: primary key                 //NOTE: we're only doing this for a single member if a compound primary key                 if (string.Compare(column.Name, Variables.tablekeycol, true) == 0)//true = ignore case during comparison                 {                     key_col_id = colIndex;                     key_col_name = column.Name;                 }             }             catch { }             //================================================================             m_idx[colIndex] = base.HostComponent.BufferManager.FindColumnByLineageID(                 input.Buffer,                 column.LineageID);         }     }     public override void PostExecute()     {         base.PostExecute();     }     public override void Input_ProcessInputRow(InputBuffer Row)     {         int colsCount = m_idx.Length;         int cColLineageKey;         if (colsCount > 0)
        {
            try
            {
                //stuff the errocode into the error_id column
                Row.Buffer[m_idx[i_error_id]] = Row.Buffer[m_idx[i_error_code_id]];
            }
            catch (Exception ex)
            {
            }
 
            try
            {
                //get the value for the "primary key" column
                Row.Buffer[m_idx[i_record_id]] = Row.Buffer[m_idx[key_col_id]];
            }
            catch (Exception ex)
            {
            }
 
            try
            {
                //get the value for the "primary key" column
                Row.Buffer[m_idx[i_record_id_dsc]] = key_col_name;
            }
            catch (Exception ex)
            {
            }
 
            try
            {
                //get the error description
                Row.Buffer[m_idx[i_error_dsc]] = (ComponentMetaData.GetErrorDescription(int.Parse(Row.Buffer[m_idx[i_error_code_id]].ToString())));
            }
            catch (Exception ex)
            {
            }
 
            try
            {
                //get the name and value of the column that failed.
                if (i_error_column_id != null && i_error_column_id > 0 && i_error_column_id  0)
                        {
                            if (colmap.TryGetValue(cColLineageKey, out columnName))
                            {
                                //use the lineage_id to pull the column name
                                //columnName should be set
                                if (cColLineageKey != null && cColLineageKey > 0 && columnName != null && columnName.Length > 0)
                                {
                                    if (colpositions.TryGetValue(columnName, out currentposition))
                                    {
                                        //use the lineage_id to pull the column name
                                        //current position should be set
                                    }
                                }
                                if (cColLineageKey != null && cColLineageKey > 0 && currentposition >= 0)//&& currentposition != null)
                                {
                                    if (colidsbyposition.TryGetValue(currentposition, out cColLineageKey))
                                    {
                                        //use the lineage_id to pull the column name
                                        //current position should be set
                                    }
                                }
                            }
                            else
                            {
                                cColLineageKey = cColLineageKey + 1;
                            }
                        }
                        else
                        {
                            //probably a "source error output"
                            cColLineageKey = cColLineageKey + 1;
                            //MAJOR MAJOR MAJOR HACK
                            //apparently, we do NOT persist the ORIGINAL LINEAGEID from source to output, so we need to... adjust... the number.
                            // this is EXCEPTIONALLY RISKY, but since MS "adjusts" the output rows for errors to have be "different" from the "it works!" destination
                            // we don't have much of a choice.  In reviewing them #'s it appears they consistently increment for errors, so we need to increment the
                            // index here to find the right value.  Horrible stuff.  Likely to break.  Enjoy.
                        }
                    }
 
                    //Retrieve from the column names dictionary and place column name in error info
                    if (i_column_nm != null && i_column_nm > 0 && i_column_nm  0)
                        {
                            if (colnames.TryGetValue(cColLineageKey, out value))
                            {
                                //use the lineage_id to pull the column name
                                Row.Buffer[m_idx[i_column_nm]] = value;
                            }
                        }
                    }
                    //get the missing column value for the key found at the identified "error column"
                    //had issues where the column blew up because of data type conversion issues, so try/catch is here to help handle this
                    try
                    {
                        if (i_error_data != null && i_error_data > 0 && i_error_data  0)
                            {
                                if (colids.TryGetValue(cColLineageKey, out colvalue))
                                {
                                    //use the lineage_id to pull the column name
                                    //NOTE: "bad" data MAY be totally thrown out here, which is why we're using the try/catch
                                    //if the custom CozyRoc row processor dies due to formatting errors then this will throw an exception
                                    //we're just going to ignore that and roll on by
                                    //probably worth revisiting at a later date to see if we can get at the bad data anyway
                                    Row.Buffer[m_idx[i_error_data]] = Row.Buffer[m_idx[colvalue]].ToString();
 
                                }
                            }
                        }
                    }
                    catch (Exception vEx)
                    {
                    }
 
                }
                else
                {
                }
            }
            catch (Exception ex)
            {
            }
        }
 
    }
 
}

There is a famous blog post written by Joel Spolsky titled “The Joel Test.” It describes some key parts of what make a given development shop a great (or scary) place to work. It’s also a key metric used on careers.stackoverflow.com, where every employer posting a job is encouraged to rate themselves, and post their own score on “The Joel Test.”

I use “The Joel Test” as one of the ways that I critique a potential employer, what sort of value they place on developers, and software development as a practice. I’m especially passionate about software, more than any other professional or personal pursuit in my life, and I want to work for places that value the work I do for them just as highly as I value the work myself. As such, I think “The Joel Test” is a pretty good indicator of that aspect of a workplace.

Since I tend to agree with values put forth in “The Joel Test,” I look for a place that scores highly. After working inside NUBIC for just a few months, I’m pretty happy with what I see here. However, rather than providing a score for NUBIC (I obviously think we score very high), I’ll just layout what we do here:

1. Do you use source control? We have an internal Git server, and we also publish much of our code on GitHub.
2. Can you make a build in one step? We have a Jenkins CI server that is tied to our Git project repositories. It automatically runs our test suite when new code is committed to the master branch. It also sends out automated emails when a build fails.
3. Do you make daily builds? As often as code is committed, it is built and tested. For projects under active development it’s common to see multiple builds per day.
4. Do you have a bug database? We use a combination of Redmine for internal projects, and GitHub for open source projects.
5. Do you fix bugs before writing new code? We prioritize bugs over new features. For the purpose of this question I’m defining bugs as things that are broken in the system, as opposed to simply inconvenient, or inefficient. However, we definitely label inconvenient or inefficient code as a bug if it is preventing meaningful work from getting done (rightfully so), and will prioritize it as such.
6. Do you have an up-to-date schedule? We scope and schedule work for all of our projects, and keep clients in the loop at every step of the process, whether we’re ahead or behind schedule.
7. Do you have a spec? The vast majority of projects don’t have specs for every feature; some of them are as simple as, “Can you change this thing from red to blue, and move it to the left by 5 pixels?” which I suppose qualifies as a spec, but may be communicated verbally as opposed to going through a formal process. However, anytime there’s major work to be done, or any place that we feel has lots of ambiguity, we work it out verbally or on a whiteboard, and then commit the details of that session back to the bug tracking system. After a project launch we also produce documentation for internal purposes, as well as training documents and screencasts for the benefit of end users.
8. Do programmers have quiet working conditions? This is one of my favorite things about NUBIC. Though we do work largely in a set of cubicles, we accomplish a quiet working environment in a several ways. First, and foremost, we respect each other by taking conversations that we expect to last more than a minute or two away from the common workspace, and into side offices with closed doors that keep the sound contained. Second, most of us also have noise-cancelling headphones that we use to either listen to our favorite music while coding, or simply to soften the ambient office noise (which is nearly silent most of the time anyway). Third, we prefer asynchronous communication (email + IM) over coming to a person’s desk unannounced. Unless you’re on the support rotation for the week, email can safely be checked just 2-3 times a day, and on a typical day I will have fewer than two IM conversations. We also like to eat lunch together often, which sometimes serve as conversation starters for projects, in place of formal meetings.
9. Do you use the best tools money can buy? We’re not talking about the most expensive tools – we’re talking about the best tools to do the job. Besides personal preferences, the tools we’re given are a help, not a hindrance to our work. For example, I personally prefer FogBugz for issue tracking over GitHub, due to its integrated, and more advanced project management features, but GitHub is by no means a hindrance to the workflow. There are also regular, open discussions that occur regarding tool choice. Programmers are not only free to experiment with various tools, but encouraged to do so, bringing their experiences back to the group so everyone can benefit. We maintain a coding library that holds maybe 100 books that have been collected over the years, and cover topics ranging from various programming languages, to guides on design, typeface choice, and the art of software craftsmanship. As for the day-to-day tools, we’re lucky to have a solid, flexible hardware setup. Here’s a photo of the standard issue desk: 15″ MacBook Pro + Apple Cinema Display, and push-pin friendly walls for hanging up notes or creating a makeshift kanbanboard.
   

   MacBook Pro, Apple cinema display, whiteboard for prototyping

10. Do you have testers? In addition to test- and behavior-driven development practices that each developer employs, we have people who specialize in user experience and UI design. We also encourage Usability Testing, of the sort outlined in Steve Krug’s “Rocket Surgery Made Easy”. Finding a coworker interested in pair programming is also very easy. In NUBIC, responsibility for testing software is a partnership between developers, clients, and groups that verify projects on staging systems, as well as anyone else who is simply interested in assisting a given project. We do not, however, have dedicated testers whose job it is to do nothing but think of ways to break the system (in order to prevent it). There’s been quite a bit of discussion about this, and we’re hoping that as our group grows, we’ll be able to add dedicated testers. The reasons for wanting dedicated testers are many, but I think the greatest benefit to having them is that having someone whose job it is to think of ways to break things is fundamentally a different role than a developer (whose job it is to think of ways to make things work). Both sides are important.
11. Do new candidates write code during their interview? Developers are expected to be able to demonstrate their ability to implement a few common algorithms, as well as design a small system, live, during the interview. If you’re awesome at software, it’s not a big deal. It basically proves that you have three things: a solid background in algorithms, an ability to reason about a system and accept feedback from others on its design, and an ability to ask intelligent questions of the client about what they want.
12. Do you do hallway usability testing? Some people do this more than others, but anytime you’re implementing a new feature where you’d like some feedback, it’s encouraged that you consult with people around you. The culture in NUBIC is such that I’ve never run into a person who wasn’t willing to give you five minutes of their time to examine something you’re working on. This often turns into a discussion of several different approaches to implementation, and helps lead to software that benefits from multiple perspectives, without adding a bunch of wasted time to the project schedule in formal meetings.

On top of that, one of my favorite things about working at Northwestern is the focus on collaboration, and leveraging the large community of brilliant people around you. Northwestern is one of the leading research universities in the country, and they have no shortage smart people that are passionate about what they do.

The panoramic views of Lake Michigan aren’t bad either.

What has not hit the press yet is the issue of HIPAA potential violations.  It is important to remember that inter HITECH, covered entities are responsible for breaches even if they  didn’t know, and by reasonable diligence would not have known.  In other words, if your phone sent PHI to the phone company, you are potentially liable for $100-50,000 per violation (probably will be interpreted as each compromised message) and up to $1.5 million total.

Sprint, AT&T and T-Mobile admit to use Carrier IQ.  Verizon says it does not.  I’m sure that my hospital is not ready to ban all non-Verizon phones…yet.

This is not unique to mobile phones.  The new Kindle Fire provides web browsing, so you might use it to access your web-based EHR, right?  However, the actual rendering of the web page is done in the Amazon Cloud and a compressed version of the page is sent to your device.  From a security standpoint, this means that Amazon must  execute what amounts to a man-in-the-middle attack on your secure browsing session.  Whether Amazon looks at your data, or not, is irrelevant.  They can do it at any time, and you would be none the wiser.

Thanks to Matt who pointed out that EFF had a nice evaluation of security on the Kindle Fire.  It turns out that HTTPS sessions bypass the cloud rendering engine, so things are not as dire as I had thought.  Google also assures that they are not logging content.  Perhaps, but since they are logging the URL and session token, they can probably reproduce much of the content at a later date.

Ultimately, this is why domain specific privacy (e.g. health privacy, mail privacy or video rental privacy)  laws are doomed to fail.  Our digital lives are far too interconnected for any service provider to create separate services to comply with the separate regulations.  As a health care organization, we cannot generally share location data more precise than a 3-digit zip code.   The phone company can collect, and give to advertisers, my location down to a few feet!  I’m sure that the phone company can tell, if they wanted, what chronic medications I take simply based on which lab I go to each month.  We need a fundamental right of digital privacy.  That right need to be based on the concept of information use, rather than on information access.  Anyone who uses my digital data should admit that they use it and justify why.

A good analogy for this is credit ratings.  Anyone who uses a credit rating to grant or deny a service is required to say what information they used to make the decision.  If it appears that someone used information that they should not have, like age or race, they need to be able to explain how they reached the decision without using that data.  The same should be true of all of our digital “breadcrumbs” that was scatter across the digital landscape as we attempt to live our digital lives.

As we increasingly use electronic media to accelerate the spread of scientific and medical knowledge, we need to think seriously about how we insure that the information we spread is the most correct, not just the loudest, truthiest, sexiest, or best financed.

While I have not tried every stylus on the marked, I have tried a number.  The major problem with the rubber tipped styli is that the friction seems to change as the rubber gets worn.  iFaraday stylus uses a cloth tip that glides smoothly and does not seem to change as much as the rubber.  The iFaraday Artist, Firm Dome is the best I have found by far.

Of course, nothing is perfect.  I would love to see the firm dome available in the Rx stylus, which has a cap to cover the tip when not in use.  I would also love to see the Rx cap be clip on, rather than screw on.  Even so, this has become my go-to stylus for note taking, sketching, or making marginal notes in papers I read on the iPad.