Here’s a quick reading list of source materials that I’m using to learn how to do it (blog posts to follow as I document the process of building the system internally):

• The Continuous Delivery book
• The Continuous Delivery blog
• Eric Ries’ 5-step primer to Continuous Deployment
• Case Study: Continuous deployment makes releases non-events
• IMVU’s Approach to Integrating Quality Assurance with Continuous Deployment
• A series on the principle of “The 5 whys”
• The Pareto principle, a.k.a. the 80/20 rule

These things couple very well with additional practices that NUBIC embraces as part of its software development process, including:

• Test Driven Development
• Behavior Driven Development
• Continuous Integration
• Releasing much of our code as open source
• Kaizen – continuous improvement philosophy
• Doing well on the “Joel Test”

Finally, ThoughtWorks Studios has a commercial product called Go for automated release management. A couple of people from ThoughtWorks also happen to be the authors of the book on Continuous Delivery.

This post discusses just one way that sensitive information is transparently taken out of your hands, and put into places you didn’t expect, or weren’t obvious from the use of a given technology.

My example is this: putting username/password combinations in a URL, as outlined in this question on StackOverflow (you’ll probably want to read through this, for some background). While the implementation question posed is a simple one, and may make certain programming tasks simpler, it’s turned out to be a bad idea, because of the potential of that information leaking into unintended places.

As a follow up to that StackOverflow question, I received an email from Anthony Castillo (you can follow his blog at inquirious.com), that delved deeper into this problem, specifically in an iOS context. The discussion is germane to anyone thinking about the unintended consequences of transferring data over networks:

Hi Jeff,

[...]

I asked a question similar to one of yours on SO (regarding https://username:password@service.com type urls).

I’m trying to decide if it is safe *enough* for me to use that format for an iOS project I’m working on. Just wondering if you went with it or if you ended up doing something else.

 

I replied…

Hey Anthony, [...]

The short answer to your follow up is “no – it’s not considered good enough”. Security, to some extent, is a matter of managing risk. Username+password in HTTPS links potentially increases your risk with basically no reward. Any logs on systems between you and the destination system that capture that information (or even non-secure logs on your own machine), effectively makes using HTTPS useless for this purpose, since you wind up putting your credentials in the clear.

No bueno.

I originally wrote this question because I was trying to automate a connection between a script on my local machine, and my source repository, thereby preventing a password prompt form coming up at all, so the script could run unattended. The problem I was trying to solve was automating a web app deployment. However, I found that this wasn’t the best thing to do, for the following reasons:

1. It’s better for security by far to use key-based authentication (rather than username+password), and connect over SSL for deployments and other automated processes, rather than HTTPS. The reasons for this are many, but the benefits are basically that automation can happen pretty easily after you get this setup, and because you’re not using passwords, the encryption is generally considered less likely to be broken, since your “secret key” is a high-security key, rather than a human-chosen password.
2. Having fully automated deployments was troublesome without also having an automatic rollback process. In my case, if a developer wasn’t involved in the deployment, that was a bigger problem than the inconvenience of having to be physically present to enter one’s password. [...]

Back to your question, HTTPS connections are, as I understand them, tunnels directly between the client and the server, so theoretically security should be okay. However, you never know how many machines that request goes through before that tunnel is established (your ISP/wireless provider, for example, is one place that might get a hold of the URL with the username+password in it). Ideally, whatever code library you’re using to make the connection should drop the username+password from the URL while establishing the tunnel, however, there’s no guarantee, and it’s just better never to store your username+password in plain text anywhere that you don’t absolutely have to do so, especially since there are much better, and more secure, alternatives that are just as easy to setup. In any cases where you’ve determined that it is absolutely necessary to store credentials in plain text (I can’t think of any in the modern world in which we live), it’s critical that the file system on which it is stored is a system completely within your control, and not accessible via the Internet, lest someone get access to it. This includes not only the file system itself, but any locations/systems, onsite or off, to which that file system is backed up. In the real world, guaranteeing such a situation is difficult at best, and in practical terms, is probably closer to impossible.

As for your iOS app, that’s a bit trickier, depending on your situation, because you’re not talking about one key, and one server, you’re talking about a bunch of users, and a server they connect to. However, I would think that you could achieve key-based authentication for your users by giving them an API/application key that they could copy/paste into your iOS app that would essentially accomplish the same thing. If you need to go over HTTPS, you can pass the key in as a POST parameter to the session, and allow secure data transfer between your app and the server, in addition to the key acting as authentication (this user is who they say they are). I’m not an iOS expert, but one of my colleagues mentioned that iOS offers some encrypted storage on the device to store sensitive information such as credentials, and that it’s been available to iOS devices at least since iOS v4 (give or take). [..]

Anthony replies…

Hi Jeff, [...]

 

After reading your email, I started digging around the web a bit more. Please let me know what you think of the following:

Basic Access Authentication - this is the “username:password@service.com” format that we have been discussing. This is as opposed to Digest Access Authentication. (There may be others as well.)

• Basic authentication can be used with either HTTP or HTTPS.
• With HTTP, it is definitely insecure.
• When used with HTTPS, that implies SSL encryption over the whole connection.

So…I think it’s okay to use basic access authentication, as long as it’s over HTTPS. (1) (2)

However, the point you make about not knowing how many machines a request goes through before reaching the server leaves me with slight doubts.

Perhaps we have slightly different choices here because you have control over both the client and server sides of the connection. Hence your ability to choose a protocol involving an API key.

As for me, I am building an app against a third-party web service. This means I have to go along with whatever protocol they have in place. In this case, it’s username/password authentication over HTTPS.

Maybe it isn’t completely secure, but it might still be the best I can do given my choices. (I think that’s my conclusion.) I don’t believe there is a way for me to force a different authentication method on the service to which I’m connecting.

Regarding encrypted storage on iOS, that is indeed a standard feature offered by Apple. They have what is known as a “Keychain” feature that allows you to store and encrypt data on an app by app basis. I basically let the OS handle the details of that for me. (Pretty nice.)

And finally…

Anthony,4

In your situation, it sounds like you’re right – if this is the only authentication method provided (and they don’t publish per-user API keys) you’re kind of up the creek on that one. You should still avoid putting the username/password in the URL itself if at all possible, depending on the limitations of the API requests. I would, if possible, contact the vendor and ask that they add it [key-based authentication]. In the mean time, I think you’re right that you’re stuck, according to what you’ve outlined.

The digest authentication looks like your best best, but isn’t considered as secure as key-based authentication (as noted in the wikipedia article you linked to), but it looks like you already know that. [...]

So, it’s always a good idea to think about where your information is not only stored, but where it’s transferred to, and what that might mean in terms of that information getting out of your control inadvertently. Thanks to Anthony Castillo for the deeper discussion, and a specific example to which we could apply the principle.

It’s not the users’ fault. Honestly, it’s not.

When answering a technical support question, have you ever asked someone, “Did you read the manual?” Well, put away your superiority complex for a moment, and realize that your users are wondering why they need a manual in the first place.

Manuals stink, plain and simple, so stop using them whenever possible. If you’ve got a complex application, website (or really any training process whatsoever), and you feel that you aren’t receiving the respect you deserve for writing that 900-page, 100% comprehensive training manual, stop spending time on trying to improve the manual, and instead change the system.

• Ask why – first and foremost you need to understand why the user is having a problem with your application, then you need to correct the flaw that is causing the problem in the first place, thereby eliminating the need for the user to ask the question at all. A great method accomplishing this is 5 Whys. During the process of asking “why” it’s important to always be gracious about honest feedback, and curious about how people arrive at their state of confusion. Once you’ve figured out what’s at the root of the problem, it’s usually a trivial thing to change it.
• Show, don’t tell – create a short training video that shows people how to use it, rather than trying to explain it via text and pictures. If your training video can’t correctly explain it in less than three minutes, your app is either too complex, or your video is trying to do too much. Either fix your app, or sharpen the focus of your video. Great examples of awesome instructional videos are the videos that introduce SquareSpace. They are short, focused on a single topic each, and (in the case of SquareSpace) linked directly from the pages in which the related question might be raised in the user’s mind. A user is editing a webpage and wants to know how to add an image? The video for editing pages is linked from the page editing screen. Simple. It’s true that they still maintain a searchable collection of videos that any user can simply watch, but the fact of the matter is that pretty much no one is going to go through this library and watch all the videos first. Users will typically try something, and only when they fail, will they ask for help.
• Protect users from accidents - There are many times that users will do things that they don’t know are dangerous until it’s too late, and they can’t go back! Whenever possible, provide an “undo” function that allows users to fix mistakes with a simple click or keystroke. This method is often far superior then shifting all responsibility to the user, and presenting them with, “Are you sure?! You cannot undo this!” sorts of messages. Those messages make users fearful, cause them to stop and call you for help making a decision about what to do, and ultimately shift blame to the user when simply providing an “undo” function largely avoids the problem from happening the first place. Even the most seasoned users will occasionally make mistakes. These people aren’t “dumb,” and they’re just human after all. Do you really want to have to recover lost data, or blame them for the mistake, when your system could simply protect users from such accidents in the first place?
• Automate it – sometimes people make mistakes when doing repetitive tasks, because humans aren’t as good at doing highly repetitive things accurately 100% of the time, as compared to computers. This problem is exacerbated by processes that have multiple steps, where a mistake in any one of the steps can cause the whole process to break down. Try helping the users of your site or application by pre-filling in values for forms, automatically inserting reasonable default values, or better yet, just completely automate the process whenever possible. If there’s no reason that a human really needs to be involved in a process, take them out of the loop and save everyone some time and energy.
• Language is imprecise – step-by-step instructions, no matter how detailed and precise, no matter how carefully worded, are difficult to follow. Users gets lost in lengthy instructions, misunderstand or misinterpret technical terms, and people simply don’t want to read instructions anyway. Providing users with a glossary of terms (thinking that the manual should explain itself) isn’t really the answer either. So, use pictures instead of words when possible, and video instead of pictures when possible. The complications of interpreting language is part of why IKEA’s assembly instructions contain no words, only pictures.

The introductory page that explains how to avoid damaging your new furniture during assembly, and what to do if you need help or are confused. Pretty clear, yes? (1) put a carpet or rug under the pieces while assembling them, (2) if you're confused, look in the manual for a picture that shows what to do, and (3) call IKEA. Note that the last picture isn't a person on a phone calling IKEA - it's literally a handset connected to IKEA. When I see this, I think only two words: "phone IKEA". The implication is uncomplicated, and clear. Also note this caption of those four pictures took an entire paragraph. Not very efficient, friendly, or helpful, is it?

TLDR:

• Download the sample package (Dynamic DFT Error Handler Sample Project)
• Run the SQL script
• Open up the package and make sure your connections are set up appropriately

Management of Bad Rows in SSIS

For ETL, SSIS does a fine job at letting you manage the basics of copying one column of data in some source table to another column of data in destination table.  Assuming all goes well, you wind up extracting/transforming/loading that data.

If things don’t go well, however…

Exception handling is a central part of any development task and usually winds up representing a significant chunk of your time and code. You wind up covering any number of “what ifs” like:

• What if I failed to connect to a system?
• What if I expected data and didn’t get any?
• What if my expected data type overflowed?
• What if something totally unanticipated happened?

If you’ve used SSIS for ETL you’re accustomed to the idea of data flow paths inside of a transformation.  You connect a source component to a destination component via either a green line (“good output”) or a red line (“bad / error output”).  This is great stuff.  Say you query some rows from a source database table and want to send the rows to a destination database table – you simply wire up the green line from the source to the destination and map the columns.  Done.  Walk away.

But what about the implied red line for bad rows?  What if you actually have an issue with the transformation?  Two immediate reasons come to mind:

• The data was truncated in some way (cast my Oracle number(20,0) to a SQL int)
• Some other unanticipated error occurred (for the sake of explanation, let’s say a primary key violation on insert)

Usually what you’d do with a static transformation is simply use row redirection to handle the exception.  A common solution is to log your error information to a shared error log table for later review.  By attaching the appropriate error output to your destination you “channel” the row information to that destination so you have a hope of figuring out what happened and what you can do about it.

SSIS usually works really well for these situations, with the exception of two nagging challenges you’ll see come up a lot in discussion forums:

• “My row failed – how do I get the error description?”
• “My row failed – how do I tell which row failed?”

Error description is fairly straight forward and I’m not going to get into it too much – there’s a great step-by-step example at (http://consultingblogs.emc.com/jamiethomson/archive/2005/08/08/1969.aspx) which is very instructive.

Error row identifier, though, is a bit more complex because of the way SSIS works.

Error Columns and Lineage IDs

Let’s say you have a row with an integer column “employee_id” which is the primary key on a table.  What you see is a single presentation of that column “employee_id” – it’s labeled that way throughout your data transformation flow, so to you it’s “the same” throughout the flow.  What SSIS sees internally, however, is something completely different.  If you dig a bit you’ll find you have a unique representation of this column at each point throughout the flow of your SSIS package.  That single “column” (“employee_id”) has to be treated uniquely at each input, output, and error output for each step.  Beyond needing to understand how to treat flow direction (ex: input column vs output column), the column itself may change data types, names, or even value as it flows through your package.  SSIS needs to keep track of that “column” at each point throughout the flow and treat it as though it’s unique.  So how does it do that?  LineageID.

There’s a great article on SQL Server Central (http://www.sqlservercentral.com/articles/Integration+Services+(SSIS)/65730/ ) that touches on some of this.  The article describes lineageid as

It’s an integer and it’s unique throughout the data flow. When buffers are reused, the Lineage ID doesn’t change – it’s the same column at the input and output. When buffers are copied, a new column is created – which gets a new (unique) Lineage ID.

That means that as the column “employee_id” flows through the DFT, it gets a unique Lineage ID - for each input and output copy of itself.  And, typically, you have…

• An input column
• An output column for “good” data
• An output column for errors

Taking the “employee_id” example from the “OLE DB Source” step in our DFT we’d have:

• Input (ID = 33)
• Source Error Output (Lineage ID 35)
• Good Output (Lineage ID = 34)

Great!  No problem.  As long as we know the LineageIDs related to our steps we can back track to determine the mapping to “column name” and voila – we know which row failed.  We can simply look up the column by LineageID using “FindColumnByLineageID” in a script task (http://msdn.microsoft.com/en-us/library/microsoft.sqlserver.dts.pipeline.wrapper.idtsbuffermanager100.findcolumnbylineageid.aspx).  Magic.

Not so fast.  One small, but critical catch.  Metadata about a task step is only available within the scope of that task step. Meaning – once we get past “OLE DB Source” I can see “Lineage ID,” but I can’t easily track back to determine the mapping of Lineage ID to column name.  So – if you want to write out error row information (specifically “column name”) in a second DFT (to your error log, for example) there’s no way to look up that name – because the metadata about LineageID is no longer in scope – it’s only available to the prior step.  Incredibly frustrating.

Getting Error Column Information With Static DFTs

For static packages this can be addressed a few ways. The general strategy is to map the Lineage IDs / IDs to column information at design time and then use that information to look up the information you need.

Couple of quick links you may find handy.

• How to Find Out Which Column Caused SSIS to Fail? (http://blogs.msdn.com/b/helloworld/archive/2008/08/01/how-to-find-out-which-column-caused-ssis-to-fail.aspx)
• Error Output’s Description (Component on CodePlex) (http://eod.codeplex.com/ )
• eLog (http://ssisctc.codeplex.com/wikipage?title=eLog&referringTitle=Home)

Again – for static packages, these can mostly if not completely solve the issue and leave you in a far better position to determine which rows failed.  I’m not going to go into these since you can read up online.

So What About a Dynamic DFT?

Note that the links I provided above address design time gathering / mapping of column information.   What do you do about a runtime situation?  We started digging into the CozyRoc dynamic DFT about a year ago.  Basic dynamic mappings worked great.  You can easily remap columns at runtime and, assuming all goes well, you’re done.  But if things don’t go well – what then?

We need to catch and log those bad rows.  But – we can’t map columns / Lineage ID information at design time because that negates the entire point of using a dynamic DFT – you won’t know any of the required information. It’s just not there.  Now that issue with the resolution of metadata from prior steps comes into play.  We can’t generate column information at design time and we can’t inspect metadata from ancestor steps within a DFT.  They’re out of scope.

I’ll admit that when I first looked at this I was stumped.  And incredibly frustrated.  There was this great opportunity to really let SSIS rock using CozyRoc’s dynamic DFT, but the inability to handle bad rows in a data warehousing solution is a showstopper (keep in mind the issue here is an SSIS design constraint, not a CozyRoc fault).  Following the examples for handling static mappings online (thank you very much, above-linked article authors), we had the notion that we should be able to pull some of the DFT information out at runtime and approach the problem somewhat similarly.

• Upon startup, obtain a list of all columns, their IDs, and their Lineage IDs
• Store that list in a collection
• Using the IDs / Lineage IDs from the errors to look up the corresponding record in our collection
• Profit

I rang up CozyRoc and discussed the situation with their engineers.  They immediately understood my intentions and mailed me back a quick sample of some code that exploited a fantastic capability of their dynamic DFT – the ability to add script to the DFT itself. (Thanks, CozyRoc!)  Not code via a script task within the DFT, but on the DFT directly.

CozyRoc DFT+ (http://www.cozyroc.com/ssis/data-flow-task) notes that you can apply script on the DFT by accessing…

• Advanced tab – specifies advanced task options.
• Script page – specifies data flow task script, which is used for Setup tab customization.

Aha.  And the magic snippet they supplied me…

public void OnColumnAdded(IDTSComponentMetaData100 component, bool isInput, string colName)
//do stuff

Great!  They provided event hooks for the dynamic column mapping!  So now I can detect when a column is added to the DFT flow, add it to my reference collection of column information, and then access that collection within the DFT to derive column information critical to error logging.

This will let me take “Lineage ID” 12345 at any point throughout the flow and figure out that it was column “employee_name_concat” or whatever and log that.  We’re in business.

Something to note here.  Handling row truncation behavior is trickier when you’re doing this dynamically.  You can now longer manually address the need to “redirect on truncation” on a column by column basis, so you just extend the magic DFT+ column binding event to do it for you.

if (!isInput)
{
      IDTSOutputColumn100 column = component.OutputCollection[0].OutputColumnCollection[colName];
      column.TruncationRowDisposition = DTSRowDisposition.RD_RedirectRow;
      column.ErrorRowDisposition = DTSRowDisposition.RD_RedirectRow;
}

Done. Setting row disposition behavior accomplished.

From there we wrote up the nastier parts of the whole exercise – the entire collection lookup mechanism to derive column information.  We did that as a script task within the body of the DFT.

The script task pulls rows out of the buffer and evaluates row position to resolve the Lineage ID / ID and determine…

• Source column name (EX: “first_name”)
• Source primary key name (EX: “employee_id”)
• Source primary key value (EX: “12345″)
• Error description (using ComponentMetaData.GetErrorDescription)
• Error data (so we can quickly eyeball the offending column)

You’ll note I said “primary key name” – we felt it was “good enough” for the moment to avoid dealing with compound keys.  That’s definitely a shortcoming, but for the time being we felt that was acceptable since it matched our existing static ETL error handling process.  It’s definitely something that needs to be addressed, though.  We also cheat by explicitly passing in the primary key as an element of the process (we derive it at an earlier step) – again, in consulting speak, an “opportunity for improvement.”

Putting it All Together

Now that we’ve touched on the ideas, let’s see it work.  Rather than walk you through the entire step-by-step process of building a package I’m going to suggest you Dynamic DFT Error Handler Sample Project.  I’ll quickly touch on the major points on how the sample works.

The download includes some SQL scripts to set up…

• [etl_proxy_table, etl_proxy_table_stg, etl_proxy_table_src] We use some fake placeholder “proxy” tables  so you can set up data bindings in the DFT+.  CozyRoc also suggests you use THUNK_COLUMNs to do this, but I’ve found using these placeholder tables to be very helpful.  The reason we use these is that the magic OnColumnAdded method only fires when a column is actually added to the DFT. If you statically map any of the columns the entire error handling approach will fail because we won’t have those “static” columns added to our column collection.  Huge thank-you to CozyRoc for clueing me in on that.
• [etl_errors] our error logging table. YMMV, but remember if you change this you also need to adjust the scripts in the DFT.
• [demo_source_table, demo_dest_table] our source and destination tables.  We’re big Simpsons fans over here, so I’ve provided appropriate sample data.

The overall package has a few steps:

• ["Set Table Information"] - A cheater Script Task to mimic pulling table configuration information.  In a production scenario you’d likely want to provide configuration elements from either a config file or, better yet, a configuration table.
• ["SQL Get First table_keycol name"] - An Execute SQL task which we’ll use to pull out primary key information from our destination table.  This just uses INFORMATION_SCHEMA to look up your target table and pull back the first column for the primary key.  If you use unique constraints or something else, just tweak the SQL or overwrite the destination variable.
• ["Truncate Destination Table"] - A second Execute SQL task to truncate our destination table (for a full load)
• ["Data Flow Task Plus"] - A CozyRoc DFT+ task for our dynamic loading.  The brains of the operation.

We also have variables.  In our production deployment we have lots and lots of variables.

The major points here are:

• table_colmap is a System.Object that is our collection of column names, IDs, and Lineage IDs for all columns in our DFT.  I scoped this to our DFT+ task because it’s specific to that task, but you could get away with scoping it to the package.
• Everything else.  We’re more or less mimicking the variables we used in previous articles.

Let’s move on to the DFT.  Open up the DFT+.  You’re going to see two main paths:

• We had an issue obtaining the source data.  (right side) Yes.  This does happen.  Case in point – you have a date of “-4444 AD” in Oracle.  The OLEDB driver we use for Oracle really doesn’t like that.  Or even a 44 digit numeric.
• We had an issue writing to the destination table. (left side)

Let’s give it a whirl and see what happens.

I’ve intentionally created opportunities for problems.

The destination columns will have conversion issues with

• email_addr length
• first_nm length
• last_nm length
• awesomeness (rating) size

If we run the package and review our error log we’ll see failures related to the highlighted columns.  (Note that I’ve removed some elements of the exception log here solely for formatting)

“You’re failing, Seymour! What is it about you and failure?”

There you go – row exceptions being logged for various issues with data from the dynamic DFT.

How Denali Should Fix This

We’re eagerly anticipating Denali for several reasons, but one fantastic piece of news is that SSIS in Denali should let us bypass most if not all of the issues with LinageID.  As Jorg Klein notes in one of his blog posts (http://sqlblog.com/blogs/jorg_klein/archive/2011/07/22/ssis-denali-ctp3-what-s-new.aspx):

SSIS always mapped columns from source to transformations or destinations with the help of lineage ids. Every column had a unique metadata ID that was known by all components in the data flow. If something changed in the source this would break the lineage ids and raised error messages like: The external metadata column collection is out of synchronization with the data source columns.
To fix this error you would re-map all broken lineage ids with the “Restore Invalid Column References Editor”.
In Denali lineage-ids are no longer used. Mappings are done on column names, which is great because you can now use auto map on column names and even copy/paste pieces of another data flow and connect them by mapping the corresponding column names.

Fan.  Tastic.  Couldn’t come soon enough.  Granted, you’ll have to upgrade to Denali to make use of this, but there are so many other compelling reasons to migrate (http://www.brentozar.com/sql/sql-server-denali-2011-2012/) that this is just icing on the cake.

Appendix – Code

This code is provided in the download, but for quick access / reference I’m also including it here.

DFT+ Column Collection Script

using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
 
/*
 
 Add references to ...
     CozyRoc.SSISPlus.2008
     Microsoft.SqlServer.DTSPipelineWrap
     Microsoft.SQLServer.DTSRuntimeWrap 
 
 */
 
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using CozyRoc.SqlServer.SSIS.Attributes;
using CozyRoc.SqlServer.SSIS;
 
using System.Collections;
using System.Collections.Generic;
 
namespace ST_44af5cee356540e294c47d0aa17d41ed.csproj
{
    [System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]
    [DataFlowColumnAdded("OnColumnAdded")]//CozyRoc annotation
    public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
    {
 
        #region VSTA generated code
        enum ScriptResults
        {
            Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
            Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
        };
        #endregion
 
        public void OnColumnAdded(IDTSComponentMetaData100 component, bool isInput, string colName)
        {
 
            try
            {
                if (!isInput)
                {
                    IDTSOutputColumn100 column = component.OutputCollection[0].OutputColumnCollection[colName];
                    column.TruncationRowDisposition = DTSRowDisposition.RD_RedirectRow;
                    column.ErrorRowDisposition = DTSRowDisposition.RD_RedirectRow;
                }
                if (isInput)
                {
 
                    IDTSInputColumn100 column = component.InputCollection[0].InputColumnCollection[colName];
                    Dictionary colmap = new Dictionary();
                    Variables variables = null;
 
                    try
                    {
                        Dts.VariableDispenser.LockOneForWrite("User::table_colmap", ref variables);
                        if (variables["User::table_colmap"].Value.GetType() == colmap.GetType())
                        {
                            colmap = (Dictionary)variables["User::table_colmap"].Value;
                        }
                        else
                        {
                        }
                        colmap.Add(column.ID, column.Name);
                        variables["User::table_colmap"].Value = colmap;//put the column collection back into the variable
                    }
                    catch (Exception exi)
                    {
                    }
                    finally
                    {
                        variables.Unlock();
                    }
                }
            }
            catch
            {
            }
 
        }
 
        public void Main()
        {
 
            Dts.TaskResult = (int)ScriptResults.Success;
        }
    }
}

Error Row Handler Script

/* Microsoft SQL Server Integration Services Script Component
*  This is CozyRoc Script Component Plus Extended Script
*  Write scripts using Microsoft Visual C# 2008.
*  ScriptMain is the entry point class of the script.*/
 
using System;
using System.Text;
 
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
 
using System.Collections;
using System.Collections.Generic; //for our dictionaries / lists
 
/*
 * HOW IT WORKS:
 * ======================================================================
 * Thanks to CozyRoc's great sample code (thanks, CozyRoc! :), we're able to rip through the
 * set of columns and find the error info critical for our logging / fixing.  We get the basic
 * column info on PreExecute() and store the column names, column lineage IDs, and column relative
 * position ("index") in two separate dictionaries for later.  We use LineageID as the key for those
 * and the later on during Input_ProcessInputRow to look up those names and IDs so we can pull back
 * data from the buffer and then also UPDATE the buffer to overwrite our custom error info columns
 *
 * Dictionary 1: Set of column names ("colnames") key: LineageID, value: column.Name
 * Dictionary 2: Set of column relative positions ("colids") key: LineageID, value colIndex
 *
 * PreExecute - set up objects for later.  IDs for columns, dictionaries, variables, etc.
 * Input_ProcessInputRow - the "real work" of adjusting / setting the values in the columns
 *
 * SETUP - READ THIS OR IT WON'T WORK
 * ======================================================================
 * REQUIRED INPUT COLUMNS
 * -------
 * We anticipate the following input columns being present (sent to the script task as inputs)
 *
 * Standard "Error Output" columns from tasks
 * ------
 * ErrorColumn      MSFT - The Lineage ID for the error column
 * ErrorCode        MSFT - The SSIS error code
 *
 * Additional error columns specific to our purposes.  You can reuse these or update the column names
 * ------
 * error_id         CUSTOM - Same as the SSIS error code, but we need them for our table
 * column_nm        CUSTOM - The name of the column where the error occurred
 * record_id_dsc    CUSTOM - the column name for the "primary key" column (EX: employee_id)
 * record_id        CUSTOM - the value/ID for the "primary key" column so you can look up the row later
 *                              EX:"12345" in column "employee_id"
 *
 * error_id         CUSTOM - the SSIS error (same as ErrorCode, but for my purposes we left it here)
 * error_dsc        CUSTOM - the human-readable description of the SSIS error EX: "The data was truncated."
 *
 * REQUIRED VARIABLES
 * -------
 * NOTE: You MUST set these up as a read-only variables within your script task.
 *
 * Package Variable: @colmap (dictionary) - the collection of column names and IDs for our dynamic columns
 *                                        - this is set in the outer DFT+ OnColumnAdded()
 *                                        - we use this to pull out the full list of columns since we can't get ahold
 *                                        - of the prior step's column IDs/LineageIDs when we're in this script task
 *
 * Package Variable: @table_keycol (string) - the name of the column that represents your primary key
 *                                              EX: "employee_id"
 *
 * This is a cheap hack, but for my situation I'm OK with that.  We don't necessarily know what a "key"
 * column is at this point - primary key, I mean here.  So to get around that we set that value in a variable
 * within the overall package.  We then use that variable to say "oh, that's the key column" later and retrieve
 * the column name and the column value so we can write out our primary key reference info.  You'll see the
 * obvious limitation - we don't support compound primary keys.  But neither does my logging table, so...
 * 
 
*/
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
 
    private int[] m_idx;
 
    private string key_col_name;        //"name" of primary key column.  EX: "employee_id".
    private int key_col_id;             //Relative column index / position of our "primary key" column
    //single primary key column.  Does not handle compound primary keys.  Retrieve this from a package variable since we want to handle this
    //dynamically and can't automatically determine it from within the package at runtime
 
    private Dictionary colnames;           //collection to store our colnames for later use within row processing section
    private Dictionary colpositions;
    private Dictionary colidsbyposition;
    private Dictionary colids;           //collection to store our column ids for later use within row processing section
    private Dictionary colmap;
 
    //Internal column tracking numbers.
    //You could probably avoid using these as separate variables, but...
    // 1. I'm not that clever
    // 2. I really, really wanted to explicitly watch them as they moved around
    private int i_error_code_id;
    private int i_error_column_id;
    private int i_error_id;
    private int i_column_nm;
    private int i_record_id;
    private int i_record_id_dsc;
    private int i_error_dsc;
    private int i_error_data;
 
    StringBuilder _sbColIDs = new StringBuilder();
    StringBuilder _sbErrorCols = new StringBuilder();
 
    private bool isSourceErrorOutput = false;// = true;
    private string _OLEDBSourceType = "";
 
    public override void PreExecute()
    {
        base.PreExecute();
 
        colnames = new Dictionary();
        colpositions = new Dictionary();
        colids = new Dictionary();
        colidsbyposition = new Dictionary();
 
        colmap = new Dictionary();
 
        try
        {
            if (Variables.tablecolmap.GetType() == colmap.GetType())
            {
                colmap = (Dictionary)Variables.tablecolmap;
            }
        }
        catch (Exception exi)
        {
        }
 
        IDTSInput100 input = base.ComponentMetaData.InputCollection[0];
        IDTSVirtualInput100 virtInput = input.GetVirtualInput();
        int colsCount = virtInput.VirtualInputColumnCollection.Count;
        m_idx = new int[colsCount];
        for (int colIndex = 0; colIndex < colsCount; colIndex++)         {             IDTSVirtualInputColumn100 column = virtInput.VirtualInputColumnCollection[colIndex];             //================================================================             //pull out the error codes and column IDs             if (string.Compare(column.Name, "ErrorColumn", true) == 0)             {                 i_error_column_id = colIndex;             }             if (string.Compare(column.Name, "ErrorCode", true) == 0)             {                 i_error_code_id = colIndex;             }             if (string.Compare(column.Name, "error_id", true) == 0)             {                 i_error_id = colIndex;             }             if (string.Compare(column.Name, "column_nm", true) == 0)             {                 i_column_nm = colIndex;             }             if (string.Compare(column.Name, "record_id", true) == 0)             {                 i_record_id = colIndex;             }             if (string.Compare(column.Name, "record_id_dsc", true) == 0)             {                 i_record_id_dsc = colIndex;             }             if (string.Compare(column.Name, "error_dsc", true) == 0)             {                 i_error_dsc = colIndex;             }             if (string.Compare(column.Name, "error_data", true) == 0)             {                 i_error_data = colIndex;             }             //add our column names to our list for later use             colnames.Add(column.LineageID, column.Name); //column.LineageID used to look up index of error column name in row             colids.Add(column.LineageID, colIndex); //column.LineageID used to look up index of error column index position in row             colidsbyposition.Add(colIndex, column.LineageID);             try             {                 colpositions.Add(column.Name, colIndex);             }             catch { }             try             {                 //is this column the "key" column we're using to identify the key values for the row? EX: primary key                 //NOTE: we're only doing this for a single member if a compound primary key                 if (string.Compare(column.Name, Variables.tablekeycol, true) == 0)//true = ignore case during comparison                 {                     key_col_id = colIndex;                     key_col_name = column.Name;                 }             }             catch { }             //================================================================             m_idx[colIndex] = base.HostComponent.BufferManager.FindColumnByLineageID(                 input.Buffer,                 column.LineageID);         }     }     public override void PostExecute()     {         base.PostExecute();     }     public override void Input_ProcessInputRow(InputBuffer Row)     {         int colsCount = m_idx.Length;         int cColLineageKey;         if (colsCount > 0)
        {
            try
            {
                //stuff the errocode into the error_id column
                Row.Buffer[m_idx[i_error_id]] = Row.Buffer[m_idx[i_error_code_id]];
            }
            catch (Exception ex)
            {
            }
 
            try
            {
                //get the value for the "primary key" column
                Row.Buffer[m_idx[i_record_id]] = Row.Buffer[m_idx[key_col_id]];
            }
            catch (Exception ex)
            {
            }
 
            try
            {
                //get the value for the "primary key" column
                Row.Buffer[m_idx[i_record_id_dsc]] = key_col_name;
            }
            catch (Exception ex)
            {
            }
 
            try
            {
                //get the error description
                Row.Buffer[m_idx[i_error_dsc]] = (ComponentMetaData.GetErrorDescription(int.Parse(Row.Buffer[m_idx[i_error_code_id]].ToString())));
            }
            catch (Exception ex)
            {
            }
 
            try
            {
                //get the name and value of the column that failed.
                if (i_error_column_id != null && i_error_column_id > 0 && i_error_column_id  0)
                        {
                            if (colmap.TryGetValue(cColLineageKey, out columnName))
                            {
                                //use the lineage_id to pull the column name
                                //columnName should be set
                                if (cColLineageKey != null && cColLineageKey > 0 && columnName != null && columnName.Length > 0)
                                {
                                    if (colpositions.TryGetValue(columnName, out currentposition))
                                    {
                                        //use the lineage_id to pull the column name
                                        //current position should be set
                                    }
                                }
                                if (cColLineageKey != null && cColLineageKey > 0 && currentposition >= 0)//&& currentposition != null)
                                {
                                    if (colidsbyposition.TryGetValue(currentposition, out cColLineageKey))
                                    {
                                        //use the lineage_id to pull the column name
                                        //current position should be set
                                    }
                                }
                            }
                            else
                            {
                                cColLineageKey = cColLineageKey + 1;
                            }
                        }
                        else
                        {
                            //probably a "source error output"
                            cColLineageKey = cColLineageKey + 1;
                            //MAJOR MAJOR MAJOR HACK
                            //apparently, we do NOT persist the ORIGINAL LINEAGEID from source to output, so we need to... adjust... the number.
                            // this is EXCEPTIONALLY RISKY, but since MS "adjusts" the output rows for errors to have be "different" from the "it works!" destination
                            // we don't have much of a choice.  In reviewing them #'s it appears they consistently increment for errors, so we need to increment the
                            // index here to find the right value.  Horrible stuff.  Likely to break.  Enjoy.
                        }
                    }
 
                    //Retrieve from the column names dictionary and place column name in error info
                    if (i_column_nm != null && i_column_nm > 0 && i_column_nm  0)
                        {
                            if (colnames.TryGetValue(cColLineageKey, out value))
                            {
                                //use the lineage_id to pull the column name
                                Row.Buffer[m_idx[i_column_nm]] = value;
                            }
                        }
                    }
                    //get the missing column value for the key found at the identified "error column"
                    //had issues where the column blew up because of data type conversion issues, so try/catch is here to help handle this
                    try
                    {
                        if (i_error_data != null && i_error_data > 0 && i_error_data  0)
                            {
                                if (colids.TryGetValue(cColLineageKey, out colvalue))
                                {
                                    //use the lineage_id to pull the column name
                                    //NOTE: "bad" data MAY be totally thrown out here, which is why we're using the try/catch
                                    //if the custom CozyRoc row processor dies due to formatting errors then this will throw an exception
                                    //we're just going to ignore that and roll on by
                                    //probably worth revisiting at a later date to see if we can get at the bad data anyway
                                    Row.Buffer[m_idx[i_error_data]] = Row.Buffer[m_idx[colvalue]].ToString();
 
                                }
                            }
                        }
                    }
                    catch (Exception vEx)
                    {
                    }
 
                }
                else
                {
                }
            }
            catch (Exception ex)
            {
            }
        }
 
    }
 
}

There is a famous blog post written by Joel Spolsky titled “The Joel Test.” It describes some key parts of what make a given development shop a great (or scary) place to work. It’s also a key metric used on careers.stackoverflow.com, where every employer posting a job is encouraged to rate themselves, and post their own score on “The Joel Test.”

I use “The Joel Test” as one of the ways that I critique a potential employer, what sort of value they place on developers, and software development as a practice. I’m especially passionate about software, more than any other professional or personal pursuit in my life, and I want to work for places that value the work I do for them just as highly as I value the work myself. As such, I think “The Joel Test” is a pretty good indicator of that aspect of a workplace.

Since I tend to agree with values put forth in “The Joel Test,” I look for a place that scores highly. After working inside NUBIC for just a few months, I’m pretty happy with what I see here. However, rather than providing a score for NUBIC (I obviously think we score very high), I’ll just layout what we do here:

1. Do you use source control? We have an internal Git server, and we also publish much of our code on GitHub.
2. Can you make a build in one step? We have a Jenkins CI server that is tied to our Git project repositories. It automatically runs our test suite when new code is committed to the master branch. It also sends out automated emails when a build fails.
3. Do you make daily builds? As often as code is committed, it is built and tested. For projects under active development it’s common to see multiple builds per day.
4. Do you have a bug database? We use a combination of Redmine for internal projects, and GitHub for open source projects.
5. Do you fix bugs before writing new code? We prioritize bugs over new features. For the purpose of this question I’m defining bugs as things that are broken in the system, as opposed to simply inconvenient, or inefficient. However, we definitely label inconvenient or inefficient code as a bug if it is preventing meaningful work from getting done (rightfully so), and will prioritize it as such.
6. Do you have an up-to-date schedule? We scope and schedule work for all of our projects, and keep clients in the loop at every step of the process, whether we’re ahead or behind schedule.
7. Do you have a spec? The vast majority of projects don’t have specs for every feature; some of them are as simple as, “Can you change this thing from red to blue, and move it to the left by 5 pixels?” which I suppose qualifies as a spec, but may be communicated verbally as opposed to going through a formal process. However, anytime there’s major work to be done, or any place that we feel has lots of ambiguity, we work it out verbally or on a whiteboard, and then commit the details of that session back to the bug tracking system. After a project launch we also produce documentation for internal purposes, as well as training documents and screencasts for the benefit of end users.
8. Do programmers have quiet working conditions? This is one of my favorite things about NUBIC. Though we do work largely in a set of cubicles, we accomplish a quiet working environment in a several ways. First, and foremost, we respect each other by taking conversations that we expect to last more than a minute or two away from the common workspace, and into side offices with closed doors that keep the sound contained. Second, most of us also have noise-cancelling headphones that we use to either listen to our favorite music while coding, or simply to soften the ambient office noise (which is nearly silent most of the time anyway). Third, we prefer asynchronous communication (email + IM) over coming to a person’s desk unannounced. Unless you’re on the support rotation for the week, email can safely be checked just 2-3 times a day, and on a typical day I will have fewer than two IM conversations. We also like to eat lunch together often, which sometimes serve as conversation starters for projects, in place of formal meetings.
9. Do you use the best tools money can buy? We’re not talking about the most expensive tools – we’re talking about the best tools to do the job. Besides personal preferences, the tools we’re given are a help, not a hindrance to our work. For example, I personally prefer FogBugz for issue tracking over GitHub, due to its integrated, and more advanced project management features, but GitHub is by no means a hindrance to the workflow. There are also regular, open discussions that occur regarding tool choice. Programmers are not only free to experiment with various tools, but encouraged to do so, bringing their experiences back to the group so everyone can benefit. We maintain a coding library that holds maybe 100 books that have been collected over the years, and cover topics ranging from various programming languages, to guides on design, typeface choice, and the art of software craftsmanship. As for the day-to-day tools, we’re lucky to have a solid, flexible hardware setup. Here’s a photo of the standard issue desk: 15″ MacBook Pro + Apple Cinema Display, and push-pin friendly walls for hanging up notes or creating a makeshift kanbanboard.
   

   MacBook Pro, Apple cinema display, whiteboard for prototyping

10. Do you have testers? In addition to test- and behavior-driven development practices that each developer employs, we have people who specialize in user experience and UI design. We also encourage Usability Testing, of the sort outlined in Steve Krug’s “Rocket Surgery Made Easy”. Finding a coworker interested in pair programming is also very easy. In NUBIC, responsibility for testing software is a partnership between developers, clients, and groups that verify projects on staging systems, as well as anyone else who is simply interested in assisting a given project. We do not, however, have dedicated testers whose job it is to do nothing but think of ways to break the system (in order to prevent it). There’s been quite a bit of discussion about this, and we’re hoping that as our group grows, we’ll be able to add dedicated testers. The reasons for wanting dedicated testers are many, but I think the greatest benefit to having them is that having someone whose job it is to think of ways to break things is fundamentally a different role than a developer (whose job it is to think of ways to make things work). Both sides are important.
11. Do new candidates write code during their interview? Developers are expected to be able to demonstrate their ability to implement a few common algorithms, as well as design a small system, live, during the interview. If you’re awesome at software, it’s not a big deal. It basically proves that you have three things: a solid background in algorithms, an ability to reason about a system and accept feedback from others on its design, and an ability to ask intelligent questions of the client about what they want.
12. Do you do hallway usability testing? Some people do this more than others, but anytime you’re implementing a new feature where you’d like some feedback, it’s encouraged that you consult with people around you. The culture in NUBIC is such that I’ve never run into a person who wasn’t willing to give you five minutes of their time to examine something you’re working on. This often turns into a discussion of several different approaches to implementation, and helps lead to software that benefits from multiple perspectives, without adding a bunch of wasted time to the project schedule in formal meetings.

On top of that, one of my favorite things about working at Northwestern is the focus on collaboration, and leveraging the large community of brilliant people around you. Northwestern is one of the leading research universities in the country, and they have no shortage smart people that are passionate about what they do.

The panoramic views of Lake Michigan aren’t bad either.

What has not hit the press yet is the issue of HIPAA potential violations.  It is important to remember that inter HITECH, covered entities are responsible for breaches even if they  didn’t know, and by reasonable diligence would not have known.  In other words, if your phone sent PHI to the phone company, you are potentially liable for $100-50,000 per violation (probably will be interpreted as each compromised message) and up to $1.5 million total.

Sprint, AT&T and T-Mobile admit to use Carrier IQ.  Verizon says it does not.  I’m sure that my hospital is not ready to ban all non-Verizon phones…yet.

This is not unique to mobile phones.  The new Kindle Fire provides web browsing, so you might use it to access your web-based EHR, right?  However, the actual rendering of the web page is done in the Amazon Cloud and a compressed version of the page is sent to your device.  From a security standpoint, this means that Amazon must  execute what amounts to a man-in-the-middle attack on your secure browsing session.  Whether Amazon looks at your data, or not, is irrelevant.  They can do it at any time, and you would be none the wiser.

Thanks to Matt who pointed out that EFF had a nice evaluation of security on the Kindle Fire.  It turns out that HTTPS sessions bypass the cloud rendering engine, so things are not as dire as I had thought.  Google also assures that they are not logging content.  Perhaps, but since they are logging the URL and session token, they can probably reproduce much of the content at a later date.

Ultimately, this is why domain specific privacy (e.g. health privacy, mail privacy or video rental privacy)  laws are doomed to fail.  Our digital lives are far too interconnected for any service provider to create separate services to comply with the separate regulations.  As a health care organization, we cannot generally share location data more precise than a 3-digit zip code.   The phone company can collect, and give to advertisers, my location down to a few feet!  I’m sure that the phone company can tell, if they wanted, what chronic medications I take simply based on which lab I go to each month.  We need a fundamental right of digital privacy.  That right need to be based on the concept of information use, rather than on information access.  Anyone who uses my digital data should admit that they use it and justify why.

A good analogy for this is credit ratings.  Anyone who uses a credit rating to grant or deny a service is required to say what information they used to make the decision.  If it appears that someone used information that they should not have, like age or race, they need to be able to explain how they reached the decision without using that data.  The same should be true of all of our digital “breadcrumbs” that was scatter across the digital landscape as we attempt to live our digital lives.

As we increasingly use electronic media to accelerate the spread of scientific and medical knowledge, we need to think seriously about how we insure that the information we spread is the most correct, not just the loudest, truthiest, sexiest, or best financed.

While I have not tried every stylus on the marked, I have tried a number.  The major problem with the rubber tipped styli is that the friction seems to change as the rubber gets worn.  iFaraday stylus uses a cloth tip that glides smoothly and does not seem to change as much as the rubber.  The iFaraday Artist, Firm Dome is the best I have found by far.

Of course, nothing is perfect.  I would love to see the firm dome available in the Rx stylus, which has a cap to cover the tip when not in use.  I would also love to see the Rx cap be clip on, rather than screw on.  Even so, this has become my go-to stylus for note taking, sketching, or making marginal notes in papers I read on the iPad.

Download “Cozy Roc Parallel Loop Demo Files” contains:

• setup_sql.txt
• SequentialLoop.dtsx
• ParallelLoop.dtsx

To use the demo files you’ll need to have at least the evaluation copy of the CozyRoc components installed (you can get 32bit and x64 versions from the CozyRoc site http://www.cozyroc.com/products)

Loops in SSIS

SSIS provides a very handy loop task – you supply a collection (of type object) and iterate through that object, executing steps or processes for each item in the collection.

Microsoft’s description (http://msdn.microsoft.com/en-us/library/ms139956.aspx) of the task:

“The For Loop container defines a repeating control flow in a package. The loop implementation is similar to the For looping structure in programming languages. In each repeat of the loop, the For Loop container evaluates an expression and repeats its workflow until the expression evaluates to False.”

Great – we can now loop through a set of data.

• For a given group of something (a collection)
• Iterate through the collection (a variable / instance)
• For each instance, execute a process

In the case of ETL Assistant we use this to do the following:

• We have a concept of a scheduling “group” – a set of source::destination table mappings. Let’s say I want to manage a set HR-related items (department list, employee list, address information, etc.) as a group (easier than managing individual tables).  I can put them into a “group” (collection).  Let’s call this “HR tables.”  I can do the same with a set of patient information (patient / person list, encounter /  visit information, and possibly some other patient demographic information).  Let’s call this “Patient tables.”
• I can, for each group, pull back a list of tables (instances)
• For each table I can execute a dynamic ETL on them to pump data from a source to a destination (EX: Oracle::employee -> SQL Server::employee)

The loop task does a great job managing simple collections and executing an operation per item.

The problem?  I’m now executing all of this serially.

This means if I have a fairly beefy server I’m still potentially sitting idle while I do a simple set of ETL operations.  You have several ways to address this, but one I’ve found attractive is to convert from a serial ForEach loop to a parallel ForEach loop using the Parallel Loop Task from CozyRoc.  This will let us do n-parallel executions of a given operation.  If you have a 64 core host, for example, and the diagram above represented tables you wanted to load from a remote source, you could execute A, B, and C loads in parallel.

Let’s get back to that example using the HR tables (department list, employee list, address information).  I can create a “group” (“HR”), then places these three tables into the HR group.  When I run a process to pull over the HR group I reference the group, pull back the three table references and place them into a collection.  I iterate through the collection and for n items in the collection, execute a task.

The Sequential ForEach Loop in SSIS

Let’s do some quick setup steps to prep this test scenario

--create a test schema
CREATE SCHEMA cozyroc AUTHORIZATION dbo
GO 
 
--this is our "group" table
--EX: HR
CREATE TABLE cozyroc.etl_groups (
	group_id INT IDENTITY(1,1) NOT NULL,
	group_nm VARCHAR(100) NOT NULL,
	group_dsc VARCHAR(255)
CONSTRAINT [PK_cozyroc_etl_groups_group_id] PRIMARY KEY CLUSTERED
(
	[group_id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
 
--this is our "table" table
--EX: employees, addresses, etc.
CREATE TABLE cozyroc.etl_tables (
	table_id INT IDENTITY(1,1) NOT NULL,
	table_nm VARCHAR(100) NOT NULL,
	table_dsc VARCHAR(255)
CONSTRAINT [PK_cozyroc_etl_tables_table_id] PRIMARY KEY CLUSTERED
(
	[table_id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
 
--this is our associative table to store
--  the mapping from group::table
--  I'm using this in the example because in later
--  posts we'll allow the table to be "grouped"
--  multiple times
CREATE TABLE cozyroc.etl_group_tables (
	group_table_id INT IDENTITY(1,1),
	group_id INT NOT NULL,
	table_id INT NOT NULL
CONSTRAINT [PK_cozyroc_etl_group_tables_group_table_id] PRIMARY KEY CLUSTERED
(
	[group_table_id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]
 
--insert a sample group
INSERT INTO cozyroc.etl_groups (group_nm, group_dsc)
VALUES ('HR', 'HR Group')
 
--insert some sample tables
INSERT INTO cozyroc.etl_tables (table_nm, table_dsc)
VALUES ('Employees', 'Employees Table')
INSERT INTO cozyroc.etl_tables (table_nm, table_dsc)
VALUES ('Departments', 'Departments Table')
INSERT INTO cozyroc.etl_tables (table_nm, table_dsc)
VALUES ('Addresses', 'Addresses Table')
 
--blindly cross join everything
INSERT INTO cozyroc.etl_group_tables (group_id, table_id)
SELECT
	g.group_id, t.table_id
FROM
	cozyroc.etl_groups g,
	cozyroc.etl_tables t
 
--now let's also create a logging table
-- this is a placeholder for more complex
-- operations
CREATE TABLE cozyroc.parallel_test (
	log_id INT IDENTITY(1,1) NOT NULL,
	group_table_id INT NOT NULL,
	group_id INT NOT NULL,
	table_id INT NOT NULL,
	execution_dts datetime2(7) DEFAULT GETDATE()
CONSTRAINT [PK_cozyroc_parallel_test_log_id] PRIMARY KEY CLUSTERED
(
	[log_id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

This will have created several tables:

• cozyroc.etl_groups (“HR”)
• cozyroc.etl_tables (“employees,” “addresses,” etc.)
• cozyroc.etl_group_tables (mapping “employees” to the “HR” group, for example)
• cozyroc.parallel_test (our fake table we’re using to test the parallel loop)

You don’t need all of this for the CozyRoc Parallel Loop Task to work, but I’m trying to just introduce some examples we’re going to use in later posts related to dynamic ETL.

Now let’s create a package

1. Launch BIDS and create a new SSIS project
2. Create an OLEDB connection to the server and database where you created the tables (in my case, that’s a database called “cozyroc” on localhost)
3. For convenience, make sure you have the “variables” panel open (SSIS -> Variables)
4. Drag a few things onto the workspace:
   1. an Execute SQL Task
   2. a ForEach Loop Container task
   3. drag another Execute SQL Task into the ForEach Loop Container (make sure it’s placed inside of the container)
5. Create a package-level variable called “mylistoftables” – make it of type “Object”
6. Click on the parallel loop task so it’s highlighted – now create a variable called “iter” and make it of type “int32.”  Clicking on the ForEach Loop and then creating the variable will scope “iter” to the loop – make sure the scope of the variable is correct.
7. Now let’s create another variable also scoped to the ForEachLoop.  Let’s call it “SQL_insert” and make it of type String.  We’re going to set this up to hold a sample insert statement so we can watch the loop in action.
   1. For the “sql_insert” variable, set EvaluateAsExpression to “True”
   2. Open the Expression and enter the following:

      "insert into cozyroc.parallel_test (group_table_id, group_id, table_id)
      select gt.group_table_id, gt.group_id, gt.table_id
      from cozyroc.etl_group_tables gt
      where gt.group_table_id = " + (DT_WSTR, 100) @[User::iter]

8. Connect the first SQL Task to the ForEach Loop

Your package should now look something like…

SSIS ForEach Loop Package Setup

Now let’s set up the SQL Task

1. Double-click the SQL Task to open the configuration
2. On the “General” panel,
   1. Set the “Result Set” property to “Full Result Set”
   2. Set the Connection property to point to your database (ex: “localhost”)
   3. Set the SQLSourceType to “Direct Input”
   4. Set the SQLStatement to “select * from cozyroc.etl_group_tables”
   5. 
3. Now go to the “Result Set” panel
1. Set “Variable Name” to point to your collection “User::mylistoftables” – this is where we’re going to store the results of the query
2. 

What did we just do here?  We told the SQL Task to execute a query (“get me everything from the coyroc.etl_group_tables table”) and then store the results in our “mylistoftables” object.  Pretty straight forward.

Let’s proceed to setting up the loop

1. Double-click on your ForEach Loop Container to open the properties panels
2. On the “Collection” panel
   1. Set the “Enumerator” to ForEach ADO Enumerator
   2. Set the ADO object source variable to your collection “User::mylistoftables”
   3. 
3. Now on the “Variable Mappings” panel
   1. Set the variable to “User::iter” and the Index to “0″ (first column of the result set)
   2. 

In this step we told the ForEach Loop Container to loop through the “mylistoftables” collection – on the first table – and set the “User::iter” variable to the first column as it loops.  Keep in mind – and this is critical for later – you’re using the User::iter variable scoped to the ForEach Loop Container.

Alright – we’re almost done setting up the basic loop.  Now we just have to wire up a task that the loop executes.  In the SQL Task within your loop, set the loop to execute the “sql_insert” statement

1. Double-click the second SQL Task to open the configuration
2. On the “General” panel,
   1. Set the Connection property to point to your database (ex: “localhost”)
   2. Set the SQLSourceType to “Variable”
   3. Set the SQLStatement to your “User::sql_insert” variable

Run the package and, if there are no errors, pop over to SQL Server Management Studio for a minute.

Run a query to quickly look at the results of the package:

SELECT * FROM cozyroc.parallel_test

You should see something like…

log_id	group_table_id	group_id	table_id	execution_dts
1	1	1	1	2011-11-04 13:15:00.9500000
2	2	1	2	2011-11-04 13:15:00.9800000
3	3	1	3	2011-11-04 13:15:01.0100000

Note the dates and times. See how there are slight differences in the dates? The dates are clearly following a pattern where later group_table_ids have later dates? This is the result of the loop running sequentially.

Converting to the CozyRoc Parallel Loop Task

Let’s upgrade this to a Parallel Loop Task.  Hang on – things are about to get weird.

First things first.  Let’s quickly throw on some more components and variables as well as tweak some other bits.

1. Drag a “Parallel Loop Task” onto the canvas
2. Delete the link from your first SQL Task to the ForEach Loop.  Where we’re going we don’t need that flow.
3. Now connect that first SQL Task to the Parallel Loop Task
4. Double-click on the Parallel Loop Task to open up the configuration panel
   1. Click on the Package Connection property and set the connection to “<New Connection>.”  When the dialog box opens, make sure “Connection to Current Package” is checked and hit “OK.”  We just told the Parallel Loop Task to talk to this package when executing.  Right – this basically just became a “meta package” with execution steps.  Think of this like it’s own self-referencing parent-child package.
   2. 
   3. Now – still inside the configuration panel of the Parallel Loop Task – click on the ForEachLoop configuration item – a new popup should appear.  Click on the name of your ForEach Loop within this package.
   4. 
5. Your final Parallel Loop Task configuration should look something like this
   1. 

Now we’re cooking.  Only a few simple changes left.

1. Disable the main ForEach Loop.  We no longer manage it – the Parallel Loop Task does.  It enables/disables this as it fires each instance of this package.  If we left the loop enabled things would get very messy – you’d have sequential instances of the loop firing within each parallel instance of this package.  Loops in loops – very loopy.
2. Create a new package-level variable called “Iter” of type “int32.”   That’s right – we have a package and a loop variable now.

That’s it.  That’s really all you have to do to take a sequential loop and turn it into a parallel loop. Your final package should look something like this:

Give it a quick test run and then go back and re-run that query to look at the dates and times.

SELECT * FROM cozyroc.parallel_test

Your execution dates and times should now be much, much closer to each other if not completely identical:

log_id	group_table_id	group_id	table_id	execution_dts
4	2	1	2	2011-11-04 13:36:06.3230000
5	3	1	3	2011-11-04 13:36:06.3300000
6	1	1	1	2011-11-04 13:36:06.3300000

See that?  Granted, this is a fairly pointless test case, but you get the idea.  By default the Parallel Loop Task iteration setting is set to “-1″ (as many cores as you have).  You may want to play with this (or better yet expose it as a runtime configuration property) depending on your situation.

Next up we’re going to step into how the CozyRoc DFT+ component can make your life easier by side-stepping SSIS’s age-old static design-time column mapping problem. Combined with the Parallel Loop Task, that’s when things really start to get interesting.

Further reading:

• SSIS For Loop Container (http://msdn.microsoft.com/en-us/library/ms139956.aspx)
• SQL Server Integration Services – SSIS – For Loop Container Samples (http://www.sqlis.com/post/For-Loop-Container-Samples.aspx)
• CozyRoc’s Parallel Loop Task (http://www.cozyroc.com/ssis/parallel-loop-task)

Reportedly the spoksman for one Congressman argued “if providers have been paid for systems they already had in place, that seems to be an inexcusable waste of taxpayer dollars. It makes no sense for HHS to pay physicians for systems they already have.”

This statement seems to ignore several obvious facts.  The first is that EHRs cost money.  If institutions that had already invested received no payments, but those who waited did receive payments, the effect would be to give a competitive advantage to the technology laggards.  It would create a clear message for providers to never invest their own money in IT.  It would effective say, “wait long enough and the government will do it for you.”  That is not a message we want to convey.

Second, to receive payments, one must not only install EHRs but to use them in a “meaningful” way.  It is no surprise that organizations with a large head start reached meaningful use first.  If organizations that had invested years of effort and millions of dollars did not reach meaningful use before those starting from scratch, we would need to seriously question the criteria.

Finally,  HITECH is not about whether providers buy EHRs, but whether they really use them.   This is only stage 1.  Stages 2 and 3 are significantly harder.  Stages 2 and 3 require significant improvement and expansions of the ways that EHRs are used, even for the established leaders. Even the HIT leaders are investing millions of dollars to reach Meaningful Use.  I know of one national leader in HIT, which already had a paperless EHR, that estimated it would roughly break even on Meaningful Use.

It is not reasonable or appropriate to evaluate the entire program based on the who crosses the Stage 1 finish line first.