{"id":165,"date":"2007-11-19T01:16:01","date_gmt":"2007-11-19T08:16:01","guid":{"rendered":"http:\/\/blogs.law.harvard.edu\/cqtwo\/2007\/11\/19\/data-a-dirty-secret\/"},"modified":"2007-11-19T01:16:01","modified_gmt":"2007-11-19T08:16:01","slug":"data-a-dirty-secret","status":"publish","type":"post","link":"https:\/\/archive.blogs.harvard.edu\/cqtwo\/2007\/11\/19\/data-a-dirty-secret\/","title":{"rendered":"Data: A dirty secret"},"content":{"rendered":"<p>Data quality is a constant issue in identity solutions.\u00a0 Usually, this is what happens:\u00a0 we tell the client that they need to clean up their identity data before we start doing any integration work.\u00a0 They assure us that this will happen.\u00a0 We have been burned by this in the past, so we write it into the contract language to try and protect ourselves.\u00a0 Doesn&#8217;t matter: the data cleansing project is so unsexy that it never comes to the top of the priority pile and so it doesn&#8217;t get done.\u00a0 The identity integration, the pipework, is complete before the data, the water that&#8217;s supposed to flow through the pipes, is clean.\u00a0 The client sponsor is furious that the identity work isn&#8217;t complete, but it&#8217;s a data quality issue and so there&#8217;s a delay at the end of the project while data cleansing happens.<\/p>\n<p>Almost every organization I&#8217;ve ever looked at has bad identity data.\u00a0 Even if you go through a data cleansing process, over time the quality will decay.\u00a0 Based on anecdotal evidence, I think it&#8217;s around three or four years to get to the point where you have to do another data cleansing exercise.\u00a0 For an organization with a relatively static employee base, it might be at the longer (four year) end of that range; for a dynamic business with lots of hires\/fires, acquisitions\/divestitures, and the like, probably less than three years.<\/p>\n<p>For example, we recently did a simple integration of a file directory with an HR database.\u00a0 We were able to automatically match with confidence 60% of the accounts in the file directory with actual people.\u00a0 (Actually, it was 56%; 4% were service accounts that don&#8217;t map to people.)\u00a0 The remaining 40% required some level of manual intervention: 12% were possible matches that needed to be reviewed manually.\u00a0 Another 8% simply didn&#8217;t have enough information to make a judgement on.\u00a0 And fully 20% were unmatched; we have good information on them but they don&#8217;t match people in the HR system.\u00a0 At some point, after a manual review, those accounts will be closed down and we&#8217;ll listen for the complaints to figure out who they belong to.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Data quality is a constant issue in identity solutions.\u00a0 Usually, this is what happens:\u00a0 we tell the client that they need to clean up their identity data before we start doing any integration work.\u00a0 They assure us that this will &hellip; <a href=\"https:\/\/archive.blogs.harvard.edu\/cqtwo\/2007\/11\/19\/data-a-dirty-secret\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1116,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[117],"tags":[],"class_list":["post-165","post","type-post","status-publish","format-standard","hentry","category-identity"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8jQA6-2F","_links":{"self":[{"href":"https:\/\/archive.blogs.harvard.edu\/cqtwo\/wp-json\/wp\/v2\/posts\/165","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive.blogs.harvard.edu\/cqtwo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive.blogs.harvard.edu\/cqtwo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/cqtwo\/wp-json\/wp\/v2\/users\/1116"}],"replies":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/cqtwo\/wp-json\/wp\/v2\/comments?post=165"}],"version-history":[{"count":0,"href":"https:\/\/archive.blogs.harvard.edu\/cqtwo\/wp-json\/wp\/v2\/posts\/165\/revisions"}],"wp:attachment":[{"href":"https:\/\/archive.blogs.harvard.edu\/cqtwo\/wp-json\/wp\/v2\/media?parent=165"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/cqtwo\/wp-json\/wp\/v2\/categories?post=165"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/cqtwo\/wp-json\/wp\/v2\/tags?post=165"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}