{"id":63,"date":"2012-04-10T11:22:42","date_gmt":"2012-04-10T15:22:42","guid":{"rendered":"http:\/\/blogs.law.harvard.edu\/chovanec\/?p=63"},"modified":"2012-04-11T18:05:29","modified_gmt":"2012-04-11T22:05:29","slug":"r-clusters-and-more-computers","status":"publish","type":"post","link":"https:\/\/archive.blogs.harvard.edu\/chovanec\/r-clusters-and-more-computers\/","title":{"rendered":"R, parallelization and large datasets"},"content":{"rendered":"<p>When we have a task that would take a long time, we can usually think about parallelization. In this post I will show how to deal with an issue when you have large shared data set (but not that big so you would need <a href=\"http:\/\/en.wikipedia.org\/wiki\/MapReduce\">MapReduce<\/a>).<\/p>\n<p>Let&#8217;s first start with how to set up cluster in R:<\/p>\n<p><b>Cluster set-up using doSNOW<\/b><br \/>\nRevolution Analytics pulled out doMC; therefore, I am using doSNOW.<\/p>\n<p><code><br \/>\nlibrary(foreach)<br \/>\nlibrary(doSNOW)<\/code><br \/>\n<code><br \/>\nnumberofcores &lt;- 4<br \/>\n<\/code><code><br \/>\nmakeCluster(numberofcores)<br \/>\nregisterDoSNOW(cl)<br \/>\n<\/code><code><br \/>\nforeach (ind=1:1000) %dopar% foo_with(bigdata)<br \/>\n<\/code><code><br \/>\nstopCluster(cl)<br \/>\n<\/code><\/p>\n<p>There are two issues here. This code gives us an error message that the function foo_with and you are transporting a lot of data what causes slow down.<\/p>\n<p><b>Solution for both problems<\/b><br \/>\nPush data into your cluster by:<br \/>\n<code><br \/>\nclusterExport(cl, bigdata)<br \/>\n<\/code><\/p>\n<p>Function can be either pushed by clusterExport or we can use clusterApply or clusterApplyLB<br \/>\n<code><br \/>\nclusterApplyLB(cl, array, foo_with_rewritten,...)<br \/>\n<\/code><\/p>\n<p>This blog post shows the solution in between simple SNOW (or different) cluster computing just MC or similar and cluster that needs MapReduce.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>When we have a task that would take a long time, we can usually think about parallelization. In this post I will show how to deal with an issue when you have large shared data set (but not that big so you would need MapReduce). Let&#8217;s first start with how to set up cluster in [&hellip;]<\/p>\n","protected":false},"author":4584,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-63","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/archive.blogs.harvard.edu\/chovanec\/wp-json\/wp\/v2\/posts\/63","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive.blogs.harvard.edu\/chovanec\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive.blogs.harvard.edu\/chovanec\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/chovanec\/wp-json\/wp\/v2\/users\/4584"}],"replies":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/chovanec\/wp-json\/wp\/v2\/comments?post=63"}],"version-history":[{"count":20,"href":"https:\/\/archive.blogs.harvard.edu\/chovanec\/wp-json\/wp\/v2\/posts\/63\/revisions"}],"predecessor-version":[{"id":124,"href":"https:\/\/archive.blogs.harvard.edu\/chovanec\/wp-json\/wp\/v2\/posts\/63\/revisions\/124"}],"wp:attachment":[{"href":"https:\/\/archive.blogs.harvard.edu\/chovanec\/wp-json\/wp\/v2\/media?parent=63"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/chovanec\/wp-json\/wp\/v2\/categories?post=63"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/chovanec\/wp-json\/wp\/v2\/tags?post=63"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}