{"id":12,"date":"2004-07-21T09:06:27","date_gmt":"2004-07-21T13:06:27","guid":{"rendered":"http:\/\/blogs.law.harvard.edu\/rlucastemp\/2004\/07\/21\/hint-preprocessing-mongo-xml-files"},"modified":"2004-07-21T09:06:27","modified_gmt":"2004-07-21T13:06:27","slug":"hint-preprocessing-mongo-xml-files-for-use-with-xmlsimple","status":"publish","type":"post","link":"https:\/\/archive.blogs.harvard.edu\/rlucastemp\/2004\/07\/21\/hint-preprocessing-mongo-xml-files-for-use-with-xmlsimple\/","title":{"rendered":"[HINT] Preprocessing mongo XML files for use with XML::Simple"},"content":{"rendered":"

<\/a><\/p>\n

If you are a reasonable Perlista, the first thing you will do when you
\nhave to do some modest but non-trivial munging of data locked up in XML
\nis to use XML::Simple.  The API is nearly perfect (absent the lack
\nof some defaults that could be more helpfully set for strictness) for
\npurposes of comprehensibility and transparency.<\/p>\n

However, if you prototype on a small document, and then try to use your
\ncode on a much bigger XML document, you will find the drawback:
\ntree-building is costly, and you may spend the vast majority of your
\nprogram’s time parsing in the document.  One handy solution is to
\npreprocess your XML — just run XML::Simple’s XMLin sub, and use
\nData::Dumper to spit out the structure that results to a file. 
\nWhen you want to use it, you can simply “eval” it, for it defines a
\nnative Perl structure, and you can use the remainder of your code
\nunchanged.  This resulted for me in a 2x – 10x speedup for certain
\ndocuments and certain sizes.<\/p>\n

However — now imagine that you have some real torture-test data — 10
\nMB, heavily nested monstrosities of XML.  The Dumper output of the
\nparsed tree is now working on 100 MB!  Slurping this in and
\nevaling it is now the real problem.<\/p>\n

Here’s an idea: rather than slurping and evaling, try inlining it at
\nthe compilation stage.  That’s right — make use of Perl’s much
\nmore efficient way of slurping and evaling a filehandle with a pipe:<\/p>\n

cat preprocessed_xml.dd myscript.pl | perl<\/span><\/p>\n

It’s somewhat unorthodox, but entirely functional.  Combined with
\njudicious use of gzip, this could be a very efficient way to get
\nlittle-changing XML documents into perl quickly — often very important
\nwhen doing dev work for which numerous iterations are required and for
\nwhich a minutes-long parse stage would adversely affect progress.<\/p>\n

Update:<\/span> It occurred to me that
\nusing Storable or a Cache::* module might be faster yet.  At this
\npoint, my work proceeds with tolerable speed using Data::Dumper, plus I
\nlike using Dumper so that I can edit the output structures by hand if
\nneed be.  But perhaps you should try those modules if you need
\neven better performance, or cringe at the hackishness of catenating
\nfiles piped to perl.<\/p>\n","protected":false},"excerpt":{"rendered":"

If you are a reasonable Perlista, the first thing you will do when you have to do some modest but non-trivial munging of data locked up in XML is to use XML::Simple.  The API is nearly perfect (absent the lack of some defaults that could be more helpfully set for strictness) for purposes of comprehensibility […]<\/p>\n","protected":false},"author":1180,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-12","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/archive.blogs.harvard.edu\/rlucastemp\/wp-json\/wp\/v2\/posts\/12","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive.blogs.harvard.edu\/rlucastemp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive.blogs.harvard.edu\/rlucastemp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/rlucastemp\/wp-json\/wp\/v2\/users\/1180"}],"replies":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/rlucastemp\/wp-json\/wp\/v2\/comments?post=12"}],"version-history":[{"count":0,"href":"https:\/\/archive.blogs.harvard.edu\/rlucastemp\/wp-json\/wp\/v2\/posts\/12\/revisions"}],"wp:attachment":[{"href":"https:\/\/archive.blogs.harvard.edu\/rlucastemp\/wp-json\/wp\/v2\/media?parent=12"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/rlucastemp\/wp-json\/wp\/v2\/categories?post=12"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/rlucastemp\/wp-json\/wp\/v2\/tags?post=12"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}