{"id":3963,"date":"2016-06-12T16:23:30","date_gmt":"2016-06-12T20:23:30","guid":{"rendered":"http:\/\/blogs.harvard.edu\/sj\/?p=3963"},"modified":"2016-06-17T12:40:35","modified_gmt":"2016-06-17T16:40:35","slug":"archiving-web-links-ideal-global-networks-of-caches-and-mirrors","status":"publish","type":"post","link":"https:\/\/archive.blogs.harvard.edu\/sj\/2016\/06\/12\/archiving-web-links-ideal-global-networks-of-caches-and-mirrors\/","title":{"rendered":"Archiving Web links: Building global layers of caches and mirrors"},"content":{"rendered":"<p>The Web is highly distributed and in flux; the people using it, even moreso. \u00a0Many projects exist to optimize its use, including:<\/p>\n<ol>\n<li><strong>Reducing\u00a0storage and bandwidth<\/strong>: \u00a0compressing parts of the web;\u00a0deduplicating files that exist in many places, replacing many\u00a0with pointers to a single copy of the file [Many <a href=\"http:\/\/mozilla.com\">browsers<\/a> &amp; servers, <a href=\"http:\/\/www.dropbox.com\">*<\/a><a href=\"http:\/\/www.box.com\">Box<\/a>]<\/li>\n<li><strong>Reducing latency and long-distance bandwidth<\/strong>: \u00a0caching popular parts of the web locally around the world\u00a0[<a href=\"http:\/\/akamai.com\">CDNs<\/a>, <a href=\"http:\/\/aws.amazon.com\">clouds<\/a>, &amp;c]<\/li>\n<li><strong>Increasing robustness &amp;\u00a0permanence of links<\/strong>:\u00a0caching linked pages (with timestamps or snapshots, for dynamic pages) <strong>[<\/strong><a href=\"http:\/\/mementoweb.org\/about\/\">Memento<\/a>, <a href=\"https:\/\/archive.org\/web\/\">Wayback Machine<\/a>, <a href=\"http:\/\/perma.cc\">perma<\/a>, <a href=\"http:\/\/amberlink.org\/\">amber<\/a><strong>]<\/strong><\/li>\n<li><strong>Increasing\u00a0interoperability <\/strong>of naming\u00a0schemes for describing or pointing to things\u00a0on the Web, so that it&#8217;s easier to cluster similar things and find copies or\u00a0versions of them <strong>[<\/strong>HvdS&#8217;s <a href=\"http:\/\/www.dlib.org\/dlib\/november15\/vandesompel\/11vandesompel.html\">15-year overview<\/a> of advancing interop<strong>]<\/strong><\/li>\n<\/ol>\n<p>This week I was thinking about the 3rd point. What would a comprehensively\u00a0backed-up Web of links look like? \u00a0How resilient can we make references to all of the failure modes we&#8217;ve seen and imagined? \u00a0Some threads for a map:<\/p>\n<ol>\n<li>Links\u00a0should include\u00a0<strong><a href=\"http:\/\/timetravel.mementoweb.org\/about\/\">timestamps<\/a><\/strong>, important ones should request\u00a0archival<strong>\u00a0<a href=\"http:\/\/perma.cc\">permalinks<\/a><\/strong>.\n<ul>\n<li>When creating a\u00a0reference, sites should notify each of the major cache-networks, asking them to store a copy.<\/li>\n<li><a href=\"http:\/\/robustlinks.mementoweb.org\/about\/\">Robust links<\/a> can embed information about where to find a cache in the <strong>a<\/strong> tag that generates the link (and possibly a fuzzy content hash?).<\/li>\n<li>Permalinks can\u00a0use an identifier system that allows\u00a0searching for the page across any of the nodes of the local\u00a0network, and across the different cache-networks. (Browsers can know how to attempt to find a copy.)<\/li>\n<\/ul>\n<\/li>\n<li>Sites should have a coat of <strong><a href=\"http:\/\/amberlink.org\/\">amber<\/a><\/strong>: a local cached snapshot of anything linked from that site, stored on their host or a nearby\u00a0<em>supernode<\/em>. \u00a0So as long as that site is available, snapshots of what it links to are, too.\n<ul>\n<li>We can comprehensively track whether sites have\u00a0signalled they have an amber\u00a0layer.\u00a0\u00a0If a site isn&#8217;t yet caching what they link to, readers can encourage them to do so or connect them to a supernode.<\/li>\n<li><strong>Libraries<\/strong> should host <strong>amber supernodes<\/strong>: caches for sites that can&#8217;t host those snapshots on their host machine.<\/li>\n<\/ul>\n<\/li>\n<li><strong>Snapshots<\/strong> of entire\u00a0websites should be <a href=\"http:\/\/archive.org\/web\">archived<\/a> regularly\n<ul>\n<li>Both public snapshots for search engines and private ones for long-term archives.<\/li>\n<\/ul>\n<\/li>\n<li>A global network of <strong>mirrors<\/strong> (a la [<a href=\"https:\/\/www.clockss.org\/clockss\/Home\">C<\/a>]<a href=\"https:\/\/www.lockss.org\/\">LOCKSS<\/a>) should maintain copies of permalink and snapshot databases\n<ul>\n<li>Consortia of libraries, archives, and publishers should commit to a broad\u00a0geographic distribution of mirrors.\n<ul>\n<li>mirrors should be available within any country that has expensive interconnects with the rest of the world;<\/li>\n<li>prioritization should lead to a <strong>kernel<\/strong> of the cached web that is stored in &#8216;<a href=\"https:\/\/commons.wikimedia.org\/wiki\/File:Svalbard_Global_Seed_Vault_(23273281972).jpg\">seed bank<\/a>&#8216; style archives, in the\u00a0most secure vaults and other\u00a0venues<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>There should be a clear way to scan for <strong>fuzzy matches<\/strong> for a broken link. Especially handy for anyone updating a large archive of broken links.\n<ul>\n<li>Is the base directory there? Is the base URL known to have moved?<\/li>\n<li>Are distant-timestamped versions of the file available? \u00a0[<em>some robustlink\u00a0implementations\u00a0do this already<\/em>]<\/li>\n<li>Are there exact matches elsewhere in the web for a\u00a0[rare]\u00a0filename? \u00a0Can you find other documents with the same content hash? [i<em>f a hash was included in the link<\/em>]<\/li>\n<li>Are there known ways to contact the original owner of the file\/directory\/site?<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p><span style=\"text-decoration: underline\">Related questions<\/span>: What other aspects of robustness need consideration? How are people\u00a0making progress at each layer? \u00a0What more is needed to have a mesh of archived links at every\u00a0scale? For instance,\u00a0WordPress supports a chunk\u00a0of the Web; top CDNs cache\u00a0more than that. What other players can make this happen? \u00a0What is needed for them to support this?<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Web is highly distributed and in flux; the people using it, even moreso. \u00a0Many projects exist to optimize its use, including: Reducing\u00a0storage and bandwidth: \u00a0compressing parts of the web;\u00a0deduplicating files that exist in many places, replacing many\u00a0with pointers to a single copy of the file [Many browsers &amp; servers, *Box] Reducing latency and long-distance [&hellip;]<\/p>\n","protected":false},"author":1202,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[211,14968,6034,213,209,709],"tags":[78849,78848],"class_list":["post-3963","post","type-post","status-publish","format-standard","hentry","category-international","category-knowledge","category-meta","category-metrics","category-popular-demand","category-wikipedia","tag-amber","tag-perma"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7iVvB-11V","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/3963","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/users\/1202"}],"replies":[{"embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/comments?post=3963"}],"version-history":[{"count":9,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/3963\/revisions"}],"predecessor-version":[{"id":3972,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/posts\/3963\/revisions\/3972"}],"wp:attachment":[{"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/media?parent=3963"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/categories?post=3963"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/archive.blogs.harvard.edu\/sj\/wp-json\/wp\/v2\/tags?post=3963"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}