Stephen Foskett, Pack Rat » XML Archives – Stephen Foskett, Pack Rat http://blog.fosketts.net Understanding the accumulation of data Fri, 10 Feb 2012 17:40:43 +0000 en hourly 1 http://wordpress.org/?v= Cool Google Spreadsheet XML/XPath Mojo http://blog.fosketts.net/2010/07/02/cool-google-spreadsheet-importxml-xpath/ http://blog.fosketts.net/2010/07/02/cool-google-spreadsheet-importxml-xpath/#comments Fri, 02 Jul 2010 20:41:12 +0000 Stephen http://blog.fosketts.net/?p=3320 Google Spreadsheet sure isn’t as responsive for a power user like me, but I love the ability to share information with others and cooperatively edit a workbook. It’s become our main tool for planning the Gestalt IT Tech Field Day events. I was thrilled to discover that Google’s spreadsheet supports the importXML tag, which allows it to automatically gather information from other web sites. Let’s take a look at how it works!

HTML == XML?

Most web users have heard of HTML, but XML is a much geekier thing. Like HTML, XML is a way of “marking up” a text document to provide hidden clues about the content and how to display it. In fact, modern HTML is (sort of) a subset or category of XML, and HTML documents (the kind you access with your web browser) can be accessed by many XML tools.

Both HTML and XML enclose text in tags. For example, a paragraph would be surrounded by <p> and </p> tags, while a table starts and ends with <table> and </table>. These can be nested within each other, and are commonly deeply nested indeed. Web and XML documents typically have many layers of <div> and <span> tags, for example, and include lists within lists as well.

This somewhat-organized mess creates a “path” through the document leading to pieces of information. This concept is called XPath. For example, a twitter profile page embeds your name deep in the <html><body> path under <div id=”side”>, <div id=”profile” class=”section profile-side”>, <address>, <ul class=”about vcard entry-author”>, <li>, and <span class=”fn”>.

Raw HTML is often organized like this, and this can work to our advantage. If a computer program wanted to pull the full name of a Twitter user out of their profile page, it could look for a <span> element with the class property set to “fn”. It could also look for the <address> tag and pull out some information about the user in the vcard microformat.

importXML

All this becomes very interesting indeed when considering the importXML function in Google Spreadsheet. It will parse a URL as an XML file automatically, and you can tell it to look in an XPath for data. This is very powerful indeed!

Twitter Followers

I think an example will make it clearer. Let’s look up the Twitter follower count of a user. We create a spreadsheet and enter a twitter username in cell A1. Then we put the following formula in cells B1 and C1 and the count magically appears!

B1:

=if(C1<>"",right(C1,len(C1)-10),"")

C1:

=if(A1<>"",importXML("http://mobile.twitter.com/"&A1,"//a[@ href='http://mobile.twitter.com/"&lower(A1)&"/followers']"),0)

This formula looks for a <a href> with the URL including the user and twitter followers, which is a unique HTML element in every mobile Twitter profile. This returns a string like “Followers:1234″, so we use another formula to strip that part out.

Updated 1/22/12 after Twitter screwed up the main page. Good thing the mobile site still works!

LinkedIn Connections

Let’s try something else. How would we pull the number of connections a person has in LinkedIn using importXML? Here’s a function!

=value(substitute(importXML(A3,"//dd[@class='overview-connections']/p/strong"),"500+","500"))

This is a little more complicated. We’re doing the same thing, taking a url from cell A3 that corresponds to the person’s public LinkedIn page, and outputting the content of the <dd class=”overview-connections”> tag. But we’re also using the SUBSTITUTE() function to take the plus sign off a “500+” response and converting it into a value for calculation.

Updated 1/22/12 for new LinkedIn Format

Alexa and Klout

Here are a few more examples. I bet you can follow along now.

Alexa traffic rank:

=value(importXML("http://www.alexa.com/search?q="&E3,"//div[@class='row']/span/a[@href][1]"))

Klout score:

=value(substitute(importXML("http://klout.com/"&C3,"//span[@class='value']"),"klout score",""))

Limitations in Google Spreadsheets

Before you go thinking you can run off and create awesome web applications like this, know that there are some serious limitations. First, Google limits the use of importXML to 50 per workbook. This means you can’t import from hundreds of sources in the same spreadsheet, or even in multiple sheets in the same workbook. Next, importXML is pretty opaque in everyday use. You have to do a lot of trial and error to get the XPath right, and it fails often with #N/A, breaking calculations.

But it’s still pretty useful when creating a spreadsheet that needs to pull in information from outside sources. You can grab all sorts of data this way, from current stock quotes to weather or sports metrics. You have the whole Internet at your disposal – let’s get creative!

© sfoskett for Stephen Foskett, Pack Rat, 2010. | Cool Google Spreadsheet XML/XPath Mojo
This post was categorized as Everything, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

]]> http://blog.fosketts.net/2010/07/02/cool-google-spreadsheet-importxml-xpath/feed/ 3 Long-Term Versus Longer-Term Archiving http://blog.fosketts.net/2008/12/02/long-term-archiving/ http://blog.fosketts.net/2008/12/02/long-term-archiving/#comments Tue, 02 Dec 2008 14:46:03 +0000 Stephen http://blog.fosketts.net/?p=1163 How will you retain records for the long haul? It depends on how you define “long”. Nearly everyone (individual and business alike) has certain records to retain for years, and some may need retention for decades or centuries. How can you accomplish this?

First, consider whether to store records as atoms or bits. You can convert paper to data or vice versa, and there are pros and cons to both:

  • Properly handled physical (paper or film) records should last for hundreds of years and can remain readable without software or devices. But they’re hard to search (you need an index), and paper is bulky, heavy, and difficult to work with.
  • Digital records can either be stored offline or kept “alive,” but questions remain about their long-term reliability and readability. Living records can be easy to search and use, and digital storage can be very space-efficient, but data tends to pile up “out of sight.”

Long-term storage of records on physical media is proven – think about papyrus, tablets, gold or nickel discs, film, and paper. But will digital media fare as well? Data tapes and disks can degrade over time, and manufacturer reliability specs are based on accelerated testing, not actual experience. Regardless of media type, careful handling can extend media life.

But will you still be able to read it? Tapes and optical disks require additional hardware to read, while disk drives are paired with their read heads. Software applications are needed to read and interpret data (backup, archiving, compression, encryption, deduplication, database) as well. What about content format? Should you use ASCII, XML, PDF/A?

I’ll be presenting a webinar on this topic tomorrow, Wednesday, December 3, at 2:00 PM Eastern time. Register on-line at the AIIM web site and join me for the discussion!

© sfoskett for Stephen Foskett, Pack Rat, 2008. | Long-Term Versus Longer-Term Archiving
This post was categorized as Enterprise storage, Personal. Each of my categories has its own feed if you'd like to filter out or focus on posts like this.

]]> http://blog.fosketts.net/2008/12/02/long-term-archiving/feed/ 1