What to do about information scarcity

On July 31, 2009, the Basex blog finally ticked me off. I unsubscribed. Their crime? Harping on, and on, and on about information overload.

Basex invented the term.

Basex sponsor annual studies.

“Information overload continues to increase.”

“Information Overload is never far from our thoughts here at Basex but, with the cost of the problem looming at some $900 billion per annum…”

Blather, blather.

Look, I’m a busy manager, responsible for developing and operating an online service. I regularly process around 100 incoming email per day, send around 50, subscribe to 83 blogs, read 2 websites daily, follow 406 people on Twitter, are Friends with 55 people on Facebook, have 77 connections and 4 group memberships on LinkedIn, and I’m not suffering from information overload. Surely if it were a real problem, I’d be suffering some form of breakdown.

And yet I find myself seeking even more information. Based on a quick perusal of my last month’s search activity on Google, I average around 8 searches per day, opening an average of three to four pages per search. So, in contrast to suffering information overload, I’m actively seeking more information.

Sampling the concept of information overload

In preparing to write this post, I scanned the last 300 posts on Twitter about “information overload,” roughly corresponding to tweets over the last 24 hours. What are people saying?

• Quite a few declare their own information overload.
• I found a link to the Information Overload Research Group.
• Quite a few people quote Clay Shirky’s pithy phrase: “It’s not information overload, it’s filter failure.”
• A number of people point to David Allen’s article The Problem is not information overload, which argues that purported information overload is really a decision problem (Allen uses the term: “meaning assignment”).
• Engaged Learning write a post that expands on Shirky’s filtering theme: “So instead of throwing our hands in the air and saying we suffer from information overload, we must learn to dynamically adjust our filters and focus.”
• Steve Mollman wrote a piece entitled How can we cope with information overload? It contextualises the problem, quotes Clay Shirky, but turns out to shill for a predictive filtering iPhone app called My6Sense.
• Blueprint 2020 attempts to provide six steps to “avoid” information overload.
• BNET Australia attempts to attract flow with the provocative title, Which Is Worse for Your Brain: Texting or Pot? But it betrays its heading’s promise by not only not discussing comparative mental states; worse, failing to mention pot even once in its article; yet even worse, turns into one of those completely disconnected, disjointed ten-point advice columns. Mercifully, it stops short of ten, offering only five points. 
• Finally, I found my way to Steven Sagmeister’s TED address, The power of time off.

So, two years after the seminal assertion, “It’s not information overload, it’s filter failure”, and four years after the initial publication of David Allen’s, “The Problem is not information overload”, the information overload meme is still buzzing.

Defining information

Departing from both folk understanding and information theoretic definitions, Eli Goldratt defines “information” as “the answer to an asked question.” 

“Everything else,” declares Goldratt, “is simply data.” The implication is that much of the “information” consumed each day is not, actually, information—but is, in fact, data. This data becomes information only when it begins to answer questions that you have asked.

The challenge then, is to ask questions. To ask better questions. To ask relevant, actionable, significant questions. To create a personal sense of what is important. In other words, to create a personal filter.

The more profound the question, the more significant the answer.

Data, entertainment, serendipity and information

In any given moment, the vast majority of digitally-available extant text is simply data. It answers questions I don’t ask, don’t care about, won’t read.

Much of the information delivered via blogs and virtually all information from news media is simply entertainment. (See Dave Pollard’s perspective that news media is unactionable; and therefore entertainment.) I let this stuff wash over me because I enjoy reading it. I enjoy being entertained. It’s not just data—I find it interesting. But it’s also not information, because it is irrelevant to my work, my family, my life.

There are times when an article, a post, an essay, a video delivers something unexpected. A point of view with the potential to update my personal filter. Something that causes me to ask new questions, to learn that something else is actually important to my world-view. This is serendipity.

Finally, there is information. Satisfying answers to questions asked.

Data, entertainment, serendipity and information forms a filter of ascending value.

The real problem

When viewing media consumption through this filter, the real problem becomes apparent. Despite the massive volume of data potentially available, the significant amount of data being attended to, the amount of entertainment and the occasional encounter with serendipity: the real problem is the continuing scarcity of truly significant information. 

Really important information is still scarce. Richard Hamming had it right: We need to ask questions that matter.

Fact-checking Nick Carr's ass

Nick Carr really needs a fact checker. He recently opined:

“Never before in history have people paid as much for information as they do today.”

He challenges his readers to:

“Do the math. Sit down right now, and add up what you pay…”

So I did. And I found Nick to be wrong. Quite wrong.

Problematic aspects of Carr’s piece

It’s a delicious irony that the man who asserts that the internet is shallowing our brains demonstrates his thesis with a slap dash thought piece rife with holes. Here are just some of the holes in Carr’s piece:

1. Carr fails to define his terms. Is he talking about inflation-adjusted pricing relative to any time in history? Or as a percentage of income? For an average individual in society? Relative to annual income? Or relative to bit of information?
2. Carr lumps together the cost of information, information access, communications and entertainment as if each of these quite distinct items are all “information.”
3. While Carr is happy to include the cost of subscription services, he excludes information purchased directly (books, music, software etc).
4. While he includes internet access, he excludes the cost of the computing devices themselves.
5. Carr assumes “us,” “we,” and “you” are always information consumers. He doesn’t include the cost of acquiring, producing, publishing or communicating information.
6. He makes no allowance for forgone information. For example, I remember my mother limiting the photographs she took because it would cost too much to develop them; whereas my family now snaps hundreds of photographs every year. Cost is no longer a factor.
7. Carr provides absolutely no evidence to support his assertion. (Although he links to a dodgy bit of analysis from Gizmodo.)

So, I’ve taken up Nick’s challenge. I compared my “information expenditure” in 2009 with my father’s “information expenditure” in 1985. When I did, I found Carr’s assertion mistaken. Information expenditure is significantly less than it was 24 years ago. 

Evidence

Subjects: Loryn Jenkins (2009), Elwyn Jenkins (1985)

I have interpreted Carr’s assertion opening assertion as meaning: “As a proportion of gross annual income, never before in history have people paid as much for information as they do today.”

I chose to compare my family’s 2009 “information expenditures” with my parental family in 1985. While any single point-in-time comparison is merely anecdotal, this is as valid a single-point comparison can be due to similarity in our respective stages in life, responsibilities in life, level of education and mental outlook.

• In 1985, my father was a 32 year old high-school teacher. Our family was single income, with three children (ages 11, 9, 7) and a mother preparing to re-enter the workforce. We lived in Albany, Western Australia. My father was pursuing a Bachelor of Education (Honours) at Murdoch University, and was an avid consumer and producer of information, and a computing afficionado. He spear-headed introducing computers into the class-room, achieving a 2:1 student:computer ratio because he believed that computing would significantly enhance a student’s education. Dad self-published some student’s fiction that year, enlisting students to sell the book door-to-door around town.
• In 2009, I was a 34 year old CTO in a software development company. My family is single income, with two children (ages 12, 11) and a wife preparing to re-enter the workforce. We lived in Sydney, New South Wales. I hold a Bachelor of Arts (Honours) degree from Macquarie University. I introduced significant improvements into my employer’s software development process, drawing both from experience and knowledge acquired from software development and applied psychology literature. I began publishing a small blog, read by 2,000 people originating from 70 different countries.

Scope

Carr’s claim is about the cost of information, but his summary challenges the reader only to calculate subscription costs. Despite including information access costs, he doesn’t include the costs of the computer hardware or software with which that information is also accessed. Nor does he include information products directly purchased. 

While I would argue that information per se excludes information access, communication and entertainment, I have opted to follow Carr’s challenge and include them all:

• information access - computer, software, telecommunications, travel-to-get-information
• entertainment - books, music, movies, tv shows
• communications - email, instant messaging, blogs, rss
• information - web, rss, books

Information gathering

My information comes directly from my family’s Quicken record of expenditures from 2009. My father’s information comes from his recollection of expenditures and income in 1985, gathered during a telephone call this evening.

The data

In 1985, Elwyn Jenkins’ family spent around 32% of gross annual income on “information;” whereas in 2009, my family spent around 10% of gross annual income on “information.” Here’s what the break-down looks like.

From Nick Carr’s original list of categories, I don’t subscribe to Cable TV, Satellite radio, Netflix, wifi hotspot access or TiVO. Additionally to his list, I included the purchase of books, movies (theatre), DVDs, computer hardware and peripherals, computer software, hand-held game consoles, fax, domain names, website and children’s books. Here’s the break-down as a percentage of my gross annual income.

Information Expenditures by Loryn Jenkins’ Family, 2009, as a percentage of gross income

• Internet service - 0.63%
• Cell phone service - 2.08%
• Landline phone service - 0.32%
• Desktop computer - 1.91%
• Online computer backup - 0.16%
• 1200 DPI Color Laser Printer - 0.26%
• Hand-held game consoles - 0.44%
• Console games - 0.42%
• Music, TV Shows, Movies, Apps - 0.53%
• Movie Rentals - 0.60%
• DVD Rentals - 0.12%
• Fax - 0.29%
• Professional Books - 0.84%
• Children’s books - 0.26%
• Domain name purchases - 0.18%
• Website operating costs - 0.07%
• Total “Information” Expenditure - 10.09%

The information categories in my father’s 1985 expenditure have been selected to be analogous with the classes of expenditure in my 2009 data.

Information Expenditures by Elwyn Jenkins’ Family, 1985, as a percentage of gross income

• Landline phone and long distance - 8.31%
• Postage - 0.02%
• Desktop Computer - 2.31%
• Modem purchase - 1.23%
• Dot Matrix Printer - 2.46%
• Word Processing Software - 0.62%
• VHS Tape Rental - 0.40%
• Encyclopaedia Britannica - 0.65%
• Newspaper subscription - 0.22%
• Children’s books - 0.18%
• Professional Books - 5.91%
• Fuel Expenditure, drive to uni - 9.23%
• Total “Information” Expenditure - 30.53%

Commentary - Elwyn Jenkins’ expenditure

I now know why my mother and father knelt beside their bed every week, with hand-written family budget and cash projections, identifying which bills to pay and which expenditures to make. After taking taxation into account, the family in which I grew up spent an astounding 45% of disposable income on “information.”

Long distance calling in Australia in the mid-1980s was very expensive. My family had a rather outsized budget for that because my father organised weekly conference calls, frequently including long-distance guest speakers. Moreover, bulletin board services being accessed induced timed long-distance calls.

In 1985, my father’s computing purchases included a “high-resolution” dot matrix printer, a Commodore 128D which had a 64-character wide color screen. In 1985, Dad purchased the Commodore 128D for himself, handing down to me and my brother the Commodore 64. (Hey, we had fun typing in machine code from magazine print-outs to get our own games.)

I recall the evening an Encyclopaedia Britannica sales rep called on my parents. He explained the educational value of the Encyclopaedia Brittanica; and my father choosing the three-year subscription plan to pay for the enormous cost of those books, so that his children could access the information to support our education. At the time, an encyclopedia set was an essential investment for any educated family.

My father reports purchasing an absurdly high number of books: two to three per week. I suspect this data may be inflated somewhat. Nevertheless, even if he only purchased half the reported volume, it doesn’t materially affect this analysis. 

You may have cause to question the inclusion of fuel in this list. Living in Albany, Western Australia, my father was 4.5 hours drive from the nearest university. In pursuing his B.Ed. (Honours) course, he was forced to drive to university weekly. In following Nick Carr’s inclusion of information access costs, it’s only fair to include my father’s information access costs too.

Commentary - Loryn Jenkins’ expenditure

My list of “information expenditure” includes many more categories than does my father’s, and a few categories that are absent.

The most notable absent category is the encyclopedia—I will simply never purchase an encyclopedia for my family. Not because my family is less well educated; on the contrary, my children have far greater access to information than I ever did as a child.

Also absent is postage. While my mother sent a letter to her mother every week—she couldn’t often afford to place an international call to her mother in New Zealand—she compensated by sending letters. If I want to communicate now, I can quite freely afford to call anyone, anywhere in the world via phone or Skype or email; as I have done in communicating with my brother, who lived in Atlanta, GA from 1998 through 2009.

Categories that did not exist on my father’s list include cell phone services, online computer backup, music, hand-held game consoles, console games, fax, domain name purchases and website operating costs.

Analysis

The 1985 expenditures and the 2009 expenditures have been reported as a percentage of gross family income. You might wonder whether the comparison is distorted by a CTO (of a small software development company) pulling in greater income than a high-school teacher in the mid-1980s.

To answer that question, I quickly Alt-tabbed to Safari, and entered a google search, “cpi index australia 1985”. The search page returned me a link to a website that allowed me to specify my starting and ending years, and then produced the CPI inflation index (quarterly and annually) plotted between 1985 and 2009. (Something my father would never have attempted to do on a rainy Saturday evening in 1985.)

The answer is that in inflation-adjusted terms, my father’s reported 1985 income is 35% greater than my 2009 income. In comparing costs in inflation-adjusted terms, my father’s 1985 expenditures are 421% of my 2009 expenditures. Turning that around, I spend only 24% of what my father did on “information expenditure” at the same stage in life.

So whether you examine it as a percentage of gross annual income, or in inflation-adjusted terms, my father out-spent my purchase of information between 3 and 4 times.

While my father lived in a small country town in the most isolated state in Australia, and needed to drive to the nearest city for his education, I live in a far-flung city on the global stage. I mean this in physical, psychological and communication terms. Psychologically and in communication terms, I’m closer to Mark Bernstein (Boston), Mark Anderson (London), Dave Winer (Berkely, California), Joel Spolsky (New York), Paul Graham (Boston) and Slava Pestov than I do my brother-in-law who lives just two hours drive from my house and works in the mining industry. In terms of my capacity to gain knowledge, I can acquire university-level information from MIT OpenCourseWare, iTunesU, Google Scholar and Google Books. I can reach out and communicate with researchers and programmers and authors and other knowledge creators regardless of where they are located.

Conclusion

Given that my father, at the same stage in life, outspent me between 3 and 4 times in his purchase of information, information access, information-based entertainment and communications, Nick Carr’s charge that we spend more on information than ever before is proven false.

Furthermore, the quality of information, information access, information-based entertainment and communications is far greater in 2009 than in 1985. My father in 1985 had limited capacity to communicate with peers internationally. He had no capacity to publish information to thousands of people in 70 different countries. He had no capacity in 1985 to pull CPI indexes at will on a Saturday evening. He had no accounting software with which to summarise his expenditure. He had no spreadsheet software with which to calculate the numbers I’m marshalling here. He had no capacity to stroll down memory lane by looking through a satellite view of his home-town, like I’ve just done (my bike route from home to school). And he certainly had no capacity to communicate this all directly to you, gentle reader.

So I agree with Carr that the quantity of information available, being consumed, and being produced, is much greater now. But I thoroughly disagree that with the assertion that “Never before in history have people paid as much for information as they do today.” Wrong. Quite wrong.

My father paid 3 to 4 times more, for more limited access to information in 1985 than I do in 2009; and no capacity to unilaterally publish information like I’m doing right now. Information abounds.