John Henry Reviews Documents

Integreon has an interesting discussion on a recent study pitting humans against machines.  No this isn't about supercomputers and Jeopardy! It's something much practical:

The underlying study by a trio of recognized experts in cognitive science, information management, and e-discovery, Herb Roitblat, Anne Kershaw, and Patrick Oot, is described in detail in their journal article, Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, published in the January 2010 issue of theJournal of the American Society for Information Science and Technology [link available at the Posse List].

It raises - and partially answers - the important question whether we are approaching a breakthrough in terms of the capability of automated review tools to render ‘consistent’ and ‘correct’ decisions, as measured against an existing standard, while classifying documents in a legal discovery context. The study pitted two teams of contract attorneys against two commercial electronic discovery applications to review a limited set - 5,000 documents - culled from a collection of 1.6 million documents. The larger collection had been reviewed two years earlier by attorney teams in connection with a Second Request relating to Verizon’s acquisition of MCI. The authors’ hypothesis was that “the rate of agreement between two independent [teams of] reviewers of the same documents will be equal to or less than the agreement between a computer-aided system and the original review.”

The study set out to test whether an automated review tool would show similar levels of agreement with classifications made by the original reviewers as did the two contract teams. The two re-review teams agreed with the original review on about 75% of document classification decisions; the commercial automated applications fared slightly better.

There a number of obvious (and not so obvious) flaws in the study, which the Integreon post nicely lays out. My first reaction is that "rate of agreement" is a lousy benchmark, since the measure conflates too many significant variables.

I'm also fascinated by this quest for the document review holy grail: total automation. Contrary to lean principles, these managers seek to automate the process without fully understanding how it works manually. Just exactly how and why do review document reviewers make different calls?

And what about a hybrid approach?

A potential hybrid model would have senior attorneys review representative sets of documents and the tool analyze features of the reviewed documents to identify and auto-tag “like” documents in the larger collection. As the review proceeded, the tool would ‘percolate’ to the review team’s attention subsets of documents from the collection dissimilar from those already reviewed. Based on the reviewers’ decisions as to these documents, the tool continues to apply tags to more of the collection.

The attraction of this approach is two-fold: human attorneys are still making initial determinations but the application magnifies the effect of their determinations by propagating decisions to similar documents throughout the larger collection. It has been suggested that, in the proper context, this approach would permit a single attorney to “review” a vast collection of documents in several hours. A test of that claim is warranted and, if the premise were proved, it would be impressive and could directly influence the increased use of automation in review, even if, for all the reasons stated above, wide adoption of such processes would take a while.

As a lawyer who likes to tightly control processes, I'll admit the attraction of this approach. As one moves down the hierarchy in any litigation team, deep knowledge of the client and issues is inevitable lost. If technology can leverage the knowledge of the most engaged, the better the result, theoretically.

(cross-posted at California E-Discovery Law)

Does Automation Diminish Our Basic Skills?

 Photo Credit: Rui Caldeira

Photo Credit: Rui Caldeira

Pilot Patrick Smith has another interesting article on cockpit automation and flight safety, something this blog has considered before.

Has automation reduced pilots' basic "stick and rudder" skills?  His answer:  "Probably, yes."

But the more interesting discussion is how automation has grafted a new technological skill set onto basic flying:

[A]utomation is merely a tool. You still need to tell the airplane what to do, when to do it, and how to do it. There are, for example, no fewer than six different ways that I can program in a simple climb or descent on my 757, depending on preference or circumstances. The automation is not flying the plane. The pilots are flying the plane through this automation.

A fitting metaphor for other knowledge work.  Technology hasn't changed what we do, as much as changed how we interface with machines to get it done.  The tools have changed.  The work, fundamentally, has not.

Of course, interfaces are complicated and can even add to our overall workload:

If you ask me, the modern cockpit hasn't sapped away a pilot's skills so much as overloaded and overburdened them, in rare instances leading to a dangerous loss of situational awareness.

A danger for all of us.  Alarms, notifications, badges, and our ever-expanding landscape of electronic inputs, distract us from real work.  Whether that's landing a plane, or delivering a project.

This has given birth to a meta-skill: the ability to sift, filter, and organize the elements of our work.  Our first challenge, then, is to maintain situational awareness in a complicated world.

Update:  Interesting post on maintaining situational awareness in e-discovery.

The Aesthetics Of Order

Interesting and elegant conceptual photography from Ursus Wehrli:

This video shows how one of the photos sets was made. Watch as the artist organizes a group of sunbathers, and all their gear, into well-ordered groups.

Any lessons here?

Organizing takes work! [Suggestion: the next time you have to organize people and beach equipment on a hot day, don't wear a three piece suit].

Some things are more functional and useful in a disordered state. Think of your desk covered with support materials while engaging in a major project. What's the point of going to the beach to lie in a line and segregate beach toys into individual piles? Maybe certain aspects of your work would be more productive and fun if you allowed more disorder.

Apparent disorder can actually be the true state of order. Again think about the sunbathers. True order is when each sunbather is gathered alongside his or her own beach toys and umbrellas. Not when all the umbrellas are grouped together. Most of us put our pens into one area of a drawer, in a group. Maybe true order is to scatter the pens around the office. Several on the desk, one in a notebook, one in the briefcase, and so on.

Finally, order can be beautiful. But disorder -- even with the same constituents which could be ordered -- may be more beautiful.  Such is the night sky.

(via Kottke)

To Be Profitable: Focus On The Customer, Not Profits.

(updated below) I finally had the chance to watch the Steve Jobs presentation on the iPhone 4 antennae issue. You can watch it here. I was impressed by this statement in Job's opening remarks:

We want to make ... all our users happy.  If you don't know that about Apple, you don't know Apple. We love making our users happy. That's what drives us to make these products in the first place.

Look, everyone has an opinion about Apple. And I have no special insight into how things actually work inside the organization. But it's fair to say that Apple's customers tend to be a very happy and devoted bunch. And its huge profits are clearly the result of staying customer-driven, by consistently turning out products people want.

As a result of focusing on the customer, rather than directly on profits, they're very profitable. Students of Lean will understand that this is not, in fact, a paradox. For long-term success in Apple's particular market, "Customer Focus" cannot be just an empty marketing slogan.

Stephen Covey alludes to this in Principle Centered Leadership. Businesses focused on profits will, in the long term, cease to be profitable. Businesses focused on higher principles - the reason they're in business in the first place -- will thrive. Apple seems to be a good example of this in practice.

Bonus Steve Jobs: To follow up on an earlier post about creative problem solving, I noticed when he said: "We want to find out what the real problem is before we start to come up with solutions."

Update:  Is buying Apple a mythical experience?  See this interesting post from Alexis Madrigal at The Atlantic.  (via Kottke)

(Photo credit: Apple Store)

The World's Got Talent

Among the many ways the web has changed the world, I'd like to add one more. For me, at least, it's engendered a greater appreciation for the variety and extent of human ability. Sure, some of what gets posted to the web is of questionable taste (or worse), but I can recall dozens of times reading someone's writing or watching a video and being, not just impressed, but surprised a fellow human was even capable of doing that.

David Letterman used to have a segment called Stupid Human Tricks, and maybe he still does. For the most part, they were, in fact, pretty stupid tricks. The web is full of those too.

But what I'm referring to is genuine talent: artistic and creativeathleticliterary, and intellectual. Okay, so the slip 'n slide video wasn't real.  But the web is a massive repository of human ability. And it's easily accessible through a computer and phone.

In doing this, it's helped me to realize that "ordinary" people can do "extraordinary" things. The world is full of talent. Human ability is everywhere. And never underestimate human potential.

On that note, please enjoy this video:

5 Things Science Tells Us About Motivation

What does science tell us about motivating people to do good work?  Here's an interesting ten minute video combining illustrations with a Daniel Pink lecture, and discussing five key findings:

  1. For rewarding simple straightforward tasks, money is a good motivator.
  2. When a task involves more than rudimentary cognitive ability -- some conceptual, creative thinking -- monetary rewards actually result in poorer performance.
  3. If you don't pay people enough, they won't be motivated to do a job.
  4. The best way to motivate people is to pay them enough to take the issue of money off the table.  Allow them to think about the work, rather than the money.
  5. Three factors lead to better performance and personal satisfaction: (1) Autonomy, (2) Mastery, and (3) Purpose.

Watch the video for a full explanation of the three factors and some examples involving real organizations.  This reminded me of the ideas underlying Google's "20-percent time."  And for a great example of these ideas in action, I highly recommend checking out Netflix's presentation on its "Freedom and Responsibility Culture."

(txs, Elan!)

An Overflowing Inbox Is Not A Kanban System

From David Allen (no link available):

It is a residue from the industrial and agricultural world, I think, when the things to be done were much more physically self-evident. The "piles" in most offices nowadays seem just meager attempts to reproduce the self-evidence of the crops, the machinery, the things to be made and moved of bygone times. But self-evidence is not forthcoming without more discrete focus and more self-directed thinking.

From a lean perspective, I think what people are looking for is a kanban system and visual controls to manage their work in the office. When the inbox starts to overflow, time to sort through it. When the mailbox gets overloaded with email, and the anxiety level gets high enough, time to scan the messages.

But these are emotional rather than rational cues.

Better to have cues built on an optimal worklow.  This means emptying all the buckets on a daily basis, and using your organizational system to cue when to take action on items. Get the mailbox down to zero three times daily, even if this means putting some of those emails in an @Action folder, until you have time to fully process them. Then cue up the work based on deadlines (i.e. customer demand) and maintaining continuous flow.

Overflowing buckets also indicate excessive inventory and poor processes. Time to rework your system.

And your inbox usually operates as a push system, rather than a pull system.

The point here is to use real cues instead of the artifical and misleading physical cues that emerge in an office environment.  The height of paper stacked on your credenza doesn't (or shouldn't) tell you what to do and when to do it.

When Does Choosing A Better Computer Become Wasteful?

No, I'm not referring to green computing devices. Though, apparently computers account for 2% of the world's carbon emissions. I'm swapping my nearly four year-old PC notebook for a new 15" MacBook Pro. Can you say upgrade? Like many buyers, I'm tempted to get the fastest possible machine with the most memory, given my budget.

But I keep thinking about over-processing. It's wasteful to get a tool that's more powerful than what's needed for the job. Here's some of my thinking.

  • How fast? I do minimal multimedia work. Mostly, I access databases and documents on a local network and remotely, create text-based documents, and work on the web. But time is money (my time ends up being my clients' money, to be precise). So, I decided to get the fastest available processor along with a solid state drive. I can always upgrade RAM, but predict 4 GB will be plenty for 95% of my work.
  • How much memory? With a 500 GB hard drive I can save data for years to come without worrying about usable disk space. But I've only got 65 GB of data now. So I decided to get the 128 GB drive. I can always upgrade when I near capacity. And who knows what cloud storage options will look like then.

The hardest decision was whether to get a solid state drive. Ultimately, I chose one because they're more reliable (no moving parts) and run cooler (no motor). The result is a more efficient machine, with the related benefit of a longer battery life. I decided to go with the Apple OEM drive rather than with a third party upgrade. There may be better after-market drives out there, but I'd rather avoid any potential problems with warranties and service. If there's a problem, it's Apple's to fix. Period.

Now, it wasn't that hard to identify the right machine for today's work. The over-processing analysis would have been easy from that standpoint. But predicting the appropriate tool for two to four years from now? Given the extraordinary rate of change in consumer electronics and the web -- who knows what we'll all be doing then. That's what made this a challenge.

Has anyone else experienced this challenge when buying a computer? From an enterprise IT perspective, our firm certainly has, and larger organizations must have it even worse.

5 (Other) Reasons To Go To The Gemba

The gemba is the place where the work is done. Lean managers "go to the gemba" to see it for themselves (genchi genbustu). This might be done during a continuous improvement (kaizen) project. Recently, I've had the pleasure of spending many hours in the gemba with our support staff. We are going firm-wide with a new document workflow, which we've been testing with a pilot group for the last six months. During our sessions, we've identified a lot of waste and generated great ideas for improving our processes.

I've also discovered -- or maybe rediscovered -- five other reasons for going to the gemba.

  1. Reaffirm respect for people. As a manager, it's easy to get preoccupied with your own work demands and forget about the day-to-day contributions others make to the organization. Seeing your people in action reminds you of their talent and dedication.
  2. Communicate. Organizations usually use email, intranet posts, and large meetings to communicate their goals and plans. But one-on-one meetings and small groups allow for more frank and focused discussions. And some people aren't comfortable talking at formal meetings. The gemba might be just the right context for a critical interaction. It also gives people the chance to ask questions, in person and in real-time. Sometimes the gemba is the first chance you get to really explain why you are doing a particular project.
  3. Remind people that you care. This might sound overly sentimental, but spending time with your people reminds them that you genuinely value their work and talent. It also reinforces the message that you appreciate their ideas and contributions to designing work processes.
  4. See other important issues. You might observe a problem that demands an immediate response. Perhaps it's a major form of waste with a quick or even an on-the-spot solution. Sometimes a person's worklife can be radically improved just by realizing they need a new $6.00 tool. And though hopefully there aren't any safety problems, but if there are, this is a chance to correct them before someone gets hurt.
  5. Learn something new. By watching and discussing the work with your people, it's guaranteed you'll learn something unexpected about your organization, your industry, and the work.

It's hard to take time away from your work to go to the gemba. But these reasons make it well worth it.