Note-Taking Jujitsu, Or How I Make Sense Of What I Read

Note-taking is a problem. It's an interesting problem, but still a problem. Many people have switched over from paper books to digital copies. I am certainly one of the early adopters in this trend, having wrangled Graeme Smith and his sister into facilitating a first iteration of Amazon's Kindle to be delivered to my house in Kandahar.

My colleague Felix Kuehn and I used Kindle versions of books heavily in our research for An Enemy We Created. Using those references in footnotes was difficult at the time: the format was so new that established footnoting styles (APA/Chicago etc) hadn’t developed the standards for referencing kindle documents. All this was made harder by the fact that Kindle copies of books added a whole new problem into the mix by abandoning page numbers for ‘Kindle location numbers’. This changed a few years later, and current users probably won’t have this problem, but if you go look at the footnotes for An Enemy We Created, you’ll still find that many, if not most, of the references are to Kindle locations and not page numbers. In fact, I think our book was probably the first serious history work to rely so extensively on digital Kindle references in the footnotes; I remember having discussions with our publisher about it.

 
 

All this isn’t to say paper copies don't have their uses. But some books just aren't available in digital format. I'll get into the workaround for that later. The best way to make this less of a problem is to gently nudge publishers to issue their books on a kindle format.1 But I am already getting off track.

All this seemed to come to a head this past week, where a podcast I hosted together with Matt Trevithick took up the topic of notes and note-taking. Mark Bernstein, our guest on the show, wrote a really excellent book on the topic some years ago entitled The Tinderbox Way. I’d strongly recommend you read if you’re involved in knowledge work in any way. Here’s a short excerpt defining the importance and use patterns for notes:

“Notes play three distinct roles in our daily work:

•Notes are records, reminding us of ideas and observations that we might otherwise forget.

•Shared notes are a medium, an efficient communication channel between colleagues and collaborators.

•Notes are a process for clarifying thinking and for refining inchoate ideas.

Understanding often emerges gradually from the accumulation of factual detail and from our growing comprehension of the relationships among isolated details. Only after examining the data, laying it out and handling it, can we feel comfortable in reaching complex decisions.”2

Later in the week, Maria Popova (of Brainpickings fame) was on Tim Ferriss’ podcast to talk about her website, her reading and her workflow. Both Tim and Maria expressed frustration over the lack of tools for people wanting to download and interact with their Kindle clippings:

“I highlight in the kindle app on the iPad, and then Amazon has this function that you can basically see your kindle notes on the desktop on your computer. I go to those, I copy them from that page, and I paste them into an Evernote file to have all my notes on a specific book in one place. But sometimes I will also take a screengrab of a kindle page with my highlighted passage, and then email that screengrab into my Evernote email, because Evernote has, as you know, Optical Character Recognition, so when I search within it, it’s also going to search the text in that image. I don’t have to wait till I’ve finished the book.

The formatting is kind of shitty in the kindle notes on the desktop(…) if you copy them, they paste into Evernote with this really weird formatting. (…) It’s awful. If you want to fix it you have to do it manually within Evernote. (…) There is no viable solution that I know.”3

She then goes on to some more detailed points of how this doesn’t work, and Tim commiserates, suggesting that maybe they should hire some people to fix this problem. But the good thing is that there are solutions. The problems Maria and Tim bemoan are things that every other Kindle user has had to deal with since day one, so thankfully there are a number of workarounds that simplify the process of reading, annotating and sifting within one’s notes of a book or document.4

So notes are important, we get that. But how do we use them to their utmost? How do we even gather them together and store them? How do we use them for our writing, for our thinking? These are all important questions which I don’t feel have been properly answered, and where those answers have been given, they’re buried or hidden somewhere out on the internet.

I want this post to get into the weeds about how to get your materials off a Kindle device, how to store it usefully on a Mac (my apologies, PC/Linux users), and how to repurpose those notes to be creative, to write, and to think.

This post has three parts:

  1. Storage
  2. Clipping & Splitting
  3. Discovery & Meaning

It will by necessity be an overview of some useful tools and options for researchers, but if you leave comments I can probably expand on individual points/sections in follow-up posts if needed.

1. Storage

This is a problem that wasn’t explicitly raised in the things that motivated this post, but it’s something I get asked frequently. Maria and Tim both seem to be avid Evernote users, and I know many others also use this, but there are other options. It’s worth starting here because the tools will determine what you can do with your notes.

I’ve offered advice to other Mac users on what software to use for research projects that require a certain deftness in handling large quantities of sometimes disparate materials. The same applies to people who are just trying to keep track of the things they read, trying to draw together connections, and to derive meaning from it all. I’ll get into the meaning-creation in the final section, but for the moment, let me briefly describe our four options for file/note storage as I see it.5

  1. Finder/PathFinder. This is the lowest-tech option. Basically, once you split your files up (see section two) you store them in folders and refer to them that way. I don’t find this option very attractive or useful, because it’s like a filing cabinet. Your ability to discover connections and to remember what’s in those folders is pretty limited. I don’t recommend this at all, but from conversations with other researchers and writers, it seems this is the default option.
  2. Evernote. I include this here because it’s part of a workflow that we’ll cover later on. Evernote is great for all the reasons you can read about on their site. It syncs across all your mobile and desktop devices, it OCRs images so you can search for text captured inside photos you upload into your library of notes.
  3. DevonThink. This is my default ‘bucket’ for information, documents and notes. You can read up on the many (MANY) things that DevonThink Pro Office or DTPO (the version you should get, if you’re getting this software) does. Not only does DTPO store your documents, but it allows you to access that information in a number of extremely useful formats. There is a mobile app, too, though it could do with a bit more work. The most interesting feature of DTPO is its search and discovery functionality (using some magic sauce algorithms). They don’t make as much of this on their website as they used to, but I’d strongly recommend you check out these two articles (one, and two) by Steve Berlin Johnson which explain a little of the wonderful things DevonThink can do for your notes. As with the next recommendation, it’s not cheap. But powerful doesn’t always come cheap. It’s a solid investment if you spend the time getting to know this piece of software.
  4. Tinderbox. I discussed this at some length on the Sources & Methods podcast with Mark Bernstein, so I’d recommend you listen to that as your first port of call. Tinderbox is not an everything-bucket in the way that Evernote and DevonThink are, and I use it slightly differently, but it’s a great place to actually do the work of thinking, organising and writing once you have something (i.e. a project of some sort) for which you want to use all your notes. I’ll explain more about this in section three.

I’d recommend getting to know the different bits of software to get a sense of what they can do. DevonThink has a handy section of their website where you can see how people use it in their work lives. Tinderbox has something similar, with some case studies of usage.

For DevonThink, it’s generally good to keep your ‘buckets’/databases of files separated by topic. I have a mix of these kinds of databases (50 in total): some are country-specific, some are project-specific (i.e. to contain the research that goes into a book or a long report), and some are topic-specific (i.e. I have one for clippings and notes relating to Mathematics, one for things relating to Cardiology etc). I’d also recommend you give Steve Berlin Johnson’s book Where Good Ideas Come From a read, particularly chapter 4.

Given the learning curve with some aspects of the workflow that follows, you might want to consider introducing these pieces of software one-by-one, or as needed. That way you’re using only what you understand and can implement things without being too overwhelmed by the novelty of the systems. It took me years (almost a decade) to implement and iterate the systems described below, and I’m still not finished modifying as the tools change.

2. Clipping & Splitting

This section is all about getting materials off mobile devices and onto your computer where you can put them into some sort of overarching database.

Accessing Your Amazon Kindle Clippings

First let’s sort out how best to get notes from a kindle onto your Mac. Don’t use Amazon’s website. It’s going to create all sorts of problems for you in terms of formatting.

First thing’s first: sync your kindle to the cloud. Just turn on the wifi/3G and select the “Sync” option. This will ensure all your highlights are backed up to the cloud.

Then plug your Kindle into your computer via USB. Then go into the “Documents” folder, and search for a file called “My Clippings.txt”. If you’ve been using your kindle for a while, it’s probably going to be quite large. Nevertheless, copy that file to your desktop. Feel free to eject your Kindle from your laptop now. We won’t be needing it any more.

 

An example of what you might see when you open your "My Clippings.txt" file

 

If you open the txt file that is now saved to your desktop, you’ll find all your clippings and annotations preserved in a useful plaintext format. This may solve your problems straightaway, in which case, congratulations: you now have all your annotations in a useful format that you can use however you wish.

If you want to take it to the next level, though, you’ll want to split this file up. At the moment, you have a very large plaintext file which contains all your notes. You’re likely to have notes from a wide variety of topics and books in here, so it doesn’t make sense for you to keep them all in a single location. The ideal solution is for you to have a single file for every clipping, a single file for every annotation.6

This is where Split-ter.scpt comes in. I’m afraid I don’t know who to credit for this wonderful piece of code. I downloaded it somewhere on the internet some years back and can’t seem to find a link to the author either in the code or elsewhere online. (Whoever you are, thank you!)

This script works with another piece of software mentioned above — DevonThink Pro Office. For now, I’ll ask you to ignore that bit, and focus on what’s happening to the file. I use the script to convert our “My Clippings.txt” file into multiple files. It goes in, finds a delimiter (any piece of text or syntax that repeats itself in the original file) and creates a new note/file every time it comes across this delimiter. In this way, you’ll quite quickly from the file shown above to something like this:

Now you have a note for every annotation and/or clipping. This is then something you can dump into Evernote, or keep in DevonThink. Again, more about the difference between these programmes in the next section. (Note, that you can use Tinderbox to split up the “MyClippings.txt” file as well using the “Explode” tool).

UPDATE (a little later on Friday night): Seb Pearce has just let me know that there are other options available for dealing with the 'My Clippings.txt' file. Check them out on his site.

The second problem raised on the Tim Ferriss podcast was Amazon’s limitations for clippings. This differs from publisher to publisher, it seems, so there’s no way of predicting it. An unfortunate rule of thumb: the more useful the book, the more likely the publisher has locked it down. When you’re making clippings inside the book, Amazon gives you no notification that you’ve reached the book’s limitations. But when you go to check your “My Clippings.txt” file to start using your notes, then you may find the note says:

"<You have reached the clipping limit set by the publishers>"

All the work you’ve done selecting pieces of text are for nothing, it would seem. The publisher has prevented you from using your book.

One solution is to remove the DRM from the book before you put it on your kindle. This is legal so long as you’re not sharing the book with other people (as this process would theoretically allow you to do).7 Follow this link to find out how to de-DRM your Kindle and iBooks documents. You can also visit libgen.org to download an already-DRMed copy of the book you’ve purchased. These will often be in .epub format so you’ll have to convert these over to a .mobi format if you want to use them on your kindle device. (To convert from .epub to .mobi, use the free Calibre cross-platform software.)

If you read a de-DRMed copy of a kindle book on your kindle device, there will be no limitations as to how much you can annotate. The publishers limitations will all be gone. So that’s one option.

For those who aren’t comfortable removing the DRM on your books, you can get all your annotations out, but it comes with a little bit of hassle.

Here’s an example of what I mean (screenshot from my DevonThink library). I was reading in Hegghammer’s excellent Jihad in Saudi Arabia and making highlights (at 4:06am, apparently) but at some point I hit the limit imposed by the publisher.

 
 

The workaround to bypass this limit from the publisher is to first export all your notes out of your “MyClippings.txt” file. So all your clippings are saved, even though some of them may not work. Let’s say, for the sake of argument, that the final three notes aren’t working because of the publisher’s limitatations. That’s the case in the screenshot above. What you do is (again, once you’ve backed up the clippings txt file) delete three of the earlier clippings that you already have. Then you sync your Kindle to the server and it will think that you have clipped three less quotes, so these will then become available (both in the myclippings.txt file and on the website. Like I said, it’s a bit fiddly. I would much rather remove the DRM completely and not have this hassle at all, though when you do that Amazon will not sync your clippings to the cloud and to their kindle.amazon.com database. You’ll have to export them using the tools I mentioned above.

Keeping Up With The Joneses, or How to Use Instapaper to Clip Web Articles

This may be something completely idiosyncratic to my own workflow, but I don’t enjoy reading articles in a web browser. I’d also prefer not to be hijacked into reading all these articles. For instance, when I’m in Tweetbot/Twitter or Facebook and I see a link that I like, I will almost never read that article then and there. Rather, I’ll send it to my Instapaper queue.

First, a quick word about Instapaper vs Pocket. I use Instapaper. I started off with them, switched over to Pocket for about two years, and now I’m back with Instapaper. They’re both more or less the same. Instapaper happens to be what I’ve chosen for myself because of their handy Kindle service. (If you have articles in your queue, you can have Instapaper send the most recent articles to your Kindle at a particular time (i.e. first thing in the morning) which you can then clip and archive to your heart’s content.) Both Pocket and Instapaper work with what follows, so just pick one and stick to it. I’d recommend Instapaper because they allow for the sharing of the full texts of articles and because of the Kindle digest feature.

I find I have so much to stay on top of and keep tracking online, I can’t just click around and read things as and when I see them online. I schedule time apart for reading of my Instapaper queue (and for reading books on my Kindle) and only read during those times. (I do the same with email, only checking and responding to email between the hours of 12-1pm and 5-6pm each day. The rest of the day email is off and disabled. I even deleted my email account on my iPhone as inspired by this medium.com post.)

My workflow with web articles is to follow as much as possible via RSS. I prune the sites I’m following every three months, but in general the number is stable around 650. I use Newsblur as my RSS reader, and every time I find an article I’d like to read (later), I use the handy ‘send to instapaper’ bookmarklet. This sends the article to my Instapaper queue.

The same goes for twitter. I follow enough people on Twitter for it to be impossible for me to read every post that passes through my stream. I will dip once or twice a day, however, to see what people are saying. I use two services to monitor my Twitter and Facebook streams to pick out the most-shared articles to ensure that I don’t miss the big sharks of the day. They’re both free, and I’d strongly recommend you signing up and getting their daily summaries of what people were talking about on Twitter that day. News.me has been around for a while and I trust their article selection. Nuzzel is newer, but it seems to have a few more options. I guess you could probably do with picking only one of the two.

After reading articles on my Kindle (or sometimes on a mobile device like my iPad or iPhone), you can clip the article if you want to save it (just like making a clipping inside a book, only the entire article is saved).

 

This is what you see in an article when you click to "Clip This Article" on a kindle...

 

Then your clippings will be captured in the ‘MyClippings.txt’ file as explained above and you can export them directly to DevonThink or Evernote or Tinderbox. (The main downside to doing things this way is that when the kindle clips it, all formatting is lost (including paragraph breaks)).

Alternatively, you can ‘Favourite’ the article. I use this setting because it then sends the article and URL to my @stricklinks twitter account, something I created to share the best things I was reading. It also saves the full text of the article to Pinboard (a service I’ve already written about on my blog here) and to Evernote. (I use If This Then That to facilitate this.)

Once I’m done reading, I can go into Evernote and all my articles are waiting for me to be sorted through. Because I use DevonThink as my everything-bucket, and because all the sorting and discoverability features are there, I have a separate stage of exporting my notes out of Evernote into DevonThink. I’ve already probably taken you a little too far down the rabbit-hole of my workflow, but this is an important stage because otherwise you can’t do anything with your notes.

Luckily, someone has written a script which makes this possible. Many many thanks to the good people at Veritrope for updating the script every time updates to the software get released. It’s fairly self-explanatory. You select the notes that you want to export, choose which DevonThink folder you want to export to and then it goes to work. It can occasionally be buggy and stop half-way through, but usually a little trial-and-error will let you pinpoint which Evernote note is causing the problem and you can transport that one over manually.

I usually do an export session to bring everything from Evernote into my DevonThink inbox once a week. This way the number of clippings doesn’t get too out of control, and I’m not constantly playing around with this during the week. You might find this all is overkill, but it has become an essential part of my workflow to store the various things I’m reading on a daily basis.

Pillaging Your Hard Copies, AKA Living the Paperless Dream

You may have hardcover copies of books that you want to use as part of this system. One way to use them is to scan the books into your DevonThink library. DevonThink Pro Office comes with an OCR package (via ABBYY FineReader) so whatever you scan can then become searchable and useful.

In the past, particularly with books I’ve purchased in Afghanistan and Pakistan that are unlikely (read: never) to be made available as electronic versions, I take a Stanley knife to the bindings, feed the pages into my ScanSnap scanner which scans both sides and compiles all the scans into a single PDF document that is searchable on my laptop. The whole process is destructive of the book, but it gives the text inside a new life. Given how fast the new ScanSnap models work (around 25 pages per minute, both sides), this is an attractive way to get digital access to materials that are only available in paper form.

You can highlight text within the resulting PDFs and then later export your clippings from those PDFs as notes into DevonThink. There’s another useful script to help with that. It only works with the free Skim PDF reader, but that’s my default PDF reader so it works out well.

For more on paperless workflows, check out David Sparks’ Field Guide on the topic.

3. Discovery & Meaning

If you made it this far, congratulations. This is the section where all the fiddling with export starts to take on some meaning. After all, we’re not reading and exporting these notes purely because we are hoarders or to fetishise the art of collection (though in some cases, that may be what’s going on). No, we are taking notes because we are trying to understand difficult topics, because we are trying to solve important problems.

Discovering Links and Connections

The Steve Berlin Johnson articles referenced earlier are an essential first stop, particularly in demonstrating how DevonThink can add some serendipity into how you use your individual notes. To give you an example of how this works, here’s a screenshot from my ‘TalQaeda’ database that I put together while working on An Enemy We Created:

 
 

In the upper part you can see a bunch of notes relating to the Haqqani family. The lower left part is the contents of a note (Note: exported from Instapaper). The bottom right list of documents (under “See Also”) is a list of notes that may be related to this particular quote. This is the magic algorithmic sauce I mentioned earlier that makes DevonThink so powerful.

If I click through to some of those suggested notes, I’m taken to similar quotes on the same topics, two PDFs of reports (of dubious analytic value, but that’s a separate issue), three clippings from Kindle books where people are making reference to the relationship between the Haqqanis and al-Qaeda (the subject of the original note). Note that I didn’t have to pre-tag documents for this ‘see also’ functionality to work its magic. It analyses inside the text and makes its suggestions based on the similarities it identifies. (Needless to say, it’s not simply a matter of matching individual words. Some of the suggested notes don’t mention al-Qaeda or the Haqqanis by name, but they are implied; DevonThink catches this all).

Once you start to build up a decent database of notes (my Afghanistan database has just under 65 million words of notes, including 12,800+ PDFs) this ‘See Also’ functionality really allows for some unexpected links to be made, especially when you’re at the stage of writing up a project/book. One note will lead to another note, which will lead to another note. If you follow these trails of notes (like breadcrumbs) you can develop a pretty idiosyncratic picture.

I do not know of a manual method which allows for this kind of process.

DevonThink has an extremely robust search function which allows you to find things along similar principles (including a very useful ‘fuzzy spelling’ option, perfect when checking my database for notes on someone whose first name could be spelt Mohammad, Muhammad, Mohammed or any of the other variations).

Figuring Out What It All Means

Once you have an idea of the outlines of the topic, once you’ve been taking notes for a while, your database in DevonThink is probably starting to fill with useful information.

If you’re writing a book, though, you’ll want to start writing alongside this gathering process. (Check out Michael Lopp’s overview of the process of writing a large research book, which, to my mind, is fairly accurate.)

I don’t find DevonThink a particularly pleasant place to write, so I do that elsewhere. Before I write things out in long form, I usually do some outlining, particularly if it’s something where the dense collection of factual detail is important to the development of the argument (as was the case with An Enemy We Created). For this, I find Tinderbox indispensable for working up an overview of what I know, for figuring out how I’m going to structure it, and for helping me put together my first draft.

Tinderbox can display notes in a number of different ways. You can view your documents as outlines, as maps, or even as timelines:

 
 

In this image you can see the information arranged as an outline, but here (below) you see the same information organised as a map (mirroring the actual layout of the map of those districts in a particular part of Kandahar):

 
 

Just to show you that it can handle complexity, here’s a map created by Felix to help him figure out how people involved in militant Islamism were/are connected across different geographical sectors:

It's complicated...

I’ll often use Tinderbox maps to store outlines for how I’ll write a particular section or chapter, making notes inside the document, dragging quotes in from DevonThink to supplement the argument that’s being constructed.

Getting to the point where you can actually start writing on the basis of your notes is the whole point of all of this. Technology is useful, but mainly when directed at a specific problem or goal. All the tips, tricks and software described in this post has helped me write books, reports and (coming soon!) even my doctoral thesis/PhD. I have encountered only a few (barely a handful) researchers who use their computers for this collation, sifting and discovery process. There’s no way to keep it all in your head. Here’s hoping more people start adopting these tools…

Footnotes:

  1. For many years, Amazon offered users the ability to let publishers know that you wanted to see title X or Y on a Kindle format, but they failed to make this piece of interaction useful by keeping track of what you'd requested of publishers (so as then to be able to let you know when it was finally released in Kindle format).
  2. Excerpt From: Mark Bernstein. “The Tinderbox Way.” iBooks.
  3. Selective transcript from around the 50-minute mark in the podcast audio. Needless to say, the rest of this blogpost constitutions a ‘viable solution’.
  4. Most of these are derived from other people, I should say. I try to give credit where I can, but sometimes I can’t remember where I first read something or who first recommended a particular tool or trick.
  5. Yes yes, I know, I’m going to leave out some mentions for useful software here. This is an overview, and I’m just trying to describe some options for what might work in certain situations.
  6. A clipping is when you have selected and copied a passage from the book for safe-keeping, and an annotation is when you yourself write a note connected to a particular passage.
  7. Needless to say, don’t take legal advice from me.

Sources & Methods: Podcast Follow-Up

We're five episodes old! The small podcast I started together with Matt Trevithick is coming along nicely. In our most recent episode, we talk with programmer and note-taker Mark Bernstein. Mark is the force behind the notetaking and outlining software, Tinderbox, much beloved by knowledge workers. This episode is about note-taking, its uses and why people need to think reflexively about the work they're doing.

It seems to have struck a chord with listeners: we've had three times as many as usual. That could also have been helped by a mention over at The Atlantic from James Fallows:

Mark Bernstein is the creator of intriguing idea-organizing Mac software called Tinderbox, which I've mentioned over the years. I have never met him but have often corresponded; three years ago, he was a guest blogger here. This week, in a podcast interview for the Sources and Methods site, he talks not so much about his software but about the larger question of how thinking interacts with the tools of the electronic age. If you find the podcast provocative, you might well be interested in the book The Tinderbox Way, which is equal parts guide to Bernstein's Tinderbox program and meditation on the right and wrong approach to "information farming." As you'll gather from the podcast and see in the book, the kind of farming he has in mind is nothing like mega-scale factory farming and very much like an artisanal plot.

The Director of Teaching and Learning, for the Bedford/St. Martin's imprint of Macmillan Education publishing house wrote a blogpost about the episode in which he recommended educators give it a listen:

There's a lot in the discussion that maps on to teaching writing, teaching research, teaching thinking, and faculty development for those professors who want to help students get better at writing, research, and thinking. The interview can be assigned in time points for students, or one might scroll to to a point and play a snippet as a way to launch a discussion. For students especially, this discussion focuses on the role of noting, of seeing and recording, and in the act of doing so, of thinking, organizing, and find order. In a way, it's about slowing down, of taking the time to start a system that will serve a learner as a writer, and over time, as they change as writers, learn more, know more, and will find it more and more useful to be able to go back into their reading and writing history to recall, reorganize, and rethink, note taking as a key element for revision.

I've added a number of podcasts to my regular queue in Overcast and in case you could use some recommendations, the following are almost always worth a listen, especially if you like the kinds of things you hear on Sources and Methods:

Our First Publication: "I Am Akbar Agha", Memoir of a Taliban Insider

I_am_Akbar_Agha-cover Very happy to announce the publication of First Draft Publishing's inaugural book, I Am Akbar Agha. Felix and I have been working on this initiative for a while, and it's nice to have it see the light of day.

Akbar Agha's book is the latest in an (increasingly) long line of memoirs by individuals associated with the (Afghan) Taliban. So far we have insider accounts from Zaeef, Mutawakil, Mujhda, Mustas'ad, Abu al-Walid al-Masri and rumours of works being written by figures like Mawlawi Qalamuddin and others.

As for what's in the book, I'll refer you to the blurb on the back:

Following in the tradition of Mullah Zaeef’s My Life With the Taliban, Akbar Agha’s memoir tells a story of war, friendship and political intrigue. Starting in 1980s Kandahar, the difficulties and successes of the mujahedeen come through clearly as Akbar Agha struggles to administer a group of fighters. He details the different groups fighting in Kandahar, their cooperation and the scale of the Soviet Union’s efforts to crush them. Not directly a participant in the Taliban government that ruled post-1994, Akbar Agha offers a sometimes-critical account of the administration built by many of his former fighters. After the fall of the Islamic Emirate in 2001, Akbar Agha was involved in the Jaish ul-Muslimeen opposition group and for the first time he has revealed his account of what happened in the kidnapping of UN aid workers. I Am Akbar Agha ends with an analysis of the problems afflicting Afghanistan and outlines a vision for the political future of the country post-elections and post-2015. Anand Gopal has written an introduction to the book.

 

If you're interested in Afghanistan and want to delve a little deeper into the Taliban's (pre-)history, you'll enjoy this book. Lots of new material.

We also published an exclusive interview with Akbar Agha over on our blog.

Anand Gopal has written a great foreword to introduce and contextualise the book. If you're interested in learning more about the book but don't know whether to commit to buying it, I'd recommend downloading the free preview/sample of the text via iBooks or the Amazon Kindle Store where you can read the full text of Anand's foreword.

If you end up reading it, please leave a review on Amazon and Goodreads as a way of supporting our publishing initiative to get more of these primary source texts translated and made available to a wider audience.

Buy a paperback or electronic copy here: LINK

Find out more about First Draft Publishing here: LINK

Sources & Methods, or why I started a podcast

 
logo-blackjpg
 

If you go over to sourcesandmethods.com you'll find the inaugural episode of a new podcast I'm doing together with Matt Trevithick. Seriously, go there now and have a listen. You'll be able to subscribe from the iTunes directory etc in about a week or so, but for now if you want to hook it up to your preferred podcast client, use the RSS feed provided.

Matt and I wanted to do this partly as an excuse to talk to interesting people, but also out of a sense that there is a certain type of person who gets interviewed less than they ought (if at all). So here's hoping that we keep interviewing interesting people, esp those who aren't already saturating the media.

Head on over and give it a try. Anand talks about things he's never talked about in a public forum (as far as I'm aware).

An Ankified Urdu Frequency Dictionary

Urdu Frequency Dictionary Until recently I had been trying to spend more time in Pakistan's megatropolis, Karachi. As part of this move I had been trying to learn Urdu. There are a variety of excellent study materials for Urdu, but I won't write about those today. Rather, I want to offer up a resource for serious students of the language.

Readers of my previous language posts will know about Anki. For those who don't know, it's basically software that allows you to memorise pieces of information (such as foreign language vocabulary).

Frequency dictionaries are wonderful things. They present words of a language in the order of frequency used (usually in writing). They are assembled by amassing huge databases/corpora of text and these are analysed to see which words are used most often.

For a beginning learner of a language, they can be a real help. You start with the most frequently used words and work your way out to the ones you'll encounter less.

Nothing like this exists for Urdu. Or so I thought. I was passed a series of scanned PDF images of an old frequency dictionary published in Canada in 1969. This was made on the basis of an analysis of newspaper copy/texts. Obviously the language used is a bit dated, but as a solid start, this is a good selection. (For those with deep pockets, you can search bookfinder.com for "An Urdu Newspaper Word Count" by Mohammad Abd-Al-Rahman Barker but I'll warn you that the cheapest copies available are $100+ USD).

Unfortunately, in the part of the dictionary I was sent, the words are listed in alphabetical (by Urdu) order. This means that the order is not ideal. There are, however, 9,956 words in this collection. If you're serious about Urdu you could do a lot worse than learning all of them. You'll skew your vocabulary a little towards the literary side, but that's not necessarily a bad thing.

I had someone type up the whole dictionary into Anki and add a spoken audio file for every word. (Many thanks to Affan Ahmad for this massive labour.) You can even set Anki to deliver you words randomly served from the frequency dictionary.

So, without any further ado, the files are here. They are split up because Anki became a bit difficult when inputting the files, but you can combine them on your own computer into a single "Urdu" folder.

UrduAnki1

UrduAnki2

Enjoy! And please post any feedback below in the comments. I mainly wanted to get this out into the public so people can use it (rather than gathering dust on my laptop).

Two new co-authored reports on Afghanistan

21 Just a short post. Two reports that I co-authored have just been published. They were both finished a few months back, but they're not so time-sensitive that this will make much of a difference.

The first is for Chatham House, written together with Felix Kuehn. You can read the executive summary here, and download the full report here. The central point we were trying to get across is that a political settlement in Afghanistan must be about more than just 'talks with the Taliban'. That ship has sailed, and new realities mean it's important to bring all parties into a discussion about the future. I remain skeptical as to internal and external parties' ability to make this happen, but here's hoping...

The second report, much longer, was mainly an effort of Felix Kuehn and Leah Farrall but I contributed some things on the sidelines. This was expert witness testimony in the case of US vs. Talha Ahsan and US vs. Babar Ahmad. You can read some of the background to the case here and here. The report we were tasked with writing related to Talha and Babar's activities in Afghanistan during the 1990s, and the extent to which this equated with support for or 'membership' in al-Qaeda. Felix and I have already written a decent amount on the topic, but it was great to team up with Leah to dive into the foreign fighters' side a great deal more.

You can read our report here, starting on page 148. It's a long report, but there's a lot of new material in there which has never been published (as far as I'm aware).

I'd also recommend reading through the judge's statement Talha's sentencing memo. I don't quite understand why there hasn't been more media coverage of this trial, and how the government were more or less told their case was extremely rickety. Perhaps it's because of all the other things going on at the moment.

UPDATE: Edited on August 11 to reflect error in identifying the sentencing memo.

How to learn a language to fluency: interview with Gabe Wyner

fluentforever Gabe Wyner has a book out today, and I took the time to interview him about his method. It relies on several things I've mentioned on this blog before -- namely Lang-8, Anki, spaced repetition, mnemonics etc -- so I'm trying out something new: posting the audio of the interview I did a few days ago.

Gabe's book is worth your time, especially if you're getting back into learning languages, or if you're starting for the first time. If you have less patience for books and the written word, you can watch his course over on CreativeLive. It's 18-hours of instruction (in an classroom environment) on how to learn a language using his method. All the steps outlined in his book are expanded upon in this course. I have watched it all, and can attest to its value.

The interview can be listened to on Soundcloud or directly below:

[soundcloud url="https://api.soundcloud.com/tracks/161778415" params="auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=true" width="100%" height="150" iframe="true" /]

The Best Books I Read in 2013

This year was a big one. 79 books in total, and still a few days to go. I only have one Afghanistan-related pick this year, and that's Vanessa M. Gezari's The Tender Soldier. Gezari's tale of incompetence, misunderstanding and tragedy packs a punch. Some of the middle sections of the book -- profiling those involved higher up in the Human Terrain Teams' management -- could probably have been ditched. Speed through them, though, and you have a story whose complexity and strangeness has the eerie ring of reality. Definetely the best book I've read on Afghanistan in recent years.

The rest of my favourites are a bit all over the place:

Donna Tartt's The Goldfinch needs no selling by me, but it's a real return to true form. As always with Tartt, be careful when you start as it is incredibly gripping and you'll be neglecting work, friends and family in order to keep reading. Also, be careful: this one's a bit of an emotional kick-in-the-guts.

Jon Moallem wrote Wild Ones: A Sometimes Dismaying, Weirdly Reassuring Story About Looking at People Looking at Animals in America as a way of reconnecting with the natural world while his daughter grew up, and it was one of the most unexpectedly enjoyable books I read this year. Stories about conservation, humans and how we interact with all the other species on the planet. The book is broadly structured in three parts, covering polar bears, butterflies (Lange's Metalmark) and the whooping crane. Mooallem looks at the ways in which people are involved in efforts to save these three species, often ending up telling us more about humans than the animals that are nominally his subject. He tells a hopeful tale, for the most part, and although his slightly saccharine ending wasn't really to my taste, it probably is correctly pitched for an American audience. We don't understand the effects of our actions in the world, or when we think we do, we don't anticipate second and third-order consequences. It's not a new idea, but it was memorably told in this well-written book.

Lion Kimbro's How to Make a Complete Map of Every Thought You Think (available for free here) isn't new, and it definitely wins the strangest-book-of-2013 prize. Kimbro decides he wants to map out his thoughts 24-7 for a few months and the book is a description of the process he used to map it all out. It's written in a stream-of-consciousness style, and there's an awful lot about pens and paper and binders and organisational systems and information hygiene (for want of a better term). There's a lot worth taking away from the experiment, though, and if you can make it through the book you'll have learnt a few tricks on the way.

I LOVED Matt Potter's Outlaws Inc., the first book I've read where a piece of technology (in this case the IL-76 cargo plane) is a lead character, but WHAT a story. Potter takes you through the world of air cargo transport and the post-Soviet airmen who fly the planes. It's perhaps a bit long, but that's probably me being unfair. I was up all night reading this book. Definitely recommend it for shining a light on something I hadn't really thought about but that is very much a part of trade and international aid.

Guy Deutscher's The Unfolding of Language wasn't an easy read, but it's the smartest (and easy-to-understand) book on linguistics I've read. It offers an overview of how languages work, and how they change over time (fragmenting and joining together in a myriad of strange ways). I'm looking forward to reading Deutscher's other book in 2014.

I've been a big fan of Marie (aka puredoxyk) for about a year now. I did a 2-month experiment switching to a polyphasic sleep cycle earlier this year -- that's 2-4 hours of sleep only per day, depending on various things -- and I couldn't have done it without her book Ubersleep: Nap-Based Sleep Schedules and the Polyphasic Lifestyle. (She's also very nice/helpful online). This book offers clear, useful advice on shifting to a polyphasic sleep cycle. If you're interested in sleep modification (and being able to sleep only 2 or 4 hours per day without medium-long-term issues), give this book a read. If nothing else, it'll expand your sense of what is possible, and that's always a good thing.

Finally, two health-related books. The past couple of years have been filled with various health issues (some mine, and some of others close to me) and I read more about health, fitness and diet than usual. Anti-Cancer by David Servan-Schreiber was one of the more helpful books on cancer that I read. It could probably use a bit of updating, but it is clear, offers evidence with references to follow up with further reading, along with useful lists of the basic ingredients of the so-called 'anti-cancer' diet. Was also glad to see that emphasis is placed on the mind-body connection. Could easily have been left out with all the focus on diet and nutrition. Full Catastrophe Living: Using the Wisdom of Your Body and Mind to Face Stress, Pain, and Illness by Jon Kabat-Zinn has a bit of a hippy title, but it's full of really important and powerful techniques. I'd strongly recommend it to anyone with ongoing chronic pain/illness issues.

Those were the best. Drop your book recommendations for 2014 in the comments below. I'm going to try for 100...

Taliban Time Travel, or How Our Understanding Is Almost Always Two Years Old

I've been noticing something on twitter and in the public debate surrounding the Afghan Taliban over the past few days. An interview of Motassim in the Guardian last week was shared widely online, and people particularly seemed to find his comments about Mullah Mohammad Omar's loss of control of the Taliban interesting. To those following the Taliban closely this isn't news. It's been the case for two or three years at least. What is interesting is that it has taken that long for that kind of analysis and comment to become accepted by mainstream commentators and to become part of the public debate on the Afghan Taliban.

My very unscientific guess is that there's usually a lag of 6-18 months from a trend starting to emerge within the Taliban to the point where those outside the movement start to notice it. And from there there's another 1-2 years before a particular feature or analytical point becomes accepted and part of public discourse.

Needless to say, this incredibly slow dispersal of understanding makes it hard for analysis and action to mesh together usefully.

Book of the Week: 'Al-Shabab in Somalia'

shabab An excellent overview of the history of the Somali al-Shabab group, one with many lessons or reminders of Afghanistan (at least for this reader).

This is a short book, based on some reports written for FFI and others, and in that it has the virtue of concision. Hansen covers al-Shabab's history starting with early proto-Islamist movements and groups started several decades ago. It was the best explanation of where the networks that make up al-Shabab come from that I've read, although it may just be that I haven't been following this too closely since the last time I was last in Mogadishu a couple of years ago.

It was packed with stories and trends that reminded me of Afghanistan, both in the way the international actors chose to respond and intervene, and also in the development of al-Shabab itself. For this reason I'd strongly recommend this book to those working in and on Afghanistan. You'll find a rich vein of material that you can bring back to enrich your understanding of the Taliban and/or the past decade or three. (Needless to say, I'm the last person to suggest that everything is the same in every country, and that there aren't hundreds of reasons why comparisons aren't useful in a this-happened-in-somalia-so-it-must-be-the-same-in-afghanistan.)

There are lots of names and places mentioned, and if you're not familiar with at least the bare outlines of the plot so far as well as some of the key players, you might find it confusing. I wish there was some sort of reference in the back to allow you to keep track of all the different people mentioned.

As always, I wasn't really sure I got a sense of the leaders of al-Shabab (or their fighters) as people in this book, but maybe that's one step too far and one in which it's harder to offer anything that isn't highly subjective or just unrepresentative. Perhaps we can look forward to a book of al-Shabab's songs and poems from Hurst in the future?

Overall, though, an impressive collation of information. Hansen has done us all a service in spending time in Somalia doing fieldwork and in taking the time to put this book together.

Buy it here.

Translators Sought -- Job Opening

I'm about to start a large archive translation project (together with Felix Kuehn and Anand Gopal). We're looking for some high-quality translators to work full-time on a project translating old newspapers and magazines from Pashto (and a small amount of Dari) into English.

The work would be 5 days per week, but you will be free to work from home. I.e. there will almost certainly not be any office for you to work from.

The project will take place over the course of a year, with a possible extension to a second year. We would initially hire you for a trial period before committing to hire you for the full year.

If you're interested, please head on over to the form here and fill it out. We'll get back to you in due course, if and when we're ready to start the hiring process.

UPDATE (JULY): We've stopped taking applications and are working on finalising the hiring process. Thanks for all your interest!

Learning to Code

Previous posts have been about languages and how to learn them. Not all languages are for communication with other people, though. It is a truism that more and more of our lives are lived through various technologies -- be it computers, 'smart' phones or other appliances -- but we often aren't too good at understanding how those things work. I've been trying to remedy this by getting a better understanding of the back end through programming languages. Not only has it been an interesting intellectual exercise, but I have found practical applications for the skills I have learnt. Recently, for example, I wrote a piece of code that crawled through webpages, saving only the parts of text that I needed to a separate database.

There is a huge variety of things that you can try out here, so I'll just offer some suggestions for things that I've found useful along the way. Most of this is aimed at complete beginners. I'll assume that's where you are as well.

Python is considered by many (if not most) to be the best place to start as a novice programmer. It teaches lots of transferrable skills that can be applied to other languages that you might want to pick up.

My top recommendation would be to enrol in Udacity's CS101 course. Udacity is a relative newcomer to the scene, but I've found the parts of this course that I've done (I'm still working my way through) to be excellent. It has LOTS of practice, frequent testing of your ability to solve problems along the way, and is not dull to watch at all (as some of these courses are). What's more, by the end of the course you will have built your own search engine using the skills you've learnt. It's free, so you have no excuse. Go sign up.

(If that's not for you, although I'm not sure why that would be the case, then visit Codeacademy or Learn Python The Hard Way for alternatives.)

In case you're interested in learning Ruby, you can try the following, many of which are designed for young children to be able to use and follow along, so, again, no excuses...

RubyMonk

TryRuby

Hackety Hack

Rails for Zombies

CodeAcademy Ruby

LearnStreet.com Ruby

Now go try some of those out...

Following Pakistan's Elections

Saba Imtiaz, a freelance journalist based in Karachi, has started a daily newsletter of updates relating to Pakistan's upcoming elections -- scheduled to take place this summer. She explains what she'll be offering in a recent blogpost:

Along with a round-up of the headlines and commentary from English, Urdu and Sindhi news sources, I’ll also be writing smaller profiles of candidates and major news issues, as well as doing smaller data dumps on voting patterns

 

View some of the updates here and subscribe to the newsletter via email here and via RSS here.

If you follow Colin Cookman's excellent Daily New Briefs then you'll probably want to subscribe to Saba's election update.