Monday, March 3, 2008

What Does It Take To Become A Grandmaster Developer?

See how much of the following sequences of letters and numbers you can memorize in the next 20 seconds:
  • T, E, X, A, S, O, H, I, O, V, E, R, M, O, N, T, R, H, O, D, E, I, S, L, A, N, D
  • 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41

Done yet?

Now, imagine that you were blindfolded and asked to recite them to someone else.

How well do you think you would do?

You'd probably be in one of these six groups:

Group 1 would get perfect recall without too much effort while Group 6 would really struggle to get a measly 20% recall. An interesting observation is that this wide range in results wouldn't be largely affected by the general memory abilities of the person. That is, the results are not based on innate or raw talent.

What's happened?

People in Group 1 didn't read the first sequence letter by letter. Somehow the words "TEXAS", "OHIO", "VERMONT", and "RHODE ISLAND" just jumped out to them because they recognized them as states in the USA. Then in the next flash of insight, they might have noticed the states are listed from largest to smallest in terms of area. Some might have even picked out that these are the states involved in the upcoming presidential primaries on Tuesday.

When approached with the number sequence, they might have read the first few numbers and then realized it was a list of prime numbers up to 41.

When asked to memorize it, they stored something like this in their mind:

"SELECT [LettersInWord] FROM [StatesInUpcomingPrimary] ORDER BY [StateSize]" which has 3 "chunks" to remember.

The second might have been stored as

"SELECT [Number] FROM [PrimeNumbers] WHERE [Number] <= 41" which also has about 3 "chunks."

All told, people in Group 1 had to consciously remember about 6 "chunks" of information. That's less numbers than a typical telephone number contains and it's what led to the perfect recall.

People in Group 6 probably took a much different approach. They might have just seen the sequences as a random collection of letters and numbers. Maybe they live in a country far away from the USA; maybe their native language is Chinese. To them, every letter and number was its own "chunk." When asked to remember the sequences, they might have been able to successfully recall 7-8 of these small "chunks."

That is, the people who remembered less overall information actually remembered more "chunks!" So in terms of "raw memory talent," Group 6 won even though they remembered only 20% of what Group 1 perfectly recalled.

Essentially Group 1 had a more efficient "chunking" algorithm.

Chunking

When talking about the mind of a chess grandmaster, Philip Ross writes:

"When confronted with a difficult position, a weaker player may calculate for half an hour, often looking many moves ahead, yet miss the right continuation, whereas a grandmaster sees the move immediately, without consciously analyzing anything at all.

By measuring the time it takes to commit a new chunk to memory and the number of hours a player must study chess before reaching grandmaster strength, Simon estimated that a typical grandmaster has access to roughly 50,000 to 100,000 chunks of chess information. A grandmaster can retrieve any of these chunks from memory simply by looking at a chess position, in the same way that most native English speakers can recite the poem 'Mary had a little lamb' after hearing just the first few words

The one thing that all expertise theorists agree on is that it takes enormous effort to build these structures in the mind. Simon coined a psychological law of his own, the 10-year rule, which states that it takes approximately a decade of heavy labor to master any field. Even child prodigies, such as Gauss in mathematics, Mozart in music and Bobby Fischer in chess, must have made an equivalent effort, perhaps by starting earlier and working harder than others.

Thus the first thing we learn about mastery is that it takes a lot of "chunks" (approximately 50,000 - 100,000) to get there.

Rating Systems

The next advantage that other fields have is a generally accepted rating system. Chess students can look at FIDE ELO chess ratings to see which people to study:

Golf has its "Official World Golf Rankings:"

But what about developers? What would be the inputs into the rating algorithm? It's hard to make good comparisons in our industry, especially when we like to hide our failures. Still, there are some reasonable categories to look at. Sonnentag, Niessen, and Volmer give a reasonable first approximation of what the input categories might be along with some interesting observations:

1.) Requirements Analysis and Design Tasks

"A common assumption is that the decomposition process is guided by a knowledge representation, which is built up with experience and stored in long-term memory. When such a knowledge representation is not available, experts develop plans through a bottom up, backward strategy
...
When software designers were asked about strategies they would recommend to an inexperienced programmer working on the same design problem, high performers reported twice as many strategies as moderate performers... experienced programmers did not necessarily have more plans available, but focused on the most salient parts of the plan and flexibly switched between plans. In contrast, novices preferred a strategy with which plans are implemented in a more linear fashion"

2.) Programming and Program Comprehension

"... poorer performers exclusively focused on program terms or on domain terms without building connections between the two 'worlds.' Thus, it seems that connecting various domains and relating them to each other is a crucial for arriving at a comprehensive understanding of the program and the underlying problem. "

3.) Testing and Debugging

"A thinking-aloud study showed that high performers needed less time for debugging, made fewer errors, searched more intensively for problems, and showed more information evaluation activity. Thus, it seems that high performers have a better representation of the program and potential problems, which helps them to actively search for problems and to evaluate information.

4.) Knowledge Representation and Recall

"Professional programmers' metacognitive knowledge was much more detailed and more comprehensive than students' knowledge. For example, a professional programmer provided a comment such as 'The main program is not very well divided; the logic has been scattered over different parts of the program; the updating paragraph has too much in it. The control function of the program should be implemented in one paragraph; now the program has much depth in itself and scanning is difficult", whereas the students' comments were more general and diffuse."

5.) Communication and Cooperation

"... exceptional software designers showed superior communication skills. In fact, much of the exceptional designers' design work was accomplished while interacting with other team members. High performers spent a lot of time educating other team members about the application domain and its mapping into computational structures... They found that individual performance of software developers is facilitated significantly by a 15-minute talk with a senior designer about their task. These results hold for complex tasks, not just for simple ones, indicating that high task complexity makes it more necessary and more valuable for the programmer to communicate with a more senior person."

I don't know how all of this would boil down to a mathematical formula to give a decent numeric rating. But this deficiency doesn't matter too much. Just thinking about how to improve in each of the 5 areas mentioned above could probably go a long way.

Effects on Code

Is all of this discussion putting us into a mental ivory tower? Does it offer help to a developer whose boss is frustrated and fervently asking "Why isn't someone coding yet (WISCY)?" I think there is great potential. It all depends on how efficient you make your chunking algorithm. Moreover, you have to be careful.

Spolsky writes:

A very senior Microsoft developer who moved to Google told me that Google works and thinks at a higher level of abstraction than Microsoft. "Google uses Bayesian filtering the way Microsoft uses the if statement," he said.

A foolish application of this statement would be to believe that you could exclusively live at the higher levels without understanding the lower levels. Chunking works effectively via hierarchical storage. Chunks stand on the shoulders of other chunks. I can guarantee you that star Googlers understand "if" statements and almost everything below that level very well. In a world of leaky abstractions, you typically have no choice, as Anders says, but to "add another level of abstraction." Early on, I thought it was always possible to "raise the level of abstraction." I was wrong. Abstraction is important to effective chunking, but it still requires that you have solid lower level chunks to build on top of do well. This is especially true when problems arise and you have to debug.

Another technique is giving meaningful names to chunks will often yield good results. For example, chess grandmasters Bobby Fischer and Gary Kasparov both won important tournament games using the "King's Indian Attack" opening move. Developers can copy this tactic. You might chunk this:

as the "SyncRoot Variation with CompareExchange Cheap Locking."

Chunking this:

could be as simple as "One Line Singleton" which is much easier to chunk than the roughly equivalent "Thread-safe Lazy Loading Singleton:"

Instead of "Array Sum via Classic For Loop Iteration:"

you could have "Array Sum Using LINQ":

The latter is in line with Anders' recent statement that future versions of C# will be about getting the programmer to declare "more of the what, and less of the how" with respect to how a result gets computed. A side effect of this is that your "chunks" tend to be more efficient for the runtime, and more importantly, your brain.

The code examples so far have been fairly straightforward given their names. Some people might be saying that "you're just calling 'chunks' what are called 'design patterns' or even 'micropatterns'." To this, I would respond "not exactly." For example, even though I had heard of the principle of least privilege, it wasn't until I saw a code snippet from Jeremy Miler that I realized you could be extra precise when defining an immutable class:

Reading Jeremy's code effectively enriched my definition of an "immutable class" chunk to have even more attributes of "least privilege." I didn't see this a year ago.

I have a long way to go, but I think that one aspect of being able to quickly churn out great code is by having a huge reservoir of chunks like I mentioned above. The good news is that you don't have to remember the syntax. You can quickly recreate it from the idea or name. I think the benefit you get is nearly the same as chess students get from specifically named moves.

Effects on Hiring

One interesting playground for chunking is interviewing people. Developer interviews should always include some sort of "chunking" test at the FizzBuzz level or higher.

Simple chunks like conditional statements, loops, pointers, and recursion should be almost as effortless as breathing. As Spolsky wrote:

"If the basic concepts aren’t so easy that you don’t even have to think about them, you’re not going to get the big concepts... You see, if you can’t whiz through the easy stuff at 100 m.p.h., you’re never gonna get the advanced stuff."

Any modern business program is inherently complicated on its own. Since a typical person can only keep "7 plus or minus 2" chunks in their head at once, it's imperative that they are good sized chunks in order to get things done.

Instead of looking for "years of experience," it'd be nice if there was something better like "years actively pursuing developer grandmaster status," but that's a bit of a stretch to ask for.

Four Stages of Chunk Competence

Each one of the 50,000+ chunks on the way mastery will probably go through the four stages of competence. Here's a concrete example:

  1. Unconscious incompetence - 10 years ago, I never even thought about software running in Turkey, let alone any issues that might come up there.
  2. Conscious incompetence - 5 years ago, I finally came to understand the basics about Unicode and internationalization and realized that my code wouldn't work quite right, but I didn't know how to best handle it.
  3. Conscious competence - 2 years ago, with the help of FxCop, I learned how to correctly handle character comparison in .net, but only recently learned how to correctly handle parsing and formatting locale specific information.
  4. Unconscious competence - I'm not there yet, but I wouldn't be surprised if I produce some painful facial expression when I see a zero parameter ".ToLower()" call in code due to the potentially bad side effects I've experienced from its use.

The progression above is what led me to devise "The Turkey Test." It was a long process. That is probably typical for any reasonable sized chunk. I'm assuming that chunk acquisition has the following learning curve:

The good news is that it's relatively easy to make fast initial improvements if you're "deliberately focused" which requires a high conscious effort. The bad news is that mastering the chunk takes a long time. Even worse, you might never even start if you stick to only the things you already know.

Reading good blogs, books, or papers can help. But the best resource by far is an environment where coworkers can safely rocket you through the above learning curve. This is why I really value intense code reviews by my peers at work. Good critiques have let me accelerate my learning process as a result and get slightly better. It's just simply way more efficient that way. It'd sort of be like the young Gary Kasparov playing a game with Bobby Fischer and getting beaten badly. It'd hurt the pride initially, but it would probably help him learn much faster than reading books (as long as he kept good notes of the game).

Is There Hope?

You want to hit the high notes and be a level 5 expert developer. These are actually reasonable goals. But you need to be realistic. If the 50,000+ chunk number is required to be an expert, it takes a lot of time. Realize that even after a decade of being a focused programmer, you're still just a teenager. Like a large number of you, I started "dabbling with computers" when I was a kid. If you count my first Applesoft BASIC program as experience, then I have 17 years of experience. But I have a long way to go.

Don't get me wrong, I agree that raw "years of experience" is a myth.

My problem is that I haven't been as focused as I could have been. I only realized that after diving into "chunking." John Cloud summarizes things well:

"Ericsson's primary finding is that rather than mere experience or even raw talent, it is dedicated, slogging, generally solitary exertion — repeatedly practicing the most difficult physical tasks for an athlete, repeatedly performing new and highly intricate computations for a mathematician — that leads to first-rate performance. And it should never get easier; if it does, you are coasting, not improving. Ericsson calls this exertion "deliberate practice," by which he means the kind of practice we hate, the kind that leads to failure and hair-pulling and fist-pounding. You like the Tuesday New York Times crossword? You have to tackle the Saturday one to be really good

Take figure-skating. For the 2003 book Expert Performance in Sports, researchers Janice Deakin and Stephen Cobley observed 24 figure skaters as they practiced. Deakin and Cobley asked the skaters to complete diaries about their practice habits. The researchers found that elite skaters spent 68% of their sessions practicing jumps — one of the riskiest and most demanding parts of figure-skating routines. Skaters in a second tier, who were just as experienced in terms of years, spent only 48% of their time on jumps, and they rested more often. As Deakin and her colleagues write in the Cambridge Handbook, "All skaters spent considerably more time practicing jumps that already existed in their repertoire and less time on jumps they were attempting to learn." In other words, we like to practice what we know, stretching out in the warm bath of familiarity rather than stretching our skills. Those who overcome that tendency are the real high performers." (Emphasis added)

That's probably one of the most truthful statements about improving that exists. The only thing I would clarify is that your conscious chunks shouldn't be getting easier; the chunks that were foggy in your conscious mind should probably get easier as they drift to your unconscious mind. However, you must refuse to coast. It's about picking chunks "not because they are easy, but because they are hard."

The way to the top is filled with things getting harder. There are no shortcuts. Sorry.

I wish that this wasn't the truth. It would be nice if you could achieve grandmaster status by learning only 101 simple tweet-sized messages in a month, a year, or even 5 years. But it's just not true.

There is some mixed hope however. One benefit from improving is that it becomes clearer where to find better information. That is, more signal in a noisy world. You get more of a "gut feel" of the feeds that should make the top tier in your reader and more importantly, which ones shouldn't.

The downside, however, is that even if you only read the "good stuff," it will not be easy. If it's easy, you're not pushing hard enough.

Right now, I know I'm fairly weak on designing great classes that are easily testable. Through researching, I've found that Rebecca Wirfs-Brock has done some great work in this area with her idea of "responsibility driven design." But sometimes it's hard to understand what she has written. She tends to write in more highbrow places like IEEE Software that are reasonably good sources of software design information. Unfortunately, these types of peer-reviewed journals are almost treated as if they have "cooties" by most of the blogging world, so it's harder to find posts on the ideas they contain.

Last month, with the generous support of my employer, I joined the ACM and subscribed to their fantastic Digital Library. One thing I like about the library and academic papers is that they force me to get out of my comfort zone. Almost none of the articles I've been reading have covered code in C# or even .net at all (even though I really like it and think it's a good platform). But this is a good thing; it's not about being an expert in other languages and platforms, just rather knowing that they exist and what they do well.

A Sustainable Road to Grandmaster Status

Is there a path towards grandmaster status that still allows for a social life and avoids 80+ hour work weeks? Is there a path that lets you invest the considerable time and energy into your kids and treat them as the gifts that they are? Is there a road that lets you stop and smell the roses?

It is possible, but you need realistic expectations. Stay focused, but realize that it might take 25 - 30 years, especially if, like me, you haven't been deliberate in training. Anything worth doing takes practice and persistence. As Fred Brooks observed on a New Orleans restaurant menu:

Good cooking takes time. If you are made to wait, it is to serve you better, and to please you.

Come to grips with the fact that even a great book might only yield 6 - 7 good, lasting chunks. Make use of the great human resources around you, especially if they have similar focus. If you hear that they read a good book, ask them what surprised them the most compared to what they expected the book to cover. Odds are there are only a few chunks. The question evokes more efficient learning. Keep reading yourself and share with others that care.

Look for passion, especially in others that aren't like you at all. Do you know someone that always raves about Python, Ruby, or Scala? Ask them about what's painful to do in those languages and seek hard to understand those points. You'll probably learn something interesting. For the same reason, find problems with your favorite language and be able to articulate them in a professional manner. You'll probably be a better developer as a result.

Don't be alarmed if junior programmers don't understand what you mean when you use your "code defensively" chunk on them. If it took you 15 years worth of war stories in the trenches to get it, it'll probably take the best ones at least 5, even with your guidance. Help them anyway. By doing so, you'll begin to repay the huge debt you took out when learning the profession.

When you see a great design or piece of code created by someone else, ask them how long it took to produce. You might be suprised that it took several rewrites and refactorizations to get there. This gives you hope that things are possible. When people, especially junior people, ask you how long it you to create something, don't try to hide the fact if it took several iterations and false starts to get there. This will also help you solidify the chunks that got you there and perhaps help you store a more efficient path for the future.

Branch out into unfamiliar territory that looks interesting on a regular basis, even if it's just for 15 minutes a day. If your employer encourages this, you'll probably grow faster.

When faced with a decision of choosing what to invest time into learning, don't overlook practicing the ones that "[you] hate, the kind that [lead] to failure and hair-pulling and fist-pounding." You're likely to improve faster that way.

Once you made the decision, always make your studies "deliberate" and "effortful." Philip Ross expands on this:

Ericsson argues that what matters is not experience per se but "effortful study," which entails continually tackling challenges that lie just beyond one's competence. That is why it is possible for enthusiasts to spend tens of thousands of hours playing chess or golf or a musical instrument without ever advancing beyond the amateur level and why a properly trained student can overtake them in a relatively short time. It is interesting to note that time spent playing chess, even in tournaments, appears to contribute less than such study to a player's progress; the main training value of such games is to point up weaknesses for future study.

Even the novice engages in effortful study at first, which is why beginners so often improve rapidly in playing golf, say, or in driving a car. But having reached an acceptable performance--for instance, keeping up with one's golf buddies or passing a driver's exam--most people relax. Their performance then becomes automatic and therefore impervious to further improvement. In contrast, experts-in-training keep the lid of their mind's box open all the time, so that they can inspect, criticize and augment its contents and thereby approach the standard set by leaders in their fields.

Good software takes at least 10 years to develop because it takes programmers at least 10 years to develop good proficiency in programming and the problem domain.

Don't forget that all of this presupposes that you enjoy software development. I'm privileged to have a "hobby" that also pays the bills. But, if you don't honestly enjoy the field, you probably won't care about wanting to go down this road in the first place. As Daniel Goleman alludes to, a key driver that lends itself towards efficient chunking is passion:

"The most contentious claim made by Dr. Ericsson is that practice alone, not natural talent, makes for a record-breaking performance. 'Innate capacities have very little to do with becoming a champion,' said his colleague, Dr. Charness. 'What's key is motivation and temperament, not a skill specific to performance. It's unlikely you can get just any child to apply themselves this rigorously for so long.'" (Emphasis added)

It will take lots of time, but don't forget to have fun and take plenty of breaks. Your body needs it:

"When we train Olympic weight lifters, we find we often have to throttle back the total time they work out," said Dr. Mahoney. "Otherwise you find a tremendous drop in mood, and a jump in irritability, fatigue and apathy."

All of the above was to just convey that "Software Grandmaster" status can only occur after at least 10 years of very deliberate focus on areas that stretch you. Maybe this post might have qualified as a 1/50,000th to 1/100,000 of the way towards getting you there.

Then again... maybe not. It's your move.

Let me know through comments.


kick it on DotNetKicks.com

45 comments:

Madcreator said...

Considering where I am at in my programming lifetime this is exactly what I needed to read, and coincidentally one of the more ironic things I have ever experienced. I dropped out of CS to pursue a psychology degree, spent 4 years learning all of the innate details...the chunks...involved with psychology, spent an entire quarter in a 400 level class which dealt with why Grandmasters can so easily identify a real chessboard configuration from a random configuration, and then when I've almost forgotten about it I receive an example of this concept that I know all too well, describing the skill of computer programming which I am only starting out on the path of learning. I am grateful to have read this, because I look at the things I still don't know about programming and it overwhelms me, but I look at what I didn't know 6 months ago and I can see how far I have come. I guess with anything you choose to learn it takes time to understand the concepts, and once you cross that peak of fully understanding it, it is then yours to build off of.

Jeff Moser said...

Madcreator: very interesting. I'm curious, what 400 class was it? Does it have a web page?

Madcreator said...

I looked it up and found out the same class is still taught by the same professor. I took it at the University of Washington, and the class is called "Psychology of Reasoning". I probably still have the book lying around if you want to read it, I could UPS it to you.

Jeff Moser said...

Madcreator: Thanks for the offer. The book might be a bit too beyond where I am right now to be worth the effort. But this leads to a practical application of the article.

For the benefit of me and anyone who might read the comments, what surprised you the most, or stuck out about the course that went against your initial expectations?

Looking forward to the response!

Petar said...

Great post, couldn't agree more. We seem to forget how hard it is to become good at something.

Jason Kealey said...

Very, very interesting read. It's great to see some extensive research in a concise blog post.

Personally, it will encourage me to put my IEEE Software subscription to good use and talk more about the chunks which I learn in there, on my blog.

Thanks!

Madcreator said...

I'd have to say you basically nailed the general theory behind the chess analogy you were describing. I re-read the section on expertise and couldn't find anything important that you had missed.

One aspect of the class that stands out in my mind were the specific types of algorithms humans utilize in order to solve problems. For instance humans will initially analyze their initial state and their goal state, and the attributes of the two states. They will then attempt to work forwards from their initial state towards the goal state, and if this fails they will attempt to work backwards from the goal state to the initial state.

What I found interesting is the way in which humans are able to discover novel solutions when simple solutions can't be found. One theory behind this approach is that humans will create an abstract model of the problem set, including meaningful variables for each problem detail, and then sort of retreat into a searching mode while their subconscious filters through known models to see if the variables between the known model and the problem model fit. If a close match comes to mind, humans will align the variables of each model to determine fit, and either continue refining the model until they are confident of a match, or they reject the model as not being a correct match. Once a model is selected as being a close match to the problem set, it can be adapted to the specifics of the problem in order to facilitate the final solution.

Continuing with what you were describing earlier it is easy to see how someone with more expertise would naturally have more models as well as more advanced models with which to map to the problem set.

Thomas Danecker said...

Wow! Impressive post! The best post I read since... a long time ago.
Currently I'm also at the level where I read papers at ACM's Digital Library, but I've to do some basic courses first to fully understand the topics I'm interesed in (and their implications).

Abhijith said...

Great article!

Jon von Gillern said...

Excellent Article.

Sudhanshu said...

You have written a very good post. A good aggregation of ideas. Some of the points really echoed with me, from personal experience.

...experienced programmers did not necessarily have more plans available, but focused on the most salient parts of the plan and flexibly switched between plans...

...connecting various domains and relating them to each other is a crucial for arriving at a comprehensive understanding...

...Abstraction is important to effective chunking, but it still requires that you have solid lower level chunks to build on top of do well...

...The way to the top is filled with things getting harder...

...You might be suprised that it took several rewrites and refactorizations to get there...

...don't overlook practicing the ones that "[you] hate, the kind that [lead] to failure and hair-pulling and fist-pounding."...

...It will take lots of time, but don't forget to have fun and take plenty of breaks...

It would be a good idea to expand upon what might be a "good representation of a program" as you mentioned in the Testing & Debugging section.

Anonymous said...

Thank you for this really excellent article.

Steve Pavlina, in this article:

http://www.stevepavlina.com/blog/2005/07/how-to-get-from-a-7-to-a-10/

puts it well. "If you think you’re at a 7, you’re really at a 3 maximum." He goes on to say that the 10s are so far out there we can't even imagine them.

In a few areas, like chess, where there are ratings and public performances, we can see this. In programming we can't.

Guitarm said...

I'm 45 years old. I've been programming on and off for approx 25 of these years. The thing I've learned about this profession is that the most important thing to learn is what I like to call context switching. Today I was kind of "Thrown out" as a teacher for a Java J2EE class I was supposed to hold. I've come to understand that it is immensely difficult to explain things that you've actually worked with and constructed in real life to an audience that consists of both people with experience (that obviously affects them in their thinking) and people with no experience that needs a blueprint to follow.

I see myself as a nice human beeing with a good intent of teaching both facts (like the logical flow of a computer language) and the experience from how these systems affects people in their daily work.

It's very easy to read code. I'ts amazingly difficult to "read" and understand people...

Tom said...

My favorite quote from this is: "And it should never get easier; if it does, you are coasting, not improving." I've discovered this in programming and also in other disciplines such as Taijiquan -- you may get better at it but it never really "gets easy" in the way beginners might expect.

Steven Romej said...

Great post. I read it this morning and fired it up again when I got home.

Jeff Moser said...

Petar: Indeed, it's a long road.

Jason Kealey: I really appreciate the "concise" comment. When I hit 4100 words, I was worried no one would read the post. Congrats on the IEEE subscription. I encourage you post on anything great you read there.

Madcreator: Fantastic, detailed comments. Thank you. I'm sure others will benefit from them as well. Just to clarify, the idea of "more expertise" implies "more models" came from the Sonnentag, Niessen, and Volmer chapter that I quoted from.

Thomas Danecker: Thanks! I'm humbled by your comments. Keep trying on the ACM Digital Library papers. It's a largely untapped goldmine.

Abhijith, Jon Von Gillern: Thanks for the encouragement.

Sudhanshu: Thanks for being specific on what echoed with you. I encourage you to read the Sonnentag, Niessen, and Volmer chapter I linked to on Google Books for details. It's a good read.

"anonymous:" I went ahead and read the Steve Pavlina article on your suggestion. It was an interesting article. It seemed that Steve didn't offer much suggestions on how to specifically improve though. What were your practical take aways from it?

Guitarm: Indeed, software is all about people. I tried to highlight the people aspect of chess grandmasters, but there are many details. With your experience, you're in a great position to share some of the extended chunks you've learned throughout the past 25 years.

Tom: I really liked that quote too. You might be interested in reading the whole Time Magazine article it came from that I linked to.

Steven Romej: Wow! Thanks for taking the time to stop by twice. I really appreciate it.

Also, thanks to the many comments on reddit on the article. Hopefully the graph is presentable now :)

nathanweisz said...

Great read. A must for aspiring programmers who should be warned that it will be very difficult to become an expert programmer.

I too (used to) have a passion for programming. But after graduating and seeing how the industry works, seeing how one never seems to gain any mastery, since programming languages keep springing up and being updated, I decided to shift careers, and went into Project Management instead. I don't know if my gamble will pay off. I don't feel the "natural high" in project management when a project is completed compared to when I see the code "work" after I've programmed it. I guess you'ld call this cowardly, exchanging one's passion for something convenient, but sometimes, one can't help but be practical. The chances of becoming an excellently paid programmer for me is slim given my age and years of development experience. Where I'm at, a mediocre project manager will probably be paid more than an above average developer. But the really excellent developers are top-billed. =)

Anonymous said...

developers playing chess?? bollocks.

similar to deducting how fast someone runs after watching him swim.

My girlfriend dresses very well and is very pretty, so i suppose she's a great coocker. Why not? those are attirbutes often related to women.

This article is pure crap, most of the complementary comments are from noobs to development.

IF you want to better your programming by playing chess go ahead, but you would better off trying to learn from more experienced programmers than from chess masters.

Jeff Moser said...

Nathanweisz: Interesting perspective. You mentioned that programming language shifts as being a hindrance to mastery. How much of our industry do you think trancends programming language/platform?

"anonymous": Perhaps I don't understand your comment. Maybe you can clarify? I was trying to convey in the article that there are some fundamental characteristics towards mastery in *any* field. If I lead you to believe that developers need to play chess, then that was unintentional. You say "would [be] better off trying to learn from more experienced programmers." I'm curious; what are you looking to learn? How would you categorize it? How would you remember it? How would you apply what you've learned to your work?

Matt said...

I find it hard to believe that the Anonymous reader actually read the post. They assume what you were saying was to play chess to get better at programming... which is of course not even close to what you actually communicated.

Grandmaster developers also can't just blindly make assumptions about problems (or reading material) and expect to learn.

For myself, I'm struggling with the decision to accept a position doing .NET development for a significant raise when I do detest the framework and platform compared to what I use now (Ruby). I know that it won't hurt my skill and certainly will stretch my patience and focus (Ruby really is very easy to develop in), but I don't know if it's worth the investment.

To be a Grandmaster, can one be a snob? I want to keep working with open source technologies in my happy *nix-land. Is it really improvement to master things that just require hyper-verbosity for simple tasks? (I also read your article about the Turkey Test.)

I do want to expand my experience from primarily Ruby to include more C/C++, Erlang, Io, and maybe even Java, but does that really help on the way to Grandmaster? I've developed in C# and .NET before and I know it can be a hassle to do very simple things, especially for someone used to the cleanliness and flexibility of Ruby.

Sorry, you must pardon my internal struggle, though your comments would be appreciated.

Anonymous said...

Excellent. Now I know where I stand, what to do and how to go.

A4 said...

Great thought-provoking article, thank you!

Jeff Moser said...

Matt: What would you consider "painful" in Ruby? No language or platform is perfect. Perhaps look to see where C# beats Ruby as a way to stretch your understanding.

As for the Turkey Test, how does Ruby handle the same problems? The main thrust of that post was that regardless of programming language, you have to be aware of the issue.

I had to look up Io. I hadn't heard of it before; interesting. I would highly encourage anyone to be at least aware of the interesting aspects of lots of languages: be it macros in Lisp to query expressions/comprehensions/LINQ in C# 3, it's helpful for understanding aspects of languages. It's sort of how someone begins to learn a lot about their own native tounge only when they study a "foreign" language. Koine Greek is highly inflected language (verb changes a lot depending on the subject) whereas English isn't so much. Lots of programming examples are like that. So, indeed, go for it. It can't hurt too much, especially if you're purposeful.

You might want to look at C# 3.0's use of lambdas and especially "Extension Methods" to see how can do Rails-esque things like "10.Days().Ago()" I admit though, not quite as elegant as Rails does it. However, there is a decent Ruby compiler coming (IronRuby) that will let you use Ruby code from C# and vice versa. At the very least it's interesting. For more Rails-ish stuff, look at the Boo language too.

Longwinded way to say do what you love, but make sure you don't discount things simply out of fear. That said, I could learn a lot from studying more of languages like Ruby and even more obscure ones like J. If you know you'll never like a job and are only doing it for the money, don't bother. You probably won't be happy.

"Anonymous", A4: Glad you liked it. Thanks for stopping by.

Stephen said...

Ten years to become competent? What does that mean for C#? Micro$oft dropped VB, and was the sole provider. What good is it to become competent, only to have the rug pulled from beneath you?

Early on, i was a polyglot. I considered myself fluent in 37 computer languages. But these days, inhaling another language isn't nearly so easy. Have i learned all the easy ones, and now have to painfully slog through the harder ones? Maybe. It may be that i was fluent at Hello World, but demand more these days.

On the other hand, back then, i wrote a multi-threading operating system... Maybe inhaling new languages stopped when my photographic memory became translucent.

For me, Hello World in BASIC to an operating system was 7 years, but could have been 5.

Great developers are NOT highly paid unless they're also great at something else.

tautologico said...

Very interesting article. I want to comment on something a little offtopic. You talk about C# and how it will be more about the whats and less about the how. Well, I think C# may get far absorbing features from functional programming languages, but it doesn't beat using a proper functional language. So, as you like .net, I'd like to suggest you take a look at F#, if you didn't already. It's a great language for the .NET platform.

Joe said...

One of the best articles I've ever read.

Well done!

- www.JoeLevi.com

aylwin wong said...

This is one of the great article I have ever read.
And I agree the approach that you gave to become grandmaster.
With the analogy of chess and golf player, you make the point very clearly, that we should not only focusing on our field but also learn from gurus from other field as well.
That is, the idea behind great invention. Newton learn gravitation from apple's tree, Wrights learn to construct plane from birds, and Michaelangelo can paint human very well because he learn anatomy!!
So certainly you got the point there....

brian said...

A Taoist may say

"Learn, then Unlearn.
Do then Non-Do"

Jeff Moser said...

stephen: What do you mean by "Micro$oft dropped VB?" Are you talking about VB6? The reason is that Paul Vick and company are very much alive with VB.

What did you write the multithreading OS in? What was your scheduling algorithm?


tautologico: I've been quite interested in F# since college (~4 years). I've made a few references to it on this blog. My first encounter with it was when I was taking a compilers course that was using Standard ML. I thought F# would give me a debugger that SML didn't have. I didn't end up using F# then, but I've kept up a bit with what Don Syme and now Luke are doing with the language. I really like how it lets me use the neat attributes of ML and Scheme in .net.

Joe: Thanks!

Alwin Wong: Thanks for the comments and references. I think however that the Newton apple story isn't true. I think Voltaire made it up to make for a good story.

Brian: How would you apply that?

Anonymous said...

Fantastic article! Most relevant to many developers including myself. I forwarded this blog post to all my team members (first time in my career!); I plan to re-read, understand and implement the ideas and thoughts into my team's daily routine going forward.
Thanks for such great work!

zhen said...

Great article! I won't waste any more words on how great it is, it's been said plenty...

One thing I'd like to know more about is, in those over 10 tedious years of conscious practice, what is it to motivate the future masters to move on in their unwavering determination?

You just briefly mentioned the notion of passion. Can you expand on that topic a little? Like the nature of passion; Can one develop his/her passion and so on. Maybe that deserves another post or two.

Thanks and looking forward to your new post!

Alfas said...

I agreed with most you wrote till section about coding. I stooped reading carefully here. Sorry, but my first impression was that you are giving examples of bad code, where you actually presenting them as good code :D. However, latter on the section about hope, was the part I agree.
To clear my confusion I want to ask. Are you still developing, creating complex parts of software, or these days are gone long time ago?
Anyway it is nice and useful blog to read.

Mike Petry said...

I couldn't sleep so I wandered down stairs to read a few blogs until I got sleepy. The insomnia really set-in after I read this blog as provoked thoughts raced through my mind. I thought about the things I had done right over the past ten years, the never ending quest to learn and discover. After ten years, I wouldn't classify myself as a grand master but I have earned myself the right to remain employed in a field that I truly enjoy while earning a decent living for my family.

Lisa said...

Brilliant post. I can see myself passing on this post to new developers 10 years from now when I'm not so new.

Jeff Moser said...

Anonymous: Thanks for the kind words. Let me know if improvements that you discover.

zhen: Thanks for the comment. "Tedious" is subjective, right? It can be frustrating at times, but you have to have the "motivation and temperament" as described in the second to last quote. There has to be an attraction to the field. This leads to have the child-like curiosity of asking "why?" and wanting to explore for fun. I believe that this is the start for what ultimately leads to passion for the field that drives you to want to improve.

One subtle way of developing the passion is look to the history of our field and see what motivated the people that made great improvements in the past 50+ years. I speak more on this in the next post. Does that answer your question at all?

Mike: Glad I helped you fall asleep :). Ten years is no magical number; it's all, like you said, about constant improvement. The question that I have to ask myself is what are my #2-#4 items on the "Four Stages of Competence," and perhaps more importantly, what do I need to do so that more of the #1's become at least #2. Any thoughts on that?

Lisa: The best thing to do is be a role model. With conscious focus, you can be a tremendous asset to helping the next wave like I mentioned.

Anonymous said...

i loved your article. and to be fair. i don't understand anything in coding or anything that has to do with programming.
but the idea you transmit about becoming a grandmaster works for everything. and it just gave me the push i needed. in fact i'm a comics student. so you could say that drawing isn't exactly coding. but still if you want to learn to draw. stick with what you can't draw and try to see new techniques, other styles and listen to grand master and you will improve.
i'm just lucky to have stumbled upon this page.

anyway a fantastic read. not just for programmers.

Jeff Moser said...

anonymous: Interesting connection to comics. You're right, a lot of the ideas span all domains (e.g. chess to programming to figure skating). Keep "focusing on the jumps" of comics :-) Thanks for stopping by.

Anonymous said...

"the students' comments were moral general and diffuse"

This makes more sense as originally written: "more general and diffuse". :-)

Jeff Moser said...

anonymous: Great catch! Typo fixed :)

Thanks!

lodewijk said...

This was a very interesting read! I must say I like your way of writing, the amount of references, the storytelling involved.
Being a beginning programmer (about a year of truly working on it) it really "got to me".

Personally I think the Edison part and the interview parts were very nice. The Edison part because it seemed that I bore myself when I'm coasting, and always seek challenge (the FIRST thing I wanted to program was a program to chat with, it was way to hard in hindsight but I did it) and the interview part because it showed that me I would pass those tests!

A great and motivational piece, congratulations.

Cihan Tek said...

I really like your style of writing and read most of your articles. This one was the most interesting one so far. It's obvious that you did good research before writing it. Everything you've written applies not only to CS, but also on any other field.

But reading it made me pessimistic about becoming a expert in software development. Especially the sentence "But the best resource by far is an environment where coworkers can safely rocket you through the above learning curve". This is %100 correct, but not everyone has the opportunity to work with people who can do that.

I've been working on embedded systems software for a long time and often faced problems for which solutions cannot be found on any book or on the web. So i really needed someone that could, "rocket me through the above learning curve". But because of the country in i live, i wasn't able to find that "someone".

So what do you think, is it possible to be a "master" on your own? Does working on increasingly difficult tasks and effortful studying will work alone for obtaining the required number of chunks? :)

Jeff Moser said...

lodewijk: I like the Ericsson part of deliberate practice too -- but it's hard for me to accept since it can be painful (although needed :-)). Thanks for stopping by!

Cihan Tek: Thanks for reading it. You're right, a lot of this applies well beyond CS since many of the people quoted aren't programmers. I was just looking for parallels in other disciplines.

As for doing it alone -- realize it's harder. At the very least, look at the work of people you respect if you can. Perhaps look at the code of an open source project you like (e.g. SQLite for embedded work) and see how much your code differs from that.

Perhaps start a blog and ask for feedback. I've learned a lot from all the generous feedback here.

Does that help?

Anonymous said...

A problem I've always encountered in life is my inability to focus. I want to learn everything there is to know about everything. Unfortunately, I will likely never know everything about one thing. I will use this article as inspiration. I now know it's ok not to know everything, and that we can only hope to improve.

goingkiloer said...

There's a good amount of overlap between this post and a talk by Kathy Sierra - here.
There's some amount of interesting new material too..

Jeff Moser said...

Anonymous: I've struggled with that as well. There are a lot of "shiny things" out there. My rule of thumb is to pick something I think is really interesting and then go deep on it enough to write a blog post on it. This forces me to stay focused.

goingkiloer: Good link. Kathy has great advice.