Skip to content
November 8, 2011 / Ben Chun

Evaluating Codecademy

Congratulations to Codecademy on raising a $2.5M seed round. It’s great to see some attention and cash being paid to online options for learning computer science or at least programming. Codecademy is the first attempt I’ve seen to bring badges and other game mechanics into this subject area. Khan Academy seems poised to do it since they have both CS video lectures and a gamified learning system, but they’re pretty focused on math right now and their exercises all seem to be limited to integer or multiple-choice answers.

While I don’t doubt Codecademy they can turn the technology they have into some kind of business (perhaps after a pivot or two), I’m not sure if the actual lessons on offer are any better than, say, Learn Python the Hard Way or Eloquent JavaScript. This is not to put down anyone’s curriculum, but just to say that this subject is hard to present. People have been trying for a long time and no one’s really figured out how to dial in the difficulty, pace, and number/type of exercises that will work for a wide range of learners.

I tried out Codecademy with my 10th grade class last week. The students have some JS experience but are mostly new to programming overall. We started with the first course written by Ryan Bubinski, one of Codecademy’s founders. All was smooth sailing until Lesson 4, where some of my students stumbled.

Although they had just apparently learned (or at least typed in stuff) about variables in Lesson 3, some students couldn’t complete the instruction “Declare and set a variable named number to the value 42.” What was wrong?

There was a hint available, but they couldn’t understand it: “To declare and set a variable in a single line, you use var variableName = variableValue, where variableName is the name of the variable, and variableValue is the value you want to assign to the variable. For this exercise, the variable name is number and the value is 42.”

Many of the students who got stuck at this point were trying to literally enter, var variableName = variableValue. In fact, they had been directly copying all the other code up to that point with some success.

That hint seems reasonable when you just read it, but if a student needs a hint at this point in the lesson, maybe the hint should be, “Go back to the previous lesson and do it again while paying attention to the meaning of the symbols you’re typing.” And as long as we’re presenting this as a web app, why not offer a click-through for that or even require completion of more exercises on the earlier concept?

The next stumbling block I saw broadly in the class was substrings. Everyone can complete the exercise that says, “To get the substring, you can type “hello”.substring(0,2). Try that now.” Again, they all just type in what is says to type, sometimes without comprehending the text at all.

The next exercise says, “This time we’re going to get the first three letters of your name. See if you can save those letters to a variable named three using the substring() method from above.” Whoa! Huge leap there. Not only are we hoping for them to have understood the concept of substring, we’re also hoping they remember how to assign a value to a variable. And as if that wasn’t enough, it seems we’re also hoping to gloss over the part about how a statement gets evaluated to a value before assignment takes place.

Some kids literally have no idea what the instructions at this point mean. “See if you can save those letters to a variable named three using the substring() method from above.” They haven’t ever seen an example of a function’s return value being assigned to a variable. If they don’t get it, should we keep making them guess?

Either way, they guess away and eventually get it or get someone to help them do it. Then, right at the very moment of success, the interface jumps them on to the next exercise. It’s hard to know what you did that was correct. Particularly if a student has been working and working away in the editor at something, why should it disappear as soon as it’s right? There’s no time to reflect on the knowledge they just gained, nor an opportunity to try variations or improvements on the solution. The interface encourages the very sort of “click and forget” behavior that works against learning.

While this pedagogical problem with the site is pervasive, Ryan’s initial course does a pretty good job of using the interpreter as a learning environment. I can’t say the same for the next course offered: JavaScript Quick Start Guide. Now to be fair, this is targeted at folks who already know how to program. But it might as well just be an HTML file, because it doesn’t take advantage of the interpreter at all.

Many screens are just reading with no typing code at all. We hit enter, and the system assumes we understood, assumes we could contextualize the information.

Then we get to Lesson 3, Exercise 3: “Bang Operator”. The instructions say “Work through the example and then hit enter to continue.” And this is what’s in the editor window:

function sad() {
  // If a value is not true, return the string ':('.
  
}

Anyone have any idea what to do here? Yeah, me neither. This is my best guess:

function sad(s) {
  // If a value is not true, return the string ':('.
  if(!s) return ':(';
}

But that wasn’t the desired answer. So my top students gave up there for the day, me along with them.

Update 11-9-2011: They fixed the bug in Lesson 3, Exercise 3 to match what I assumed was the original intention. The structural problems with reinforcement remain. In the new DOM and jQuery course, Lesson 2, Exercise 4 it asks you to select the elements from a particular class. But you can select the elements from any class — as soon as you push enter it drops you right into the next exercise, which isn’t an exercise at all, just a demo of adding elements to the DOM. All you can do there is click “Run” and you’re now two lessons away from where you got confused in the first place.

About these ads

15 Comments

Leave a Comment
  1. Dave Naffziger / Nov 8 2011 11:06 pm

    Lame. Completing exercises without comprehension can be too easy and results in zero actual knowledge gain.

    CS is hard. Teaching it is harder. But cookbooks teach nothing other than direction following.

    I expect great things from developers of online curricula, but codeacademy is apparently not in that segment yet.

  2. Zach, Codecademy Co-Founder / Nov 9 2011 8:03 am

    Hi Ben,

    I’m Zach, one of the cofounders of Codecademy. Would love to follow up with you here – we’re trying to make things as easy as possible for our users. You can reach me at contact (at) codecademy (dot) com.

    Thanks!

    Zach

    • Ben Chun / Nov 9 2011 12:06 pm

      Hey Zach – I sent you an email. I’m not sure if “easy as possible” is the right goal. How about “most effective for learning”?

  3. Tim / Nov 9 2011 10:57 am

    I just tried Codecademy and I have just about precisely the same reaction. At minimum, there needs to be a way to scroll back (or access) *all* of your previous work, and to get more than the one hint on a step. (Because otherwise you just get stuck.) Plus, the UI itself isn’t very good — as least in Chrome. Often the editor will not accept a backspace erase, and you have to start the lesson over again. Finally, when you select “Run” the site will just send you to the next step, without any indication (in print) that the code was correct.

  4. Brit Cruise / Nov 9 2011 1:06 pm

    Great article Ben,

    I just interviewed with Zach as joining CodeCademy as content organizer and brought up some of the same concerns. I think codecademy can change the way CS is taught worldwide, however it’s still in it’s infancy. I think your feedback is priceless as it’s empirical – mainly what I think is that errors need to be FUN on codecademy.

    This means that we need more dynamic error reporting to help people know “where” they should be looking and “what” the TYPE of problem is. ie.

    Instead of:

    ERROR: Unexpected token )

    should be

    ERROR: Spelling mistake here —->

    2)
    Also, They also need to re-word and simplify instructions and in some cases add extra steps.

    For example they say:

    “You can use the following signs as well in if statements as well as elsewhere:
    > (greater than), < (less than), !== and != (not equal to)."

    Here they are throwing too much at us. First they should move = vs != to it's own step.
    then they should trim it to say ONE thing we care about:

    "The IF statement can do more than check whether two things are equal. It can also compare numbers."

    3)
    I also think the page jump after pressing enter is a problem. On the Ipad it actually just jumps out of view, and even when it is working it's frustrating. I think they should have smooth scrolling at all times so we never "jump" out of context.

    Thanks,

    Brit

  5. gasstationwithoutpumps / Nov 9 2011 8:59 pm

    Sounds terrible to me. A half-decent book and an interpreter would be better.

    • Bob Doyle / Jan 13 2012 6:21 pm

      Got a book you can recommend? I knew coding back in the 80’s, knew Fortran, Pascal, even dabbled on the 8502. Thought Java through Codecademy would be “easy” and a “refresher”, but it is a nightmare. There is no “hints”, no step by step walk through if you don’t get “their” answer. And no idea to know what “their answer” is with the vaguely worded question.

      • Ben Chun / Jan 13 2012 9:13 pm

        If you generally know how to program and want to learn Java (not JavaScript) then I recommend the book Head First Java by Sierra and Bates. If you want to learn JavaScript, try http://eloquentjavascript.net

  6. Andrés Monroy-Hernández / Dec 20 2011 4:39 pm

    I like what Codeacademy is trying to do, but as with any approach, one size does not fit all and one single method is not enough. I remember a lot of programming books having the same approach (copy & paste), and they were rewarding at first but at some point one reaches a wall. Having a mentor around is still needed.

    Always, a pleasure to read your observations from the ground :-)

  7. Philip Walker / Jan 13 2012 3:48 am

    Certainly I would echo criticisms of some of the content which is not well organised.

    I call as my witness the following two exercises from the JavaScript Quick Start Guide:

    3.3, as you mention, which assumes knowledge of the return keyword without prior introduction, and also assume a more rigorous understanding of functions which are introduced in chapter 5;

    and 3.6, which uses the || logical disjunction (OR), when the introduction is in 4.3.

    Since both of these are JavaScript distinctives, it is not sensible to assume that a complete newcomer to JavaScript will know about them, even if the newcomer is an experienced programmer.

    One of the things I learnt quite early on in programming, from TurboPascal, was the importance of introducing functions and variables in the correct order. That kind of tidiness of mind is not only important in programming, but in teaching as well.

    This, though, is a criticism of content production which needs a lot of quality checking, rather than the technique, which clearly contributes an effective mechanism for teaching, or at least embedding, understanding of a programming language. Whether it develops the ‘tidy mind’ and similarly transferrable skills I suppose remains to be seen.

  8. Spencer / Jan 26 2012 1:00 pm

    “The interface encourages the very sort of “click and forget” behavior that works against learning.”

    This is EXACTLY how I felt. I am brand new to JavaScript and programming for that matter. I tried some Python guides online but those were tough to get through. I fell like I do one exercise and then move right on to the next. I knew that I would forget about it if I moved on but there wasn’t really anything I could do about that knowledge, I only know how to do that specific example!

    I do like the badge and point system, though. It kind of gets you hooked until the rest drags you down.

  9. jtg / Feb 29 2012 4:57 pm

    Maybe they should include tutorials on how an actual computer works. I’m not a programmer (but signed up on Code Academy) but I was told by an experienced software engineer that before you learn to code you should understand how the computer works. This seemed to make sense to me. There are videos on a number of topics on the following site. I’ve found some of them very helpful.

    http://nptel.iitm.ac.in/courses.php?disciplineId=106

  10. reconix / Mar 20 2012 6:32 am

    From start to finish I thought codecademy was awesome ! Truly awesome.
    Yes a huge part of the interface is point and click, quickly to the next one, I actually proffered that as in some of the earlier lessons I already understood what was going on wile as at the same time I was like oh, that’s actually a better way of doing things. Yet you still take on all of the information you need. Personally I think the next part of the learning process is up to you.

    Here’s what works for me when following big tutorials or guides.

    Download one of the following and install the one you find comfortable with, don’t install the one you think is the best but the one you’re best suited with.

    http://www.wampserver.com

    http://www.ampps.com/

    http://www.apachefriends.org/en/xampp.html

    There’s versions for Linux, Windows and mac users

    Create a new folder(directory) in the corresponding web root of one of the packages you downloaded.. example: www folder

    Create DOC folder, SOURCE folder and TEST folder

    DOC FOLDER:
    In documents wright up tutorials for yourself that you known you’ll understand, obviously if what you’ve read you understand completely then just copy paste. If you come across a certain tutorial where you follow along in example but would like to reflect back on what you’ve done post it in the documents section.

    SOURCE FOLDER:
    in here you place any downloadable content from tutorials corresponding by it’s own folder and text file explaining what the tutorial is and does plus where it’s from.

    TEST FOLDER:
    This is where you test out all the hard work you’ve learned. Obviously you’ll come across many many problems but, once you solve them, write them down in the documents section. If it’s taking longer than 30min to an hour to figure some thing out you should probably seek help.

    That’s a basic overview of how I always like to do it, I know it’s very unnecessary in most cases but it works for me and I learn more by also trying to integrated what ever I’ve learned into an example website.

    Interesting to know if anyone else has peculiar learning habits.

Trackbacks

  1. Evaluating Codecademy: A Response « And Yet It Moves
  2. Questioning Code Year « And Yet It Moves

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: