Assessing What Matters in School: Intelligence and Beyond

The latest Educational Leadership has a good piece on how to create better types of assessments / tests to fully measure the capabilities of a student. The author, Robert Sternberg, is president of the American Psychological Association, one of the most prestigious posts in the field, and he starts the article relaying an anecdote about how both he and the preceding president received a C in their intro psych college course. This irony serves as the jumping off point for a rebuke of the traditional multiple-choice and fill-in-the-blank assessments in schools and on the SAT which "assess primarily remembered knowledge and analytical skills applied to this knowledge."

When I look at the skills and concepts I have needed to succeed in my own field, I find a number that are crucial: creativity, common sense, wisdom, ethics, dedication, honesty, teamwork, hard work, knowing how to win and how to lose, a sense of fair play, and lifelong learning. But memorizing books is certainly not one of them.

But how do you test someone’s creativity or wisdom or practical abilities? Sternberg has designed a test which assess these things and others. Here are two examples of his "Success Intelligence Model" questions:

In science, we might ask (1) What is the evidence suggesting that global warming is taking place (analytical)? (2) What do you think the world will be like in 200 years if global warming continues at its present rate (creative)? (3) What can you, personally, do to help slow down global warming (practical)? and (4) What responsibility do we have, if any, to future generations to act on global warming now before it gets much worse (wisdom)?

In mathematics, we might ask (1) What is the interest after six months on a loan of $4,000 at 4 percent annually (analytical)? (2) Create a mathematical problem involving interest on a loan (creative); (3) How would you invest $4,000 to maximize your rate of return without risking more than 10 percent of the principal (practical)? and (4) Why do states set maximum rates of interest that lenders can charge, and should they do so (wisdom)?

This seems good for school use, but how to scale it to the level of a nationwide test like SAT? Don’t know. Sternberg has good reasons for working on this stuff. First, he thinks a more holistic test is fairer. Multiple choice tests are biased in favor of those cognitively suited to the task — if you’re good at one multiple choice test, you’ll be good at another, regardless of content. So it primarily tests your multiple-choice-taking skill, instead of the content of the multiple-choice test. Second, he says it is a far better predicator of freshman year college grades. Third, today’s times call for a range of intelligences, not just raw analytical or memorization ability, so it’s probably a better life success predicator, too.

By the way, school assessments failing to be a reliable predictor of career / life success isn’t just limited to psychology. Law bar exams are notoriously disconnected from reality. One famous example is when the former dean of Stanford Law School, Kathleen Sullivan, who’s argued cases in front of the Supreme Court, failed the California Bar Exam a year or two ago.

(hat tip: Eide Neurolearning)

The Illusion of Knowledge

The illusion of knowledge is worse than knowledge itself.

One risk when receiving a broad, liberal arts education, replete with generalist courses like “Questions of Civilization,” is that it’s tempting to believe that the two days your Civ class spends on the Koran is all you need to know. In other words, you spend a couple days on the Koran, and then move on, and in your mind you check the “Koran” box, and don’t feel it necessary to really dig into it later on.

Sure, perhaps that limited exposure actually motivates you to dig into the text in a way blind ignorance would not. But I’m not so sure.

I think chipping away in superficial, very high level way at a massively complex topic or book can almost be worse than spending no time on it at all. I’d rather have college students walk around knowing they were completely ignorant about the Koran than think they even understand something.

I prefer knowingly ignorant to superficially informed. Of course it’s possible to be both superficially informed and aware of that fact, which is the best of these worlds, but I wouldn’t place this bet on a typical college campus.

Helping Students Find Their Way

Seth Roberts posts an email from someone who thinks college-age students need outside help to figure out the best career path:

I believe a large fraction of people around ages 16-22 are ignorant of what kinds of work environments and activities will make them happy and productive later in life. Current classroom-based training structures do not provide exposure to work environments. The cultural and social pressures from media, family and friends can be overwhelming and can often lead to people being very confused, and hence, making poor choices. I’ve seen that people tend to get very limited and highly biased information that leads to making training choices and work choices early in their life that are often not well matched for the person’s individual genius. By mid 20’s and 30’s, getting out of these poor choices is extremely difficult, as financial requirements as one ages grow and available time to retrain diminishes. Expectations of experience grow as one gets older, and the neural ability to quickly learn and master new skills diminishes, especially much later, after 40 or 50 years. All of these factors point toward a critical need to have experienced, outside input into making early choices about career paths, and what types of experiences individuals would benefit from most. Such advice is available, and can be found – but it is not commonly accepted that expert outside opinion is the best source for career and training choices for young people. Kids get it mostly from their parents and friends – neither of which are consistently accurate, trained in normal psychology, or unbiased in their assessments. …

All good points. I would add the following:

1. Parents and friends are indeed most consulted on this front, as the emailer says, but maybe for good reason: they know you better than any outsider ever will. That said, he is right that students underweight bias. The strongest bias in parental advice is probably their focus on risk mitigation. I believe you should take risks when you’re young because the cost of failure is low. I believe you should try more experimental jobs, or jobs for which there’s a lot of uncertainty about how it will work out.

2. Peer advice is also a double edged sword. Their familiarity with who you are can be helpful. However, a lot of college-age students simply project what they would do onto your situation. (The most explicit example of this is advice prefaced with, "If it were me, I would…") I think this is a developmental thing — it’s really hard to analyze options from somebody else’s perspective.

3. Lack of imagination also contributes to young people choosing work environments and activities that don’t make them happy. It’s hard to think of different types of jobs. Especially in non-vocational schools, we have limited exposure to professions beyond what our parents do and what our friends’ parents do. So, I think simply exposing students to more types of jobs would go a long way.

4. Your career path is not chosen at age 22 and then set in stone. We should de-emphasize the importance of your first job and celebrate the fact that switching careers is possible.

What We Wish We’d Known in College

Keith Gessen of n+1 magazine spoke at Scripps / Claremont a few weeks ago. n+1 a relatively new literary journal and political magazine started by a bunch of 30-something intellectuals. Here’s A.O. Scott’s fantastic profile from awhile back of the editors and their venture.

They give freshmen in college a free copy of their pamphlet "What We Should Have Known," a transcript of eleven editors discussing what they wish they’d known in college, specifically about books. What books should they have read, what should they have read earlier, what should they have not read, etc. It’s interesting if a bit esoteric — I hadn’t heard of most of the philosophers and authors they talk about — but there are some good nuggets which I excerpt below.

On advice to an 18 year-old:

Caleb Crain: If I were speaking to an 18 year-old, I’d say, "Don’t worry. Don’t be precocious." But the flip side of that is, this is the only life you’ll get, and it won’t come again. So, I don’t think you should be precocious, and I don’t think you should beat yourself up for not having published a book at the age of 28, but I think that a young person should keep a journal, and read seriously, and, you know, think about everything that happens.

On becoming an intellectual, in the postscript by Keith Gessen, love the last sentence:

I am certain that the most valuable parts of these talks lie beyond those debates. They are the moments when the panelists reveal their deep uncertainty — Meghan Falvey walking in a mist of abstraction on the way to her dorm from Phenomenology, Existentialism, and something else with a P — and how each of them has struggled to read and think their way out of that uncertainty. It turns out that in order to become an intellectual, you must first become a pseudo-intellectual. But to have the courage, in the meantime, of your uncertainty — to remain open to things, and serious about them — would be a pretty good way to go through college, and not just college.

One of the more surprising parts of the transcript was when most announced they shouldn’t have gone to college, the very place where they were exposed to most of the books they had been discussing:

Ilya Bernstein: I do have one regret. I regret having gone to college!

Rebecca Curtis: Me too!

Siddhartha Deb: I do too. I think it’s a waste.

Mark Grief: It would be much better if you were released into the world when you were 18, and instead you’re kept in this juvenile  detention for a further four years, in which you’re equipped with things which, frankly, you’d be able to understand much better later —

Ilya Bernstein: I would have done much more if I’d been out of college.

Mark Greif: And to have an entire nation of people going to college, right? That’s ridiculous.

Siddhartha Deb: I think we should go straight to work.

Ilya Bernstein: It was actually like a hiatus in my life. I did stuff before college and I did stuff after college, but what the hell did I do in college?

Benjamin Kunkel: It’s summer camp.

Ilya Bernstein: Four years of summer camp.

Mark Greif: It’s like being buried alive, or something, right?

Ilya Bernstein: But you enjoy it.

Mark Greif: Of course, but you come back from the dead, and you start the chronology over again…It’s life before and after Christ…before and after college. And it shouldn’t be like that.

On the definition of an essayist:

Mark Greif: Essayist! That’s interesting. You know, you go through life not really knowing who you are, and one day, somebody calls you an essayist. Out of all the pathetic categories that I read growing up, I knew there was no bigger joke than an essayist. Someone who couldn’t write something long enough to actually grab hold of anyone, someone without hte imagination to write fiction, someone without the romantic inspiration to write poetry, and someone who would never make any money to be published. I’m an essayist!

What Teachers Can Learn from Prof. Pitney

I took an Introduction to American Politics honors class with Professor John J. Pitney this past semester. He is a masterful teacher and this post will capture the lessons I drew on how to effectively engage a class. I hope it’s useful for other teachers reading this.

Be respected as an authority on the material: In any place where students are intellectually curious, they first want to be assured that you know your stuff. At most good high schools or colleges, it’s assumed teachers know the material. But effective teachers will provide background on how and why they know what they’re talking about. As students, we’re trained to be skeptical, so convince us.

Tell stories. This is a universal Good Thing for effective communicating, no less in formal teaching. His stories are all the more vivid since he was there (earlier in his career) — in D.C., in Albany, in the back room, wherever. 1) Make a statement, 2) Illustrate with a story, 3) Repeat.

Be weird and wacky. Pitney stomped and jumped all over the classroom. He did weird impersonations. He raised his voice, lowered his voice. He laughed. He showed odd videos. All this made him memorable. Weird is good.

Pitney_2 Get personal. He sent us a picture of him and his son over Halloween; he told us stories about family; he opened up. We relate to warm, human beings more than we do to cold, only-business PhDs.

Send email and write a blog – In short, communicate on grounds most comfortable to students, namely the internet. The primary out-of-class communication channel was email. The class had a blog to which we had to post.

Ask for feedback half-way into the semester. He requested anonymous feedback from students on how the course could be improved. It’s astonishing that teachers don’t take student feedback on teaching more seriously: by the time a student’s in college, he’s been exposed to dozens of teachers and varying styles. We have ideas. Consider them. Instead of waiting till the end of the semester, Pitney asked for it early on so he could incorporate the feedback while still teaching us.

Don’t listen to all the student feedback – have non-negotiables. He calls at students at random to answer questions. This makes students uncomfortable and people ask that he not do it, but he says, "I’m preparing you for the real world. People are going to call on you. People are going to make you uncomfortable." Or he obsesses about following Strunk and White writing style. People complain (I, for example, think he should note that minimalist writing is one kind of writing but not always the best). He doesn’t listen to us. He sticks to his principles.

Close each class with questions for next time. Each class ended with, "Next time we’ll discuss X, Y, and Z. Think about the following. [Insert three rhetorical questions here.]" This gets students thinking — even if at an unconscious level — about the next class meeting.

Allow students to personalize the course. Each class opened with, "Anything for the good of the order?" This was an opportunity for students to say something on their mind. Or most frequently, to comment on a story in the news. For essays, he provided multiple topic options and always allowed students to propose their own.

Be knowledgeable about the institution. Pitney knew the College well. If a topic fell outside his area of expertise, he referred us to another member of the faculty.

Let your emotion and passion show. At good liberal arts colleges, the professors love teaching. They’re passionate about it. Pitney displayed this energy at every turn. On the last class of the semester, he closed saying, "I’m going to end this course the way I’ve ended every class I’ve taught since grad school. I’m going to read a brief quote from A Man for All Seasons." His voice started to crack, and he read the quote:

God made the angels to show him splendor, as he made animals for innocence and plants for their simplicity. But man he made to serve him wittily in the tangle of his mind.

And without another word, he picked up his bag and left. For a few seconds, the class sat silently, contemplating the suddenness with which the class had come to a close. There was an unspoken consensus: We had been in the presence of a master.

What $80k More Gets You in Education

In his review of new pop economics books, Brad DeLong, a professor at Berkeley, says:

We here at Berkeley charge one-third the tuition of Stanford. Our students like a place where their peers don’t regard themselves as rich. We lose students who believe that the extra $80,000 is well worth paying to get Stanford’s smaller class sizes and better physical plant.

It’s a point I’ve thought about when I meet students at Claremont who, thanks to the $46,000 annual tuition, are taking on student loans which they’ll be paying off for years after graduation.

Sure, the small, private college atmosphere offers unparalleled facilities, access to top faculty, and overall very high quality of life. But is it really worth the extraordinary financial cost, particularly if it’s one you’ll be shouldering (and thus constrained by) several years after graduation?

I have my doubts. So I have enormous respect for those who choose a slightly less prestigious school for one which offers more financial aid, or those who go to public institutions to save money and keep their options open in the years after school.

Especially since it doesn’t matter a whole lot where you went to undergrad, I’m puzzled at those familes who bankrupt themselves in order to send Johnny to the most elite school he can get into.

A Portable Gym Which Exercises the Mind

James Flynn writes an essay about education and intelligence, which I will read later, but Arnold Kling pulls out this paragraph:

The best chance of enjoying enhanced cognitive skills is to fall in love with ideas, or intelligent conversation, or intelligent books, or some intellectual pursuit. If I do that, I create within my own mind a stimulating mental environment that accompanies me wherever I go. Then I am relatively free of needing good luck to enjoy a rich cognitive environment. I have constant and instant access to a portable gymnasium that exercises the mind. Books and ideas and analyzing things are possessions easier to access than even the local gym.

An interesting thought, to which Arnold responds:

Fine.  But take something you are not good at.  Imagine someone giving me one of these lectures:

Arnold, the best chance of enjoying enhanced fishing skills is to fall in love with fishing, or casting, or filleting, or being in a rowboat on some smelly river at 5 o’clock in the morning.

Arnold, the best chance to enjoy penmanship is to fall in love with neat handwriting, or nicely-formed letters, or taking forever to express written thoughts.

Thanks very much–enjoyed hearing your advice. Now, if you’ll just excuse me, I’m going to do something useful with the rest of my day. The Reading Instruction gurus argue that competence comes first, enjoyment second.  That feels more right to me.

Feels more right to me, too.

Hook-Up Culture in College

The mainstream media obsesses about American colleges and the college experience. One of its points of interest is the so-called “hook-up culture” among college youth, a phrase notoriously hard to pin down but that basically refers to indiscriminate, casual sexual activity instead of long-term dating with a single partner.

The stories about the rise in hook-ups are almost always breathless and unbelievable. College parties are depicted as Girls Gone Wild retreats. Romantic pick-up lines are out, efficient phrases like “wanna fuck?” are in. Dinner and a movie are out, dark corners at parties in the basement are in.

I’ve never been sure whether the coverage of hook-up culture is extensive because it actually exists or if it’s more a reflection of: a) a journalist’s own voyeuristic desires, b) a journalist’s interest in portraying my generation as hedonistic and materialistic, c) a journalist’s interest in feeding us villains/victims/heroes stories, with teen girls being the victims, or d) shoddy research due to a journalist’s laziness (this includes not being more skeptical of teens’ claims about their social life – they/we tend to overstate sexual activity and drinking/drug use according to studies).

Now that I’m in college, my perspective is that hook-up culture is indeed exaggerated in the popular press, though I’m still unclear as to exactly why. Granted, I go to an academic liberal arts college which probably has less “Girls Gone Wild” flavor than a state school. And my sole perspective shouldn’t, in theory, be as credible as the commentator who checks out many campuses and gathers anecdotes. Still, in talking with people and in my own experience so far, large amounts of sex, drugs, and alcohol happen on my campus and others, but it’s far from the free-for-all that you read about in the papers.

Last week’s Wall Street Journal had an article ($) which epitomized the media’s overreach on this topic. Jeff Zaslow wrote about hook-up culture in the villains/victims vein. It contained one of those classic anecdotes designed to shock:

Obviously, boys no longer have to call girls on Wednesday for a Saturday date. Now, college boys seeking weekend hookups send girls “U busy?” text messages at 2 or 3 a.m., and girls routinely rouse themselves and go, according to Ms. Stepp’s research. Many girls spend the next day clutching their cellphones, waiting in vain for the boy to call.

Yeah, right. Look, there’s no doubt that hook-up culture is alive and well on college campuses, but exaggerating the case does nobody any good.


Here’s my review of Female Chauvinist Pigs in which author Ariel Levy says girls need to step up and stop perpetuating raunch culture. Here’s my post on “life, sex, and relationships” orientation in college.

College Offers a Gazillion Social Interactions

This may be the single best thing college does in preparing you for the real world: it aggregates (in a constrained physical environment) thousands of people who are similar to you in age but not necessarily in interests or background, giving you tons of practice interacting with all sorts of people in all sorts of settings.

The social life part of the college experience is discussed mainly in the context of having fun. But the more time I spend in college the more I realize that the social interactions — the sheer practice that comes from the 24/7 pressure-cooker of living, sleeping, working, drinking, thinking with tons of other people — it’s these interactions which are probably more valuable than the academics. Since most professions demand people skills more than anything else, the socialization process in college does a nice job at letting you discover and develop your people skills in a range of diverse environments.

And the people who you practice socializing with usually go on to be friends, or at least weak ties, many years after college, providing the base for a professional network.

One Difference Between College and Real World

How much does college prepare you for the real world? That’s a question I’ll be thinking about in the coming months and years.

One big difference between college and the real world is that college is an information-rich environment which makes it very easy to track your progress (and be motivated) day-by-day.

In college you constantly receive reports on your progress. You turn in assignments, you receive grades. Rarely does a week go by without some affirmation or refutation of effort from an all-knowing expert (professor, advisor, whoever).

In the real world, best I can tell, the information you receive from your “market” (customers, boss, whoever) is far more ambiguous. Anyone who’s built a company knows that months can go by without clear feedback about whether you’re on the right track. Indeed, sometimes it takes months of unyielding effort with your head down before you figure out whether you’re creating something of value.

The most successful people I’ve met in the real world have a tolerance for ambiguity and are self-motivated enough to take care of business even if there aren’t routine, external validations or challenges.

So do college students get spoiled by the constant information delivery and assessments that’s part of structured education? Is there a risk that such an explicit reward system will retard a student’s ability to be intrinsically motivated? Will a student, upon graduation, be able to apply consistent effort without receiving a decisive “A” or “B” for each of his tasks?

(thanks to my friend Cal Newport for sparking this idea)