In 1964 the eminent physicist Richard Feynman served on the State of California's Curriculum Commission and saw how the Commission chose math textbooks for use in California's public schools. In his acerbic memoir of that experience, titled "Judging Books by Their Covers," Feynman analyzed the Commission's idiotic method of evaluating books, and he described some of the tactics employed by schoolbook salesmen who wanted the Commission to adopt their shoddy products. "Judging Books by Their Covers" appeared as a chapter in "Surely You're Joking, Mr. Feynman!" -- Feynman's autobiographical book that was published in 1985 by W.W. Norton & Company.

To introduce a series of articles about corruption in schoolbook-adoption proceedings, we present here (with permission from W.W. Norton & Company) an extended excerpt from Feynman's narrative.

As our "Annals of Corruption" series unfolds, readers will see that Feynman's account is as timely now as it was when he wrote it. State adoption proceedings still are pervaded by sham, malfeasance and ludicrous incompetence, and they still reflect cozy connections between state agencies and schoolbook companies.

Judging Books by Their Covers

Richard P. Feynman

I didn't pay much attention to what he said.

But the next day I got a telephone call from a pretty famous lawyer here in Pasadena, Mr. Norris, who was at that time on the State Board of Education. He asked me if I would serve on the State Curriculum Commission, which had to choose the new schoolbooks for the state of California. You see, the state had a law that all of the schoolbooks used by all of the kids in all of the public schools have to be chosen by the State Board of Education, so they have a committee to look over the books and to give them advice on which books to take.

It happened that a lot of the books were on a new method of teaching arithmetic that they called "new math," and since usually the only people to look at the books were schoolteachers or administrators in education, they thought it would be a good idea to have somebody who uses mathematics scientifically, who knows what the end product is and what we're trying to teach it for, to help in the evaluation of the schoolbooks.

I must have had, by this time, a guilty feeling about not cooperating with the government, because I agreed to get on this committee.

Immediately I began getting letters and telephone calls from schoolbook publishers. They said things like, "We're very glad to hear you're on the committee because we really wanted a scientific guy . . ." and "It's wonderful to have a scientist on the committee, because our books are scientifically oriented . . ." But they also said things like, "We'd like to explain to you what our book is about . . ." and "We'll be very glad to help you in any way we can to judge our books . . ." That seemed to me kind of crazy. I'm an objective scientist, and it seemed to me that since the only thing the kids in school are going to get is the books (and the teachers get the teacher's manual, which I would also get), any extra explanation from the company was a distortion. So I didn't want to speak to any of the publishers and always replied, "You don't have to explain; I'm sure the books will speak for themselves."

I represented a certain district, which comprised most of the Los Angeles area except for the city of Los Angeles, which was represented by a very nice lady from the L.A. school system named Mrs. Whitehouse. Mr. Norris suggested that I meet her and find out what the committee did and how it worked.

Mrs. Whitehouse started out telling me about the stuff they were going to talk about in the next meeting (they had already had one meeting; I was appointed late). "They're going to talk about the counting numbers." I didn't know what that was, but it turned out they were what I used to call integers. They had different names for everything, so I had a lot of trouble right from the start.

She told me how the members of the commission normally rated the new schoolbooks. They would get a relatively large number of copies of each book and would give them to various teachers and administrators in their district. Then they would get reports back on what these people thought about the books. Since I didn't know a lot of teachers or administrators, and since I felt that I could, by reading the books myself, make up my mind as to how they looked to me, I chose to read all the books myself. . . .

I was overwhelmed.

"It's all right, Mr. Feynman; we'll get someone to help you read them."

I couldn't figure out how you do that: you either read them or you don't read them. I had a special bookshelf put in my study downstairs (the books took up seventeen feet), and began reading all the books that were going to be discussed in the next meeting. We were going to start out with the elementary schoolbooks.

It was a pretty big job, and I worked all the time at it down in the basement. My wife says that during this period it was like living over a volcano. It would be quiet for a while, but then all of a sudden, "BLLLLLOOOOOOWWWWW!!!!" -- there would be a big explosion from the "volcano" below.

The reason was that the books were so lousy. They were false. They were hurried. They would try to be rigorous, but they would use examples (like automobiles in the street for "sets") which were almost OK, but in which there were always some subtleties. The definitions weren't accurate. Everything was a little bit ambiguous -- they weren't smart enough to understand what was meant by "rigor." They were faking it. They were teaching something they didn't understand, and which was, in fact, useless, at that time, for the child.

I understood what they were trying to do. Many [Americans] thought we were behind the Russians after Sputnik, and some mathematicians were asked to give advice on how to teach math by using some of the rather interesting modern concepts of mathematics. The purpose was to enhance mathematics for the children who found it dull.

I'll give you an example: They would talk about different bases of numbers -- five, six, and so on -- to show the possibilities. That would be interesting for a kid who could understand base ten -- something to entertain his mind. But what they turned it into, in these books, was that every child had to learn another base! And then the usual horror would come: "Translate these numbers, which are written in base seven, to base five." Translating from one base to another is an utterly useless thing. If you can do it, maybe it's entertaining; if you can't do it, forget it. There's no point to it.

Anyhow, I'm looking at all these books, all these books, and none of them has said anything about using arithmetic in science. If there are any examples on the use of arithmetic at all (most of the time it's this abstract new modern nonsense), they are about things like buying stamps.

Finally I come to a book that says, "Mathematics is used in science in many ways. We will give you an example from astronomy, which is the science of stars." I turn the page, and it says, "Red stars have a temperature of four thousand degrees, yellow stars have a temperature of five thousand degrees . . ." -- so far, so good. It continues: "Green stars have a temperature of seven thousand degrees, blue stars have a temperature of ten thousand degrees, and violet stars have a temperature of . . . (some big number)." There are no green or violet stars, but the figures for the others are roughly correct. It's vaguely right -- but already, trouble! That's the way everything was: Everything was written by somebody who didn't know what the hell he was talking about, so it was a little bit wrong, always! And how we are going to teach well by using books written by people who don't quite understand what they're talking about, I cannot understand. I don't know why, but the books are lousy; UNIVERSALLY LOUSY!

Anyway, I'm happy with this book, because it's the first example of applying arithmetic to science. I'm a bit unhappy when I read about the stars' temperatures, but I'm not very unhappy because it's more or less right -- it's just an example of error. Then comes the list of problems. It says, "John and his father go out to look at the stars. John sees two blue stars and a red star. His father sees a green star, a violet star, and two yellow stars. What is the total temperature of the stars seen by John and his father?" -- and I would explode in horror.

My wife would talk about the volcano downstairs. That's only an example: it was perpetually like that. Perpetual absurdity! There's no purpose whatsoever in adding the temperature of two stars. Nobody ever does that except, maybe, to then take the average temperature of the stars, but not to find out the total temperature of all the stars! It was awful! All it was was a game to get you to add, and they didn't understand what they were talking about. It was like reading sentences with a few typographical errors, and then suddenly a whole sentence is written backwards. The mathematics was like that. Just hopeless!

Then I came to my first meeting. The other members had given some kind of ratings to some of the books, and they asked me what my ratings were. My rating was often different from theirs, and they would ask, "Why did you rate that book low?" I would say the trouble with that book was this and this on page so-and-so -- I had my notes.

They discovered that I was kind of a goldmine: I would tell them, in detail, what was good and bad in all the books; I had a reason for every rating.

I would ask them why they had rated this book so high, and they would say, "Let us hear what you thought about such and such a book." I would never find out why they rated anything the way they did. Instead, they kept asking me what I thought.

We came to a certain book, part of a set of three supplementary books published by the same company, and they asked me what I thought about it.

I said, "The book depository didn't send me that book, but the other two were nice."

Someone tried repeating the question: "What do you think about that book?"

"I said they didn't send me that one, so I don't have any judgment on it."

The man from the book depository was there, and he said, "Excuse me; I can explain that. I didn't send it to you because that book hadn't been completed yet. There's a rule that you have to have every entry in by a certain time, and the publisher was a few days late with it. So it was sent to us with just the covers, and it's blank in between. The company sent a note excusing themselves and hoping they could have their set of three books considered, even though the third one would be late."

It turned out that the blank book had a rating by some of the other members! They couldn't believe it was blank, because [the book] had a rating. In fact, the rating for the missing book was a little bit higher than for the two others. The fact that there was nothing in the book had nothing to do with the rating.

I believe the reason for all this is that the system works this way: When you give books all over the place to people, they're busy; they're careless; they think, "Well, a lot of people are reading this book, so it doesn't make any difference." And they put in some kind of number -- some of them, at least; not all of them, but some of them. Then when you receive your reports, you don't know why this particular book has fewer reports than the other books -- that is, perhaps one book has ten, and this one only has six people reporting -- so you average the rating of those who reported; you don't average the ones who didn't report, so you get a reasonable number. This process of averaging all the time misses the fact that there is absolutely nothing between the covers of the book!

I made that theory up because I saw what happened in the curriculum commission: For the blank book, only six out of the ten members were reporting, whereas with the other books, eight or nine out of the ten were reporting. And when they averaged the six, they got as good an average as when they averaged with eight or nine. They were very embarrassed to discover they were giving ratings to that book, and it gave me a little bit more confidence. It turned out the other members of the committee had done a lot of work in giving out the books and collecting reports, and had gone to sessions in which the book publishers would explain the books before they read them; I was the only guy on that commission who read all the books and didn't get any information from the book publishers except what was in the books themselves, the things that would ultimately go to the schools.

This question of trying to figure out whether a book is good or bad by looking at it carefully or by taking the reports of a lot of people who looked at it carelessly is like this famous old problem: Nobody was permitted to see the Emperor of China, and the question was, What is the length of the Emperor of China's nose? To find out, you go all over the country asking people what they think the length of the Emperor of China's nose is, and you average it. And that would be very "accurate" because you averaged so many people. But it's no way to find anything out; when you have a very wide range of people who contribute without looking carefully at it, you don't improve your knowledge of the situation by averaging.

At first we weren't supposed to talk about the cost of the books. We were told how many books we could choose, so we designed a program which used a lot of supplementary books, because all the new textbooks had failures of one kind or another. The most serious failures were in the "new math" books: there were no applications; not enough word problems. There was no talk of selling stamps; instead there was too much talk about commutation and abstract things and not enough translation to situations in the world. What do you do: add, subtract, multiply, or divide? So we suggested some books which had some of that as supplementary -- one or two for each classroom -- in addition to a textbook for each student. We had it all worked out to balance everything, after much discussion.

When we took our recommendations to the Board of Education, they told us they didn't have as much money as they had thought, so we'd have to go over the whole thing and cut out this and that, . . . . When the senate budget committee got to it, the program was emasculated still further. Now it was really lousy! I was asked to appear before the state senators when the issue was being discussed, but I declined: By that time, having argued this stuff so much, I was tired. We had prepared our recommendations for the Board of Education, and I figured it was their job to present it to the state -- which was legally right, but not politically sound. I shouldn't have given up so soon, but to have worked so hard and discussed so much about all these books to make a fairly balanced program, and then to have the whole thing scrapped at the end -- that was discouraging! The whole thing was an unnecessary effort that could have been turned around and done the opposite way: start with the cost of the books, and buy what you can afford.

What finally clinched it, and made me ultimately resign, was that the following year we were going to discuss science books. I thought maybe the science would be different, so I looked at a few of them.

The same thing happened: something would look good at first and then turn out to be horrifying. For example, there was a book that started out with four pictures: first there was a windup toy; then there was an automobile; then there was a boy riding a bicycle; then there was something else. And underneath each picture it said, "What makes it go?"

I thought, "I know what it is: They're going to talk about mechanics, how the springs work inside the toy; about chemistry, how the engine of the automobile works; and biology, about how the muscles work."

It was the kind of thing my father would have talked about: "What makes it go? Everything goes because the sun is shining." And then we would have fun discussing it:

"No, the toy goes because the spring is wound up," I would say. "How did the spring get wound up?" he would ask.

"I wound it up."

"And how did you get moving?"

"From eating."

"And food grows only because the sun is shining. So it's because the sun is shining that all these things are moving." That would get the concept across that motion is simply the transformation of the sun's power.

I turned the page. The answer was, for the wind-up toy, "Energy makes it go." And for the boy on the bicycle, "Energy makes it go." For everything, "Energy makes it go."

Now that doesn't mean anything. Suppose it's "Wakalixes." That's the general principle: "Wakalixes makes it go." There's no knowledge coming in. The child doesn't learn anything; it's just a word!

What they should have done is to look at the wind-up toy, see that there are springs inside, learn about springs, learn about wheels, and never mind "energy." Later on, when the children know something about how the toy actually works, they can discuss the more general principles of energy.

It's also not even true that "energy makes it go," because if it stops, you could say, "energy makes it stop" just as well. What they're talking about is concentrated energy being transformed into more dilute forms, which is a very subtle aspect of energy. Energy is neither increased nor decreased in these examples; it's just changed from one form to another. And when the things stop, the energy is changed into heat, into general chaos.

But that's the way all the books were: They said things that were useless, mixed-up, ambiguous, confusing, and partially incorrect. How anybody can learn science from these books, I don't know, because it's not science.

So when I saw all these horrifying books with the same kind of trouble as the math books had, I saw my volcano process starting again. Since I was exhausted from reading all the math books, and discouraged from its all being a wasted effort, I couldn't face another year of that, and had to resign.

Sometime later I heard that the energy-makes-it-go book was going to be recommended by the curriculum commission to the Board of Education, so I made one last effort. At each meeting of the commission the public was allowed to make comments, so I got up and said why I thought the book was bad.

The man who replaced me on the commission said, "That book was approved by sixty-five engineers at the Such-and-such Aircraft Company!"

I didn't doubt that the company had some pretty good engineers, but to take sixty-five engineers is to take a wide range of ability -- and to necessarily include some pretty poor guys! It was once again the problem of averaging the length of the emperor's nose, or the ratings on a book with nothing between the covers. . . .

I couldn't get through to him, and the book was approved by the board. . . .

Then the question came up, "Will the schools be getting the books at the regular time, or could they, perhaps, get them a little earlier, in time for the coming term?"

One publisher's representative got up and said, "We are happy that you accepted our bid; we can get it out in time for the next term."

A representative of the publisher that lost out was also there, and he got up and said, "Since our bids were submitted based on the later deadline, I think we should have a chance a bid again for the earlier deadline, because we too can meet the earlier deadline."

Mr. Norris, the Pasadena lawyer on the board, asked the guy from the other publisher, "And how much would it cost for us to get your books at the earlier date?"

And he gave a number: It was less!

The first guy got up: "If he changes his bid, I have the right to change my bid!" -- and his bid is still less!

Norris asked, "Well how is that -- we get the books earlier and it's cheaper?"

"Yes," one guy says. "We can use a special offset method we wouldn't normally use . . ." -- some excuse why it came out cheaper. The other guy agreed: "When you do it quicker, it costs less!"

That was really a shock. It ended up two million dollars cheaper. Norris was really incensed by this sudden change.

What happened, of course, was that the uncertainty about the date had opened the possibility that these guys could bid against each other. Normally, when books were supposed to be chosen without taking the cost into consideration, there was no reason to lower the price; the book publishers could put the prices at any place they wanted to. There was no advantage in competing by lowering the price; the way you competed was to impress the members of the curriculum commission.

By the way, whenever our commission had a meeting, there were book publishers entertaining curriculum commission members by taking them to lunch and talking to them about their books. I never went.

It seems obvious now, but I didn't know what was happening the time I got a package of dried fruit and whatnot delivered by Western Union with a message that read, "From our family to yours, Happy Thanksgiving -- The Pamilios."

It was from a family I had never heard of in Long Beach, obviously someone wanting to send this to his friend's family who got the name and address wrong, so I thought I'd better straighten it out. I called up Western Union, got the telephone number of the people who sent the stuff, and I called them.

"Hello, my name is Mr. Feynman. I received a package . . ."

"Oh, hello, Mr. Feynman, this is Pete Pamilio" and he says it in such a friendly way that I think I'm supposed to know who he is! I'm normally such a dunce that I can't remember who anyone is.

So I said, "I'm sorry, Mr. Pamilio, but I don't quite remember who you are . . ."

It turned out he was a representative of one of the publishers whose books I had to judge on the curriculum commission.

"I see. But this could be misunderstood."

"It's only family to family."

"Yes, but I'm judging a book that you're publishing, and maybe someone might misinterpret your kindness!" I knew what was happening, but I made it sound like I was a complete idiot.

Another thing like this happened when one of the publishers sent me a leather briefcase with my name nicely written in gold on it. I gave them the same stuff: "I can't accept it; I'm judging some of the books you're publishing. I don't think you understand that!"

One commissioner, who had been there for the greatest length of time, said, "I never accept the stuff; it makes me very upset. But it just goes on."

But I really missed one opportunity. If I had only thought fast enough, I could have had a very good time on that commission. I got to the hotel in San Francisco in the evening to attend my very first meeting the next day, and I decided to go out to wander in the town and eat something. I came out of the elevator, and sitting on a bench in the hotel lobby were two guys who jumped up and said, "Good evening, Mr. Feynman. Where are you going? Is there something we can show you in San Francisco?" They were from a publishing company, and I didn't want to have anything to do with them.

"I'm going out to eat."

"We can take you out to dinner."

"No, I want to be alone."

"Well, whatever you want, we can help you."

I couldn't resist. I said, "Well, I'm going out to get myself in trouble."

"I think we can help you in that, too."

"No, I think I'll take care of that myself." Then I thought, "What an error! I should have let all that stuff operate and [kept] a diary, so the people of the state of California could find out how far the publishers will go!". . . .

Richard P. Feynman (1918-1988) was one of the premier physicists of the second half of the 20th century. He worked on atomic weapons at Los Alamos during World War 2, then spent several years on the faculty of Cornell University, and then (in 1950) became a professor at the California Institute of Technology. He remained at Caltech for the rest of his career. Feynman worked on the theory of superfluidity, predicted the existence of quarks, and was one of the three men whose contributions to quantum electrodynamics were commemorated by the 1965 Nobel Prize in physics. The two other men were Julian Schwinger and Sin-Itiro Tomonaga.

Editor's postscript

We don't know of any other case in which state functionaries have given high marks to a book whose pages were blank, but all the other follies and outrages described in Feynman's narrative are familiar.

During the past ten years we have studied adoptions operated by state agencies in California and elsewhere, and we have seen -- again and again -- the same practices that Feynman observed in 1964. As a rule, state textbook-adoption proceedings are bureaucratic shams. As a rule, the evaluation committees assembled by state agencies are manifestly unqualified to appraise textbooks or to render any meaningful advice.

Recall Feynman's observation that "usually the only people to look at the books were schoolteachers or administrators in education." That is still true. In a typical case, a state agency creates an evaluation committee that consists of schoolteachers and school-district officials, perhaps augmented by an ordinary citizen or two. Then the agency charges this committee with the task of appraising some number of books in a given subject, even though the committee lacks anyone who possesses expert knowledge of the subject in question. The members of the committee then engage in a silly, scripted ritual, producing collective judgments about books that few (if any) members have actually tried to read -- and they eventually contrive some unexplained, unsupported, collective recommendations, declaring that the books should be accepted for use in schools. They don't write individual reports, so the taxpayers cannot know who did what -- i.e., the taxpayers cannot determine who (if anyone) inspected and endorsed any particular book. No one is responsible for anything.

If a state agency really wanted to obtain legitimate evaluations of textbooks, the agency could achieve this by using a process that is very well known: Send each book to a knowledgeable reviewer who will appraise it, who will write a report to set forth and explain his appraisal, and who will sign his name to his report. This is the process employed by the book-review editors of newspapers, magazines and professional journals throughout the land. It works, and it can be repeated to any desired extent: To obtain several appraisals of a given book, simply send the book to several reviewers.

As a rule, however, state agencies don't want legitimate evaluations of the textbooks that publishers submit for adoption, because the agencies are allied with the publishers. The adoption proceedings staged by these agencies are not designed to help school districts, to protect students, or to serve the interests of taxpayers. Rather, they are designed to serve the interests of the publishers, to generate approvals and certifications for the publishers' books, and to help the publishers sell those books to local schools.

This is why we continue to see, in state after state, the same absurdities that Feynman saw thirty-five years ago. In state after state, the legitimate reviewing of books is shunned. In state after state, incompetent committees engage in group-grope "evaluation" shams which embody the emperor's-nose fallacy -- the nonsensical notion that a question can be answered by averaging a bunch of guesses proffered by people who don't know what they are talking about. In state after state, corrupt proceedings are corrupted further by publishers' "presentations": The publishers send salesmen to describe, explain and promote their books to the evaluation committees, to other state functionaries, and even to members of state boards of education. Feynman refused to hear such pitches, because he held that a book should be able to speak for itself. He was right, of course -- but in many states, evaluation committees are routinely subjected to "presentations" by publishers' pitchmen.

Some state agencies are so strongly allied with publishers that the agencies help the publishers to market their books to unwary school districts. Consider, for example, what goes on in Florida. Each year, the Florida Department of Education issues a document called the Florida Catalog of State Adopted Instructional Materials, which includes many anonymous descriptions of textbooks and other materials that have been approved for use in Florida's public schools. The descriptions actually are advertisements. They are promotional claims written by the companies that sell the materials, but this fact is concealed. Florida school districts are led to believe that the glowing descriptions printed in the Catalog represent appraisals developed by the state's evaluation teams. This matter will be described at greater length in another issue of The Textbook Letter.