Numerical Computing

My research involves computer simulations, so I spend a lot of time implementing numerical algorithms. I view a numerical technique as a tool for solving a problem, not as something intrinsically interesting itself. But as a numerical modeler it pays to become familiar with your tools, so I have.

The first question is which programming language to use? I use FORTRAN, and I’m sure that makes me a dinosaur in the eyes of many. But FORTRAN is still common among physicists, and has served me well since I first learned it as a senior in high school. If you look in the solution manual for the 4th edition of Intermediate Physics for Medicine and Biology, you will find a few programs written in FORTRAN. Russ Hobbie humored me by letting me write these in FORTRAN rather than C, although C appears in the textbook itself.

Numerical Recipes:
The Art of Scientific Computing,
by Press et al.

I tend to avoid software like Matlab, Mathematica, and Maple as being to “black-boxy.” I like to tinker with the guts of my program, and often you can’t do that with high-level software packages. Perhaps younger scientists disagree. I do use Matlab for graphics.

When I face a numerical problem that is new to me, the first place I look is Numerical Recipes: The Art of Scientific Computing, by Press, Teukolsky, Vetterling, and Flannery . The copy on my bookshelf is the 2nd edition of Numerical Recipes in FORTRAN 77: The Art of Scientific Computing. In the preface Press et al. write

We call this book “Numerical Recipes” for several reasons. In one sense, the book is indeed a “cookbook” on numerical computation. However there is an important distinction between a cookbook and a restaurant menu. The latter presents choices among complete dishes in each of which the individual flavors are blended and disguised. The former—and this book—reveals the individual ingredients and explains how they are prepared and combined.

Numerical Methods That Work,
by Forman Acton.

For a guide to some of the lore of numerical computing, I recommend two delightful books by Forman Acton: Numerical Methods that Work, and Real Computing Made Real: Preventing Errors in Scientific and Engineering Calculations. In Real Computing, Acton writes

This book addresses errors of the third kind. You’ve never heard of them? But you've made them; we all make them every time we write a computer program to solve a physical problem.

Errors of the first kind are grammatical—we write things that aren’t in our programming language. The compiler finds them.

Errors of the second kind are our mistakes in programming the algorithms we sought to use. They include n−1 errors, inversions of logical tests, overwriting array limits (in Fortran) and a lot of other little technical mistakes that just don’t happen to be ungrammatical. We have to find them.

Then, Mirabile visu, the program runs—and even gives the correct answers to the two test problems we happen to have already solved.

Errors of the third kind are the ones we haven’t found yet. They show up only for as-yet-untested input values—often for quite limited ranges of these parameters. They include (but are not limited to) loss of significant digits, iterative instabilities, degenerative inefficiences in algorithms and convergence to extraneous roots or previously docile equations. Since some of these errors occur only for limited combinations of parameters inputs they may never disturb our results; more likely some of them will creep into our answers, but so quietly that we don't notice them—until our bridge has collapsed!

Real Computing Made Real,
by Forman Acton.

In the rest of the book, Acton serves up a feast of tricks, tips, and techniques. Even if you don’t particularly like numerical methods, you will enjoy these books. Readers of Intermediate Physics for Medicine and Biology will find them useful when trying to write a computer program to solve the Hodgkin and Huxley equations or implement the Fourier method for computed tomography.

I end with a quote from Acton’s book that he attributes to Richard Hamming. It is one of my favorite quotes, and one I believe is worth repeating:

The purpose of computing is insight, not numbers.