Programming Massively Parallel Processors: A Hands-on Approach

Very good book

Although not evident from the title, this book is mainly about CUDA, NVIDIA's method for programming their GPUs, but the title is perhaps justified by the existence of chapters on half a dozen or so other packages for programming GPUs, including an important one on using GPUs on MPI lusters. Most of the examples are in C, but with a chapter on FORTRAN. As other reviewers have already mentioned, the book is either excessively wordy or wonderfully detailed, depending on your mode of learning. For me, it was too wordy: there were sometimes two or three pages to explain an algorithm that was immediately clear from the figure. I would add that the authors have a curious view of what readers might know or not know: they give a long explanation of IEEE floating-point formats, which I believe are well known among application programmers, but mention only in passing with no explanation such less well known concepts as atomic operations and Amdahl's law. Various generations of NVIDIA chips are mentioned by code names that are probably not familiar to those of us who just buy a graphics card, although an appendix at the end gives some details. All in all, I found this a very useful introduction to CUDA, and especially to problems of memory access management on GPU boards, which I knew nothing about beforehand. Nonetheless, it was clear that one would need access to the detailed library manuals and spec sheets for the particular chip one was using before undertaking a real coding project, as there seem to be rather significant differences between chips in how one might try to maximize efficient use of the multiple execution units, memory blocks, and different cache sizes on the different chips. I expect that to have life beyond the current generation of chips, an application would need to query the system for these properties and optimize accordingly at run time. The book has some hints on how to do this.

So much effort went into writing this text, but better guidance was needed. True, there are typos as many have said, but not as many as I find in most textbooks and research papers.Before I get started, I have one strong compliment for the text: exercises at the ends of the chapters, where they appear, are very well thought out and get right to the point.The worst part of this book is its index, which gets an unnegotiable F grade from me. So much is missing. Try looking for the gridDim built-in variable; it's not there. blockDim is, but not gridDim. Or look up "constant memory"; that's not there under "memory", "constant", or otherwise. In fact although most of parallel programming is about overcoming memory bottlenecks, the index doesn't say much about memory in general. Indexing books is very hard and is usually left to professional indexers; there really is such a trade. But for this book the authors needed to pay more attention to the end product.The overall arrangement is illogical, starting from an enormous abstraction and then adding detail. Better to begin with something small and concrete which can be understood, and build up.At several points a non-programmer was permitted to do the typing, which is like allowing a pilot to fly while waking out of general anesthesia. See Figure 10.4 at lines 1 and 5.Many concepts are never brought together in some kind of summary that one would ever turn to as a reference, such as special variables. One can also find seeming conflicts that need clarification, such as Table 5.1's "local" memory for automatic array variables that are "stored into the global memory" on the next page. Both turn out to be true, but the presentation is not ideal.References are sloppily missing in certain cases, such as ELLPACK on page 224.At several points the text completely derails with outrageous metaphors. For instance page 85, where the authors try to explain standard queries of an implementation's capabilities by informing us that "most American hotel rooms come with hair dryers". The illustration's purpose is to publish a thicker book that looks "more valuable", but the truth is that any programmer who read to page 85 knows that most libraries have functions for querying their parameters and limits. There's no need to digress into toothpaste in Asia.If you must take a course that is based on this text, things get worse for you: the authors made presentation slides, and you bear the risk that your instructor may use them in the classroom. These are not well thought out from either a clarity or instructional standpoint, and they are very cluttered with confusing details. One example is how tiled matrix multiplication is explained; one slide actually draws three 4x4 matrices with row, column, and matrix name (!!!) labels in every single cell, giving the poor class 3 x (4 x 4) x 3 = 144 extra trivialities to distract them on that slide alone.Another pedagogical problem with their slide example, which the book also mishandles, is that they've chosen to illustrate the use of submatrices by dividing a 4 x 4 problem into a 2 x 2 problem. Although this is the smallest case they could select, the difficulty is that the human brain comprehends these 2 x 2 divisions in terms of "corners" (top left, etc.) instead of columns and rows. The examples should have been 9 x 9 divided into 3 x 3.In fairness, Kirk and Hwu did not have the best of resources available to assemble this book from. nVIDIA's "CUDA C Programming Guide" has no index whatsoever, and the table of contents doesn't seem to have much to do with what one is trying to find a second time. There's a reason nVIDIA doesn't sell print copies of that text on Amazon; people would review it.

Like the authors, the book is clear, down to earth, willing to speak truth to power, and if you listen to them (read the book) you will do better science. A serially executing program is a not-too-special case of a massively parallel solution.Code samples and extensive graphics will help you on your transition. The history from an NVIDIA insider is fascinating, and the team used this material to teach people that have since done excelent work.

I gave five starts to this book since the book comes with excellent conditions. I recommend this product for programmer that want to learn general programming concept about cuds

Good book on CUDA.

I am teaching a Parallel Programming for Master students. The APIs I am teaching are MPI/CUDA/OpenMP these are enough for a student to grasp and appreciate parallel programming.For the CUDA course I have selected this book to start and fill in with material of other books. For instance when this books describes how CUDA maps to multidimension data like an image there should be an example. The best example is the julia example of the book CUDA by example of course modified so as to teach coalescing in the writes and what happens when DIM is not divisible by the number of threads.After two years I will be able to present the material I am teaching. I think this will be helpful.

This book is a bit uneven in quality. Some of the material seems well written, while other sections are disorganized, marred by cut-and-paste errors and typos, omit basic definitions, and offer vague and sloppy analogies. I find that I frequently need to seek more clarity from other free sources on the internet. That generally works so well that I find myself almost never consulting this book as a first resort to answer a specific technical question.

Sicuramente l'argomento non è per tutti, visto la tecnologia, quindi di difficile comprensione se non si hanno delle basi sulla programmazione parallela e in particolare su CUDA. Ritengo questo libro di alta qualità senza ombra di dubbio, ma salta alcuni passaggi basilari che bisogna colmare con la propria esperienza.

El libro esta muy bien, pero algunas figuras tienen una resolución deplorable o están mal. De todos modos recomiendo su compra.

This second edition of PMPP extends the table of contents of the first one, almost doubling the number of pages (in the 2nd ed. its ~500 pages. I have the paper version.)The book can be separated roughly in 4 parts: the first, and more important, deals with parallel programming using Nvidia's CUDA technology: this takes about the first 10 chapters and Ch. 20; the second slice shows a couple of important examples (MRI image reconstruction and molecular simulation and visualization, chapters 11 and 12); the 3rd important block of chapters (chapters 14 upto 19) deals with other parallel programming technologies and CUDA expansions: OpenCL, OpenACC, CUDA Fortran, Thrust, C++AMP, MPI. Finally, spread all over the book, there are several "outlier", but nevertheless important, chapters: Ch. 7 discusses floating-point issues and its impact in calculation's accuracy; Ch. 13, "PP and Computational Thinking", discusses broadly how to think when converting sequential algorithms to parallel; and Ch. 21 discusses the future of PP (using CUDA goggles :-).I've read about half of the book (I attended Coursera's MOOC -"Heterogeneous Parallel Computing"- taught by one of the authors, Prof. W. Hwu, and waited until the 2nd edition was out to buy it), and browsed carefully the other half. Here are my...Comments----------(+++) Pluses:# There are just a few typos, here and there, but they are easy to spot (the funniest is in line 5 of ch. 1 (!), where Giga corresponds to 10^12 and Tera to 10^15, according to the authors: of course Giga is 10^9 and Tera is 10^12 - this bug is browseable with Amazon's "look inside" feature...).# CUDA is described under an application POV; many computation patterns are exemplified in CUDA, from the simplest (vector addition) to more difficult ones (matrix multiplication, image filtering with convolution kernels, scanning,...)# There is a description of several other PP technologies (OpenCL, MPI,...), what is a good and useful feature if you are evaluating or selecting a PP technology to use.# The book is quite comprehensive about current PP technologies. CUDA is the "leading actress", but if you dominate CUDA you can easily transfer your PP knowledge to other technologies. The "tête-à-tête" of CUDA with those technologies appears in the respective chapters, by showing the respective templates for basic tasks (e.g. for vector addition or matrix multiplication).# There are many discussions about the tradeoffs between memory transfer (from CPU to GPU) and speed of GPU computation, as well as about the optimization of this speed.(---) Minuses:# The figures, pictures and tables use a variety of font sizes and backgrounds (either gray, white, etc...); some fonts are very tiny and so in those cases the code is difficult to read.# The chapters with descriptions of other PP technologies (OpenCL, OpenACC, MPI,...), often written by "invited" authors (acknowledged by Kirk and Hwu), are in general succinct; and the maturity and availability (free, commercial, open-source,...) of the technologies are not discussed.# The prose is often excessive (somewhat verbose), and then the tentative to explain deeply the matters sometimes leads to confusion.# There is some heterogeneity in the book (that's OK, we are talking about "heterogeneous parallel processors and programming" here ;-) perhaps because there are two main authors with different backgrounds and several "guest authors" in the latter chapters.# It lacks a well-thought introductory chapter treating, along a pedagogical and explicative style, the subset of C commonly seen in PP computation and the CUDA extensions/templates. These matters are (lightly) in the book, but scattered in many chapters.# Browsing the CUDA programming manual we can see that there are many issues not treated (or barely mentioned) in PMPP. An appendix of 20 or 30 pages with a systematic summary of the CUDA API and C extensions would be an welcome addition to the book.Conclusion----------After having browsed (with Amazon's "look inside" feature and by reading the reader's comments) other books about PP and CUDA, I decided to get this one and I am not disappointed at all. It has a nice description of CUDA and of many parallel computation patterns and its CUDA implementation and it gives you a barebones sample of other PP technologies. PMPP can be read easily as a "straight-line" text or on a chapter-by-chapter basis (this last one was more useful for me). Recommended for guys and gals with some experience in C programming and a will of getting into PP (or in expanding their skills...)

good book! explains things by the detail