Сравнение OpenCL с CUDA, GLSL и OpenMP
На хабре уже рассказали о том, что такое OpenCL и для чего он нужен, но этот стандарт сравнительно новый, поэтому интересно как соотносится производительность программ на нём с другими решениями.
В этом топике приведено сравнение OpenCL с CUDA и шейдерами для GPU, а также с OpenMP для CPU.
Тестирование проводилось на задаче N-тел. Она хорошо ложится на параллельную архитектуру, сложность задачи растёт как O(N 2 ), где N — число тел.
В качестве тестовой была выбрана задача симуляции эволюции системы частиц.
На скриншотах (они кликабельны) видна задача N точечных зарядов в статическом магнитном поле. По вычислительной сложности она ничем не отличается от классической задачи N тел (разве что картинки не такие красивые).
Во время проведения замеров вывод на экран был отключен, а FPS означает число итераций в секунду (каждая итерация — это следующий шаг в эволюции системы).
Код на GLSL и CUDA для этой задачи был уже написан сотрудниками ННГУ.
NVidia Quadro FX5600
Версия драйвера 197.45
CUDA обгоняет OpenCL приблизительно на 13%. При этом, если оценивать теоретически возможную производительность для этой задачи для данной архитектуры, реализация на CUDA достигает её.
(В работе A Performance Comparison of CUDA and OpenCL говорится о том, что производительность ядра OpenCL проигрывает CUDA от 13% до 63% )
Несмотря на то, что тесты проводились на карточке серии Quadro, понятно, что обычный GeForce 8800 GTS или GeForce 250 GTS дадут схожие результаты (все три карточки основаны на чипе G92).
ATI Stream SDK версия 2.01
OpenCL проигрывает шейдерам на карточках от AMD так как вычислительный блоки на них имеют архитектуру VLIW, на которую (после оптимизации) могут хорошо лечь многие шейдерные программы, но компилятор для кода OpenCL (который является частью драйвера) плохо справляется с оптимизацией.
Также этот весьма скромный результат может быть вызван тем, что карточки от AMD не поддерживают локальную память на физическом уровне, а отображают область локальной памяти на глобальную.
Код с использованием OpenMP был скомпилирован при помощи компиляторов от Intel и Microsoft.
Компания Intel не выпустила своих драйверов для запуска кода OpenCL на центральном процессоре, поэтому был использован ATI Stream SDK.
Intel Core2Duo E8200
ATI Stream SDK версия 2.01
Код на OpenMP, скомпилированный при помощи MS VC++ имеет практически идентичную производительность с OpenCL.
Это ещё при том, что Intel не выпустил своего драйвера для интерпретации OpenCL, и используется драйвер от AMD.
Компилятор от Intel поступил не совсем «честно» он полностью развернул основной цикл программы, повторив его где-то 8k раз (число частиц было задано константой в коде) и получив семикратный прирост производительности также благодаря использованию SSE инструкций. Но победителей, конечно, не судят.
Что характерно, на моём стареньком AMD Athlon 3800+ код тоже запустился, но таких выдающихся результатов, как на Intel, конечно, ждать не приходится.
CUDA vs OpenCL: Which to Use for GPU Programming
Author: Dori Exterman Published On: June 7, 2021 Estimated reading time: 8 minutes
Graphic Processing Units or GPUs have become an essential part of providing processing power for high performance computing applications over the recent years. GPGPU Programming is general purpose computing with the use of a Graphic Processing Unit (GPU). This is done by using a GPU together with a Central Processing Unit (CPU) to accelerate the computations in applications that are traditionally handled by just the CPU only. GPU programming is now included in virtually every industry, from accelerating video, digital image, audio signal processing, and gaming to manufacturing, neural networks and deep learning.
GPGPU programming essentially entails dividing multiple processes or a single process among different processors to accelerate the time needed for completion. GPGPU’s take advantage of software frameworks such as OpenCL and CUDA to accelerate certain functions in a software with the end goal of making your work quicker and easier. GPU’s make parallel computing possible by use of hundreds of on-chip processor cores which simultaneously communicate and cooperate to solve complex computing problems.
CUDA vs OpenCL – two interfaces used in GPU computing and while they both present some similar features, they do so using different programming interfaces.
CUDA which stands for Compute Unified Device Architecture, is a parallel programming paradigm which was released in 2007 by NVIDIA. CUDA while using a language which is similar to the C language is used to develop software for graphic processors and a vast array of general-purpose applications for GPU’s which are highly parallel in nature.
CUDA is a proprietary API and as such is only supported on NVIDIA’s GPUs that are based on Tesla Architecture. The graphics cards which support CUDA are the GeForce 8 series, Tesla and Quadro. The CUDA programming paradigm is a combination of both serial and parallel executions and contains a special C function called the kernel, which is in simple terms a C code that is executed on a graphics card on a fixed number of threads concurrently (learn more about what is CUDA).
OpenCL an acronym for the Open Computing Language was launched by Apple and the Khronos group as a way to provide a benchmark for heterogeneous computing that was not restricted to only NVIDIA GPU’s. OpenCL offers a portable language for GPU programming that uses CPU’s, GPU’s, Digital Signal Processors and other types of processors. This portable language is used to design programs or applications that are general enough to run on considerably different architectures while still being adaptable enough to allow each hardware platform achieve high performance.
OpenCL provides portable, device- and vendor-independent programs which are capable of being accelerated on various different hardware platforms. OpenCL C language is a restricted version of the C99 language that has extensions which are appropriate for executing data-parallel codes on various devices.
CUDA vs OpenCL Comparison
OpenCL assures a portable language for GPU programming, which is adept at targeting very unrelated parallel processing devices. This in no way means that a code is guaranteed to run on all devices if at a all due to the fact that most have very different feature sets. Some extra effort has to be put in to make the code run on multiple devices while avoiding vendor-specific extension. Unlike the CUDA kernel, an OpenCL kernel can be compiled at runtime, which would add up to an OpenCL’s running time. However, On the other hand, this just-in-time compile could allow the compiler to generate code that will make better use of the target GPU.
CUDA, is developed by the same company that develops the hardware on which it executes its functions, which is why one may expect it to better match the computing characteristics of the GPU, and therefore offering more access to features and better performance.
However, performance wise, the compiler (and ultimately the programmer) is what makes each interface faster as both can fully utilize hardware. The performance will be dependent on some variables, including code quality, algorithm type and hardware type.
Implementation by Vendors
As of the time of this writing there is only one vendor for CUDA implementation and that is its proprietor, NVIDIA.
OpenCL, however, has been implemented by a vast array of vendors including but not limited to:
- AMD: Intel and AMD chips and GPU’s are supported.
- Radeon 5xxx, 6xxx, 7xxx series, R9xxx series are supported
- All CPUs support OpenCL 1.2 only
- NVIDIA: NVIDIA GeForce 8600M GT, GeForce 8800 GT, GeForce 8800 GTS, GeForce 9400M, GeForce 9600M GT, GeForce GT 120, GeForce GT 130, ATI Radeon 4850, Radeon 4870, and likely more are supported.
- Apple (MacOS X only is supported)
- Host CPUs as compute devices are supported
- CPU, GPU, and “MIC” (Xeon Phi).
This is likely the most recognized difference between the two as CUDA runs on only NVIDIA GPUs while OpenCL is an open industry standard and runs on NVIDIA, AMD, Intel, and other hardware devices. Also OpenCL provides for CPU fallback and as such code maintenance is easier while on the other hand CUDA does not provide CPU fallback, which makes developers put if-statements in their codes that help to distinguish between the presence of a GPU device at runtime or its absence.
Open-source vs commercial
Another highly recognized difference between CUDA and OpenCL is that OpenCL is Open-source and CUDA is a proprietary framework of NVIDIA. This difference brings its own pros and cons and the general decision on this has to do with your app of choice.
Generally if the app of your choice supports both CUDA and OpenCL, going with CUDA is the best option as it generates better performance results in this scenario. This is because NVIDIA provides top quality support. If some apps are CUDA based and others have OpenCL support, a recent NVIDIA card will help you get the most out of CUDA enabled apps while having good compatibility in non-CUDA apps.
However, if all your apps of choice are OpenCL supported then the decision is already made for you.
Multiple OS Support
CUDA is able to run on Windows, Linux, and MacOS, but only using NVIDIA hardware. However, OpenCL is available to run on almost any operating system and most hardware varieties. When it comes to the OS support comparison the chief deciding factor still remains the hardware as CUDA is able to run on the leading operating systems while OpenCL runs on almost all.
The hardware distinction is what really sets the comparison. With CUDA having a requirement of only the use of NVIDIA hardware, while with OpenCL the hardware is so not specified. This distinction has its own pros and cons.
Libraries are key to GPU Computing, because they give access to a set of functions which have already been finetuned to take advantage of data-parallelism. CUDA comes in very strong in this category as it has support for templates and free raw math libraries which embody high performance math routines:
- cuBLAS – Complete BLAS Library
- cuRAND – Random Number Generation (RNG) Library
- cuSPARSE – Sparse Matrix Library
- NPP – Performance Primitives for Image & Video Processing
- cuFFT – Fast Fourier Transforms Library
- Thrust – Templated Parallel Algorithms & Data Structures
- h – C99 floating-point Library
OpenCL has alternatives which can be easily built and have matured in recent times, however nothing like the CUDA libraries. An example of which is the ViennaCL. AMD’s OpenCL libraries also have an added bonus of not only running on AMD devices but additionally on all OpenCL compliant devices
This is a part of the comparison that encompasses the support, longevity, commitment, etc of each framework. While these things could be hard to measure, a look at forums give a measure of how large a community is. The number of topics on NVIDIA’s CUDA forums are staggeringly larger than AMD’s OpenCL forums. However, the OpenCL forums have been increasing in topics in recent years and one ought to also note that CUDA has been around for a larger amount of time.
CUDA allows for developers to write their software in C or C++ because it is only a platform and programming model not a language or API. Parallelization is achieved by the employment of CUDA keywords.
On the other hand OpenCl does not permit for writing code in C++, however it provides an environment resembling the C programming language for work and permits for work with GPU resources directly.
|Performance||No clear advantage, dependent code quality, hardware type and other variables||No clear advantage, dependent code quality, hardware type and other variables|
|Vendor Implementation||Implemented by only NVIDIA||Implemented by TONS of vendors including AMD, NVIDIA, Intel, Apple, Radeon etc.|
|Portability||Only works using NVIDIA hardware||Can be ported to various other hardware as long as vendor-specific extensions are avoided|
|Open Source vs Commercial||Proprietary framework of NVIDIA||Open Source standard|
|OS Support||Supported on the leading Operating systems with the only distinction of NVIDIA hardware must be used||Supported on various Operating Systems|
|Libraries||Has extensive high performance libraries||Has a good number of libraries which can be used on all OpenCL compliant hardware but not as extensive as CUDA|
|Community||Has a larger community||Has a growing community not as large as CUDA|
|Technicalities||Not a language but a platform and programming model that achieves parallelization using CUDA keywords||Does not enable for writing code in C++ but works in a C programming language resembling environment|
How To Choose
When GPU is supported it brings huge great benefits to computing power and apps. With CUDA and OpenCL being the leading frameworks as at the time of writing. CUDA being a proprietary NVIDIA framework is not supported in as many applications as OpenCL, but where it is supported, the support makes for unparalleled performance. While the OpenCL which is supported in more applications does not give the same performance boosts where supported as CUDA does.
NVIDIA GPUs (newer ones) while being CUDA supported have strong OpenCL performance for the instances CUDA is not supported. The general rule of thumb being that if on the instance a great majority your choice of apps and hardware are all OpenCL supported then OpenCL should be the choice for you.
No matter what you decide on Incredibuild can help you turbocharge your compilations and tests leading to better computing, be it in content creation, machine learning, signal processing, and tons of other computer-intensive workloads. A look at our case study with MediaPro is an example of how we can accelerate your compilations and tests to a fraction of the time (in this case more than 6 times faster).
An expert software developer and product strategist, Dori Exterman has 20 years of experience in the software development industry. As CTO of Incredibuild, he directs the company’s product strategy and is responsible for product vision, implementation, and technical partnerships. Before joining Incredibuild, Dori held a variety of technical and product development roles at software companies, with a focus on architecture, performance, advanced technologies, DevOps, release management and C++. He is an expert and frequent speaker on technological advancement in development tools.
OpenCL or CUDA Which way to go?
I’m investigating ways of using GPU in order to process streaming data. I had two choices but couldn’t decide which way to go?
My criterias are as follows:
- Ease of use (good API)
- Community and Documentation
I’ll code in C and C++ under linux.
3 Answers 3
- interfaced from your production code
- portable between different graphics hardware
- limited operations but preprepared shortcuts
- separate language (CUDA C)
- nVidia hardware only
- almost full control over the code (coding in a C-like language)
- lot of profiling and debugging tools
Bottom line — OpenCL is portable, CUDA is nVidia only. However, being an independent language, CUDA is much more powerful and has a bunch of really good tools.
- Ease of use — OpenCL is easier to use out of the box, but once you setup the CUDA coding environment it’s almost like coding in C.
- Community and Documentation — both have extensive documentation and examples, however I think CUDA has better.
- Performance — CUDA allows for greater control, hence can be better fine-tuned for higher performance.
- Future — hard to say really.
My personal experiences were:
API: OpenCL has slightly more complex api. However most time you will spent with writing kernel code, and here both are almost identical.
Community: CUDA has a much bigger community then OpenCL up til now, but this will probably about to even out.
Documentation: Both are very well documented.
Performance: We made the experience, that OpenCL drivers are not yet fully optimized.
Future: The future lies with OpenCL as it is an open standard, not restricted to a vendor or specific hardware!
Opencl vs Cuda: Similarities, Differences, and Proper Use
When it comes to parallel computing, two popular programming languages stand out: OpenCL and CUDA. Both languages allow developers to write code for parallel processing on GPUs, but there are some key differences between them. In this article, we will explore the differences between OpenCL and CUDA and help you determine which one is right for your project.
Let’s define what OpenCL and CUDA are. OpenCL stands for Open Computing Language, and it is an open standard for parallel programming across CPUs, GPUs, and other processors. CUDA, on the other hand, stands for Compute Unified Device Architecture, and it is a proprietary language developed by NVIDIA for programming their GPUs.
So, which one is better? The answer is not so simple. Each language has its own strengths and weaknesses, and the choice largely depends on your specific needs. In the next sections, we will take a closer look at the features of OpenCL and CUDA to help you make an informed decision.
OpenCL, short for Open Computing Language, is an open standard for parallel programming of heterogeneous systems. It is a framework for writing programs that execute across different platforms, including CPUs, GPUs, and other processors. OpenCL enables developers to write code in a single language that can be executed on a wide range of devices, making it a versatile and flexible tool for high-performance computing.
CUDA, short for Compute Unified Device Architecture, is a parallel computing platform and programming model developed by NVIDIA. It is designed to harness the power of NVIDIA GPUs for general-purpose computing. CUDA provides a comprehensive development environment that includes a compiler, libraries, and tools for creating high-performance applications. It allows developers to write code in C, C++, or Fortran and run it on NVIDIA GPUs, making it a powerful tool for scientific computing, machine learning, and other applications that require high-performance computing.
How To Properly Use The Words In A Sentence
When discussing the differences between OpenCL and CUDA, it’s important to use the correct terminology in order to effectively communicate your ideas. Here are some tips on how to properly use the words OpenCL and CUDA in a sentence.
How To Use Opencl In A Sentence
OpenCL stands for Open Computing Language, and it is an open standard for parallel programming across CPUs, GPUs, and other processors. When discussing OpenCL in a sentence, it’s important to provide context and explain its purpose. Here are some examples:
- OpenCL is a powerful tool for accelerating scientific simulations on GPUs.
- Developers can use OpenCL to write code that runs on a variety of different hardware platforms.
- OpenCL provides a framework for parallel programming that is accessible to a wide range of developers.
Notice how each sentence provides a clear explanation of what OpenCL is and how it can be used. This helps to ensure that readers understand the context of the discussion.
How To Use Cuda In A Sentence
CUDA stands for Compute Unified Device Architecture, and it is a parallel computing platform and programming model developed by NVIDIA. When discussing CUDA in a sentence, it’s important to provide context and explain its purpose. Here are some examples:
- CUDA is a powerful tool for accelerating deep learning algorithms on NVIDIA GPUs.
- Developers can use CUDA to write code that takes advantage of the parallel processing power of NVIDIA GPUs.
- CUDA provides a flexible and scalable platform for parallel programming that is optimized for NVIDIA hardware.
Again, each sentence provides a clear explanation of what CUDA is and how it can be used. This helps to ensure that readers understand the context of the discussion and can follow along with the comparison between OpenCL and CUDA.
More Examples Of Opencl & Cuda Used In Sentences
In order to better understand the practical applications of OpenCL and CUDA, it can be helpful to see them used in context. Here are some examples of how these technologies might be used in a sentence:
Examples Of Using Opencl In A Sentence
- By using OpenCL, the software was able to take advantage of the GPU’s parallel processing power.
- OpenCL allows developers to write code that can run on a variety of different hardware platforms.
- The OpenCL framework provides a way to write code that can be executed on both CPUs and GPUs.
- Using OpenCL, the program was able to achieve significant speed improvements on certain types of calculations.
- OpenCL is often used in scientific computing applications to speed up simulations and data processing tasks.
- With OpenCL, developers can write code that can take advantage of both CPU and GPU resources simultaneously.
- The OpenCL standard was developed by a consortium of technology companies, including Apple, AMD, and NVIDIA.
- OpenCL provides a way for software developers to write code that can be easily ported between different hardware architectures.
- One advantage of using OpenCL is that it allows developers to write code that can be executed on a wide range of devices, from smartphones to supercomputers.
- OpenCL is often used in machine learning applications to speed up the training of neural networks.
Examples Of Using Cuda In A Sentence
- CUDA is a parallel computing platform developed by NVIDIA.
- By using CUDA, the program was able to take advantage of the GPU’s massive parallel processing power.
- CUDA allows developers to write code that can be executed on NVIDIA GPUs.
- Using CUDA, the software was able to achieve significant speed improvements on certain types of calculations.
- CUDA is often used in scientific computing applications to speed up simulations and data processing tasks.
- With CUDA, developers can write code that can take advantage of both CPU and GPU resources simultaneously.
- The CUDA programming model provides a way for developers to write code that can be easily parallelized.
- CUDA is often used in machine learning applications to speed up the training of neural networks.
- One advantage of using CUDA is that it allows developers to write code that can take advantage of the specific features of NVIDIA GPUs.
- CUDA is widely used in the gaming industry to create realistic graphics and physics simulations.
Common Mistakes To Avoid
When it comes to parallel computing, OpenCL and CUDA are two of the most popular programming languages. However, many people make the mistake of using them interchangeably or assuming that they are the same thing. Here are some common mistakes to avoid:
1. Assuming Opencl And Cuda Are Interchangeable
One of the biggest mistakes people make is assuming that OpenCL and CUDA are interchangeable. While both languages are used for parallel computing, they are not the same thing. OpenCL is an open standard that can be used on a variety of devices, including CPUs and GPUs from different vendors. On the other hand, CUDA is a proprietary language developed by NVIDIA specifically for their GPUs.
2. Not Considering Hardware Compatibility
Another mistake people make is not considering hardware compatibility. OpenCL is designed to work with a variety of devices, including CPUs, GPUs, and FPGAs from different vendors. CUDA, on the other hand, only works with NVIDIA GPUs. If you are developing software that needs to work on different hardware, OpenCL may be a better choice.
3. Using Cuda-specific Features In Opencl
Some people make the mistake of using CUDA-specific features in OpenCL. This can lead to compatibility issues and make it difficult to port your code to other devices. It’s important to stick to the OpenCL standard and avoid using features that are specific to CUDA.
4. Not Optimizing For The Target Device
Finally, many people make the mistake of not optimizing their code for the target device. Each device has its own strengths and weaknesses, and it’s important to take these into account when developing your code. For example, if you are developing for a GPU, you may want to use techniques like thread-level parallelism and shared memory to maximize performance.
Tips For Avoiding These Mistakes:
- Take the time to learn the differences between OpenCL and CUDA.
- Consider hardware compatibility when choosing a language.
- Stick to the OpenCL standard and avoid using CUDA-specific features.
- Optimize your code for the target device.
When it comes to choosing between OpenCL and CUDA, the context in which they are used plays a crucial role in making the decision. While both are parallel computing platforms that allow developers to accelerate their applications, the choice between the two depends on several factors.
Factors Affecting The Choice
Some of the factors that can affect the choice between OpenCL and CUDA include:
- Hardware: The hardware on which the application is running can play a significant role in the choice between OpenCL and CUDA. For instance, if the application is running on an NVIDIA GPU, then CUDA might be the better choice. On the other hand, if the application is running on an AMD GPU, then OpenCL might be the better choice.
- Application Domain: The domain of the application can also affect the choice between OpenCL and CUDA. For instance, if the application is in the field of machine learning, then CUDA might be the better choice as it has better support for deep learning frameworks like TensorFlow and PyTorch. On the other hand, if the application is in the field of scientific computing, then OpenCL might be the better choice as it has better support for scientific computing libraries like BLAS and FFTW.
- Development Environment: The development environment can also play a role in the choice between OpenCL and CUDA. For instance, if the developer is more familiar with the CUDA programming model, then CUDA might be the better choice. On the other hand, if the developer is more familiar with the OpenCL programming model, then OpenCL might be the better choice.
Examples Of Different Contexts
Let’s take a look at some examples of different contexts and how the choice between OpenCL and CUDA might change:
|Context||Choice between OpenCL and CUDA|
|Application is running on an NVIDIA GPU||CUDA might be the better choice|
|Application is running on an AMD GPU||OpenCL might be the better choice|
|Application is in the field of machine learning||CUDA might be the better choice|
|Application is in the field of scientific computing||OpenCL might be the better choice|
|Developer is more familiar with the CUDA programming model||CUDA might be the better choice|
|Developer is more familiar with the OpenCL programming model||OpenCL might be the better choice|
As you can see, the choice between OpenCL and CUDA can depend on several factors. It is important to carefully consider these factors before making a decision.
Exceptions To The Rules
While OpenCL and CUDA have their own specific use cases, there are certain instances where the rules for using them might not apply. It is important to identify these exceptions to ensure that the best tool is used for the job at hand.
Explanation And Examples
One exception to the rule is when the hardware being used is not compatible with either OpenCL or CUDA. In such cases, the choice of tool is limited to what is available and compatible with the hardware. For example, if the hardware being used is an AMD graphics card, then OpenCL would be the only option available.
Another exception is when the software being used is not compatible with either OpenCL or CUDA. In such cases, the choice of tool is again limited to what is available and compatible with the software. For example, if the software being used is designed to work only with CUDA, then OpenCL cannot be used.
Additionally, there may be cases where the specific task to be performed is better suited for one tool over the other, despite the general rule. For example, if the task involves a lot of integer operations, then OpenCL may be the better choice, even though CUDA is generally considered to be better for floating-point operations.
It is important to keep in mind that while OpenCL and CUDA are powerful tools, they are not always the best fit for every situation. Identifying exceptions to the rules and understanding when to use each tool is crucial for achieving the best results.
As with any skill, practice is key to improving your understanding and use of OpenCL and CUDA. Here are some practice exercises to help you hone your skills:
Exercise 1: Syntax
Fill in the blanks with the correct syntax for OpenCL or CUDA:
- ______ kernel void add(int a, int b, __global int* c)
- cudaMalloc(&______, size);
- ______ void __global__ saxpy(float a, float* x, float* y)
- ______ dim3 block(8, 8, 1);
Exercise 2: Performance
Given the following code snippets, which implementation will have better performance on a GPU? Explain why.
Answer: The error in this code is the use of the bitwise AND operator (&) instead of the logical AND operator (&&) in the if statement. The correct code should be:
After exploring the differences between OpenCL and CUDA, it is clear that both technologies have their strengths and weaknesses. OpenCL is a more versatile option that allows for cross-platform development and supports a wider range of devices. CUDA, on the other hand, offers superior performance on NVIDIA GPUs and has a more mature development ecosystem.
When deciding which technology to use, it is important to consider the specific needs and constraints of your project. If cross-platform compatibility is a priority, OpenCL may be the better choice. If performance on NVIDIA GPUs is critical, CUDA may be the way to go.
- OpenCL and CUDA are both technologies used for parallel computing on GPUs.
- OpenCL is more versatile and supports a wider range of devices, while CUDA offers superior performance on NVIDIA GPUs.
- The choice between OpenCL and CUDA depends on the specific needs and constraints of your project.
Regardless of which technology you choose, it is important to continue learning about grammar and language use. Effective communication is essential in any field, and improving your writing skills can help you convey your ideas more clearly and persuasively.
Shawn Manaher is the founder and CEO of The Content Authority. He’s one part content manager, one part writing ninja organizer, and two parts leader of top content creators. You don’t even want to know what he calls pancakes.