If you don’t know what is traces based rendering regression testing<\/em>, read the appendix<\/a> before continuing.<\/p>\n\n\n\n The Mesa<\/a> community has witnessed an explosion of the Continuous Integration<\/em> interest in the last two years.<\/p>\n\n\n\n In addition to checking the proper building of the project, integrating the testing of its functional correctness has become a priority. The user space graphics drivers exhibit a wide variety of types of tests and test suites. One kind of those tests are the traces based rendering regression testing.<\/p>\n\n\n\n The public effort to add this kind of tests into Mesa’s CI started with this mail<\/a> from Alexandros Frantzis<\/a>.<\/p>\n\n\n\n At some point, we had support for replaying OpenGL, Vulkan and D3D11 traces using apitrace<\/a>, RenderDoc<\/a> and GFXReconstruct<\/a> with the in-tree tool tracie<\/em>. However, it was a very custom solution made to the needs of Mesa so I proposed to move this codebase and integrate it into the piglit<\/a> test suite. It was a natural step forward.<\/p>\n\n\n\n This is how replayer<\/a> was born into piglit.<\/p>\n\n\n\n The first step to test a trace is, actually, obtaining a trace. I won’t go into the details about how to create one from scratch. The process is well documented on each of the tools listed above. However, the Mesa community has been collecting publicly distributable traces for a while and placing them in traces-db<\/a> whose CI is copying them to Freedesktop.org’s MinIO instance<\/a>.<\/p>\n\n\n\n To make things simple, once we have built and installed piglit, if we would like to test an apitrace created OpenGL trace, we can download from there with:<\/p>\n\n\n\n The parameters are self explanatory. The downloaded trace will now exist at The next step will be to dump an image from the trace. Since it is a The dumped PNG image will be at Dumping from a trace may result in a range of different possible images. One example is when the trace makes use of uninitialized values, leading to undefined behaviors.<\/p>\n\n\n\n However, since the original aim was performing pre-merge<\/strong> rendering regression testing in Mesa’s CI, the idea is that replaying any of the provided traces would be quick and the dumped image will be consistent. In other words, if we would dump several times the same frame of a trace with the same GFX stack, the image will always be the same.<\/p>\n\n\n\n With this precondition, we can test whether 2 different images are the same just by doing a hash of its content. replayer can obtain the hash for the generated dumped image:<\/p>\n\n\n\n Now, if we would build a different commit of Mesa, we could check the generated image at this new point against the previously generated reference image<\/strong>. If everything goes well, we will see something like:<\/p>\n\n\n\n replayer<\/em>‘s compare<\/em> subcommand is the one spitting a piglit formatted test expectations output.<\/p>\n\n\n\n We can make the whole process way simpler by passing the replayer a YAML tests list file. For example:<\/p>\n\n\n\n replayer<\/em> features also the query<\/em> subcommand, which is just a helper to read the YAML files with the tests configuration.<\/p>\n\n\n\n Testing the other kind of supported 3D traces doesn’t change much from what’s shown here. Just make sure to have the needed tools installed: RenderDoc, GFXReconstruct, the VK_LAYER_LUNARG_screenshot<\/a> layer, Wine<\/a> and DXVK<\/a>. A good reference for building, installing and configuring these tools are Mesa’s GL<\/a> and VK<\/a> test containers building scripts.<\/p>\n\n\n\n replayer<\/em> also accepts several configurations to tweak how to behave and where to find the actual tracing tools needed for replaying the different types of traces. Make sure to check the replay<\/em> section in piglit’s configuration example file<\/a>.<\/p>\n\n\n\n replayer<\/em>‘s README.md<\/a> file is also a good read for further information.<\/p>\n\n\n\n replayer<\/em> is a test runner in a similar fashion to shader_runner<\/em> or glslparsertest<\/em>. We are now missing how does it integrate so we can do piglit runs<\/em> which will produce piglit formatted results.<\/p>\n\n\n\n This is done through the replay<\/a> test profile.<\/p>\n\n\n\n This profile needs a couple configuration values. Easiest is just to set the The following example features a similar run to the one done above<\/a> invoking directly replayer but with piglit integration, providing formatted results:<\/p>\n\n\n\n We can create some summary based on the results:<\/p>\n\n\n\n Creating an HTML summary may be also interesting, specially when finding failures!<\/p>\n\n\n\n Thanks a lot to the whole Mesa community for helping with the creation of this tool. Alexandros Frantzis, Rohan Garg<\/a> and Tomeu Vizoso<\/a> did a lot of the initial development for the in-tree tracie<\/em> tool while Dylan Baker was very patient while reviewing my patches for the piglit integration.<\/p>\n\n\n\n Finally, thanks to Igalia<\/a> for allowing me to work in this.<\/p>\n\n\n\n In 3D computer graphics we say \u00abtraces\u00bb<\/em>, for short, to name the files generated by 3D APIs capturing tools which store not only the calls to the specific 3D API but also the internal state of the 3D program during the capturing process: shaders, textures, buffers, etc.<\/p>\n\n\n\n Being able to \u00abrecord\u00bb the execution of a 3D program is very useful. Usually, it will allow us to replay the execution without the need of the original program from which we generated the trace, it will also allow in-depth analysis for debugging and performance optimization, it’s a very good solution for sharing with other developers, and, in some cases, will allow us to check how the replay will happen with different GPUs.<\/p>\n\n\n\n In this post, however, I focus in a specific usage: rendering regression testing<\/strong>.<\/p>\n\n\n\n When doing a regression test what we would do is compare a specific metric obtained by replaying the trace with a specific version of the GFX software stack against the same metric obtained from a different version of the GFX stack. If the value of the metric changes we have found a regression (or an improvement!).<\/p>\n\n\n\n To make things simpler, we would like to check changes happening just in one of the many elements of the software stack. The most relevant component is the user space driver<\/em>. In particular, I care about the Mesa drivers and the GNU\/Linux stack.<\/p>\n\n\n\n Mainly, there are two kinds of regression testing we can do with a trace: performance<\/em> or rendering regression testing<\/em>. When doing a performance one, the checked metric(s) usually are in terms of speed or memory usage. In the case of the rendering ones what we would do is comparing the rendered output at one (or many) point during the trace replay. This output, a bitmap image, is the metric that we will compare in between two different points of the Mesa driver. If the images differ, we may have found a regression; artifacts, improper colors, etc, or an enhancement, if the reference image is the one featuring any of these problems.<\/p>\n","protected":false},"excerpt":{"rendered":" If you don’t know what is traces based rendering regression testing, read the appendix before continuing. The Mesa community has witnessed an explosion of the Continuous Integration interest in the last two years. In addition to checking the proper building … Sigue leyendo
\n\n\n\nreplayer<\/h2>\n\n\n\n
$ replayer.py download \\\n \t --download-url https:\/\/minio-packet.freedesktop.org\/mesa-tracie-public\/ \\\n \t --db-path .\/traces-db \\\n \t --force-download \\\n \t glxgears\/glxgears-2.trace<\/code><\/pre>\n\n\n\n.\/traces-db\/glxgears\/glxgears-2.trace<\/code>.<\/p>\n\n\n\n.trace<\/code> file we will need to have apitrace installed in the system. If we do not specify the call(s) from which to dump the image(s), we will just get the last frame of the trace:<\/p>\n\n\n\n$ replayer.py dump .\/traces-db\/glxgears\/glxgears-2.trace<\/code><\/pre>\n\n\n\n.\/results\/glxgears-2.trace-0000001413.png<\/code>. Notice, the number suffix is the snapshot id from the trace.<\/p>\n\n\n\n$ replayer.py checksum .\/results\/glxgears-2.trace-0000001413.png \nf8eba0fec6e3e0af9cb09844bc73bdc8<\/code><\/pre>\n\n\n\n$ replayer.py compare trace \\\n \t --download-url https:\/\/minio-packet.freedesktop.org\/mesa-tracie-public\/ \\\n \t --device-name gl-vmware-llvmpipe \\\n \t --db-path .\/traces-db \\\n \t --keep-image \\\n \t glxgears\/glxgears-2.trace f8eba0fec6e3e0af9cb09844bc73bdc8\n[dump_trace_images] Info: Dumping trace .\/traces-db\/glxgears\/glxgears-2.trace...\n[dump_trace_images] Running: apitrace dump --calls=frame .\/traces-db\/glxgears\/glxgears-2.trace\n\/\/ process.name = \"\/usr\/bin\/glxgears\"\n1384 glXSwapBuffers(dpy = 0x56060e921f80, drawable = 31457282)\n\n1413 glXSwapBuffers(dpy = 0x56060e921f80, drawable = 31457282)\n\nerror: drawable failed to resize: expected 1515x843, got 300x300\n[dump_trace_images] Running: eglretrace --headless --snapshot=1413 --snapshot-prefix=.\/results\/trace\/gl-vmware-llvmpipe\/glxgears\/glxgears-2.trace- .\/blog-traces-db\/glxgears\/glxgears-2.trace\nWrote .\/results\/trace\/gl-vmware-llvmpipe\/glxgears\/glxgears-2.trace-0000001413.png\n\nOK\n[check_image]\n actual: f8eba0fec6e3e0af9cb09844bc73bdc8\n expected: f8eba0fec6e3e0af9cb09844bc73bdc8\n[check_image] Images match for:\n glxgears\/glxgears-2.trace\n\nPIGLIT: {\"images\": [{\"image_desc\": \"glxgears\/glxgears-2.trace\", \"image_ref\": \"f8eba0fec6e3e0af9cb09844bc73bdc8.png\", \"image_render\": \".\/results\/trace\/gl-vmware-llvmpipe\/glxgears\/glxgears-2.trace-0000001413-f8eba0fec6e3e0af9cb09844bc73bdc8.png\"}], \"result\": \"pass\"}<\/code><\/pre>\n\n\n\nPutting everything together<\/h3>\n\n\n\n
$ cat testing-traces.yml\ntraces-db:\n download-url: https:\/\/minio-packet.freedesktop.org\/mesa-tracie-public\/\n\ntraces:\n - path: gputest\/triangle.trace\n expectations:\n - device: gl-vmware-llvmpipe\n checksum: c8848dec77ee0c55292417f54c0a1a49\n - path: glxgears\/glxgears-2.trace\n expectations:\n - device: gl-vmware-llvmpipe\n checksum: f53ac20e17da91c0359c31f2fa3f401e\n$ replayer.py compare yaml \\\n \t --device-name gl-vmware-llvmpipe \\\n \t --yaml-file testing-traces.yml \n[check_image] Downloading file gputest\/triangle.trace took 5s.\n[dump_trace_images] Info: Dumping trace .\/replayer-db\/gputest\/triangle.trace...\n[dump_trace_images] Running: apitrace dump --calls=frame .\/replayer-db\/gputest\/triangle.trace\n\/\/ process.name = \"\/home\/anholt\/GpuTest_Linux_x64_0.7.0\/GpuTest\"\n397 glXSwapBuffers(dpy = 0x7f0ad0005a90, drawable = 56623106)\n\n510 glXSwapBuffers(dpy = 0x7f0ad0005a90, drawable = 56623106)\n\n\n\/home\/anholt\/GpuTest_Linux_x64_0.7.0\/GpuTest\n[dump_trace_images] Running: eglretrace --headless --snapshot=510 --snapshot-prefix=.\/results\/trace\/gl-vmware-llvmpipe\/gputest\/triangle.trace- .\/replayer-db\/gputest\/triangle.trace\nWrote .\/results\/trace\/gl-vmware-llvmpipe\/gputest\/triangle.trace-0000000510.png\n\nOK\n[check_image]\n actual: c8848dec77ee0c55292417f54c0a1a49\n expected: c8848dec77ee0c55292417f54c0a1a49\n[check_image] Images match for:\n gputest\/triangle.trace\n\n[check_image] Downloading file glxgears\/glxgears-2.trace took 5s.\n[dump_trace_images] Info: Dumping trace .\/replayer-db\/glxgears\/glxgears-2.trace...\n[dump_trace_images] Running: apitrace dump --calls=frame .\/replayer-db\/glxgears\/glxgears-2.trace\n\/\/ process.name = \"\/usr\/bin\/glxgears\"\n1384 glXSwapBuffers(dpy = 0x56060e921f80, drawable = 31457282)\n\n1413 glXSwapBuffers(dpy = 0x56060e921f80, drawable = 31457282)\n\n\n\/usr\/bin\/glxgears\nerror: drawable failed to resize: expected 1515x843, got 300x300\n[dump_trace_images] Running: eglretrace --headless --snapshot=1413 --snapshot-prefix=.\/results\/trace\/gl-vmware-llvmpipe\/glxgears\/glxgears-2.trace- .\/replayer-db\/glxgears\/glxgears-2.trace\nWrote .\/results\/trace\/gl-vmware-llvmpipe\/glxgears\/glxgears-2.trace-0000001413.png\n\nOK\n[check_image]\n actual: f8eba0fec6e3e0af9cb09844bc73bdc8\n expected: f8eba0fec6e3e0af9cb09844bc73bdc8\n[check_image] Images match for:\n glxgears\/glxgears-2.trace<\/code><\/pre>\n\n\n\npiglit<\/h2>\n\n\n\n
PIGLIT_REPLAY_DESCRIPTION_FILE<\/code> and PIGLIT_REPLAY_DEVICE_NAME<\/code> env variables. They are self explanatory, but make sure to check the documentation<\/a> for this and other configuration options for this profile.<\/p>\n\n\n\n$ PIGLIT_REPLAY_DESCRIPTION_FILE=testing-traces.yml PIGLIT_REPLAY_DEVICE_NAME=gl-vmware-llvmpipe piglit run replay -n replay-example replay-results\n[2\/2] pass: 2 \nThank you for running Piglit!\nResults have been written to replay-results<\/code><\/pre>\n\n\n\n# piglit summary console replay-results\/\ntrace\/gl-vmware-llvmpipe\/glxgears\/glxgears-2.trace: pass\ntrace\/gl-vmware-llvmpipe\/gputest\/triangle.trace: pass\nsummary:\n name: replay-example\n ---- --------------\n pass: 2\n fail: 0\n crash: 0\n skip: 0\n timeout: 0\n warn: 0\n incomplete: 0\n dmesg-warn: 0\n dmesg-fail: 0\n changes: 0\n fixes: 0\nregressions: 0\n total: 2\n time: 00:00:00<\/code><\/pre>\n\n\n\nWishlist<\/h2>\n\n\n\n
\n\n\n\n
\n\n\n\nAppendix<\/h2>\n\n\n\n