After some hunting for memory leaks, it became obvious that some rewrite of XS will be necessary. Since this will be a big undertaking I asked for some help from chromatic and garu. We came up with the following plan for v2.4.
Currently all XS is mapped to the SDL_perl module. This does not reflect the modular nature of the sdl libs. So the plan is to gradually move SDL structs and their respective functions to their correct namespaces. We will start with SDL::Rect as garu has added many tests for it. This way the build system will be a lot easier to hack since Module::Build was made for one XS per module. Moreover we will move the Build utilities to Alien::SDL, making it even simpler. The main idea is to adhere to single responsibility principle and encapsulation.
Obviously this will take some time and effort, but it will pay off in the long run. Any help will be greatly appreciated.