Working with Native Bitmap pixel buffers in Xamarin.Forms

I mentioned in my previous post that extracting pixel buffers from native Bitmap API’s can be quite tricky. In this post I would like to share the approach that I took for extracting native Bitmap pixel buffers into an collection of Xamarin.Forms.Colors objects so it can be used from a portable class library. I wrote and used a more complex version of the code mentioned in this post on my last project where I was working with image detection and color analysis for an app using Xamarin.Forms, in this project all my color analysis was done in a Portable Class Library using an abstraction over the native bitmap data.

BitmapData abstraction

In .NET you had access to an API called System.Drawing.Bitmap which encapsulates a low-level Windows API called Bitmap from GDI. The managed Bitmap class exposed a method called LockBits which in return gave you a BitmapData instance. BitmapData exposed information that allows you to manipulate the pixel buffer at a pointer level and is the fastest and recommended way to analyze and manipulate pixel information. I loved the BitmapData class but my portable class library implementation will not contain anything but a pixel buffer in ARGB and a Get/SetPixel(x,y, Color) method and a method for getting the average color of a certain area in the Bitmap to demonstrate what this can be used for

Here's the code

public class BitmapData
{
    public BitmapData(Size size, int[] pixelBuffer)
    {
        Size = size;
        PixelBuffer = pixelBuffer;
    }

    public int[] PixelBuffer { get; }

    public Size Size { get; }

    public Color GetPixel(Point point) => GetPixel(point.X, point.Y);

    public Color GetPixel(double x, double y) => Color.FromUint((uint)PixelBuffer[(int)x * (int)y]);

    public void SetPixel(Point point, Color color) => SetPixel((int)point.X, (int)point.Y, color);

    public void SetPixel(double x, double y, Color color) => PixelBuffer[(int)(x * y)] = (int)(color.A * byte.MaxValue) << 24 |
                                                                                        ((int)color.R * byte.MaxValue) << 16 |
                                                                                        ((int)color.G * byte.MaxValue) << 8 |
                                                                                        ((int)color.B * byte.MaxValue) << 0;

    public Color GetAverageColor(params Rectangle[] rectangles)
    {
        var colors = new List<Color>();
        foreach (var rectangle in rectangles)
            for (var y = rectangle.Y; y < rectangle.Y + rectangle.Height; y++)
                for (var x = (int)rectangle.X; x < (int)rectangle.X + (int)rectangle.Width; x++)
                    colors.Add(GetPixel(x, y));

        var red = (int)(colors.Average(c => c.R) * byte.MaxValue);
        var blue = (int)(colors.Average(c => c.G) * byte.MaxValue);
        var green = (int)(colors.Average(c => c.B) * byte.MaxValue);
        var alpha = (int)(colors.Average(c => c.A) * byte.MaxValue);
            
        return Color.FromRgba(red, blue, green, alpha);
    }
}

UIImage to BitmapData (iOS)

To get the pixel buffer from a UIImage instance we need to draw it to a new drawing surface by calling DrawImage on an CGBitmapContext instance. When we construct the drawing surface we specify the pixel format and provide a pointer or an array of bytes in which the data will be written to. We need to specify that the pixels will contain a byte for each component, 4 bytes per pixel, and that the byte order is 32-bit Big Endian. We can also specify whether we specify the alpha component is in the most or least significant bits of each pixel, but for this example I will put it in the end since when I was researching about this, most of the examples I found used the least significant bit to store the alpha component.

Here's the code

public BitmapData Convert(object nativeBitmap)
{
    var image = (UIImage)nativeBitmap;
    return new BitmapData(new Xamarin.Forms.Size(image.Size.Width, image.Size.Height), GetPixels(image));
}

private static int[] GetPixels(UIImage image)
{
    const int bytesPerPixel = 4;
    const int bitsPerComponent = 8;
    const CGBitmapFlags flags = CGBitmapFlags.ByteOrder32Big | CGBitmapFlags.PremultipliedLast;

    var width = (int)image.CGImage.Width;
    var height = (int)image.CGImage.Height;
    var bytesPerRow = bytesPerPixel * width;
    var buffer = new byte[bytesPerRow * height];
    var pixels = new int[width * height];

    var handle = GCHandle.Alloc(buffer);
    try
    {
        using (var colorSpace = CGColorSpace.CreateGenericRgb())
        using (var context = new CGBitmapContext(buffer, width, height, bitsPerComponent, bytesPerRow, colorSpace, flags))
            context.DrawImage(new RectangleF(0, 0, width, height), image.CGImage);

        for (var y = 0; y < height; y++)
        {
            var offset = y * width;
            for (var x = 0; x < width; x++)
            {
                var idx = bytesPerPixel * (offset + x);
                var r = buffer[idx + 0];
                var g = buffer[idx + 1];
                var b = buffer[idx + 2];
                var a = buffer[idx + 3];
                pixels[x * y] = a << 24 | r << 16 | g << 8 | b << 0;
            }
        }
    }
    finally
    {
        handle.Free();
    }

    return pixels;
}

Bitmap to BitmapData (Android)

This is pretty easy to do in Android as the Bitmap class exposes the GetPixels method to get the pixel buffer and the pixel information is conveniently stored in ARGB

Here's the code

public BitmapData Convert(object nativeBitmap)
{
    var bitmap = (Bitmap)nativeBitmap;
    var info = bitmap.GetBitmapInfo();
    var pixels = new int[info.Width * info.Height];
    bitmap.GetPixels(pixels, 0, (int)info.Width, 0, 0, (int)info.Width, (int)info.Height);
    return new BitmapData(new Xamarin.Forms.Size(info.Width, info.Height), pixels);
}

WriteableBitmap to BitmapData (Universal Windows Platform)

To do this using the Universal Windows Platform is a bit similar to iOS but is less complex. The WriteableBitmap class exposes a PixelBuffer as an IBuffer in which you can call the extension method ToArray() on to get an array of integers. The interesting part about the WriteableBitmap PixelBuffer is that it doesn't really say anywhere in the documentation (at least not directly) that the component order is BGRA, I only figured this out by reading the sample code provided in the WriteableBitmap documentation where it says in a code comment that WriteableBitmap uses BGRA format.

Here's the code

public BitmapData Convert(object nativeBitmap)
{
    var imageSource = (WriteableBitmap)nativeBitmap;
    var pixelData = GetPixelDataFromImage(imageSource).ToArray();
    return new BitmapData(new Size(imageSource.PixelWidth, imageSource.PixelHeight), pixelData);
}

private static IEnumerable<int> GetPixelDataFromImage(WriteableBitmap imageSource)
{
    const int bytesPerPixel = 4;
    var pixelHeight = imageSource.PixelHeight;
    var pixelWidth = imageSource.PixelWidth;
    var buffer = imageSource.PixelBuffer.ToArray();
    var pixels = new int[buffer.Length];

    for (var y = 0; y < pixelHeight; y++)
    {
        var offset = y * pixelWidth;
        for (var x = 0; x < pixelWidth; x++)
        {
            var idx = bytesPerPixel * (offset + x);
            var b = buffer[idx + 0];
            var g = buffer[idx + 1];
            var r = buffer[idx + 2];
            var a = buffer[idx + 3];
            pixels[x * y] = a << 24 | r << 16 | g << 8 | b << 0;
        }
    }

    return pixels;
}

I remember struggling quite a bit when I was figuring out what I just shared and I hope that some one out there might be able to make some good use of it.

Over 2 years of Xamarin

I have held radio silence for over 2 years because I am having a blast doing nothing but Xamarin based development. During this time I have been fully involved in building music apps that I actually use in my everyday life. I struggled with the pains and gained a love/hate relationship with the development tools I use. I work primarily on a Mac since the products we build only target iOS and Android and I adore Xamarin Studio, it’s super fast and simple and it reminds me of the good old bad days of Visual Studio from over a decade ago where it was pretty much just a big text editor with a few fancy features. Xamarin Studio of course has its ups and downs and sometimes it just doesn’t want to cooperate. Every time a new “stable” version is out I needed to make sure that I only update during non-busy days as I risk losing all productivity if I install an update with breaking changes, or an update that is just simply broken. Some updates do break stuff! Some breaking changes are minor and easy to fix and some are rather complex to work around the problem. I remember once, our builds broke because the command line build tooling somehow got deprecated suggesting us to move from MDTool to XBuild, although the change sounds trivial it costed me a full day of work to update all our build scripts and double check every single build configuration that they work as they should in our build servers

On top of having a full time position doing Xamarin development for a Music Services company, I also do a few side project. For these projects I use Xamarin.Forms as it shows a lot of promise. The early versions were terrible but they eventually got much better. For simple UI requirements its absolutely perfect. With Xamarin.Forms you can create a cross platform UI that you only write once using either a flavor of XAML or in code (using C# in my case). In Xamarin.Forms you can either take advantage of the sharable UI tooling so you only have to write the UI code once and have it rendered natively on iOS, Android, and Windows Phone, or you can create custom renderers that allow you fully control using the API’s of the platforms you target to implement the control or the entire screen. When you end up implementing the entire app using custom renderers then Xamarin.Forms starts making less sense for what you’re trying to build, but for Line-of-Business apps its perfect. I can really see a big future in Xamarin.Forms

Knowing and understanding the platform is always better than relying on cross platform tools. I have mostly been working on multi-platform solutions throughout my career and the experience I gained has been a great advantage for me as I understand how a lot of things work and why API’s do what they do. A good example of such cross platform issues is working with Bitmaps. In a nutshell, a bitmap is an array of 32-bit integers containing the Red-Blue-Green-Alpha pixel information. If you do cross platform development and image processing you will need to ensure that the order of the pixel information bytes are uniform so that the algorithm you work with will work across all the targetted platforms. This might come as a surprise to some but pixel information is stored differently for every platform’s Bitmap API, for example, iOS uses Red-Blue-Green-Alpha, Android uses Alpha-Red-Blue-Green, and Windows uses Blue-Green-Red-Alpha. All these platforms have API’s that allow you retrieve the pixel buffer (the array of 32-bit integers) in a single call avoiding calling using slow API’s like GetPixel(x,y) hundreds or even thousands of times. Too make things even more fun the Xamarin.Forms.Color structure has a static helper method called FromUint(uint argb) but the documentation (at the current time of writing) describes the method as Returns a new Color with the requested RGBA value and the method parameter is described as A uint that represents the ARGB value of the color and the return value is described as An RGBA color. So that was a bit confusing! Eventually Xamarin will correct these small details or developers will just have to live with them

I just ended a side project and will probably not take on any new side projects for a while so I can spend more time learning new awesome stuff, or in some case understanding old awesome stuff. I have a few hobby projects that I have been intending to open source for a while so I’ll probably finish those up and finally publish them. I had some a bit success with some open source projects such as SQL Compact Query Analyzer (50k downloads), SQL Compact Code Generator (15k downloads), and ResW File Code Generator (4k downloads), all which had very constructive and positive reviews. I always enjoyed building developer tools and I hope to get more of that done in the nearest future