Experiments in PSK-31 Synthesis

PSK-31 is a narrow-bandwidth digital mode which encodes text as an audio tone that varies phase at a known rate. To learn more about this digital mode and solve a challenging programming problem, I’m going to write a PSK-31 encoder and decoder from scratch using the C# programming language. All code created for this project is open-source, available from my PSK Experiments GitHub Repository, and released under the permissive MIT license. This page documents my progress and notes things I learn along the way.

Encoding Bits as Phase Shifts

• PSK31 messages have a continuous carrier tone

• Symbols are represented by “symbols”, each 1/31.25 seconds long

• If a symbol changes phase from its previous symbol it is a 0, otherwise it is a 1

Amplitude Modulation Silences Phase Transitions

Although a continuous phase-shifting tone of constant amplitude can successfully transmit PSK31 data, the abrupt phase transitions will cause splatter. If you transmit this you will be heard, but those trying to communicate using adjacent frequencies will be highly disappointed.

Hard phase transitions (splatter)	Soft phase transitions (cleaner)

To reduce spectral artifacts that result from abruptly changing phase, phase transitions are silenced by shaping the waveform envelope as a sine wave so it is silent at the transition. This way the maximum rate of phase shifts is a sine wave with a period of half the baud rate. This is why the opening of a PSK31 message (a series of logical 0 bits) sounds like two tones: It’s the carrier sine wave with an envelope shaped like a sine wave with a period of 31.25/2 Hz. These two tones separated by approximately 15 Hz are visible in the spectrogram (waterfall).

Encoding Text as Bits

Unlike ASCII (8 bits per character) and RTTY (5 bits per character), BPSK uses Varicode (1-10 bits per character) to encode text. Consecutive zeros 00 separate characters, so character codes must not contain 00 and they must start ane end with a 1 bit. Messages are flanked by a preamble (repeated 0 bits) and a postamble (repeated 1 bits).

How to Generate a PSK Waveform

Now that we’ve covered the major steps of PSK31 message composition and modulation, let’s go through the steps ot generate a PSK31 message in code.

Step 1: Convert a Message to Varicode

Here’s the gist of how I store my varicode table in code. Note that the struct has an additional Description field which is useful for decoding and debugging.

public struct VaricodeSymbol
{
    public readonly string Symbol;
    public string BitString;
    public int[] Bits;
    public readonly string Description;

    public VaricodeSymbol(string symbol, string bitString, string? description = null)
    {
        Symbol = symbol;
        BitString = bitString;
        Bits = bitString.ToCharArray().Select(x => x == '1' ? 1 : 0).ToArray();
        Description = description ?? string.Empty;
    }
}

static VaricodeSymbol[] GetAllSymbols() => new VaricodeSymbol[]
{
    new("NUL", "1010101011", "Null character"),
    new("LF", "11101", "Line feed"),
    new("CR", "11111", "Carriage return"),
    new("SP", "1", "Space"),
    new("a", "1011"),
    new("b", "1011111"),
    new("c", "101111"),
    // etc...
};

I won’t show how I do the message-to-varicode lookup, but it’s trivial. Here’s the final function I use to generate Varicode bits from a string:

static int[] GetVaricodeBits(string message)
{
    List<int> bits = new();

    // add a preamble of repeated zeros
    for (int i=0; i<20; i++)
        bits.Add(0);

    // encode each character of a message
    foreach (char character in message)
    {
        VaricodeSymbol symbol = Lookup(character);
        bits.AddRange(symbol.Bits);
        bits.AddRange(CharacterSeparator);
    }

    // add a postamble of repeated ones
    for (int i=0; i<20; i++)
        bits.Add(1);

    return bits.ToArray();
}

Step 2: Determine Phase Shifts

Now that we have our Varicode bits, we need to generate an array to indicate phase transitions. A transition occurs every time a bit changes value form the previous bit. This code returns phase as an array of double given the bits from a Varicode message.

public static double[] GetPhaseShifts(int[] bits, double phase1 = 0, double phase2 = Math.PI)
{
    double[] phases = new double[bits.Length];
    for (int i = 0; i < bits.Length; i++)
    {
        double previousPhase = i > 0 ? phases[i - 1] : phase1;
        double oppositePhase = previousPhase == phase1 ? phase2 : phase1;
        phases[i] = bits[i] == 1 ? previousPhase : oppositePhase;
    }
    return phases;
}

Step 3: Generate the Waveform

These constants will be used to define the shape of the waveform:

public const int SampleRate = 8000;
public const double Frequency = 1000;
public const double BaudRate = 31.25;

This minimal code generates a decipherable PSK-31 message, but it does not silence the phase transitions so it produces a lot of splatter. This function must be refined to shape the waveform such that phase transitions are silenced.

public double[] GetWaveformBPSK(double[] phases)
{
    int totalSamples = (int)(phases.Length * SampleRate / BaudRate);
    double[] wave = new double[totalSamples];
    for (int i = 0; i < wave.Length; i++)
    {
        double time = (double)i / SampleRate;
        int frame = (int)(time * BaudRate);
        double phaseShift = phases[frame];
        wave[i] = Math.Cos(2 * Math.PI * Frequency * time + phaseShift);
    }
    return wave;
}

Step 4: Generate the Waveform with Amplitude Modulation

This is the same function as above, but with extra logic for amplitude-modulating the waveform in the shape of a sine wave to silence phase transitions.

public double[] GetWaveformBPSK(double[] phases)
{
    int baudSamples = (int)(SampleRate / BaudRate);
    double samplesPerBit = SampleRate / BaudRate;
    int totalSamples = (int)(phases.Length * SampleRate / BaudRate);
    double[] wave = new double[totalSamples];

    // create the amplitude envelope sized for a single bit
    double[] envelope = new double[(int)samplesPerBit];
    for (int i = 0; i < envelope.Length; i++)
        envelope[i] = Math.Sin((i + .5) * Math.PI / envelope.Length);

    for (int i = 0; i < wave.Length; i++)
    {
        // phase modulated carrier
        double time = (double)i / SampleRate;
        int frame = (int)(time * BaudRate);
        double phaseShift = phases[frame];
        wave[i] = Math.Cos(2 * Math.PI * Frequency * time + phaseShift);

        // envelope at phase transitions
        int firstSample = (int)(frame * SampleRate / BaudRate);
        int distanceFromFrameStart = i - firstSample;
        int distanceFromFrameEnd = baudSamples - distanceFromFrameStart + 1;
        bool isFirstHalfOfFrame = distanceFromFrameStart < distanceFromFrameEnd;
        bool samePhaseAsLast = frame == 0 ? false : phases[frame - 1] == phases[frame];
        bool samePhaseAsNext = frame == phases.Length - 1 ? false : phases[frame + 1] == phases[frame];
        bool rampUp = isFirstHalfOfFrame && !samePhaseAsLast;
        bool rampDown = !isFirstHalfOfFrame && !samePhaseAsNext;

        if (rampUp)
            wave[i] *= envelope[distanceFromFrameStart];

        if (rampDown)
            wave[i] *= envelope[distanceFromFrameEnd];
    }

    return wave;
}

PSK31 Encoder Program

I wrapped the functionality above in a Windows Forms GUI that allows the user to type a message, specify frequency, baud rate, and whether or not to refine the envelope to reduce splatter, then either play or save the result. An interactive ScottPlot Chart allows the user to inspect the waveform.

• Download PSK31 Encoder: PSK31-encoder.zip

• PSK31 Encoder Source Code: PSK Experiments on GitHub

Sample PSK-31 Transmissions

These audio files encode the text The Quick Brown Fox Jumped Over The Lazy Dog 1234567890 Times! in 1kHz BPSK at various baud rates.

• PSK-31: dog31.wav
• PSK-63: dog63.wav
• PSK-125: dog125.wav
• PSK-256: dog256.wav

Non-Standard Baud Rates

Let’s see what PSK-3 sounds like. This mode encodes data at a rate of 3 bits per second. Note that Varicode characters may require up to ten bits, so this is pretty slow. On the other hand the side tones are closer to the carrier and the total bandwidth is much smaller. The message here has been shortened to just my callsign, AJ4VD.

• PSK-3: psk3.wav

Encode PSK-31 In Your Browser

After implementing the C# encoder described above I created a JavaScript version (as per Atwood’s Law).

• Try it on your phone or computer! Launch PskJS

Decoding PSK-31

Considering all the steps for encoding PSK-31 transmissions are already described on this page, it doesn’t require too much additional effort to create a decoder using basic software techniques. Once symbol phases are detected it’s easy to work backwards: detect phase transitions (logical 0s) or repeats (logical 1s), treat consecutive zeros as a character separator, then look-up characters according to the Varicode table. The tricky bit is analyzing the source audio to generate the array of phase offsets.

The simplest way to decode PSK-31 transmissions leans on the fact that we already know the baud rate: 31.25 symbols per second, or one symbol every 256 samples at 8kHz sample rate. The audio signal can be segregated into many 256-sample bins, and processed by FFT. Once the center frequency is determined, the FFT power at this frequency can be calculated for each bin. Signal offset can be adjusted to minimize the imaginary component of the FFTs at the carrier frequency, then the real component will be strongly positive or negative, allowing phase transitions to be easily detected.

There are more advanced techniques to improve BPSK decoding, such as continuously adjusting frequency and phase alignment (synchronization). A Costas loop can help lock onto the carrier frequency while preserving its phase. Visit Simple BPSK31 Decoding with Python for an excellent demonstration of how to decode BPSK31 using these advanced techniques.

A crude C# implementation of a BPSK decoded is available on GitHub in the PSK Experiments repository

Encoding PSK-31 in Hardware

Since BPSK is just a carrier that applies periodic 180º phase-shifts, it’s easy to generate in hardware by directly modulating the signal source. A good example of this is KA7OEI’s PSK31 transmitter which feeds the output of an oscillator through an even or odd number of NAND gates (from a 74HC00) to produce two signals of opposite phase.

Quadrature Phase Shift Keying (QPSK)

Unlike the 0º and 180º phases of binary phase shift keying (BPSK), quadrature phase shift keying (QPSK) encodes extra data into each symbol by uses a larger number of phases. When QPSK-31 is used in amateur radio these extra bits aren’t used to send messages faster but instead send them more reliably using convolutional coding and error correction. These additional features come at a cost (an extra 3 dB SNR is required), and in practice QPSK is not used as much by amateur radio operators.

QPSK encoding/decoding and convolutional encoding/decoding are outside the scope of this page, but excellent information exists on the Wikipedia: QPSK and in the US Naval Academy’s EC314 Lesson 23: Digital Modulation document.

PSK-31 in 2022

After all that, it turns out PSK-31 isn’t that popular anymore. These days it seems the FT-8 digital mode with WSJT-X software is orders of magnitude more popular ??

Resources

• PSK Experiments (GitHub) - Source code for the project shown on this page

• Software: digipan - A Freeware Program for PSK31 and PSK63

• Software: fldigi - Supports PSK31 and other digital modes

• Software: WinPSK - open source PSK31 software for Windows

• Software: PSKCore DLL - A Windows DLL that can be included in other software to add support for PSK31

• Software: jacobwgillespie/psk31 - Example PSK31 message generate using JavaScript

• Digital Modulation (US Naval Academy, EC314 Lesson 23) - A good description of quadrature PSK and higher order phase-shift encoding.

• PSK-31 Specification (ARRL) - theory, varicode table, and convolutional code table.

• PSK31 Description by G3PLX is the original / official description of the digital mode.

• PSK31: A New Radio-Teletype Mode (1999) by Peter Martinez, G3PLX

• PSK31 The Easy Way (1999) by Alan Gibbs, VK6PG

• Wikipedia: Varicode includes a table of all symbols

• Wikipedia: QPSK

• PSK31 Fundamentals and PSK31 Setup by Peter Martinez, G3PLX

• Varicode - West Virginia State University CS240

• Introduction to PSK31 by engineering students at Walla Walla University

• GNURadio PSK31 Decoder by VA7STH

• Simple BPSK31 - a fantastic Jupyter notebook demonstrating BPSK decoding with Python

• PySDR: Digital Modulation - a summary of signal modulation types

• A PIC-Based PSK31 exciter using a Balanced Modulator by Clint Turner, KA7OEI

Resample Time Series Data using Cubic Spline Interpolation

Cubic spline interpolation can be used to modify the sample rate of time series data. This page describes how I achieve signal resampling with spline interpolation in pure C# without any external dependencies. This technique can be used to:

• Convert unevenly-sampled data to a series of values with a fixed sample rate

• Convert time series data from one sample rate to another sample rate

• Fill-in missing values from a collection of measurements

Simulating Data Samples

To simulate unevenly-sampled data I create a theoretical signal then sample it at 20 random time points.

// this function represents the signal being measured
static double f(double x) => Math.Sin(x * 10) + Math.Sin(x * 13f);

// randomly sample values from 20 time points
Random rand = new(123);
double[] sampleXs = Enumerable.Range(0, 20)
    .Select(x => rand.NextDouble())
    .OrderBy(x => x)
    .ToArray();
double[] sampleYs = sampleXs.Select(x => f(x)).ToArray();

Resample for Evenly-Spaced Data

I then generate an interpolated spline using my sampled data points as the input.

• I can control the sample rate by defining the number of points generated in the output signal.

• Full source code is at the bottom of this article.

(double[] xs, double[] ys) = Cubic.Interpolate1D(sampleXs, sampleYs, count: 50);

The generated points line-up perfectly with the sampled data.

There is slight deviation from the theoretical signal (and it’s larger where there is more missing data) but this is an unsurprising result considering the original samples had large gaps of missing data.

Source Code

Interpolation.cs

public static class Interpolation
{
    public static (double[] xs, double[] ys) Interpolate1D(double[] xs, double[] ys, int count)
    {
        if (xs is null || ys is null || xs.Length != ys.Length)
            throw new ArgumentException($"{nameof(xs)} and {nameof(ys)} must have same length");

        int inputPointCount = xs.Length;
        double[] inputDistances = new double[inputPointCount];
        for (int i = 1; i < inputPointCount; i++)
            inputDistances[i] = inputDistances[i - 1] + xs[i] - xs[i - 1];

        double meanDistance = inputDistances.Last() / (count - 1);
        double[] evenDistances = Enumerable.Range(0, count).Select(x => x * meanDistance).ToArray();
        double[] xsOut = Interpolate(inputDistances, xs, evenDistances);
        double[] ysOut = Interpolate(inputDistances, ys, evenDistances);
        return (xsOut, ysOut);
    }

    private static double[] Interpolate(double[] xOrig, double[] yOrig, double[] xInterp)
    {
        (double[] a, double[] b) = FitMatrix(xOrig, yOrig);

        double[] yInterp = new double[xInterp.Length];
        for (int i = 0; i < yInterp.Length; i++)
        {
            int j;
            for (j = 0; j < xOrig.Length - 2; j++)
                if (xInterp[i] <= xOrig[j + 1])
                    break;

            double dx = xOrig[j + 1] - xOrig[j];
            double t = (xInterp[i] - xOrig[j]) / dx;
            double y = (1 - t) * yOrig[j] + t * yOrig[j + 1] +
                t * (1 - t) * (a[j] * (1 - t) + b[j] * t);
            yInterp[i] = y;
        }

        return yInterp;
    }

    private static (double[] a, double[] b) FitMatrix(double[] x, double[] y)
    {
        int n = x.Length;
        double[] a = new double[n - 1];
        double[] b = new double[n - 1];
        double[] r = new double[n];
        double[] A = new double[n];
        double[] B = new double[n];
        double[] C = new double[n];

        double dx1, dx2, dy1, dy2;

        dx1 = x[1] - x[0];
        C[0] = 1.0f / dx1;
        B[0] = 2.0f * C[0];
        r[0] = 3 * (y[1] - y[0]) / (dx1 * dx1);

        for (int i = 1; i < n - 1; i++)
        {
            dx1 = x[i] - x[i - 1];
            dx2 = x[i + 1] - x[i];
            A[i] = 1.0f / dx1;
            C[i] = 1.0f / dx2;
            B[i] = 2.0f * (A[i] + C[i]);
            dy1 = y[i] - y[i - 1];
            dy2 = y[i + 1] - y[i];
            r[i] = 3 * (dy1 / (dx1 * dx1) + dy2 / (dx2 * dx2));
        }

        dx1 = x[n - 1] - x[n - 2];
        dy1 = y[n - 1] - y[n - 2];
        A[n - 1] = 1.0f / dx1;
        B[n - 1] = 2.0f * A[n - 1];
        r[n - 1] = 3 * (dy1 / (dx1 * dx1));

        double[] cPrime = new double[n];
        cPrime[0] = C[0] / B[0];
        for (int i = 1; i < n; i++)
            cPrime[i] = C[i] / (B[i] - cPrime[i - 1] * A[i]);

        double[] dPrime = new double[n];
        dPrime[0] = r[0] / B[0];
        for (int i = 1; i < n; i++)
            dPrime[i] = (r[i] - dPrime[i - 1] * A[i]) / (B[i] - cPrime[i - 1] * A[i]);

        double[] k = new double[n];
        k[n - 1] = dPrime[n - 1];
        for (int i = n - 2; i >= 0; i--)
            k[i] = dPrime[i] - cPrime[i] * k[i + 1];

        for (int i = 1; i < n; i++)
        {
            dx1 = x[i] - x[i - 1];
            dy1 = y[i] - y[i - 1];
            a[i - 1] = k[i - 1] * dx1 - dy1;
            b[i - 1] = -k[i] * dx1 + dy1;
        }

        return (a, b);
    }
}

Program.cs

This is the source code I used to generate the figures on this page.

Plots were generated using ScottPlot.NET.

// this function represents the signal being measured
static double f(double x) => Math.Sin(x * 10) + Math.Sin(x * 13f);

// create points representing randomly sampled time points of a smooth curve
Random rand = new(123);
double[] sampleXs = Enumerable.Range(0, 20)
    .Select(x => rand.NextDouble())
    .OrderBy(x => x)
    .ToArray();
double[] sampleYs = sampleXs.Select(x => f(x)).ToArray();

// use 1D interpolation to create an evenly sampled curve from unevenly sampled data
(double[] xs, double[] ys) = Interpolation.Interpolate1D(sampleXs, sampleYs, count: 50);

var plt = new ScottPlot.Plot(600, 400);

double[] theoreticalXs = ScottPlot.DataGen.Range(xs.Min(), xs.Max(), .01);
double[] theoreticalYs = theoreticalXs.Select(x => f(x)).ToArray();
var perfectPlot = plt.AddScatterLines(theoreticalXs, theoreticalYs);
perfectPlot.Label = "theoretical signal";
perfectPlot.Color = plt.Palette.GetColor(2);
perfectPlot.LineStyle = ScottPlot.LineStyle.Dash;

var samplePlot = plt.AddScatterPoints(sampleXs, sampleYs);
samplePlot.Label = "sampled points";
samplePlot.Color = plt.Palette.GetColor(0);
samplePlot.MarkerSize = 10;
samplePlot.MarkerShape = ScottPlot.MarkerShape.openCircle;
samplePlot.MarkerLineWidth = 2;

var smoothPlot = plt.AddScatter(xs, ys);
smoothPlot.Label = "interpolated points";
smoothPlot.Color = plt.Palette.GetColor(3);
smoothPlot.MarkerShape = ScottPlot.MarkerShape.filledCircle;

plt.Legend();
plt.SaveFig("output.png");

2D and 3D Spline Interpolation

The interpolation method described above only considered the horizontal axis when generating evenly-spaced time points (1D interpolation). For information and code examples regarding 2D and 3D cubic spline interpolation, see my previous blog post: Spline Interpolation with C#

Resources

• 2D and 3D Spline Interpolation with C# - Blog post from Jan, 2022

• Spline Interpolation with ScottPlot - Demonstrates additional types of interpolation: Bezier, Catmull-Rom, Chaikin, Cubic, etc. The project is open source under a MIT license.

• Programmer’s guide to polynomials and splines

Show A Progress Bar as your Blazor App Loads

Today I created a Blazor WebAssembly app that shows a progress bar while the page loads. This is especially useful for users on slow connections because Blazor apps typically require several megabytes of DLL and DAT files to be downloaded before meaningful content appears on the page.

Live Demo: LJPcalc (source code)

Step 1: Add a progress bar

Edit index.html and identify your app’s main div:

<div id="app">Loading...</div>

Add a progress bar inside it:

<div id="app">
	<h2>Loading...</h2>
	<div class="progress mt-2" style="height: 2em;">
		<div id="progressbar" class="progress-bar progress-bar-striped progress-bar-animated"
			style="width: 10%; background-color: #204066;"></div>
	</div>
	<div>
		<span id="progressLabel" class="text-muted">Downloading file list</span>
	</div>
</div>

See Bootstrap’s progressbar page for extensive customization and animation options and best practices when working with progress indicators. Also ensure the version of Bootstrap in your Blazor app is consistent with the documentation/HTML you are referencing.

Step 2: Disable Blazor AutoStart

Edit index.html and identify where your app loads Blazor resources:

<script src="https://swharden.com/static/2022/05/29/_framework/blazor.webassembly.js"></script>

Update that script so it does not download automatically:

<script src="https://swharden.com/static/2022/05/29/_framework/blazor.webassembly.js" autostart="false"></script>

Step 3: Create a Blazor Startup Script

Add a script to the bottom of the page to start Blazor manually, identifying all the resources needed and incrementally downloading them while updating the progressbar along the way.

<script>
	function StartBlazor() {
		let loadedCount = 0;
		const resourcesToLoad = [];
		Blazor.start({
			loadBootResource:
				function (type, filename, defaultUri, integrity) {
					if (type == "dotnetjs")
						return defaultUri;

					const fetchResources = fetch(defaultUri, {
						cache: 'no-cache',
						integrity: integrity,
						headers: { 'MyCustomHeader': 'My custom value' }
					});

					resourcesToLoad.push(fetchResources);

					fetchResources.then((r) => {
						loadedCount += 1;
						if (filename == "blazor.boot.json")
							return;
						const totalCount = resourcesToLoad.length;
						const percentLoaded = 10 + parseInt((loadedCount * 90.0) / totalCount);
						const progressbar = document.getElementById('progressbar');
						progressbar.style.width = percentLoaded + '%';
						const progressLabel = document.getElementById('progressLabel');
						progressLabel.innerText = `Downloading ${loadedCount}/${totalCount}: ${filename}`;
					});

					return fetchResources;
				}
		});
	}

	StartBlazor();
</script>

Resources

• ASP.NET Core Blazor startup describes the Blazor.start() process and loadBootResource().

• Code here inspired by @EdmondShtogu’s comment on dotnet/aspnetcore #25165 from Feb 18, 2021

Use Maui.Graphics to Draw 2D Graphics in Any .NET Application

This week Microsoft officially released .NET Maui and the new Microsoft.Maui.Graphics library which can draw 2D graphics in any .NET application (not just Maui apps). This page offers a quick look at how to use this new library to draw graphics using SkiaSharp in a .NET 6 console application. The C# Data Visualization site has additional examples for drawing and animating graphics using Microsoft.Maui.Graphics in Windows Forms and WPF applications.

The code below is a full .NET 6 console application demonstrating common graphics tasks (setting colors, drawing shapes, rendering text, etc.) and was used to generate the image above.

// These packages are available on NuGet
using Microsoft.Maui.Graphics;
using Microsoft.Maui.Graphics.Skia;

// Create a bitmap in memory and draw on its Canvas
SkiaBitmapExportContext bmp = new(600, 400, 1.0f);
ICanvas canvas = bmp.Canvas;

// Draw a big blue rectangle with a dark border
Rect backgroundRectangle = new(0, 0, bmp.Width, bmp.Height);
canvas.FillColor = Color.FromArgb("#003366");
canvas.FillRectangle(backgroundRectangle);
canvas.StrokeColor = Colors.Black;
canvas.StrokeSize = 20;
canvas.DrawRectangle(backgroundRectangle);

// Draw circles randomly around the image
for (int i = 0; i < 100; i++)
{
    float x = Random.Shared.Next(bmp.Width);
    float y = Random.Shared.Next(bmp.Height);
    float r = Random.Shared.Next(5, 50);

    Color randomColor = Color.FromRgb(
        red: Random.Shared.Next(255),
        green: Random.Shared.Next(255),
        blue: Random.Shared.Next(255));

    canvas.StrokeSize = r / 3;
    canvas.StrokeColor = randomColor.WithAlpha(.3f);
    canvas.DrawCircle(x, y, r);
}

// Measure a string
string myText = "Hello, Maui.Graphics!";
Font myFont = new Font("Impact");
float myFontSize = 48;
canvas.Font = myFont;
SizeF textSize = canvas.GetStringSize(myText, myFont, myFontSize);

// Draw a rectangle to hold the string
Point point = new(
    x: (bmp.Width - textSize.Width) / 2,
    y: (bmp.Height - textSize.Height) / 2);
Rect myTextRectangle = new(point, textSize);
canvas.FillColor = Colors.Black.WithAlpha(.5f);
canvas.FillRectangle(myTextRectangle);
canvas.StrokeSize = 2;
canvas.StrokeColor = Colors.Yellow;
canvas.DrawRectangle(myTextRectangle);

// Daw the string itself
canvas.FontSize = myFontSize * .9f; // smaller than the rectangle
canvas.FontColor = Colors.White;
canvas.DrawString(myText, myTextRectangle, 
    HorizontalAlignment.Center, VerticalAlignment.Center, TextFlow.OverflowBounds);

// Save the image as a PNG file
bmp.WriteToFile("console2.png");

Multi-Platform Graphics Abstraction

The Microsoft.Maui.Graphics namespace a small collection of interfaces which can be implemented by many different rendering technologies (SkiaSharp, SharpDX, GDI, etc.), making it possible to create drawing routines that are totally abstracted from the underlying graphics rendering system.

I really like that I can now create a .NET Standard 2.0 project that exclusively uses interfaces from Microsoft.Maui.Graphics to write code that draws complex graphics, then reference that code from other projects that use platform-specific graphics libraries to render the images.

When I write scientific simulations or data visualization code I frequently regard my graphics drawing routines as business logic, and drawing with Maui.Graphics lets me write this code to an abstraction that keeps rendering technology dependencies out of my business logic - a big win!

Rough Edges

After working with this library while it was being developed over the last few months, these are the things I find most limiting in my personal projects which made it through the initial release this week. Some of them are open issues so they may get fixed soon, and depending on how the project continues to evolve many of these rough edges may improve with time. I’m listing them here now so I can keep track of them, and I intend to update this list if/as these topics improve:

• Strings cannot be accurately measured: The size returned by GetStringSize() is inaccurate and does not respect font. There’s an issue tracking this (#279), but it’s been open for more than three months and the library was released this week in its broken state.

  • EDIT: I concede multi-platform font support is a very hard problem, but this exactly the type of problem that .NET Maui was created to solve.
    
    

• Missing XML documentation: Intellisense can really help people who are new to a library. The roll-out of a whole new application framework is a good example of a time when a lot of people will be exploring a new library. Let’s take the Color class for example (which 100% of people will interact with) and consider misunderstandings that could be prevented by XML documentation and intellisense: If new Color() accepts 3 floats, should they be 0-255 or 0-1? I need to make a color from the RGB web value #003366, why does Color.FromHex() tell me to use FromArgb? Web colors are RGBA, should I use FromRrgba()? But wait, that string is RGB, not ARGB or RGBA, so will it throw an exception? What does Color.Parse() do?

  • Edit 1: Some of these answers are documented in source code, but they are not XML docs, so this information is not available to library users.

  • Edit 2: Is it on the open-source community to contribute XML documentation? If so, fair enough, but it is a very extensive effort (to write and to review), so a call should be put out for this job to ensure someone doesn’t go through all the effort then have their open PR sit unmerged for months while it falls out of sync with the main branch.

• The library has signs of being incomplete: There remain a good number of NotImplementedException and // todo in sections of the code base that indicate additional work is still required.

Again, I’m pointing these things out the very first week .NET Maui was released, so there’s plenty of time and opportunity for improvements in the coming weeks and months.

I’m optimistic this library will continue to improve, and I am very excited to watch it progress! I’m not aware of the internal pressures and constraints that led to the library being released like it was this week, but I want to end by complimenting the team on their great job so far and encourage everyone (at Microsoft and in the open-source community at large) to keep moving this library forward. The .NET Maui team undertook an ambitious challenge by setting-out to implement cross-platform graphics support, but achieving this goal elegantly will be a huge accomplishment for the .NET community!

Resources

• Source code for this project
• Maui.Graphics WinForms Quickstart
• Maui.Graphics WPF Quickstart
• Maui.Graphics Console Quickstart
• Maui.Graphics .NET Maui Quickstart
• https://maui.graphics

Using DataFrames in C#

The DataFrame is a data structure designed for manipulation, analysis, and visualization of tabular data, and it is the cornerstone of many data science applications. One of the most famous implementations of the data frame is provided by the Pandas package for Python. An equivalent data structure is available for C# using Microsoft’s data analysis package. Although data frames are commonly used in Jupyter notebooks, they can be used in standard .NET applications as well. This article surveys Microsoft’s Data Analysis package and introduces how to interact with with data frames using C# and the .NET platform.

DataFrame Quickstart

• A DataFrame is a 2D matrix that stores data values in named columns.
• Each column has a distinct data type.
• Rows represent observations.

Add the Microsoft.Data.Analysis package to your project, then you can create a DataFrame like this:

using Microsoft.Data.Analysis;

string[] names = { "Oliver", "Charlotte", "Henry", "Amelia", "Owen" };
int[] ages = { 23, 19, 42, 64, 35 };
double[] heights = { 1.91, 1.62, 1.72, 1.57, 1.85 };

DataFrameColumn[] columns = {
    new StringDataFrameColumn("Name", names),
    new PrimitiveDataFrameColumn<int>("Age", ages),
    new PrimitiveDataFrameColumn<double>("Height", heights),
};

DataFrame df = new(columns);

Contents of a DataFrame can be previewed using Console.WriteLine(df) but the formatting isn’t pretty.

Name  Age   Height
Oliver23    1.91
Charlotte19    1.62
Henry 42    1.72
Amelia64    1.57
Owen  35    1.85

Pretty DataFrame Formatting

A custom PrettyPrint() extension method can improve DataFrame readability. Implementing this as an extension method allows me to call df.PrettyPrint() anywhere in my code.

using Microsoft.Data.Analysis;
using System.Text;

internal static class PrettyPrinters
{
    public static void PrettyPrint(this DataFrame df) => Console.WriteLine(PrettyText(df));
    public static string PrettyText(this DataFrame df) => ToStringArray2D(df).ToFormattedText();

    public static string ToMarkdown(this DataFrame df) => ToStringArray2D(df).ToMarkdown();

    public static void PrettyPrint(this DataFrameRow row) => Console.WriteLine(Pretty(row));
    public static string Pretty(this DataFrameRow row) => row.Select(x => x?.ToString() ?? string.Empty).StringJoin();
    private static string StringJoin(this IEnumerable<string> strings) => string.Join(" ", strings.Select(x => x.ToString()));

    private static string[,] ToStringArray2D(DataFrame df)
    {
        string[,] strings = new string[df.Rows.Count + 1, df.Columns.Count];

        for (int i = 0; i < df.Columns.Count; i++)
            strings[0, i] = df.Columns[i].Name;

        for (int i = 0; i < df.Rows.Count; i++)
            for (int j = 0; j < df.Columns.Count; j++)
                strings[i + 1, j] = df[i, j]?.ToString() ?? string.Empty;

        return strings;
    }

    private static int[] GetMaxLengthsByColumn(this string[,] strings)
    {
        int[] maxLengthsByColumn = new int[strings.GetLength(1)];

        for (int y = 0; y < strings.GetLength(0); y++)
            for (int x = 0; x < strings.GetLength(1); x++)
                maxLengthsByColumn[x] = Math.Max(maxLengthsByColumn[x], strings[y, x].Length);

        return maxLengthsByColumn;
    }

    private static string ToFormattedText(this string[,] strings)
    {
        StringBuilder sb = new();
        int[] maxLengthsByColumn = GetMaxLengthsByColumn(strings);

        for (int y = 0; y < strings.GetLength(0); y++)
        {
            for (int x = 0; x < strings.GetLength(1); x++)
            {
                sb.Append(strings[y, x].PadRight(maxLengthsByColumn[x] + 2));
            }
            sb.AppendLine();
        }

        return sb.ToString();
    }


    private static string ToMarkdown(this string[,] strings)
    {
        StringBuilder sb = new();
        int[] maxLengthsByColumn = GetMaxLengthsByColumn(strings);

        for (int y = 0; y < strings.GetLength(0); y++)
        {
            for (int x = 0; x < strings.GetLength(1); x++)
            {
                sb.Append(strings[y, x].PadRight(maxLengthsByColumn[x]));
                if (x < strings.GetLength(1) - 1)
                    sb.Append(" | ");
            }
            sb.AppendLine();

            if (y == 0)
            {
                for (int i = 0; i < strings.GetLength(1); i++)
                {
                    int bars = maxLengthsByColumn[i] + 2;
                    if (i == 0)
                        bars -= 1;
                    sb.Append(new String('-', bars));

                    if (i < strings.GetLength(1) - 1)
                        sb.Append("|");
                }
                sb.AppendLine();
            }
        }

        return sb.ToString();
    }
}

Name       Age  Height
Oliver     23   1.91
Charlotte  19   1.62
Henry      42   1.72
Amelia     64   1.57
Owen       35   1.85

I can create similar methods to format a DataFrame as Markdown or HTML.

Name      | Age | Height
----------|-----|--------
Oliver    | 23  | 1.91
Charlotte | 19  | 1.62
Henry     | 42  | 1.72
Amelia    | 64  | 1.57
Owen      | 35  | 1.85

Using DataFrames in Interactive Notebooks

To get started using .NET workbooks, install the .NET Interactive Notebooks extension for VS Code, create a new demo.ipynb file, then add your code.

Previously users had to create custom HTML formatters to properly display DataFrames in .NET Interactive Notebooks, but these days it works right out of the box.

💡 See demo.html for a full length demonstration notebook

// visualize the DataFrame
df

Append a Row

Build a new row using key/value pair then append it to the DataFrame

List<KeyValuePair<string, object>> newRowData = new()
{
    new KeyValuePair<string, object>("Name", "Scott"),
    new KeyValuePair<string, object>("Age", 36),
    new KeyValuePair<string, object>("Height", 1.65),
};

df.Append(newRowData, inPlace: true);

Add a Column

Build a new column, populate it with data, and add it to the DataFrame

int[] weights = { 123, 321, 111, 121, 131 };
PrimitiveDataFrameColumn<int> weightCol = new("Weight", weights);
df.Columns.Add(weightCol);

Sort and Filter

The DataFrame class has numerous operations available to sort, filter, and analyze data in many different ways. A popular pattern when working with DataFrames is to use method chaining to combine numerous operations together into a single statement. See the DataFrame Class API for a full list of available operations.

df.OrderBy("Name")
    .Filter(df["Age"].ElementwiseGreaterThan(30))
    .PrettyPrint();

Name    Age  Height
Henry   42   1.72
Oliver  23   1.91
Owen    35   1.85

Mathematical Operations

It’s easy to perform math on columns or across multiple DataFrames. In this example we will perform math using two columns and create a new column to hold the output.

DataFrameColumn iqCol = df["Age"] * df["Height"] * 1.5;

double[] iqs = Enumerable.Range(0, (int)iqCol.Length)
    .Select(x => (double)iqCol[x])
    .ToArray();

df.Columns.Add(new PrimitiveDataFrameColumn<double>("IQ", iqs));
df.PrettyPrint();

Name       Age  Height  IQ
Oliver     23   1.91    65.9
Charlotte  19   1.62    46.17
Henry      42   1.72    108.36
Amelia     64   1.57    150.72
Owen       35   1.85    97.12

Statistical Operations

You can iterate across every row of a column to calculate population statistics

foreach (DataFrameColumn col in df.Columns.Skip(1))
{
    // warning: additional care must be taken for datasets which contain null
    double[] values = Enumerable.Range(0, (int)col.Length).Select(x => Convert.ToDouble(col[x])).ToArray();
    (double mean, double std) = MeanAndStd(values);
    Console.WriteLine($"{col.Name} = {mean} +/- {std:N3} (n={values.Length})");
}

Age = 36.6 +/- 15.982 (n=5)
Height = 1.734 +/- 0.130 (n=5)

private static (double mean, double std) MeanAndStd(double[] values)
{
	if (values is null)
		throw new ArgumentNullException(nameof(values));

	if (values.Length == 0)
		throw new ArgumentException($"{nameof(values)} must not be empty");

	double sum = 0;
	for (int i = 0; i < values.Length; i++)
		sum += values[i];

	double mean = sum / values.Length;

	double sumVariancesSquared = 0;
	for (int i = 0; i < values.Length; i++)
	{
		double pointVariance = Math.Abs(mean - values[i]);
		double pointVarianceSquared = Math.Pow(pointVariance, 2);
		sumVariancesSquared += pointVarianceSquared;
	}

	double meanVarianceSquared = sumVariancesSquared / values.Length;
	double std = Math.Sqrt(meanVarianceSquared);

	return (mean, std);
}

Plot Values from a DataFrame

I use ScottPlot.NET to visualize data from DataFrames in .NET applications and .NET Interactive Notebooks. ScottPlot can generate a variety of plot types and has many options for customization. See the ScottPlot Cookbook for examples and API documentation.

// Register a custom formatter to display ScottPlot plots as images
using Microsoft.DotNet.Interactive.Formatting;
Formatter.Register(typeof(ScottPlot.Plot), (plt, writer) => 
    writer.Write(((ScottPlot.Plot)plt).GetImageHTML()), HtmlFormatter.MimeType);

// Get the data you wish to display in double arrays
double[] ages = Enumerable.Range(0, (int)df.Rows.Count).Select(x => Convert.ToDouble(df["Age"][x])).ToArray();
double[] heights = Enumerable.Range(0, (int)df.Rows.Count).Select(x => Convert.ToDouble(df["Height"][x])).ToArray();

// Create and display a plot
var plt = new ScottPlot.Plot(400, 300);
plt.AddScatter(ages, heights);
plt.XLabel("Age");
plt.YLabel("Height");
plt

💡 See demo.html for a full length demonstration notebook

If you are only working inside a Notebook and you want all your plots to be HTML and JavaScript, XPlot.Plotly is a good tool to use.

Data may contain null

I didn’t demonstrate it in the code examples above, but note that all column data types are nullable. While null-containing data requires extra considerations when writing mathematical routes, it’s a convenient way to model missing data which is a common occurrence in the real world.

Why not just use LINQ?

I see this question asked frequently, often with an aggressive and condescending tone. LINQ (Language-Integrated Query) is fantastic for performing logical operations on simple collections of data. When you have large 2D datasets of labeled data, advantages of data frames over flat LINQ statements start to become apparent. It is also easy to perform logical operations across multiple data frames, allowing users to write simpler and more readable code than could be achieved with LINQ statements. Data frames also make it much easier to visualize complex data too. In the data science world where complex labeled datasets are routinely compared, manipulated, merged, and visualized, often in an interactive context, the data frames are much easier to work with than raw LINQ statements.

Conclusions

Although I typically reach for Python to perform exploratory data science, it’s good to know that C# has a DataFrame available and that it can be used to inspect and manipulate tabular data. DataFrames pair well with ScottPlot figures in interactive notebooks and are a great way to inspect and communicate complex data. I look forward to watching Microsoft’s Data Analysis namespace continue to evolve as part of their machine learning / ML.NET platform.

Resources

• Example notebook for this project

• Source code for this project

• Official Microsoft.Data.Analysis.DataFrame Class Documentation

• Microsoft.Data.Analysis source code

• An Introduction to DataFrame (.NET Blog)

• ExtremeOptimization DataFrame Quickstart

• Microsoft.Data.Analysis on NuGet

• Getting Started With C# DataFrame and XPlot.Plotly

• 10 minutes to pandas