SWHarden.com

The personal website of Scott W Harden

Experiments in PSK-31 Synthesis

How to encode and decode PSK-31 messages using C#

PSK-31 is a narrow-bandwidth digital mode which encodes text as an audio tone that varies phase at a known rate. To learn more about this digital mode and solve a challenging programming problem, I’m going to write a PSK-31 encoder and decoder from scratch using the C# programming language. All code created for this project is open-source, available from my PSK Experiments GitHub Repository, and released under the permissive MIT license. This page documents my progress and notes things I learn along the way.

Encoding Bits as Phase Shifts

Amplitude Modulation Silences Phase Transitions

Although a continuous phase-shifting tone of constant amplitude can successfully transmit PSK31 data, the abrupt phase transitions will cause splatter. If you transmit this you will be heard, but those trying to communicate using adjacent frequencies will be highly disappointed.

Hard phase transitions (splatter) Soft phase transitions (cleaner)

To reduce spectral artifacts that result from abruptly changing phase, phase transitions are silenced by shaping the waveform envelope as a sine wave so it is silent at the transition. This way the maximum rate of phase shifts is a sine wave with a period of half the baud rate. This is why the opening of a PSK31 message (a series of logical 0 bits) sounds like two tones: It’s the carrier sine wave with an envelope shaped like a sine wave with a period of 31.25/2 Hz. These two tones separated by approximately 15 Hz are visible in the spectrogram (waterfall).

Encoding Text as Bits

Unlike ASCII (8 bits per character) and RTTY (5 bits per character), BPSK uses Varicode (1-10 bits per character) to encode text. Consecutive zeros 00 separate characters, so character codes must not contain 00 and they must start ane end with a 1 bit. Messages are flanked by a preamble (repeated 0 bits) and a postamble (repeated 1 bits).

NUL 1010101011
SOH 1011011011
STX 1011101101
ETX 1101110111
EOT 1011101011
ENQ 1101011111
ACK 1011101111
BEL 1011111101
BS  1011111111
HT  11101111
LF  11101
VT  1101101111
FF  1011011101
CR  11111
SO  1101110101
SI  1110101011
DLE 1011110111
DC1 1011110101
DC2 1110101101
DC3 1110101111
DC4 1101011011
NAK 1101101011
SYN 1101101101
ETB 1101010111
CAN 1101111011
EM  1101111101
SUB 1110110111
ESC 1101010101
FS  1101011101
GS  1110111011
RS  1011111011
US  1101111111
SP  1
!   111111111
"   101011111
#   111110101
$   111011011
%   1011010101
&   1010111011
'   101111111
(   11111011
)   11110111
*   101101111
+   111011111
,   1110101
-   110101
.   1010111
/   110101111
0   10110111
1   10111101
2   11101101
3   11111111
4   101110111
5   101011011
6   101101011
7   110101101
8   110101011
9   110110111
:   11110101
;   110111101
<   111101101
=   1010101
>   111010111
?   1010101111
@   1010111101
A   1111101
B   11101011
C   10101101
D   10110101
E   1110111
F   11011011
G   11111101
H   101010101
I   1111111
J   111111101
K   101111101
L   11010111
M   10111011
N   11011101
O   10101011
P   11010101
Q   111011101
R   10101111
S   1101111
T   1101101
U   101010111
V   110110101
X   101011101
Y   101110101
Z   101111011
[   1010101101
\   111110111
]   111101111
^   111111011
_   1010111111
.   101101101
/   1011011111
a   1011
b   1011111
c   101111
d   101101
e   11
f   111101
g   1011011
h   101011
i   1101
j   111101011
k   10111111
l   11011
m   111011
n   1111
o   111
p   111111
q   110111111
r   10101
s   10111
t   101
u   110111
v   1111011
w   1101011
x   11011111
y   1011101
z   111010101
{   1010110111
|   110111011
}   1010110101
~   1011010111
DEL 1110110101

How to Generate a PSK Waveform

Now that we’ve covered the major steps of PSK31 message composition and modulation, let’s go through the steps ot generate a PSK31 message in code.

Step 1: Convert a Message to Varicode

Here’s the gist of how I store my varicode table in code. Note that the struct has an additional Description field which is useful for decoding and debugging.

public struct VaricodeSymbol
{
    public readonly string Symbol;
    public string BitString;
    public int[] Bits;
    public readonly string Description;

    public VaricodeSymbol(string symbol, string bitString, string? description = null)
    {
        Symbol = symbol;
        BitString = bitString;
        Bits = bitString.ToCharArray().Select(x => x == '1' ? 1 : 0).ToArray();
        Description = description ?? string.Empty;
    }
}
static VaricodeSymbol[] GetAllSymbols() => new VaricodeSymbol[]
{
    new("NUL", "1010101011", "Null character"),
    new("LF", "11101", "Line feed"),
    new("CR", "11111", "Carriage return"),
    new("SP", "1", "Space"),
    new("a", "1011"),
    new("b", "1011111"),
    new("c", "101111"),
    // etc...
};

I won’t show how I do the message-to-varicode lookup, but it’s trivial. Here’s the final function I use to generate Varicode bits from a string:

static int[] GetVaricodeBits(string message)
{
    List<int> bits = new();

    // add a preamble of repeated zeros
    for (int i=0; i<20; i++)
        bits.Add(0);

    // encode each character of a message
    foreach (char character in message)
    {
        VaricodeSymbol symbol = Lookup(character);
        bits.AddRange(symbol.Bits);
        bits.AddRange(CharacterSeparator);
    }

    // add a postamble of repeated ones
    for (int i=0; i<20; i++)
        bits.Add(1);

    return bits.ToArray();
}

Step 2: Determine Phase Shifts

Now that we have our Varicode bits, we need to generate an array to indicate phase transitions. A transition occurs every time a bit changes value form the previous bit. This code returns phase as an array of double given the bits from a Varicode message.

public static double[] GetPhaseShifts(int[] bits, double phase1 = 0, double phase2 = Math.PI)
{
    double[] phases = new double[bits.Length];
    for (int i = 0; i < bits.Length; i++)
    {
        double previousPhase = i > 0 ? phases[i - 1] : phase1;
        double oppositePhase = previousPhase == phase1 ? phase2 : phase1;
        phases[i] = bits[i] == 1 ? previousPhase : oppositePhase;
    }
    return phases;
}

Step 3: Generate the Waveform

These constants will be used to define the shape of the waveform:

public const int SampleRate = 8000;
public const double Frequency = 1000;
public const double BaudRate = 31.25;

This minimal code generates a decipherable PSK-31 message, but it does not silence the phase transitions so it produces a lot of splatter. This function must be refined to shape the waveform such that phase transitions are silenced.

public double[] GetWaveformBPSK(double[] phases)
{
    int totalSamples = (int)(phases.Length * SampleRate / BaudRate);
    double[] wave = new double[totalSamples];
    for (int i = 0; i < wave.Length; i++)
    {
        double time = (double)i / SampleRate;
        int frame = (int)(time * BaudRate);
        double phaseShift = phases[frame];
        wave[i] = Math.Cos(2 * Math.PI * Frequency * time + phaseShift);
    }
    return wave;
}

Step 4: Generate the Waveform with Amplitude Modulation

This is the same function as above, but with extra logic for amplitude-modulating the waveform in the shape of a sine wave to silence phase transitions.

public double[] GetWaveformBPSK(double[] phases)
{
    int baudSamples = (int)(SampleRate / BaudRate);
    double samplesPerBit = SampleRate / BaudRate;
    int totalSamples = (int)(phases.Length * SampleRate / BaudRate);
    double[] wave = new double[totalSamples];

    // create the amplitude envelope sized for a single bit
    double[] envelope = new double[(int)samplesPerBit];
    for (int i = 0; i < envelope.Length; i++)
        envelope[i] = Math.Sin((i + .5) * Math.PI / envelope.Length);

    for (int i = 0; i < wave.Length; i++)
    {
        // phase modulated carrier
        double time = (double)i / SampleRate;
        int frame = (int)(time * BaudRate);
        double phaseShift = phases[frame];
        wave[i] = Math.Cos(2 * Math.PI * Frequency * time + phaseShift);

        // envelope at phase transitions
        int firstSample = (int)(frame * SampleRate / BaudRate);
        int distanceFromFrameStart = i - firstSample;
        int distanceFromFrameEnd = baudSamples - distanceFromFrameStart + 1;
        bool isFirstHalfOfFrame = distanceFromFrameStart < distanceFromFrameEnd;
        bool samePhaseAsLast = frame == 0 ? false : phases[frame - 1] == phases[frame];
        bool samePhaseAsNext = frame == phases.Length - 1 ? false : phases[frame + 1] == phases[frame];
        bool rampUp = isFirstHalfOfFrame && !samePhaseAsLast;
        bool rampDown = !isFirstHalfOfFrame && !samePhaseAsNext;

        if (rampUp)
            wave[i] *= envelope[distanceFromFrameStart];

        if (rampDown)
            wave[i] *= envelope[distanceFromFrameEnd];
    }

    return wave;
}

PSK31 Encoder Program

I wrapped the functionality above in a Windows Forms GUI that allows the user to type a message, specify frequency, baud rate, and whether or not to refine the envelope to reduce splatter, then either play or save the result. An interactive ScottPlot Chart allows the user to inspect the waveform.

Sample PSK-31 Transmissions

These audio files encode the text The Quick Brown Fox Jumped Over The Lazy Dog 1234567890 Times! in 1kHz BPSK at various baud rates.

Non-Standard Baud Rates

Let’s see what PSK-3 sounds like. This mode encodes data at a rate of 3 bits per second. Note that Varicode characters may require up to ten bits, so this is pretty slow. On the other hand the side tones are closer to the carrier and the total bandwidth is much smaller. The message here has been shortened to just my callsign, AJ4VD.

Encode PSK-31 In Your Browser

After implementing the C# encoder described above I created a JavaScript version (as per Atwood’s Law).

Decoding PSK-31

Considering all the steps for encoding PSK-31 transmissions are already described on this page, it doesn’t require too much additional effort to create a decoder using basic software techniques. Once symbol phases are detected it’s easy to work backwards: detect phase transitions (logical 0s) or repeats (logical 1s), treat consecutive zeros as a character separator, then look-up characters according to the Varicode table. The tricky bit is analyzing the source audio to generate the array of phase offsets.

The simplest way to decode PSK-31 transmissions leans on the fact that we already know the baud rate: 31.25 symbols per second, or one symbol every 256 samples at 8kHz sample rate. The audio signal can be segregated into many 256-sample bins, and processed by FFT. Once the center frequency is determined, the FFT power at this frequency can be calculated for each bin. Signal offset can be adjusted to minimize the imaginary component of the FFTs at the carrier frequency, then the real component will be strongly positive or negative, allowing phase transitions to be easily detected.

There are more advanced techniques to improve BPSK decoding, such as continuously adjusting frequency and phase alignment (synchronization). A Costas loop can help lock onto the carrier frequency while preserving its phase. Visit Simple BPSK31 Decoding with Python for an excellent demonstration of how to decode BPSK31 using these advanced techniques.

A crude C# implementation of a BPSK decoded is available on GitHub in the PSK Experiments repository

Encoding PSK-31 in Hardware

Since BPSK is just a carrier that applies periodic 180º phase-shifts, it’s easy to generate in hardware by directly modulating the signal source. A good example of this is KA7OEI’s PSK31 transmitter which feeds the output of an oscillator through an even or odd number of NAND gates (from a 74HC00) to produce two signals of opposite phase.

Quadrature Phase Shift Keying (QPSK)

Unlike the 0º and 180º phases of binary phase shift keying (BPSK), quadrature phase shift keying (QPSK) encodes extra data into each symbol by uses a larger number of phases. When QPSK-31 is used in amateur radio these extra bits aren’t used to send messages faster but instead send them more reliably using convolutional coding and error correction. These additional features come at a cost (an extra 3 dB SNR is required), and in practice QPSK is not used as much by amateur radio operators.

QPSK encoding/decoding and convolutional encoding/decoding are outside the scope of this page, but excellent information exists on the Wikipedia: QPSK and in the US Naval Academy’s EC314 Lesson 23: Digital Modulation document.

PSK-31 in 2022

After all that, it turns out PSK-31 isn’t that popular anymore. These days it seems the FT-8 digital mode with WSJT-X software is orders of magnitude more popular ??

Resources


Resample Time Series Data using Cubic Spline Interpolation

This page describes how I achieve signal resampling with spline interpolation in pure C# without any external dependencies.

Cubic spline interpolation can be used to modify the sample rate of time series data. This page describes how I achieve signal resampling with spline interpolation in pure C# without any external dependencies. This technique can be used to:

Simulating Data Samples

To simulate unevenly-sampled data I create a theoretical signal then sample it at 20 random time points.

// this function represents the signal being measured
static double f(double x) => Math.Sin(x * 10) + Math.Sin(x * 13f);

// randomly sample values from 20 time points
Random rand = new(123);
double[] sampleXs = Enumerable.Range(0, 20)
    .Select(x => rand.NextDouble())
    .OrderBy(x => x)
    .ToArray();
double[] sampleYs = sampleXs.Select(x => f(x)).ToArray();

Resample for Evenly-Spaced Data

I then generate an interpolated spline using my sampled data points as the input.

(double[] xs, double[] ys) = Cubic.Interpolate1D(sampleXs, sampleYs, count: 50);

The generated points line-up perfectly with the sampled data.

There is slight deviation from the theoretical signal (and it’s larger where there is more missing data) but this is an unsurprising result considering the original samples had large gaps of missing data.

Source Code

Interpolation.cs

public static class Interpolation
{
    public static (double[] xs, double[] ys) Interpolate1D(double[] xs, double[] ys, int count)
    {
        if (xs is null || ys is null || xs.Length != ys.Length)
            throw new ArgumentException($"{nameof(xs)} and {nameof(ys)} must have same length");

        int inputPointCount = xs.Length;
        double[] inputDistances = new double[inputPointCount];
        for (int i = 1; i < inputPointCount; i++)
            inputDistances[i] = inputDistances[i - 1] + xs[i] - xs[i - 1];

        double meanDistance = inputDistances.Last() / (count - 1);
        double[] evenDistances = Enumerable.Range(0, count).Select(x => x * meanDistance).ToArray();
        double[] xsOut = Interpolate(inputDistances, xs, evenDistances);
        double[] ysOut = Interpolate(inputDistances, ys, evenDistances);
        return (xsOut, ysOut);
    }

    private static double[] Interpolate(double[] xOrig, double[] yOrig, double[] xInterp)
    {
        (double[] a, double[] b) = FitMatrix(xOrig, yOrig);

        double[] yInterp = new double[xInterp.Length];
        for (int i = 0; i < yInterp.Length; i++)
        {
            int j;
            for (j = 0; j < xOrig.Length - 2; j++)
                if (xInterp[i] <= xOrig[j + 1])
                    break;

            double dx = xOrig[j + 1] - xOrig[j];
            double t = (xInterp[i] - xOrig[j]) / dx;
            double y = (1 - t) * yOrig[j] + t * yOrig[j + 1] +
                t * (1 - t) * (a[j] * (1 - t) + b[j] * t);
            yInterp[i] = y;
        }

        return yInterp;
    }

    private static (double[] a, double[] b) FitMatrix(double[] x, double[] y)
    {
        int n = x.Length;
        double[] a = new double[n - 1];
        double[] b = new double[n - 1];
        double[] r = new double[n];
        double[] A = new double[n];
        double[] B = new double[n];
        double[] C = new double[n];

        double dx1, dx2, dy1, dy2;

        dx1 = x[1] - x[0];
        C[0] = 1.0f / dx1;
        B[0] = 2.0f * C[0];
        r[0] = 3 * (y[1] - y[0]) / (dx1 * dx1);

        for (int i = 1; i < n - 1; i++)
        {
            dx1 = x[i] - x[i - 1];
            dx2 = x[i + 1] - x[i];
            A[i] = 1.0f / dx1;
            C[i] = 1.0f / dx2;
            B[i] = 2.0f * (A[i] + C[i]);
            dy1 = y[i] - y[i - 1];
            dy2 = y[i + 1] - y[i];
            r[i] = 3 * (dy1 / (dx1 * dx1) + dy2 / (dx2 * dx2));
        }

        dx1 = x[n - 1] - x[n - 2];
        dy1 = y[n - 1] - y[n - 2];
        A[n - 1] = 1.0f / dx1;
        B[n - 1] = 2.0f * A[n - 1];
        r[n - 1] = 3 * (dy1 / (dx1 * dx1));

        double[] cPrime = new double[n];
        cPrime[0] = C[0] / B[0];
        for (int i = 1; i < n; i++)
            cPrime[i] = C[i] / (B[i] - cPrime[i - 1] * A[i]);

        double[] dPrime = new double[n];
        dPrime[0] = r[0] / B[0];
        for (int i = 1; i < n; i++)
            dPrime[i] = (r[i] - dPrime[i - 1] * A[i]) / (B[i] - cPrime[i - 1] * A[i]);

        double[] k = new double[n];
        k[n - 1] = dPrime[n - 1];
        for (int i = n - 2; i >= 0; i--)
            k[i] = dPrime[i] - cPrime[i] * k[i + 1];

        for (int i = 1; i < n; i++)
        {
            dx1 = x[i] - x[i - 1];
            dy1 = y[i] - y[i - 1];
            a[i - 1] = k[i - 1] * dx1 - dy1;
            b[i - 1] = -k[i] * dx1 + dy1;
        }

        return (a, b);
    }
}

Program.cs

This is the source code I used to generate the figures on this page.

Plots were generated using ScottPlot.NET.

// this function represents the signal being measured
static double f(double x) => Math.Sin(x * 10) + Math.Sin(x * 13f);

// create points representing randomly sampled time points of a smooth curve
Random rand = new(123);
double[] sampleXs = Enumerable.Range(0, 20)
    .Select(x => rand.NextDouble())
    .OrderBy(x => x)
    .ToArray();
double[] sampleYs = sampleXs.Select(x => f(x)).ToArray();

// use 1D interpolation to create an evenly sampled curve from unevenly sampled data
(double[] xs, double[] ys) = Interpolation.Interpolate1D(sampleXs, sampleYs, count: 50);

var plt = new ScottPlot.Plot(600, 400);

double[] theoreticalXs = ScottPlot.DataGen.Range(xs.Min(), xs.Max(), .01);
double[] theoreticalYs = theoreticalXs.Select(x => f(x)).ToArray();
var perfectPlot = plt.AddScatterLines(theoreticalXs, theoreticalYs);
perfectPlot.Label = "theoretical signal";
perfectPlot.Color = plt.Palette.GetColor(2);
perfectPlot.LineStyle = ScottPlot.LineStyle.Dash;

var samplePlot = plt.AddScatterPoints(sampleXs, sampleYs);
samplePlot.Label = "sampled points";
samplePlot.Color = plt.Palette.GetColor(0);
samplePlot.MarkerSize = 10;
samplePlot.MarkerShape = ScottPlot.MarkerShape.openCircle;
samplePlot.MarkerLineWidth = 2;

var smoothPlot = plt.AddScatter(xs, ys);
smoothPlot.Label = "interpolated points";
smoothPlot.Color = plt.Palette.GetColor(3);
smoothPlot.MarkerShape = ScottPlot.MarkerShape.filledCircle;

plt.Legend();
plt.SaveFig("output.png");

2D and 3D Spline Interpolation

The interpolation method described above only considered the horizontal axis when generating evenly-spaced time points (1D interpolation). For information and code examples regarding 2D and 3D cubic spline interpolation, see my previous blog post: Spline Interpolation with C#

Resources


Show A Progress Bar as your Blazor App Loads

How to add a progress bar to your client-side Blazor WebAssembly app to indicate page load progress.

Today I created a Blazor WebAssembly app that shows a progress bar while the page loads. This is especially useful for users on slow connections because Blazor apps typically require several megabytes of DLL and DAT files to be downloaded before meaningful content appears on the page.

Live Demo: LJPcalc (source code)

Step 1: Add a progress bar

Edit index.html and identify your app’s main div:

<div id="app">Loading...</div>

Add a progress bar inside it:

<div id="app">
	<h2>Loading...</h2>
	<div class="progress mt-2" style="height: 2em;">
		<div id="progressbar" class="progress-bar progress-bar-striped progress-bar-animated"
			style="width: 10%; background-color: #204066;"></div>
	</div>
	<div>
		<span id="progressLabel" class="text-muted">Downloading file list</span>
	</div>
</div>

See Bootstrap’s progressbar page for extensive customization and animation options and best practices when working with progress indicators. Also ensure the version of Bootstrap in your Blazor app is consistent with the documentation/HTML you are referencing.

Step 2: Disable Blazor AutoStart

Edit index.html and identify where your app loads Blazor resources:

<script src="https://swharden.com/static/2022/05/29/_framework/blazor.webassembly.js"></script>

Update that script so it does not download automatically:

<script src="https://swharden.com/static/2022/05/29/_framework/blazor.webassembly.js" autostart="false"></script>

Step 3: Create a Blazor Startup Script

Add a script to the bottom of the page to start Blazor manually, identifying all the resources needed and incrementally downloading them while updating the progressbar along the way.

<script>
	function StartBlazor() {
		let loadedCount = 0;
		const resourcesToLoad = [];
		Blazor.start({
			loadBootResource:
				function (type, filename, defaultUri, integrity) {
					if (type == "dotnetjs")
						return defaultUri;

					const fetchResources = fetch(defaultUri, {
						cache: 'no-cache',
						integrity: integrity,
						headers: { 'MyCustomHeader': 'My custom value' }
					});

					resourcesToLoad.push(fetchResources);

					fetchResources.then((r) => {
						loadedCount += 1;
						if (filename == "blazor.boot.json")
							return;
						const totalCount = resourcesToLoad.length;
						const percentLoaded = 10 + parseInt((loadedCount * 90.0) / totalCount);
						const progressbar = document.getElementById('progressbar');
						progressbar.style.width = percentLoaded + '%';
						const progressLabel = document.getElementById('progressLabel');
						progressLabel.innerText = `Downloading ${loadedCount}/${totalCount}: ${filename}`;
					});

					return fetchResources;
				}
		});
	}

	StartBlazor();
</script>

Resources


Use Maui.Graphics to Draw 2D Graphics in Any .NET Application

How to use Microsoft.Maui.Graphics to draw graphics in a .NET console application and save the output as an image file using SkiaSharp

This week Microsoft officially released .NET Maui and the new Microsoft.Maui.Graphics library which can draw 2D graphics in any .NET application (not just Maui apps). This page offers a quick look at how to use this new library to draw graphics using SkiaSharp in a .NET 6 console application. The C# Data Visualization site has additional examples for drawing and animating graphics using Microsoft.Maui.Graphics in Windows Forms and WPF applications.

The code below is a full .NET 6 console application demonstrating common graphics tasks (setting colors, drawing shapes, rendering text, etc.) and was used to generate the image above.

// These packages are available on NuGet
using Microsoft.Maui.Graphics;
using Microsoft.Maui.Graphics.Skia;

// Create a bitmap in memory and draw on its Canvas
SkiaBitmapExportContext bmp = new(600, 400, 1.0f);
ICanvas canvas = bmp.Canvas;

// Draw a big blue rectangle with a dark border
Rect backgroundRectangle = new(0, 0, bmp.Width, bmp.Height);
canvas.FillColor = Color.FromArgb("#003366");
canvas.FillRectangle(backgroundRectangle);
canvas.StrokeColor = Colors.Black;
canvas.StrokeSize = 20;
canvas.DrawRectangle(backgroundRectangle);

// Draw circles randomly around the image
for (int i = 0; i < 100; i++)
{
    float x = Random.Shared.Next(bmp.Width);
    float y = Random.Shared.Next(bmp.Height);
    float r = Random.Shared.Next(5, 50);

    Color randomColor = Color.FromRgb(
        red: Random.Shared.Next(255),
        green: Random.Shared.Next(255),
        blue: Random.Shared.Next(255));

    canvas.StrokeSize = r / 3;
    canvas.StrokeColor = randomColor.WithAlpha(.3f);
    canvas.DrawCircle(x, y, r);
}

// Measure a string
string myText = "Hello, Maui.Graphics!";
Font myFont = new Font("Impact");
float myFontSize = 48;
canvas.Font = myFont;
SizeF textSize = canvas.GetStringSize(myText, myFont, myFontSize);

// Draw a rectangle to hold the string
Point point = new(
    x: (bmp.Width - textSize.Width) / 2,
    y: (bmp.Height - textSize.Height) / 2);
Rect myTextRectangle = new(point, textSize);
canvas.FillColor = Colors.Black.WithAlpha(.5f);
canvas.FillRectangle(myTextRectangle);
canvas.StrokeSize = 2;
canvas.StrokeColor = Colors.Yellow;
canvas.DrawRectangle(myTextRectangle);

// Daw the string itself
canvas.FontSize = myFontSize * .9f; // smaller than the rectangle
canvas.FontColor = Colors.White;
canvas.DrawString(myText, myTextRectangle, 
    HorizontalAlignment.Center, VerticalAlignment.Center, TextFlow.OverflowBounds);

// Save the image as a PNG file
bmp.WriteToFile("console2.png");

Multi-Platform Graphics Abstraction

The Microsoft.Maui.Graphics namespace a small collection of interfaces which can be implemented by many different rendering technologies (SkiaSharp, SharpDX, GDI, etc.), making it possible to create drawing routines that are totally abstracted from the underlying graphics rendering system.

I really like that I can now create a .NET Standard 2.0 project that exclusively uses interfaces from Microsoft.Maui.Graphics to write code that draws complex graphics, then reference that code from other projects that use platform-specific graphics libraries to render the images.

When I write scientific simulations or data visualization code I frequently regard my graphics drawing routines as business logic, and drawing with Maui.Graphics lets me write this code to an abstraction that keeps rendering technology dependencies out of my business logic - a big win!

Rough Edges

After working with this library while it was being developed over the last few months, these are the things I find most limiting in my personal projects which made it through the initial release this week. Some of them are open issues so they may get fixed soon, and depending on how the project continues to evolve many of these rough edges may improve with time. I’m listing them here now so I can keep track of them, and I intend to update this list if/as these topics improve:

Again, I’m pointing these things out the very first week .NET Maui was released, so there’s plenty of time and opportunity for improvements in the coming weeks and months.

I’m optimistic this library will continue to improve, and I am very excited to watch it progress! I’m not aware of the internal pressures and constraints that led to the library being released like it was this week, but I want to end by complimenting the team on their great job so far and encourage everyone (at Microsoft and in the open-source community at large) to keep moving this library forward. The .NET Maui team undertook an ambitious challenge by setting-out to implement cross-platform graphics support, but achieving this goal elegantly will be a huge accomplishment for the .NET community!

Resources


Using DataFrames in C#

How to use the DataFrame class from the Microsoft.Data.Analysis package to interact with tabular data

The DataFrame is a data structure designed for manipulation, analysis, and visualization of tabular data, and it is the cornerstone of many data science applications. One of the most famous implementations of the data frame is provided by the Pandas package for Python. An equivalent data structure is available for C# using Microsoft’s data analysis package. Although data frames are commonly used in Jupyter notebooks, they can be used in standard .NET applications as well. This article surveys Microsoft’s Data Analysis package and introduces how to interact with with data frames using C# and the .NET platform.

DataFrame Quickstart

Add the Microsoft.Data.Analysis package to your project, then you can create a DataFrame like this:

using Microsoft.Data.Analysis;

string[] names = { "Oliver", "Charlotte", "Henry", "Amelia", "Owen" };
int[] ages = { 23, 19, 42, 64, 35 };
double[] heights = { 1.91, 1.62, 1.72, 1.57, 1.85 };

DataFrameColumn[] columns = {
    new StringDataFrameColumn("Name", names),
    new PrimitiveDataFrameColumn<int>("Age", ages),
    new PrimitiveDataFrameColumn<double>("Height", heights),
};

DataFrame df = new(columns);

Contents of a DataFrame can be previewed using Console.WriteLine(df) but the formatting isn’t pretty.

Name  Age   Height
Oliver23    1.91
Charlotte19    1.62
Henry 42    1.72
Amelia64    1.57
Owen  35    1.85

Pretty DataFrame Formatting

A custom PrettyPrint() extension method can improve DataFrame readability. Implementing this as an extension method allows me to call df.PrettyPrint() anywhere in my code.

💡 View the full PrettyPrinters.cs source code
using Microsoft.Data.Analysis;
using System.Text;

internal static class PrettyPrinters
{
    public static void PrettyPrint(this DataFrame df) => Console.WriteLine(PrettyText(df));
    public static string PrettyText(this DataFrame df) => ToStringArray2D(df).ToFormattedText();

    public static string ToMarkdown(this DataFrame df) => ToStringArray2D(df).ToMarkdown();

    public static void PrettyPrint(this DataFrameRow row) => Console.WriteLine(Pretty(row));
    public static string Pretty(this DataFrameRow row) => row.Select(x => x?.ToString() ?? string.Empty).StringJoin();
    private static string StringJoin(this IEnumerable<string> strings) => string.Join(" ", strings.Select(x => x.ToString()));

    private static string[,] ToStringArray2D(DataFrame df)
    {
        string[,] strings = new string[df.Rows.Count + 1, df.Columns.Count];

        for (int i = 0; i < df.Columns.Count; i++)
            strings[0, i] = df.Columns[i].Name;

        for (int i = 0; i < df.Rows.Count; i++)
            for (int j = 0; j < df.Columns.Count; j++)
                strings[i + 1, j] = df[i, j]?.ToString() ?? string.Empty;

        return strings;
    }

    private static int[] GetMaxLengthsByColumn(this string[,] strings)
    {
        int[] maxLengthsByColumn = new int[strings.GetLength(1)];

        for (int y = 0; y < strings.GetLength(0); y++)
            for (int x = 0; x < strings.GetLength(1); x++)
                maxLengthsByColumn[x] = Math.Max(maxLengthsByColumn[x], strings[y, x].Length);

        return maxLengthsByColumn;
    }

    private static string ToFormattedText(this string[,] strings)
    {
        StringBuilder sb = new();
        int[] maxLengthsByColumn = GetMaxLengthsByColumn(strings);

        for (int y = 0; y < strings.GetLength(0); y++)
        {
            for (int x = 0; x < strings.GetLength(1); x++)
            {
                sb.Append(strings[y, x].PadRight(maxLengthsByColumn[x] + 2));
            }
            sb.AppendLine();
        }

        return sb.ToString();
    }


    private static string ToMarkdown(this string[,] strings)
    {
        StringBuilder sb = new();
        int[] maxLengthsByColumn = GetMaxLengthsByColumn(strings);

        for (int y = 0; y < strings.GetLength(0); y++)
        {
            for (int x = 0; x < strings.GetLength(1); x++)
            {
                sb.Append(strings[y, x].PadRight(maxLengthsByColumn[x]));
                if (x < strings.GetLength(1) - 1)
                    sb.Append(" | ");
            }
            sb.AppendLine();

            if (y == 0)
            {
                for (int i = 0; i < strings.GetLength(1); i++)
                {
                    int bars = maxLengthsByColumn[i] + 2;
                    if (i == 0)
                        bars -= 1;
                    sb.Append(new String('-', bars));

                    if (i < strings.GetLength(1) - 1)
                        sb.Append("|");
                }
                sb.AppendLine();
            }
        }

        return sb.ToString();
    }
}
Name       Age  Height
Oliver     23   1.91
Charlotte  19   1.62
Henry      42   1.72
Amelia     64   1.57
Owen       35   1.85

I can create similar methods to format a DataFrame as Markdown or HTML.

Name      | Age | Height
----------|-----|--------
Oliver    | 23  | 1.91
Charlotte | 19  | 1.62
Henry     | 42  | 1.72
Amelia    | 64  | 1.57
Owen      | 35  | 1.85
Name Age Height
Oliver 23 1.91
Charlotte 19 1.62
Henry 42 1.72
Amelia 64 1.57
Owen 35 1.85

Using DataFrames in Interactive Notebooks

To get started using .NET workbooks, install the .NET Interactive Notebooks extension for VS Code, create a new demo.ipynb file, then add your code.

Previously users had to create custom HTML formatters to properly display DataFrames in .NET Interactive Notebooks, but these days it works right out of the box.

💡 See demo.html for a full length demonstration notebook

// visualize the DataFrame
df

Append a Row

Build a new row using key/value pair then append it to the DataFrame

List<KeyValuePair<string, object>> newRowData = new()
{
    new KeyValuePair<string, object>("Name", "Scott"),
    new KeyValuePair<string, object>("Age", 36),
    new KeyValuePair<string, object>("Height", 1.65),
};

df.Append(newRowData, inPlace: true);

Add a Column

Build a new column, populate it with data, and add it to the DataFrame

int[] weights = { 123, 321, 111, 121, 131 };
PrimitiveDataFrameColumn<int> weightCol = new("Weight", weights);
df.Columns.Add(weightCol);

Sort and Filter

The DataFrame class has numerous operations available to sort, filter, and analyze data in many different ways. A popular pattern when working with DataFrames is to use method chaining to combine numerous operations together into a single statement. See the DataFrame Class API for a full list of available operations.

df.OrderBy("Name")
    .Filter(df["Age"].ElementwiseGreaterThan(30))
    .PrettyPrint();
Name    Age  Height
Henry   42   1.72
Oliver  23   1.91
Owen    35   1.85

Mathematical Operations

It’s easy to perform math on columns or across multiple DataFrames. In this example we will perform math using two columns and create a new column to hold the output.

DataFrameColumn iqCol = df["Age"] * df["Height"] * 1.5;

double[] iqs = Enumerable.Range(0, (int)iqCol.Length)
    .Select(x => (double)iqCol[x])
    .ToArray();

df.Columns.Add(new PrimitiveDataFrameColumn<double>("IQ", iqs));
df.PrettyPrint();
Name       Age  Height  IQ
Oliver     23   1.91    65.9
Charlotte  19   1.62    46.17
Henry      42   1.72    108.36
Amelia     64   1.57    150.72
Owen       35   1.85    97.12

Statistical Operations

You can iterate across every row of a column to calculate population statistics

foreach (DataFrameColumn col in df.Columns.Skip(1))
{
    // warning: additional care must be taken for datasets which contain null
    double[] values = Enumerable.Range(0, (int)col.Length).Select(x => Convert.ToDouble(col[x])).ToArray();
    (double mean, double std) = MeanAndStd(values);
    Console.WriteLine($"{col.Name} = {mean} +/- {std:N3} (n={values.Length})");
}
Age = 36.6 +/- 15.982 (n=5)
Height = 1.734 +/- 0.130 (n=5)
💡 View the full MeanAndStd() source code
private static (double mean, double std) MeanAndStd(double[] values)
{
	if (values is null)
		throw new ArgumentNullException(nameof(values));

	if (values.Length == 0)
		throw new ArgumentException($"{nameof(values)} must not be empty");

	double sum = 0;
	for (int i = 0; i < values.Length; i++)
		sum += values[i];

	double mean = sum / values.Length;

	double sumVariancesSquared = 0;
	for (int i = 0; i < values.Length; i++)
	{
		double pointVariance = Math.Abs(mean - values[i]);
		double pointVarianceSquared = Math.Pow(pointVariance, 2);
		sumVariancesSquared += pointVarianceSquared;
	}

	double meanVarianceSquared = sumVariancesSquared / values.Length;
	double std = Math.Sqrt(meanVarianceSquared);

	return (mean, std);
}

Plot Values from a DataFrame

I use ScottPlot.NET to visualize data from DataFrames in .NET applications and .NET Interactive Notebooks. ScottPlot can generate a variety of plot types and has many options for customization. See the ScottPlot Cookbook for examples and API documentation.

// Register a custom formatter to display ScottPlot plots as images
using Microsoft.DotNet.Interactive.Formatting;
Formatter.Register(typeof(ScottPlot.Plot), (plt, writer) => 
    writer.Write(((ScottPlot.Plot)plt).GetImageHTML()), HtmlFormatter.MimeType);
// Get the data you wish to display in double arrays
double[] ages = Enumerable.Range(0, (int)df.Rows.Count).Select(x => Convert.ToDouble(df["Age"][x])).ToArray();
double[] heights = Enumerable.Range(0, (int)df.Rows.Count).Select(x => Convert.ToDouble(df["Height"][x])).ToArray();
// Create and display a plot
var plt = new ScottPlot.Plot(400, 300);
plt.AddScatter(ages, heights);
plt.XLabel("Age");
plt.YLabel("Height");
plt

💡 See demo.html for a full length demonstration notebook

If you are only working inside a Notebook and you want all your plots to be HTML and JavaScript, XPlot.Plotly is a good tool to use.

Data may contain null

I didn’t demonstrate it in the code examples above, but note that all column data types are nullable. While null-containing data requires extra considerations when writing mathematical routes, it’s a convenient way to model missing data which is a common occurrence in the real world.

Why not just use LINQ?

I see this question asked frequently, often with an aggressive and condescending tone. LINQ (Language-Integrated Query) is fantastic for performing logical operations on simple collections of data. When you have large 2D datasets of labeled data, advantages of data frames over flat LINQ statements start to become apparent. It is also easy to perform logical operations across multiple data frames, allowing users to write simpler and more readable code than could be achieved with LINQ statements. Data frames also make it much easier to visualize complex data too. In the data science world where complex labeled datasets are routinely compared, manipulated, merged, and visualized, often in an interactive context, the data frames are much easier to work with than raw LINQ statements.

Conclusions

Although I typically reach for Python to perform exploratory data science, it’s good to know that C# has a DataFrame available and that it can be used to inspect and manipulate tabular data. DataFrames pair well with ScottPlot figures in interactive notebooks and are a great way to inspect and communicate complex data. I look forward to watching Microsoft’s Data Analysis namespace continue to evolve as part of their machine learning / ML.NET platform.

Resources