PSK-31 is a narrow-bandwidth digital mode which encodes text as an audio tone that varies phase at a known rate. To learn more about this digital mode and solve a challenging programming problem, I’m going to write a PSK-31 encoder and decoder from scratch using the C# programming language. All code created for this project is open-source, available from my PSK Experiments GitHub Repository, and released under the permissive MIT license. This page documents my progress and notes things I learn along the way.
Although a continuous phase-shifting tone of constant amplitude can successfully transmit PSK31 data, the abrupt phase transitions will cause splatter. If you transmit this you will be heard, but those trying to communicate using adjacent frequencies will be highly disappointed.
Hard phase transitions (splatter)
Soft phase transitions (cleaner)
To reduce spectral artifacts that result from abruptly changing phase, phase transitions are silenced by shaping the waveform envelope as a sine wave so it is silent at the transition. This way the maximum rate of phase shifts is a sine wave with a period of half the baud rate. This is why the opening of a PSK31 message (a series of logical 0 bits) sounds like two tones: It’s the carrier sine wave with an envelope shaped like a sine wave with a period of 31.25/2 Hz. These two tones separated by approximately 15 Hz are visible in the spectrogram (waterfall).
Unlike ASCII (8 bits per character) and RTTY (5 bits per character), BPSK uses Varicode (1-10 bits per character) to encode text. Consecutive zeros 00 separate characters, so character codes must not contain 00 and they must start ane end with a 1 bit. Messages are flanked by a preamble (repeated 0 bits) and a postamble (repeated 1 bits).
+ 111011111 , 1110101 - 110101 . 1010111 / 110101111 0 10110111 1 10111101 2 11101101 3 11111111 4 101110111 5 101011011 6 101101011 7 110101101 8 110101011 9 110110111 : 11110101 ; 110111101 < 111101101 = 1010101 > 111010111 ? 1010101111 @ 1010111101 A 1111101 B 11101011 C 10101101 D 10110101 E 1110111 F 11011011 G 11111101 H 101010101 I 1111111 J 111111101 K 101111101 L 11010111 M 10111011 N 11011101 O 10101011 P 11010101 Q 111011101 R 10101111 S 1101111 T 1101101 U 101010111
V 110110101 X 101011101 Y 101110101 Z 101111011 [ 1010101101 \ 111110111 ] 111101111 ^ 111111011 _ 1010111111 . 101101101 / 1011011111 a 1011 b 1011111 c 101111 d 101101 e 11 f 111101 g 1011011 h 101011 i 1101 j 111101011 k 10111111 l 11011 m 111011 n 1111 o 111 p 111111 q 110111111 r 10101 s 10111 t 101 u 110111 v 1111011 w 1101011 x 11011111 y 1011101 z 111010101 { 1010110111 | 110111011 } 1010110101 ~ 1011010111 DEL 1110110101
Here’s the gist of how I store my varicode table in code. Note that the struct has an additional Description field which is useful for decoding and debugging.
I won’t show how I do the message-to-varicode lookup, but it’s trivial. Here’s the final function I use to generate Varicode bits from a string:
staticint[] GetVaricodeBits(string message)
{
List<int> bits = new();
// add a preamble of repeated zerosfor (int i=0; i<20; i++)
bits.Add(0);
// encode each character of a messageforeach (char character in message)
{
VaricodeSymbol symbol = Lookup(character);
bits.AddRange(symbol.Bits);
bits.AddRange(CharacterSeparator);
}
// add a postamble of repeated onesfor (int i=0; i<20; i++)
bits.Add(1);
return bits.ToArray();
}
Now that we have our Varicode bits, we need to generate an array to indicate phase transitions. A transition occurs every time a bit changes value form the previous bit. This code returns phase as an array of double given the bits from a Varicode message.
This minimal code generates a decipherable PSK-31 message, but it does not silence the phase transitions so it produces a lot of splatter. This function must be refined to shape the waveform such that phase transitions are silenced.
publicdouble[] GetWaveformBPSK(double[] phases)
{
int totalSamples = (int)(phases.Length * SampleRate / BaudRate);
double[] wave = newdouble[totalSamples];
for (int i = 0; i < wave.Length; i++)
{
double time = (double)i / SampleRate;
int frame = (int)(time * BaudRate);
double phaseShift = phases[frame];
wave[i] = Math.Cos(2 * Math.PI * Frequency * time + phaseShift);
}
return wave;
}
This is the same function as above, but with extra logic for amplitude-modulating the waveform in the shape of a sine wave to silence phase transitions.
publicdouble[] GetWaveformBPSK(double[] phases)
{
int baudSamples = (int)(SampleRate / BaudRate);
double samplesPerBit = SampleRate / BaudRate;
int totalSamples = (int)(phases.Length * SampleRate / BaudRate);
double[] wave = newdouble[totalSamples];
// create the amplitude envelope sized for a single bitdouble[] envelope = newdouble[(int)samplesPerBit];
for (int i = 0; i < envelope.Length; i++)
envelope[i] = Math.Sin((i + .5) * Math.PI / envelope.Length);
for (int i = 0; i < wave.Length; i++)
{
// phase modulated carrierdouble time = (double)i / SampleRate;
int frame = (int)(time * BaudRate);
double phaseShift = phases[frame];
wave[i] = Math.Cos(2 * Math.PI * Frequency * time + phaseShift);
// envelope at phase transitionsint firstSample = (int)(frame * SampleRate / BaudRate);
int distanceFromFrameStart = i - firstSample;
int distanceFromFrameEnd = baudSamples - distanceFromFrameStart + 1;
bool isFirstHalfOfFrame = distanceFromFrameStart < distanceFromFrameEnd;
bool samePhaseAsLast = frame == 0 ? false : phases[frame - 1] == phases[frame];
bool samePhaseAsNext = frame == phases.Length - 1 ? false : phases[frame + 1] == phases[frame];
bool rampUp = isFirstHalfOfFrame && !samePhaseAsLast;
bool rampDown = !isFirstHalfOfFrame && !samePhaseAsNext;
if (rampUp)
wave[i] *= envelope[distanceFromFrameStart];
if (rampDown)
wave[i] *= envelope[distanceFromFrameEnd];
}
return wave;
}
I wrapped the functionality above in a Windows Forms GUI that allows the user to type a message, specify frequency, baud rate, and whether or not to refine the envelope to reduce splatter, then either play or save the result. An interactive ScottPlot Chart allows the user to inspect the waveform.
Let’s see what PSK-3 sounds like. This mode encodes data at a rate of 3 bits per second. Note that Varicode characters may require up to ten bits, so this is pretty slow. On the other hand the side tones are closer to the carrier and the total bandwidth is much smaller. The message here has been shortened to just my callsign, AJ4VD.
Considering all the steps for encoding PSK-31 transmissions are already described on this page, it doesn’t require too much additional effort to create a decoder using basic software techniques. Once symbol phases are detected it’s easy to work backwards: detect phase transitions (logical 0s) or repeats (logical 1s), treat consecutive zeros as a character separator, then look-up characters according to the Varicode table. The tricky bit is analyzing the source audio to generate the array of phase offsets.
The simplest way to decode PSK-31 transmissions leans on the fact that we already know the baud rate: 31.25 symbols per second, or one symbol every 256 samples at 8kHz sample rate. The audio signal can be segregated into many 256-sample bins, and processed by FFT. Once the center frequency is determined, the FFT power at this frequency can be calculated for each bin. Signal offset can be adjusted to minimize the imaginary component of the FFTs at the carrier frequency, then the real component will be strongly positive or negative, allowing phase transitions to be easily detected.
There are more advanced techniques to improve BPSK decoding, such as continuously adjusting frequency and phase alignment (synchronization). A Costas loop can help lock onto the carrier frequency while preserving its phase. Visit Simple BPSK31 Decoding with Python for an excellent demonstration of how to decode BPSK31 using these advanced techniques.
A crude C# implementation of a BPSK decoded is available on GitHub in the PSK Experiments repository
Since BPSK is just a carrier that applies periodic 180º phase-shifts, it’s easy to generate in hardware by directly modulating the signal source. A good example of this is KA7OEI’s PSK31 transmitter which feeds the output of an oscillator through an even or odd number of NAND gates (from a 74HC00) to produce two signals of opposite phase.
Unlike the 0º and 180º phases of binary phase shift keying (BPSK), quadrature phase shift keying (QPSK) encodes extra data into each symbol by uses a larger number of phases. When QPSK-31 is used in amateur radio these extra bits aren’t used to send messages faster but instead send them more reliably using convolutional coding and error correction. These additional features come at a cost (an extra 3 dB SNR is required), and in practice QPSK is not used as much by amateur radio operators.
QPSK encoding/decoding and convolutional encoding/decoding are outside the scope of this page, but excellent information exists on the Wikipedia: QPSK and in the US Naval Academy’s EC314 Lesson 23: Digital Modulation document.
After all that, it turns out PSK-31 isn’t that popular anymore. These days it seems the FT-8 digital mode with WSJT-X software is orders of magnitude more popular ??
Cubic spline interpolation can be used to modify the sample rate of time series data. This page describes how I achieve signal resampling with spline interpolation in pure C# without any external dependencies. This technique can be used to:
Convert unevenly-sampled data to a series of values with a fixed sample rate
Convert time series data from one sample rate to another sample rate
Fill-in missing values from a collection of measurements
The generated points line-up perfectly with the sampled data.
There is slight deviation from the theoretical signal (and it’s larger where there is more missing data) but this is an unsurprising result considering the original samples had large gaps of missing data.
The interpolation method described above only considered the horizontal axis when generating evenly-spaced time points (1D interpolation). For information and code examples regarding 2D and 3D cubic spline interpolation, see my previous blog post: Spline Interpolation with C#
Spline Interpolation with ScottPlot - Demonstrates additional types of interpolation: Bezier, Catmull-Rom, Chaikin, Cubic, etc. The project is open source under a MIT license.
Today I created a Blazor WebAssembly app that shows a progress bar while the page loads. This is especially useful for users on slow connections because Blazor apps typically require several megabytes of DLL and DAT files to be downloaded before meaningful content appears on the page.
See Bootstrap’s progressbar page for extensive customization and animation options and best practices when working with progress indicators. Also ensure the version of Bootstrap in your Blazor app is consistent with the documentation/HTML you are referencing.
Add a script to the bottom of the page to start Blazor manually, identifying all the resources needed and incrementally downloading them while updating the progressbar along the way.
This week Microsoft officially released .NET Maui and the new Microsoft.Maui.Graphics library which can draw 2D graphics in any .NET application (not just Maui apps). This page offers a quick look at how to use this new library to draw graphics using SkiaSharp in a .NET 6 console application. The C# Data Visualization site has additional examples for drawing and animating graphics using Microsoft.Maui.Graphics in Windows Forms and WPF applications.
The code below is a full .NET 6 console application demonstrating common graphics tasks (setting colors, drawing shapes, rendering text, etc.) and was used to generate the image above.
// These packages are available on NuGetusingMicrosoft.Maui.Graphics;
usingMicrosoft.Maui.Graphics.Skia;
// Create a bitmap in memory and draw on its CanvasSkiaBitmapExportContext bmp = new(600, 400, 1.0f);
ICanvas canvas = bmp.Canvas;
// Draw a big blue rectangle with a dark borderRect backgroundRectangle = new(0, 0, bmp.Width, bmp.Height);
canvas.FillColor = Color.FromArgb("#003366");
canvas.FillRectangle(backgroundRectangle);
canvas.StrokeColor = Colors.Black;
canvas.StrokeSize = 20;
canvas.DrawRectangle(backgroundRectangle);
// Draw circles randomly around the imagefor (int i = 0; i < 100; i++)
{
float x = Random.Shared.Next(bmp.Width);
float y = Random.Shared.Next(bmp.Height);
float r = Random.Shared.Next(5, 50);
Color randomColor = Color.FromRgb(
red: Random.Shared.Next(255),
green: Random.Shared.Next(255),
blue: Random.Shared.Next(255));
canvas.StrokeSize = r / 3;
canvas.StrokeColor = randomColor.WithAlpha(.3f);
canvas.DrawCircle(x, y, r);
}
// Measure a stringstring myText = "Hello, Maui.Graphics!";
Font myFont = new Font("Impact");
float myFontSize = 48;
canvas.Font = myFont;
SizeF textSize = canvas.GetStringSize(myText, myFont, myFontSize);
// Draw a rectangle to hold the stringPoint point = new(
x: (bmp.Width - textSize.Width) / 2,
y: (bmp.Height - textSize.Height) / 2);
Rect myTextRectangle = new(point, textSize);
canvas.FillColor = Colors.Black.WithAlpha(.5f);
canvas.FillRectangle(myTextRectangle);
canvas.StrokeSize = 2;
canvas.StrokeColor = Colors.Yellow;
canvas.DrawRectangle(myTextRectangle);
// Daw the string itselfcanvas.FontSize = myFontSize * .9f; // smaller than the rectanglecanvas.FontColor = Colors.White;
canvas.DrawString(myText, myTextRectangle,
HorizontalAlignment.Center, VerticalAlignment.Center, TextFlow.OverflowBounds);
// Save the image as a PNG filebmp.WriteToFile("console2.png");
The Microsoft.Maui.Graphics namespace a small collection of interfaces which can be implemented by many different rendering technologies (SkiaSharp, SharpDX, GDI, etc.), making it possible to create drawing routines that are totally abstracted from the underlying graphics rendering system.
I really like that I can now create a .NET Standard 2.0 project that exclusively uses interfaces from Microsoft.Maui.Graphics to write code that draws complex graphics, then reference that code from other projects that use platform-specific graphics libraries to render the images.
When I write scientific simulations or data visualization code I frequently regard my graphics drawing routines as business logic, and drawing with Maui.Graphics lets me write this code to an abstraction that keeps rendering technology dependencies out of my business logic - a big win!
After working with this library while it was being developed over the last few months, these are the things I find most limiting in my personal projects which made it through the initial release this week. Some of them are open issues so they may get fixed soon, and depending on how the project continues to evolve many of these rough edges may improve with time. I’m listing them here now so I can keep track of them, and I intend to update this list if/as these topics improve:
Note: This section was last reviewed on April 25, 2022 and improvements may have been made since this text was written.
Strings cannot be accurately measured: The size returned by GetStringSize() is inaccurate and does not respect font. There’s an issue tracking this (#279), but it’s been open for more than three months and the library was released this week in its broken state.
EDIT: I concede multi-platform font support is a very hard problem, but this exactly the type of problem that .NET Maui was created to solve.
Missing XML documentation: Intellisense can really help people who are new to a library. The roll-out of a whole new application framework is a good example of a time when a lot of people will be exploring a new library. Let’s take the Color class for example (which 100% of people will interact with) and consider misunderstandings that could be prevented by XML documentation and intellisense: If new Color() accepts 3 floats, should they be 0-255 or 0-1? I need to make a color from the RGB web value #003366, why does Color.FromHex() tell me to use FromArgb? Web colors are RGBA, should I use FromRrgba()? But wait, that string is RGB, not ARGB or RGBA, so will it throw an exception? What does Color.Parse() do?
Edit 1: Some of these answers are documented in source code, but they are not XML docs, so this information is not available to library users.
Edit 2: Is it on the open-source community to contribute XML documentation? If so, fair enough, but it is a very extensive effort (to write and to review), so a call should be put out for this job to ensure someone doesn’t go through all the effort then have their open PR sit unmerged for months while it falls out of sync with the main branch.
The library has signs of being incomplete: There remain a good number of NotImplementedException and // todo in sections of the code base that indicate additional work is still required.
Again, I’m pointing these things out the very first week .NET Maui was released, so there’s plenty of time and opportunity for improvements in the coming weeks and months.
I’m optimistic this library will continue to improve, and I am very excited to watch it progress! I’m not aware of the internal pressures and constraints that led to the library being released like it was this week, but I want to end by complimenting the team on their great job so far and encourage everyone (at Microsoft and in the open-source community at large) to keep moving this library forward. The .NET Maui team undertook an ambitious challenge by setting-out to implement cross-platform graphics support, but achieving this goal elegantly will be a huge accomplishment for the .NET community!
The DataFrame is a data structure designed for manipulation, analysis, and visualization of tabular data, and it is the cornerstone of many data science applications. One of the most famous implementations of the data frame is provided by the Pandas package for Python. An equivalent data structure is available for C# using Microsoft’s data analysis package. Although data frames are commonly used in Jupyter notebooks, they can be used in standard .NET applications as well. This article surveys Microsoft’s Data Analysis package and introduces how to interact with with data frames using C# and the .NET platform.
A custom PrettyPrint() extension method can improve DataFrame readability. Implementing this as an extension method allows me to call df.PrettyPrint() anywhere in my code.
💡 View the full PrettyPrinters.cs source code
usingMicrosoft.Data.Analysis;
usingSystem.Text;
internalstaticclassPrettyPrinters{
publicstaticvoid PrettyPrint(this DataFrame df) => Console.WriteLine(PrettyText(df));
publicstaticstring PrettyText(this DataFrame df) => ToStringArray2D(df).ToFormattedText();
publicstaticstring ToMarkdown(this DataFrame df) => ToStringArray2D(df).ToMarkdown();
publicstaticvoid PrettyPrint(this DataFrameRow row) => Console.WriteLine(Pretty(row));
publicstaticstring Pretty(this DataFrameRow row) => row.Select(x => x?.ToString() ?? string.Empty).StringJoin();
privatestaticstring StringJoin(this IEnumerable<string> strings) => string.Join(" ", strings.Select(x => x.ToString()));
privatestaticstring[,] ToStringArray2D(DataFrame df)
{
string[,] strings = newstring[df.Rows.Count + 1, df.Columns.Count];
for (int i = 0; i < df.Columns.Count; i++)
strings[0, i] = df.Columns[i].Name;
for (int i = 0; i < df.Rows.Count; i++)
for (int j = 0; j < df.Columns.Count; j++)
strings[i + 1, j] = df[i, j]?.ToString() ?? string.Empty;
return strings;
}
privatestaticint[] GetMaxLengthsByColumn(thisstring[,] strings)
{
int[] maxLengthsByColumn = newint[strings.GetLength(1)];
for (int y = 0; y < strings.GetLength(0); y++)
for (int x = 0; x < strings.GetLength(1); x++)
maxLengthsByColumn[x] = Math.Max(maxLengthsByColumn[x], strings[y, x].Length);
return maxLengthsByColumn;
}
privatestaticstring ToFormattedText(thisstring[,] strings)
{
StringBuilder sb = new();
int[] maxLengthsByColumn = GetMaxLengthsByColumn(strings);
for (int y = 0; y < strings.GetLength(0); y++)
{
for (int x = 0; x < strings.GetLength(1); x++)
{
sb.Append(strings[y, x].PadRight(maxLengthsByColumn[x] + 2));
}
sb.AppendLine();
}
return sb.ToString();
}
privatestaticstring ToMarkdown(thisstring[,] strings)
{
StringBuilder sb = new();
int[] maxLengthsByColumn = GetMaxLengthsByColumn(strings);
for (int y = 0; y < strings.GetLength(0); y++)
{
for (int x = 0; x < strings.GetLength(1); x++)
{
sb.Append(strings[y, x].PadRight(maxLengthsByColumn[x]));
if (x < strings.GetLength(1) - 1)
sb.Append(" | ");
}
sb.AppendLine();
if (y == 0)
{
for (int i = 0; i < strings.GetLength(1); i++)
{
int bars = maxLengthsByColumn[i] + 2;
if (i == 0)
bars -= 1;
sb.Append(new String('-', bars));
if (i < strings.GetLength(1) - 1)
sb.Append("|");
}
sb.AppendLine();
}
}
return sb.ToString();
}
}
Name Age Height
Oliver 231.91Charlotte 191.62Henry 421.72Amelia 641.57Owen 351.85
I can create similar methods to format a DataFrame as Markdown or HTML.
Name | Age | Height
----------|-----|--------
Oliver | 23 | 1.91
Charlotte | 19 | 1.62
Henry | 42 | 1.72
Amelia | 64 | 1.57
Owen | 35 | 1.85
Previously users had to create custom HTML formatters to properly display DataFrames in .NET Interactive Notebooks, but these days it works right out of the box.
💡 See demo.html for a full length demonstration notebook
The DataFrame class has numerous operations available to sort, filter, and analyze data in many different ways. A popular pattern when working with DataFrames is to use method chaining to combine numerous operations together into a single statement. See the DataFrame Class API for a full list of available operations.
It’s easy to perform math on columns or across multiple DataFrames. In this example we will perform math using two columns and create a new column to hold the output.
You can iterate across every row of a column to calculate population statistics
foreach (DataFrameColumn col in df.Columns.Skip(1))
{
// warning: additional care must be taken for datasets which contain nulldouble[] values = Enumerable.Range(0, (int)col.Length).Select(x => Convert.ToDouble(col[x])).ToArray();
(double mean, double std) = MeanAndStd(values);
Console.WriteLine($"{col.Name} = {mean} +/- {std:N3} (n={values.Length})");
}
I use ScottPlot.NET to visualize data from DataFrames in .NET applications and .NET Interactive Notebooks. ScottPlot can generate a variety of plot types and has many options for customization. See the ScottPlot Cookbook for examples and API documentation.
// Register a custom formatter to display ScottPlot plots as imagesusingMicrosoft.DotNet.Interactive.Formatting;
Formatter.Register(typeof(ScottPlot.Plot), (plt, writer) =>
writer.Write(((ScottPlot.Plot)plt).GetImageHTML()), HtmlFormatter.MimeType);
// Get the data you wish to display in double arraysdouble[] ages = Enumerable.Range(0, (int)df.Rows.Count).Select(x => Convert.ToDouble(df["Age"][x])).ToArray();
double[] heights = Enumerable.Range(0, (int)df.Rows.Count).Select(x => Convert.ToDouble(df["Height"][x])).ToArray();
// Create and display a plotvar plt = new ScottPlot.Plot(400, 300);
plt.AddScatter(ages, heights);
plt.XLabel("Age");
plt.YLabel("Height");
plt
💡 See demo.html for a full length demonstration notebook
If you are only working inside a Notebook and you want all your plots to be HTML and JavaScript, XPlot.Plotly is a good tool to use.
I didn’t demonstrate it in the code examples above, but note that all column data types are nullable. While null-containing data requires extra considerations when writing mathematical routes, it’s a convenient way to model missing data which is a common occurrence in the real world.
I see this question asked frequently, often with an aggressive and condescending tone. LINQ (Language-Integrated Query) is fantastic for performing logical operations on simple collections of data. When you have large 2D datasets of labeled data, advantages of data frames over flat LINQ statements start to become apparent. It is also easy to perform logical operations across multiple data frames, allowing users to write simpler and more readable code than could be achieved with LINQ statements. Data frames also make it much easier to visualize complex data too. In the data science world where complex labeled datasets are routinely compared, manipulated, merged, and visualized, often in an interactive context, the data frames are much easier to work with than raw LINQ statements.
Although I typically reach for Python to perform exploratory data science, it’s good to know that C# has a DataFrame available and that it can be used to inspect and manipulate tabular data. DataFrames pair well with ScottPlot figures in interactive notebooks and are a great way to inspect and communicate complex data. I look forward to watching Microsoft’s Data Analysis namespace continue to evolve as part of their machine learning / ML.NET platform.