This website is a static site containing thousands of pages, and I recently had the desire to check all of them for broken links and images. Although it would be a tedious task to perform manually, I was able to automate the process using Python. This page describes the process I used to identify broken links and images and may be helpful to others using static site generator tools like Jekyll, Hugo, Eleventy, Pelican, Gatsby, other Jamstack site generators, or hand-written HTML files.
Although my website content is stored as markdown files, I found it most convenient to check the generated HTML files for broken link and image URLs. I generated my static site locally, then manually removed pagination folders so only article HTML files were present. I then wrote the following Python script which uses pathlib to locate HTML files and BeautifulSoup to analyze a href and img src attributes and saves all URLs identified as a CSV file.
importpathlibfrombs4import BeautifulSoup
defget_urls(file: pathlib.Path) ->list:
"""Return link and image URLs in a HTML file"""withopen(file, errors='ignore') as f:
html = f.read()
soup = BeautifulSoup(html, 'html.parser')
urls =set()
for link in soup.find_all('a'):
href = link.get('href')
if href.startswith("#"):
continueif"#"in href:
href = href.split("#")[0]
href =str(href).strip('/')
urls.add(href)
for image in soup.find_all('img'):
src = image.get('src')
urls.add(str(src))
returnsorted(list(urls))
defget_urls_by_page(folder: pathlib.Path) ->dict:
"""Return link and image URLs for all HTML files in a folder""" html_files =list(folder.rglob("*.html"))
urls_by_page = {}
for i, html_file inenumerate(html_files):
urls = get_urls(html_file)
urls_by_page[html_file] = urls
print(f"{i+1} of {len(html_files)}: {len(urls)} URLs found in {html_file}")
return urls_by_page
defwrite_csv(urls_by_page: dict, csv_file: pathlib.Path):
txt ='URL, Page\n'for page, urls in urls_by_page.items():
for url in urls:
txt +=f'{url}, {page}\n' csv_file.write_text(txt)
if __name__ =="__main__":
folder = pathlib.Path("public")
urls_by_page = get_urls_by_page(folder)
write_csv(urls_by_page, pathlib.Path("urls.csv"))
Running the script generated a CSV report showing every link on my website, organized by which page it’s on. It looks like my blog has over 7,000 links! That would be a lot to check by hand.
Each URL from the report was checked using HEAD requests. Note that a HEAD request to a file path only returns HTTP headers but does not actually download the file, allowing it to consume far less bandwidth GET requests. Inspecting the HTTP response code indicates whether the URL is a valid path to a file (code 200).
I used python sets to prevent checking the same URL twice. I logged good and broken URLs as they were checked, and consumed these log files at startup to allow the program to be stopped and restarted without causing it to check the same URL twice.
importpathlibimporturllib.requestimporttimedefis_url_valid(url: str) ->bool:
"""Check if a URL exists without downloading its contents""" request = urllib.request.Request(url)
request.get_method =lambda: 'HEAD'try:
urllib.request.urlopen(request, timeout=5)
returnTrueexceptExceptionas e:
print(f"ERROR: {e}")
returnFalseif __name__ =="__main__":
# load URLs from report urls = [x.split(", ")[0] for x
in pathlib.Path("urls.csv").read_text().split("\n")
if x.startswith("http")]
# load previously checked URLs url_file_good = pathlib.Path("urls-good.txt")
url_file_good.touch()
url_file_bad = pathlib.Path("urls-bad.txt")
url_file_bad.touch()
checked =set()
for url in url_file_good.read_text().split("\n"):
checked.add(url)
for url in url_file_bad.read_text().split("\n"):
checked.add(url)
# check each URLfor i, url inenumerate(urls):
print(f"{i+1} of {len(urls)}", end=" ")
if url in checked:
print(f"SKIPPING {url}")
continue time.sleep(.2)
print(f"CHECKING {url}")
log_file = url_file_good if is_url_valid(url) else url_file_bad
withopen(log_file, 'a') as f:
f.write(url+"\n")
checked.add(url)
I wrote a Python script to generate a HTML report of all the broken links on my website. The script shows every broken link found on the website and lists all the pages on which each broken link appears.
importpathliburls_and_pages = [x.split(", ") for x
in pathlib.Path("urls.csv").read_text().split("\n")
if x.startswith("http")]
urls_broken = [x for x
in pathlib.Path("urls-bad.txt").read_text().split("\n")
if x.startswith("http")]
urls_broken =set(urls_broken)
html ="<h1>Broken Links</h1>"for url_broken in urls_broken:
html +=f"<div class='mt-4'><a href='{url_broken}'>{url_broken}</a></div>" pages = [x[1] for x in urls_and_pages if x[0] == url_broken]
for page in pages:
html +=f"<div><code>{page}</code></div>"
I wrapped the output in Bootstrap to make the report look pretty and appear properly on mobile devices:
The generated report lets me focus my effort narrowly to tactically fix the broken URLs I think are most important (e.g., image URLs, URLs pointing domain names).
Testing URLs extracted from a folder of HTML files proved to be an effective method for identifying broken links and images across a large static website with thousands of pages.
This method could be employed in a CI/CD pipeline to ensure links and image paths are valid on newly created pages.
If the goal is to validate internal links only, HTTP requests could be replaced with path existence checks if the entire website is present in the local filesystem.
Treemap diagrams display a series of positive numbers using rectangles sized proportional to the value of each number. This page demonstrates how to calculate the size and location of rectangles to create a tree map diagram using C#. Although the following uses System.Drawing to save the tree map as a Bitmap image, these concepts may be combined with information on the C# Data Visualization page to create treemap diagrams using SkiaSharp, WPF, or other graphics technologies.
The tree map above was generated from random data using the following C# code:
// Create sample data. Data must be sorted large to small.double[] sortedValues = Enumerable.Range(0, 40)
.Select(x => (double)Random.Shared.Next(10, 100))
.OrderByDescending(x => x)
.ToArray();
// Create an array of labels in the same order as the sorted data.string[] labels = sortedValues.Select(x => x.ToString()).ToArray();
// Calculate the size and position of all rectangles in the tree mapint width = 600;
int height = 400;
RectangleF[] rectangles = TreeMap.GetRectangles(sortedValues, width, height);
// Create an image to draw on (with 1px extra to make room for the outline)usingBitmap bmp = new(width + 1, height + 1);
usingGraphics gfx = Graphics.FromImage(bmp);
usingFont fnt = new("Consolas", 8);
usingSolidBrush brush = new(Color.Black);
gfx.Clear(Color.White);
// Draw and label each rectanglefor (int i = 0; i < rectangles.Length; i++)
{
brush.Color = Color.FromArgb(
red: Random.Shared.Next(150, 250),
green: Random.Shared.Next(150, 250),
blue: Random.Shared.Next(150, 250));
gfx.FillRectangle(brush, rectangles[i]);
gfx.DrawRectangle(Pens.Black, rectangles[i]);
gfx.DrawString(labels[i], fnt, Brushes.Black, rectangles[i].X, rectangles[i].Y);
}
// Save the outputbmp.Save("treemap.bmp");
The previous code block focuses on data generation and display, but hides the tree map calculations behind the TreeMap class. Below is the code for that class. It is self-contained static class and exposes a single static method which takes a pre-sorted array of values and returns tree map rectangles ready to display on an image.
💡 Although the System.Drawing.Common is a Windows-only library (as of .NET 7), System.Drawing.Primitives is a cross-platform package that provides the RectangleF structure used in the tree map class. See the SkiaSharp Quickstart to learn how to create image files using cross-platform .NET code.
publicstaticclassTreeMap{
publicstatic RectangleF[] GetRectangles(double[] values, int width, int height)
{
for (int i = 1; i < values.Length; i++)
if (values[i] > values[i - 1])
thrownew ArgumentException("values must be ordered large to small");
var slice = GetSlice(values, 1, 0.35);
var rectangles = GetRectangles(slice, width, height);
return rectangles.Select(x => x.ToRectF()).ToArray();
}
privateclassSlice {
publicdouble Size { get; }
public IEnumerable<double> Values { get; }
public Slice[] Children { get; }
public Slice(double size, IEnumerable<double> values, Slice sub1, Slice sub2)
{
Size = size;
Values = values;
Children = new Slice[] { sub1, sub2 };
}
public Slice(double size, double finalValue)
{
Size = size;
Values = newdouble[] { finalValue };
Children = Array.Empty<Slice>();
}
}
privateclassSliceResult {
publicdouble ElementsSize { get; }
public IEnumerable<double> Elements { get; }
public IEnumerable<double> RemainingElements { get; }
public SliceResult(double elementsSize, IEnumerable<double> elements, IEnumerable<double> remainingElements)
{
ElementsSize = elementsSize;
Elements = elements;
RemainingElements = remainingElements;
}
}
privateclassSliceRectangle {
public Slice Slice { get; set; }
publicfloat X { get; set; }
publicfloat Y { get; set; }
publicfloat Width { get; set; }
publicfloat Height { get; set; }
public SliceRectangle(Slice slice) => Slice = slice;
public RectangleF ToRectF() => new(X, Y, Width, Height);
}
privatestatic Slice GetSlice(IEnumerable<double> elements, double totalSize, double sliceWidth)
{
if (elements.Count() == 1)
returnnew Slice(totalSize, elements.Single());
SliceResult sr = GetElementsForSlice(elements, sliceWidth);
Slice child1 = GetSlice(sr.Elements, sr.ElementsSize, sliceWidth);
Slice child2 = GetSlice(sr.RemainingElements, 1 - sr.ElementsSize, sliceWidth);
returnnew Slice(totalSize, elements, child1, child2);
}
privatestatic SliceResult GetElementsForSlice(IEnumerable<double> elements, double sliceWidth)
{
var elementsInSlice = new List<double>();
var remainingElements = new List<double>();
double current = 0;
double total = elements.Sum();
foreach (var element in elements)
{
if (current > sliceWidth)
remainingElements.Add(element);
else {
elementsInSlice.Add(element);
current += element / total;
}
}
returnnew SliceResult(current, elementsInSlice, remainingElements);
}
privatestatic IEnumerable<SliceRectangle> GetRectangles(Slice slice, int width, int height)
{
SliceRectangle area = new(slice) { Width = width, Height = height };
foreach (var rect in GetRectangles(area))
{
if (rect.X + rect.Width > area.Width)
rect.Width = area.Width - rect.X;
if (rect.Y + rect.Height > area.Height)
rect.Height = area.Height - rect.Y;
yieldreturn rect;
}
}
privatestatic IEnumerable<SliceRectangle> GetRectangles(SliceRectangle sliceRectangle)
{
var isHorizontalSplit = sliceRectangle.Width >= sliceRectangle.Height;
var currentPos = 0;
foreach (var subSlice in sliceRectangle.Slice.Children)
{
var subRect = new SliceRectangle(subSlice);
int rectSize;
if (isHorizontalSplit)
{
rectSize = (int)Math.Round(sliceRectangle.Width * subSlice.Size);
subRect.X = sliceRectangle.X + currentPos;
subRect.Y = sliceRectangle.Y;
subRect.Width = rectSize;
subRect.Height = sliceRectangle.Height;
}
else {
rectSize = (int)Math.Round(sliceRectangle.Height * subSlice.Size);
subRect.X = sliceRectangle.X;
subRect.Y = sliceRectangle.Y + currentPos;
subRect.Width = sliceRectangle.Width;
subRect.Height = rectSize;
}
currentPos += rectSize;
if (subSlice.Values.Count() > 1)
{
foreach (var sr in GetRectangles(subRect))
{
yieldreturn sr;
}
}
elseif (subSlice.Values.Count() == 1)
{
yieldreturn subRect;
}
}
}
}
A few days ago I wrote an article describing how to programmatically generate .NET source code analytics using C#. Using these tools I analyzed the source code for all classes in a large project (ScottPlot.NET). The following tree map displays every class in the project as a rectangle sized according to number of lines of code and colored according to maintainability.
In this diagram large rectangles represent classes with the most code, and red color indicates classes that are difficult to maintain.
💡 I’m using a perceptually uniform colormap (similar to Turbo)provided by the ScottPlot provided by the ScottPlot NuGet package. See ScottPlot’s colormaps gallery for all available colormaps.
💡 The Maintainability Index is a value between 0 (worst) and 100 (best) that represents the relative ease of maintaining the code. It’s calculated from a combination of Halstead complexity (size of the compiled code), Cyclomatic complexity (number of paths that can be taken through the code), and the total number of lines of code.
This page describes how to use the Microsoft.CodeAnalysis.Metrics package to perform source code analysis of .NET assemblies from a console application. Visual Studio users can perform source code analysis by clicking the “Analyze” dropdown menu and selecting “Calculate Code Metrics”, but I sought to automate this process so I can generate custom code analysis reports from console applications as part of my CI pipeline.
The code analysis XML contains information about every assembly, namespace, type, and function in the whole code base! There is a lot of possible information to extract, but the code below is enough to get us started extracting basic metric information for every type in the code base.
usingSystem;
usingSystem.IO;
usingSystem.Linq;
usingSystem.Text;
usingSystem.Xml.Linq;
usingSystem.Collections.Generic;
/// <summary>/// Display a particular metric for every type in an assembly./// </summary>void RankTypes(string xmlFilePath, string metricName = "CyclomaticComplexity", bool highToLow = true)
{
string xmlText = File.ReadAllText(xmlFilePath);
XDocument doc = XDocument.Parse(xmlText);
XElement assembly = doc.Descendants("Assembly").First();
var rankedTypes = GetMetricByType(assembly, metricName).OrderBy(x => x.Value).ToArray();
if (highToLow)
Array.Reverse(rankedTypes);
Console.WriteLine($"Types ranked by {metricName}:");
foreach (var type in rankedTypes)
Console.WriteLine($"{type.Value:N0}\t{type.Key}");
}
Dictionary<string, int> GetMetricByType(XElement assembly, string metricName)
{
Dictionary<string, int> metricByType = new();
foreach (XElement namespaceElement in assembly.Element("Namespaces")!.Elements("Namespace"))
{
foreach (XElement namedType in namespaceElement.Elements("Types").Elements("NamedType"))
{
XElement metric = namedType.Element("Metrics")!.Elements("Metric")
.Where(x => x.Attribute("Name")!.Value == metricName)
.Single();
string typeName = namedType.Attribute("Name")!.Value;
string namespaceName = namespaceElement.Attribute("Name")!.Value;
string fullTypeName = $"{namespaceName}.{typeName}";
metricByType[fullTypeName] = int.Parse(metric.Attribute("Value")!.Value!.ToString());
}
}
return metricByType;
}
Specific metrics of interest will vary, but here are some code examples demonstrating how to parse the code metrics file to display useful information. For these examples I run the code analysis command above to generate ScottPlot.Metrics.xml from the ScottPlot code base and use the code above to generate various reports.
Cyclomatic complexity is a measure of the number of different paths that can be taken through a computer program, and it is often used as an indicator for difficult-to-maintain code. Some CI systems even prevent the merging of pull requests if their cyclomatic complexity exceeds a predefined threshold! Although I don’t intend to gate pull requests by complexity at this time, I would like to gain insight into which classes are the most complex as a way to quantitatively target my code maintenance and efforts.
Similarly, ranking all my project’s types by how many lines of code they contain can give me insight into which types may benefit most from refactoring.
The Maintainability Index is a value between 0 (worst) and 100 (best) that represents the relative ease of maintaining the code. It’s calculated from a combination of Halstead complexity (size of the compiled code), Cyclomatic complexity (number of paths that can be taken through the code), and the total number of lines of code.
The maintainability index is calculated by Microsoft.CodeAnalysis.Metrics so we don’t have to. I don’t know how Microsoft arrived at their weights for this formula, but the overall idea is described here.
With a little more effort you can generate HTML reports that use tables and headings to highlight useful code metrics and draw attention to types that could benefit from refactoring to improve maintainability.
Microsoft’s official Microsoft.CodeAnalysis.Metrics NuGet package is a useful tool for analyzing assemblies, navigating through namespaces, types, properties, and methods, and evaluating their metrics. Since these analyses can be performed using console applications, they can be easily integrated into CI pipelines or used to create standalone code analysis applications. Future projects can build on the concepts described here to create graphical visualizations of code metrics in large projects.
NDepend is commercial software for performing code analysis on .NET code bases and has many advanced features that make it worth considering for organizations that wish to track code quality and who can afford the cost. The NDepend Sample Reports demonstrate useful ways to report code analysis metrics.
I often find it useful to gate frequency counters using a 1 pulse per second (1PPS) signal derived from a 10 MHz precision frequency reference. However, a divide-by-10-million circuit isn’t trivial to implement. Counter ICs exist which enable divide-by-100 by combining multiple divide-by-2 and divide-by-5 stages (e.g., MC74HC390A around $0.85 each), but dividing 10 MHz all the way down to 1Hz would require at least 4 of these chips and a lot of wiring.
You can clock a microcontroller at 10 MHz and use its timer and interrupt systems to generate 1PPS. For example, an ATTiny202 in an 8-pin SOIC package is available from Mouser (>50k stocked) for $0.51 each. Note that modern series AVRs require a special UDPI programmer.
ATTiny202 ($0.51)
ATTint826 ($0.95)
This page documents a divide-by-10-million circuit achieved with a single microcontroller to scale a 10MHz frequency reference down to 1PPS. I’m using an ATTiny826 because that is what I have on hand, but these concepts apply to any microcontroller with a 16-bit timer.
Some AVRs come in DIP packages but their pin numbers may be different than the same chip in a SMT package. To facilitate prototyping using designs and code that will work identically across a breadboard prototype and a PCB containing SMT chips, I prefer to build DIP breakout boards using whatever SMT package I intend to include in my final builds. In this case it’s ATTint826 in a SOIC-20 package, and I can easily use this in a breadboard by soldering them onto SOIC-to-DIP breakout boards.
I assembled the breakout board by hand using a regular soldering iron. When working with small packages it helps so much to coat the pins with a little tack flux to facilitate wetting and prevent solder bridges. I’m presently using Chip Quik NC191. Even if flux is advertized as “no-clean”, it’s good practice and makes the boards look much nicer to remove remaining flux with acetone and Q-tips or brushes.
FTDI breakout board for power: To test this design I’m using a FT232 breakout board just to provide easy access to GND and Vcc (5V from the USB rail).
10 MHz can oscillator: It’s not ovenized or GPS disciplined, but I’m using this as a stand-in for whatever high-precision 10 MHz frequency standard will eventually be used in this circuit. The important thing is just to know that it outputs 0-5V square waves at 10 MHz going into the EXTCLK pin of the microcontroller
Traditional AVR microcontrollers used fuse bits to set the clock source, but modern series chips can change the clock source from within code. However, modifying the clock source requires temporarily disabling the configuration change protection (CCP) system.
Disabling the CCP only lasts four clock cycles, so the immediate next statement must be assignment of the new value. I use the following function to facilitate this action.
/* Write a value to a CCP-protected register */voidccp_write(volatileregister8_t* address, uint8_t value){
CCP = CCP_IOREG_gc;
*address = value;
}
// Use internal 20 MHz clock with CKOUT pin enabled
ccp_write(&CLKCTRL.MCLKCTRLA, CLKCTRL.MCLKCTRLA | CLKCTRL_CLKOUT_bm);
Do not use compound statements when writing to the CCP register. The code below fails to change clock as one may expect by looking at the code, presumably because the combined OR operation with the assignment exceeds four clock cycles. Instead of direct assignment, use the ccp_write function described above.
// WARNING: This code does not actually change the clock source
CCP = CCP_IOREG_gc;
CLKCTRL.MCLKCTRLA = CLKCTRL.MCLKCTRLA | CLKCTRL_CLKOUT_bm;
This is how I configured my ATTiny826’s TCA0 16-bit timer to fire an interrupt every 200 ms.
Prescale: By enabling a divide-by-64 prescaler, my 10 MHz input becomes 156,250 Hz.
Top: By setting the top of my 16-bit counter at 31,250, I achieve exactly 5 overflows per second (once every 200 ms).
Interrupt: By enabling an overflow interrupt, I am able to call a function every 200 ms.
voidconfigure_1pps(){
// 10 MHz system clock with div64 prescaler is 156,250 Hz.
// Setting a 16-bit timer's top to 31,250 means 5 overflows per second.
TCA0.SINGLE.INTCTRL = TCA_SINGLE_OVF_bm; // overflow interrupt
TCA0.SINGLE.CTRLB = TCA_SINGLE_WGMODE_NORMAL_gc; // normal mode
TCA0.SINGLE.PER =31249UL; // control timer period by setting timer top
TCA0.SINGLE.CTRLA |= TCA_SINGLE_CLKSEL_DIV64_gc; // set clock source
TCA0.SINGLE.CTRLA |= TCA_SINGLE_ENABLE_bm; // start timer
}
Alternatively, multiple timers could be cascaded to achieve a similar effect. Modern AVR series microcontrollers have sections in their datasheet describing considerations for cascading two 16-bit timers to create a single 32-bit timer. Using this strategy one could set the top of the counter to 5 million and arrange an interrupt to toggle an LED, resulting in a 1Hz signal with 50% duty.
This method is called whenever the timer’s overflow interrupt is triggered. Since it’s called 5 times per second, I just need a global counter to count the number of times it was called, and set an output pin to high on every 5th invocation.
Do not forget to enable global interrupts in your start-up sequence! This is an easy mistake to make, and without calling this function the overflow function will never be invoked.
We have achieved a light that blinks exactly once per second with roughly the same precision as the 10 MHz frequency reference used to clock the microcontroller. This output signal is ready to use for precision measurement purposes, such as toggling the gate of a discrete frequency counter.
Inspecting the header file iotn826.h in my Program Files / Atmel folder was very useful for identifying named bit masks stored as enums. There is a similarly named file for every supported AVR microcontroller.
This page describes how to program Microchip’s newest series of AVR microcontrollers using official programming gear and software. I spent many years programming the traditional series of Atmel chips, but now several years after Microchip acquired Atmel I am interested in exploring the capabilities of the latest series of AVR microcontrollers (especially the new AVR DD family). Currently the global chip shortage makes it difficult to source traditional ATMega and STM32 chips, but the newest series of AVR microcontrollers feature an impressive set of peripherals for the price and are available from all the major vendors.
pressive set of peripherals for the price and are available from all the major vendors.
Older AVR microcontrollers are programmed using in-circuit serial programming (ICSP) through the RESET, SCK, MISO, and MOSI pins using cheap programmers like USBtiny. However, serial programming is not supported on newer AVR microcontrollers.
New AVR microcontrollers are programmed using the unified program and debug interface (UDPI) exclusively through the UDPI pin. UDPI is a Microchip proprietary interface requiring a UDPI-capable programmer.
Official UDPI programmers include Atmel-ICE ($129) and MPLAB Snap ($35). The Atmel-ICE is expensive but it is very well supported. The MPLAB Snap is hacky, requires re-flashing, and has a physical design flaw requiring a hardware modification before it can program AVR series chips.
There are notable attempts to create alternative programmers (e.g., jtag2updi and pymcuprog), but this journey into the land of unofficial programmer designs is fraught with abandoned GitHub repositories and a lot of added complexity and brittleness (e.g., SpenceKonde/AVR-Guidance), so to save yourself frustration in the future I highly recommend just buying an officially supported programmer. It’s also nice when you can program and debug your microcontroller from within your IDE.
UDPI programmers have a Vcc pin that is used to sense supply voltage (but not provide it), so you must power your board yourself while using one of these new programmers.
Blinking a LED is the “Hello, World” of microcontroller programming. Let’s take a look at the code necessary to blink a LED on pin 2 of an ATTiny286. It is compiled and programmed onto the chip using Microchip Studio.
PORTA.DIR sets the direction of pins on port A (0xFF means all outputs)
PORTA.OUT sets output voltage on pins of port A (0xFF means all high)
Using _delay_ms() requires including delay.h
Including delay.h requires defining F_CPU (the CPU frequency)
The ATTiny286 datasheet section 11.3.3 indicates the default clock is 20 MHz with a /6 prescaler, so the default clock is 3333333 Hz (3.3 MHz). This behavior can be customized using the Oscillator Configuration Fuse (FUSE.OSCCFG).
I power the device from the 3.3V or 5V pins on a FT232 USB breakout board. Although the topic is out of scope for this article, I find it convenient to use FTDI chips to exchange small amounts of data or debug messages between a microcontroller and a modern PC over USB without requiring special drivers.
I’m surprised how little information there is online about how to reliably program modern AVR series microcontrollers. In late 2022 there is a surprisingly large amount of “advice” on this topic which leads to dead ends and broken or abandoned projects. After looking into it for a while, here is my opinionated assessment. Mouser and Digikey have links to expensive programmers, and Amazon has links to similar items but reviews are littered with comments like “arrived bricked” and “can not program AVR series chips”. DIY options typically involve abandoned (or soon-to-be abandoned?) GitHub repositories, or instructions for Arduino-related programming. I seek to consolidate and distill the most useful information onto this page, and I hope others will find it useful.
After using $5 ICSP programmers for the last decade I almost fell out of my chair when I saw Microchip’s recommended entry-level programmer is over $180! Digikey sells a “basic” version without cables for $130, but that still seems crazy to me. Also, $50 for a ribbon cable?
I found a kit on Amazon that sells the programmer with a cable for $126. It was hard for me to press that buy button, but I figured the time I would save by having access to modern and exotic chips during the present global chip shortage would make it worth it. After a couple days of free Prime shipping, it arrived. It was smaller than I thought it would be from the product photos.
The cable that came with the device seemed a bit hacky at first, but I’m happy to have it. The female 2.54 mm socket is easy to insert breadboard jumpers into.
I’m glad this thing is idiot proof. The very first thing I did after unboxing this programmer was hook it up to my power supply rails using reverse polarity. I misread the pin diagram and confused the socket with the connector (they are mirror images of one another). This is an easy mistake to make though, so here’s a picture of the correct orientation. Note the location of the tab on the side of the connector.
Atmel ICE Pinout
Programming Connection
Black: GND
Red: Vcc - This line is used to sense power and not deliver it, so you are responsible for externally powering your board.
Blue: UPDI pin - Although a pull-up resistor on the UPDI pin is recommended, I did not find it was required to program my chip on the breadboard in this configuration.
The AVR Ice was easy to use with Microchip Studio. My programmer was detected immediately, a window popped-up and walked me through updating the firmware, and my LED was blinking in no time.
Did I really need to spend $126 for an AVR programmer? Amazon carries the MPLAB Snap for $34, but lots of reviews say it doesn’t work. After easily getting the Atmel-ICE programmer up and running I thought it would be a similarly easy experience setting-up the MPLAB Snap for AVR UPDI programming, but boy was I wrong. Now that I know the steps to get this thing working it’s not so bad, but the information here was only gathered after hours of frustration.
Here are the steps you can take to program modern AVR microcontrollers with UPDI using a MPLAB Snap:
The MPLAB Snap ships with obsolete firmware and must be re-flashed immediately upon receipt.
Microchip Studio’s firmware upgrade tool does not actually work with the MPLAB Snap. It shows the board with version 0.00 software and it hangs (with several USB disconnect/reconnect sounds) if you try to upgrade it.
You can only re-flash the MPLAB Snap using the MPLAB X IDE. Download the 1.10 GB MPLAB setup executable and install the MPLAB IDE software which occupies a cool 9.83 GB.
In the MPLAB IDE select Tools and select Hardware Tool Emergency Boot Firmware Recovery. At least this tool is helpful. It walks you through how to reset the device and program the latest firmware.
Defective may be a strong word, but let’s just say the hardware was not designed to enable programming AVR chips using UPDI. Microchip Studio will detect the programmer but if you try to program an AVR you’ll get a pop-up error message that provides surprisingly little useful information.
Atmel Studio was unable to start your debug session. Please verify device selection, interface settings, target power and connections to the target device. Look in the details section for more information.
StatusCode: 131107
ModuleName: TCF (TCF command: Processes:launch failed.)
An illegal configuration parameter was used. Debugger command Activate physical failed.
These photos were taken after I removed the resistor. I didn’t use hot air. I just touched it a for a few seconds with a soldering iron and wiped it off then threw it away.
You don’t need a microscope, but I’m glad I had one.
You can now program AVR microcontrollers using UPDI with your MPLAB Snap! Blink, LED, blink.
Can you believe this is the officially recommended action? According to the official Microchip Engineering Technical Note ETN #36: MPLAB Snap AVR Interface Modification
Symptom: Programming and debugging fails with AVR microcontroller devices that use the UPDI/PDI/TPI interfaces. MPLAB SNAP, Assembly #02-10381-R1 requires an external pull-up resistor for AVR microcontroller
devices that use these interfaces.
Problem: AVR microcontroller devices that use the UPDI/PDI/TPI interfaces require the idle state of inactivity to be at a logic high level. Internally, the AVR devices have a weak (50-100K) pull-up resistor that attempts to keep the line high. An external and stronger pull-up resistor may be enough to mitigate this issue and bring voltages to acceptable VDD levels. In some cases, this may not be enough and the pull-down resistor that is part of the ICSP protocol can be removed for these AVR microcontroller applications.
Solution: If most of the applications are AVR-centric, consider removing the R48 resistor as shown below. This completely isolates any loading on the programming data line. Additionally, a pull-up resistor to VDD in the range of 1K to 10K should be used for robustness. Pin 4 of J4 is the TPGD data line used for ICSP interfaces and it also doubles as the DAT signal for UPDI/PDI and TPI interfaces. The pull-up resistor can be mounted directly from TVDD (J4-2) to TPGD/DAT (J4-4). Alternatively, the resistor can be mounted on the application side of the circuit
for convenience.
I feel genuinely sorry for the Amazon sellers who are getting poor reviews because they sell this product. It really isn’t their fault. I hope Google directs people here so that they can get their boards working and leave positive reviews that point more people to this issue.
There is no official support for UPDI programming using a serial adapter, but it seems some people have figured out how to do it in some capacity. There was a promising pyupdi project, but it is now deprecated. At the time of writing the leading project aiming to enable UPDI programming without official hardware is pymcuprog, but its repository has a single commit dated two months ago and no activity since. Interestingly, that commit was made by buildmaster@microchip.com (an unverified email address), so it may not be fair to refer to it as an “unofficial” solution. The long term support of the pymcuprog project remains uncertain, but regardless let’s take a closer look at how it works.
I used Microchip Studio to compile my C code and generate the hex file. Now I can use pymcuprog to load those hex files onto the chip. It’s slower to program and inconvenient to drop to a terminal whenever I want to program a chip, but it works.
The new AVR series microcontrollers have lots of cool peripherals for the price and are available during a chip shortage that threatens availability of the more common traditional microcontrollers.
The Atmel-ICE is expensive, but the most convenient and reliable way to program modern AVR microcontrollers using UPDI.
The MPLAB Snap can program modern AVRs using UPDI after a software flash and a hardware modification, but its support for AVRs seems like an afterthought rather than its design priority.
You can create a makeshift unofficial UPDI programmer from a USB serial adapter, but the added complexity, lack of debugging capabilities, increased friction during the development loop, and large number of abandoned projects in this space make this an unappealing long term solution in my opinion.