SWHarden.com

The personal website of Scott W Harden

Static Site Broken Link Detection

How to check for broken links and images across large static websites
May 13, 2023
python hugo

This website is a static site containing thousands of pages, and I recently had the desire to check all of them for broken links and images. Although it would be a tedious task to perform manually, I was able to automate the process using Python. This page describes the process I used to identify broken links and images and may be helpful to others using static site generator tools like Jekyll, Hugo, Eleventy, Pelican, Gatsby, other Jamstack site generators, or hand-written HTML files.

Although my website content is stored as markdown files, I found it most convenient to check the generated HTML files for broken link and image URLs. I generated my static site locally, then manually removed pagination folders so only article HTML files were present. I then wrote the following Python script which uses pathlib to locate HTML files and BeautifulSoup to analyze a href and img src attributes and saves all URLs identified as a CSV file.

import pathlib
from bs4 import BeautifulSoup


def get_urls(file: pathlib.Path) -> list:
    """Return link and image URLs in a HTML file"""
    with open(file, errors='ignore') as f:
        html = f.read()
    soup = BeautifulSoup(html, 'html.parser')
    urls = set()

    for link in soup.find_all('a'):
        href = link.get('href')
        if href.startswith("#"):
            continue
        if "#" in href:
            href = href.split("#")[0]
        href = str(href).strip('/')
        urls.add(href)

    for image in soup.find_all('img'):
        src = image.get('src')
        urls.add(str(src))

    return sorted(list(urls))


def get_urls_by_page(folder: pathlib.Path) -> dict:
    """Return link and image URLs for all HTML files in a folder"""
    html_files = list(folder.rglob("*.html"))
    urls_by_page = {}
    for i, html_file in enumerate(html_files):
        urls = get_urls(html_file)
        urls_by_page[html_file] = urls
        print(f"{i+1} of {len(html_files)}: {len(urls)} URLs found in {html_file}")
    return urls_by_page


def write_csv(urls_by_page: dict, csv_file: pathlib.Path):
    txt = 'URL, Page\n'
    for page, urls in urls_by_page.items():
        for url in urls:
            txt += f'{url}, {page}\n'
    csv_file.write_text(txt)


if __name__ == "__main__":
    folder = pathlib.Path("public")
    urls_by_page = get_urls_by_page(folder)
    write_csv(urls_by_page, pathlib.Path("urls.csv"))

Running the script generated a CSV report showing every link on my website, organized by which page it’s on. It looks like my blog has over 7,000 links! That would be a lot to check by hand.

Checking URLs

Each URL from the report was checked using HEAD requests. Note that a HEAD request to a file path only returns HTTP headers but does not actually download the file, allowing it to consume far less bandwidth GET requests. Inspecting the HTTP response code indicates whether the URL is a valid path to a file (code 200).

I used python sets to prevent checking the same URL twice. I logged good and broken URLs as they were checked, and consumed these log files at startup to allow the program to be stopped and restarted without causing it to check the same URL twice.

import pathlib
import urllib.request
import time


def is_url_valid(url: str) -> bool:
    """Check if a URL exists without downloading its contents"""
    request = urllib.request.Request(url)
    request.get_method = lambda: 'HEAD'
    try:
        urllib.request.urlopen(request, timeout=5)
        return True
    except Exception as e:
        print(f"ERROR: {e}")
        return False


if __name__ == "__main__":

    # load URLs from report
    urls = [x.split(", ")[0] for x
            in pathlib.Path("urls.csv").read_text().split("\n")
            if x.startswith("http")]

    # load previously checked URLs
    url_file_good = pathlib.Path("urls-good.txt")
    url_file_good.touch()
    url_file_bad = pathlib.Path("urls-bad.txt")
    url_file_bad.touch()
    checked = set()
    for url in url_file_good.read_text().split("\n"):
        checked.add(url)
    for url in url_file_bad.read_text().split("\n"):
        checked.add(url)

    # check each URL
    for i, url in enumerate(urls):
        print(f"{i+1} of {len(urls)}", end=" ")
        if url in checked:
            print(f"SKIPPING {url}")
            continue
        time.sleep(.2)
        print(f"CHECKING {url}")
        log_file = url_file_good if is_url_valid(url) else url_file_bad
        with open(log_file, 'a') as f:
            f.write(url+"\n")
        checked.add(url)

I wrote a Python script to generate a HTML report of all the broken links on my website. The script shows every broken link found on the website and lists all the pages on which each broken link appears.

import pathlib

urls_and_pages = [x.split(", ") for x
                    in pathlib.Path("urls.csv").read_text().split("\n")
                    if x.startswith("http")]

urls_broken = [x for x
                in pathlib.Path("urls-bad.txt").read_text().split("\n")
                if x.startswith("http")]

urls_broken = set(urls_broken)

html = "<h1>Broken Links</h1>"
for url_broken in urls_broken:
    html += f"<div class='mt-4'><a href='{url_broken}'>{url_broken}</a></div>"
    pages = [x[1] for x in urls_and_pages if x[0] == url_broken]
    for page in pages:
        html += f"<div><code>{page}</code></div>"

I wrapped the output in Bootstrap to make the report look pretty and appear properly on mobile devices:

html = f"""<!doctype html>
<html lang="en">
  <head>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0-alpha3/dist/css/bootstrap.min.css" rel="stylesheet">
  </head>
  <body>
    <div class='container'>
      {html}
    </div>
  </body>
</html>"""

pathlib.Path("report.html").write_text(html)

The generated report lets me focus my effort narrowly to tactically fix the broken URLs I think are most important (e.g., image URLs, URLs pointing domain names).

Conclusions

  Feedback   Edit

Treemapping with C#

How to create a treemap diagram using C#
March 28, 2023
csharp graphics

Treemap diagrams display a series of positive numbers using rectangles sized proportional to the value of each number. This page demonstrates how to calculate the size and location of rectangles to create a tree map diagram using C#. Although the following uses System.Drawing to save the tree map as a Bitmap image, these concepts may be combined with information on the C# Data Visualization page to create treemap diagrams using SkiaSharp, WPF, or other graphics technologies.

The tree map above was generated from random data using the following C# code:

// Create sample data. Data must be sorted large to small.
double[] sortedValues = Enumerable.Range(0, 40)
    .Select(x => (double)Random.Shared.Next(10, 100))
    .OrderByDescending(x => x)
    .ToArray();

// Create an array of labels in the same order as the sorted data.
string[] labels = sortedValues.Select(x => x.ToString()).ToArray();

// Calculate the size and position of all rectangles in the tree map
int width = 600;
int height = 400;
RectangleF[] rectangles = TreeMap.GetRectangles(sortedValues, width, height);

// Create an image to draw on (with 1px extra to make room for the outline)
using Bitmap bmp = new(width + 1, height + 1);
using Graphics gfx = Graphics.FromImage(bmp);
using Font fnt = new("Consolas", 8);
using SolidBrush brush = new(Color.Black);
gfx.Clear(Color.White);

// Draw and label each rectangle
for (int i = 0; i < rectangles.Length; i++)
{
    brush.Color = Color.FromArgb(
        red: Random.Shared.Next(150, 250),
        green: Random.Shared.Next(150, 250),
        blue: Random.Shared.Next(150, 250));

    gfx.FillRectangle(brush, rectangles[i]);
    gfx.DrawRectangle(Pens.Black, rectangles[i]);
    gfx.DrawString(labels[i], fnt, Brushes.Black, rectangles[i].X, rectangles[i].Y);
}

// Save the output
bmp.Save("treemap.bmp");

Treemap Logic

The previous code block focuses on data generation and display, but hides the tree map calculations behind the TreeMap class. Below is the code for that class. It is self-contained static class and exposes a single static method which takes a pre-sorted array of values and returns tree map rectangles ready to display on an image.

💡 Although the System.Drawing.Common is a Windows-only library (as of .NET 7), System.Drawing.Primitives is a cross-platform package that provides the RectangleF structure used in the tree map class. See the SkiaSharp Quickstart to learn how to create image files using cross-platform .NET code.

public static class TreeMap
{
    public static RectangleF[] GetRectangles(double[] values, int width, int height)
    {
        for (int i = 1; i < values.Length; i++)
            if (values[i] > values[i - 1])
                throw new ArgumentException("values must be ordered large to small");

        var slice = GetSlice(values, 1, 0.35);
        var rectangles = GetRectangles(slice, width, height);
        return rectangles.Select(x => x.ToRectF()).ToArray();
    }

    private class Slice
    {
        public double Size { get; }
        public IEnumerable<double> Values { get; }
        public Slice[] Children { get; }

        public Slice(double size, IEnumerable<double> values, Slice sub1, Slice sub2)
        {
            Size = size;
            Values = values;
            Children = new Slice[] { sub1, sub2 };
        }

        public Slice(double size, double finalValue)
        {
            Size = size;
            Values = new double[] { finalValue };
            Children = Array.Empty<Slice>();
        }
    }

    private class SliceResult
    {
        public double ElementsSize { get; }
        public IEnumerable<double> Elements { get; }
        public IEnumerable<double> RemainingElements { get; }

        public SliceResult(double elementsSize, IEnumerable<double> elements, IEnumerable<double> remainingElements)
        {
            ElementsSize = elementsSize;
            Elements = elements;
            RemainingElements = remainingElements;
        }
    }

    private class SliceRectangle
    {
        public Slice Slice { get; set; }
        public float X { get; set; }
        public float Y { get; set; }
        public float Width { get; set; }
        public float Height { get; set; }
        public SliceRectangle(Slice slice) => Slice = slice;
        public RectangleF ToRectF() => new(X, Y, Width, Height);
    }

    private static Slice GetSlice(IEnumerable<double> elements, double totalSize, double sliceWidth)
    {
        if (elements.Count() == 1)
            return new Slice(totalSize, elements.Single());

        SliceResult sr = GetElementsForSlice(elements, sliceWidth);
        Slice child1 = GetSlice(sr.Elements, sr.ElementsSize, sliceWidth);
        Slice child2 = GetSlice(sr.RemainingElements, 1 - sr.ElementsSize, sliceWidth);
        return new Slice(totalSize, elements, child1, child2);
    }

    private static SliceResult GetElementsForSlice(IEnumerable<double> elements, double sliceWidth)
    {
        var elementsInSlice = new List<double>();
        var remainingElements = new List<double>();
        double current = 0;
        double total = elements.Sum();

        foreach (var element in elements)
        {
            if (current > sliceWidth)
                remainingElements.Add(element);
            else
            {
                elementsInSlice.Add(element);
                current += element / total;
            }
        }

        return new SliceResult(current, elementsInSlice, remainingElements);
    }

    private static IEnumerable<SliceRectangle> GetRectangles(Slice slice, int width, int height)
    {
        SliceRectangle area = new(slice) { Width = width, Height = height };

        foreach (var rect in GetRectangles(area))
        {
            if (rect.X + rect.Width > area.Width)
                rect.Width = area.Width - rect.X;

            if (rect.Y + rect.Height > area.Height)
                rect.Height = area.Height - rect.Y;

            yield return rect;
        }
    }

    private static IEnumerable<SliceRectangle> GetRectangles(SliceRectangle sliceRectangle)
    {
        var isHorizontalSplit = sliceRectangle.Width >= sliceRectangle.Height;
        var currentPos = 0;
        foreach (var subSlice in sliceRectangle.Slice.Children)
        {
            var subRect = new SliceRectangle(subSlice);
            int rectSize;

            if (isHorizontalSplit)
            {
                rectSize = (int)Math.Round(sliceRectangle.Width * subSlice.Size);
                subRect.X = sliceRectangle.X + currentPos;
                subRect.Y = sliceRectangle.Y;
                subRect.Width = rectSize;
                subRect.Height = sliceRectangle.Height;
            }
            else
            {
                rectSize = (int)Math.Round(sliceRectangle.Height * subSlice.Size);
                subRect.X = sliceRectangle.X;
                subRect.Y = sliceRectangle.Y + currentPos;
                subRect.Width = sliceRectangle.Width;
                subRect.Height = rectSize;
            }

            currentPos += rectSize;

            if (subSlice.Values.Count() > 1)
            {
                foreach (var sr in GetRectangles(subRect))
                {
                    yield return sr;
                }
            }
            else if (subSlice.Values.Count() == 1)
            {
                yield return subRect;
            }
        }
    }
}

Source Code Complexity Analysis

A few days ago I wrote an article describing how to programmatically generate .NET source code analytics using C#. Using these tools I analyzed the source code for all classes in a large project (ScottPlot.NET). The following tree map displays every class in the project as a rectangle sized according to number of lines of code and colored according to maintainability.

In this diagram large rectangles represent classes with the most code, and red color indicates classes that are difficult to maintain.

💡 I’m using a perceptually uniform colormap (similar to Turbo)provided by the ScottPlot provided by the ScottPlot NuGet package. See ScottPlot’s colormaps gallery for all available colormaps.

💡 The Maintainability Index is a value between 0 (worst) and 100 (best) that represents the relative ease of maintaining the code. It’s calculated from a combination of Halstead complexity (size of the compiled code), Cyclomatic complexity (number of paths that can be taken through the code), and the total number of lines of code.

Conclusions

References

  Feedback   Edit

.NET Source Code Analysis

How to analyze source code metrics of .NET assemblies from a console application
March 5, 2023
csharp

This page describes how to use the Microsoft.CodeAnalysis.Metrics package to perform source code analysis of .NET assemblies from a console application. Visual Studio users can perform source code analysis by clicking the “Analyze” dropdown menu and selecting “Calculate Code Metrics”, but I sought to automate this process so I can generate custom code analysis reports from console applications as part of my CI pipeline.

Performing Code Analysis

Step 1: Add the Microsoft.CodeAnalysis.Metrics package to your project:

dotnet add package Microsoft.CodeAnalysis.Metrics

Step 2: Perform code analysis:

dotnet build -target:Metrics

Note that multi-targeted projects must append --framework net6.0 to specify a single platform target to use for code analysis.

Step 3: Inspect analysis results in ProjectName.Metrics.xml

<?xml version="1.0" encoding="utf-8"?>
<CodeMetricsReport Version="1.0">
  <Targets>
    <Target Name="ScottPlot.csproj">
      <Assembly Name="ScottPlot, Version=4.1.61.0">
        <Metrics>
          <Metric Name="MaintainabilityIndex" Value="81" />
          <Metric Name="CyclomaticComplexity" Value="6324" />
          <Metric Name="ClassCoupling" Value="664" />
          <Metric Name="DepthOfInheritance" Value="3" />
          <Metric Name="SourceLines" Value="35360" />
          <Metric Name="ExecutableLines" Value="10208" />
        </Metrics>
        <Namespaces>
        ...

Parsing the Analysis XML File

The code analysis XML contains information about every assembly, namespace, type, and function in the whole code base! There is a lot of possible information to extract, but the code below is enough to get us started extracting basic metric information for every type in the code base.

using System;
using System.IO;
using System.Linq;
using System.Text;
using System.Xml.Linq;
using System.Collections.Generic;

/// <summary>
/// Display a particular metric for every type in an assembly.
/// </summary>
void RankTypes(string xmlFilePath, string metricName = "CyclomaticComplexity", bool highToLow = true)
{
    string xmlText = File.ReadAllText(xmlFilePath);
    XDocument doc = XDocument.Parse(xmlText);
    XElement assembly = doc.Descendants("Assembly").First();

    var rankedTypes = GetMetricByType(assembly, metricName).OrderBy(x => x.Value).ToArray();
    if (highToLow)
        Array.Reverse(rankedTypes);

    Console.WriteLine($"Types ranked by {metricName}:");
    foreach (var type in rankedTypes)
        Console.WriteLine($"{type.Value:N0}\t{type.Key}");
}

Dictionary<string, int> GetMetricByType(XElement assembly, string metricName)
{
    Dictionary<string, int> metricByType = new();

    foreach (XElement namespaceElement in assembly.Element("Namespaces")!.Elements("Namespace"))
    {
        foreach (XElement namedType in namespaceElement.Elements("Types").Elements("NamedType"))
        {
            XElement metric = namedType.Element("Metrics")!.Elements("Metric")
                .Where(x => x.Attribute("Name")!.Value == metricName)
                .Single();
            string typeName = namedType.Attribute("Name")!.Value;
            string namespaceName = namespaceElement.Attribute("Name")!.Value;
            string fullTypeName = $"{namespaceName}.{typeName}";
            metricByType[fullTypeName] = int.Parse(metric.Attribute("Value")!.Value!.ToString());
        }
    }

    return metricByType;
}

Querying Code Analysis Results

Specific metrics of interest will vary, but here are some code examples demonstrating how to parse the code metrics file to display useful information. For these examples I run the code analysis command above to generate ScottPlot.Metrics.xml from the ScottPlot code base and use the code above to generate various reports.

Rank Types by Cyclomatic Complexity

Cyclomatic complexity is a measure of the number of different paths that can be taken through a computer program, and it is often used as an indicator for difficult-to-maintain code. Some CI systems even prevent the merging of pull requests if their cyclomatic complexity exceeds a predefined threshold! Although I don’t intend to gate pull requests by complexity at this time, I would like to gain insight into which classes are the most complex as a way to quantitatively target my code maintenance and efforts.

RankTypes("ScottPlot.Metrics.xml", "CyclomaticComplexity");

517     ScottPlot.Plot
218     ScottPlot.Plottable.SignalPlotBase<T>
173     ScottPlot.Plottable.ScatterPlot
139     ScottPlot.Settings
120     ScottPlot.Ticks.TickCollection
118     ScottPlot.Renderable.Axis
114     ScottPlot.Drawing.Colormap
113     ScottPlot.Control.ControlBackEnd
109     ScottPlot.DataGen
99      ScottPlot.Plottable.AxisLineVector
98      ScottPlot.Plottable.Heatmap
95      ScottPlot.Tools
93      ScottPlot.Plottable.RepeatingAxisLine
91      ScottPlot.Plottable.PopulationPlot
85      ScottPlot.Plottable.AxisLine
83      ScottPlot.Plottable.AxisSpan
77      ScottPlot.Plottable.RadialGaugePlot
...

Rank Types by Lines of Code

Similarly, ranking all my project’s types by how many lines of code they contain can give me insight into which types may benefit most from refactoring.

RankTypes("ScottPlot.Metrics.xml", "SourceLines");

Types ranked by SourceLines:
4,155   ScottPlot.Plot
1,182   ScottPlot.DataGen
954     ScottPlot.Plottable.SignalPlotBase<T>
726     ScottPlot.Control.ControlBackEnd
670     ScottPlot.Ticks.TickCollection
670     ScottPlot.Settings
630     ScottPlot.Plottable.ScatterPlot
600     ScottPlot.Renderable.Axis
477     ScottPlot.Statistics.Common
454     ScottPlot.Tools
451     ScottPlot.Plottable.PopulationPlot
432     ScottPlot.Drawing.GDI
343     ScottPlot.Plottable.SignalPlotXYGeneric<TX, TY>
336     ScottPlot.Plottable.RepeatingAxisLine
335     ScottPlot.Drawing.Colormap
332     ScottPlot.Plottable.AxisLineVector
...

Rank Types by Maintainability

The Maintainability Index is a value between 0 (worst) and 100 (best) that represents the relative ease of maintaining the code. It’s calculated from a combination of Halstead complexity (size of the compiled code), Cyclomatic complexity (number of paths that can be taken through the code), and the total number of lines of code.

MaintainabilityIndex = 171 
  - 5.2 * Math.Log(HalsteadVolume) 
  - 0.23 * CyclomaticComplexity
  - 16.2 * Math.Log(LinesOfCCode);

The maintainability index is calculated by Microsoft.CodeAnalysis.Metrics so we don’t have to. I don’t know how Microsoft arrived at their weights for this formula, but the overall idea is described here.

RankTypes("ScottPlot.Metrics.xml", "MaintainabilityIndex", highToLow: false);

43      ScottPlot.Drawing.Tools
48      ScottPlot.Statistics.Interpolation.Cubic
48      ScottPlot.Statistics.Interpolation.PeriodicSpline
49      ScottPlot.Statistics.Interpolation.EndSlopeSpline
49      ScottPlot.Statistics.Interpolation.NaturalSpline
50      ScottPlot.Renderable.AxisTicksRender
54      ScottPlot.Statistics.Interpolation.CatmullRom
55      ScottPlot.Statistics.Interpolation.SplineInterpolator
57      ScottPlot.DataGen
58      ScottPlot.DataStructures.SegmentedTree<T>
58      ScottPlot.MarkerShapes.Hashtag
58      ScottPlot.Ticks.TickCollection
59      ScottPlot.MarkerShapes.Asterisk
59      ScottPlot.Plottable.SignalPlotXYGeneric<TX, TY>
59      ScottPlot.Statistics.Interpolation.Bezier
60      ScottPlot.Statistics.Interpolation.Chaikin
61      ScottPlot.Generate
61      ScottPlot.Plot
61      ScottPlot.Statistics.Finance
...

Create Custom HTML Reports

With a little more effort you can generate HTML reports that use tables and headings to highlight useful code metrics and draw attention to types that could benefit from refactoring to improve maintainability.

Conclusions

Microsoft’s official Microsoft.CodeAnalysis.Metrics NuGet package is a useful tool for analyzing assemblies, navigating through namespaces, types, properties, and methods, and evaluating their metrics. Since these analyses can be performed using console applications, they can be easily integrated into CI pipelines or used to create standalone code analysis applications. Future projects can build on the concepts described here to create graphical visualizations of code metrics in large projects.

Resources

  Feedback   Edit

Divide 10 MHz to 1PPS with a Microcontroller

How to use a microcontroller to inexpensively scale down a 10 MHz reference clock into a one pulse per second (1pps) signal
December 17, 2022
circuit microcontroller

I often find it useful to gate frequency counters using a 1 pulse per second (1PPS) signal derived from a 10 MHz precision frequency reference. However, a divide-by-10-million circuit isn’t trivial to implement. Counter ICs exist which enable divide-by-100 by combining multiple divide-by-2 and divide-by-5 stages (e.g., MC74HC390A around $0.85 each), but dividing 10 MHz all the way down to 1Hz would require at least 4 of these chips and a lot of wiring.

You can clock a microcontroller at 10 MHz and use its timer and interrupt systems to generate 1PPS. For example, an ATTiny202 in an 8-pin SOIC package is available from Mouser (>50k stocked) for $0.51 each. Note that modern series AVRs require a special UDPI programmer.

ATTiny202 ($0.51) ATTint826 ($0.95)

This page documents a divide-by-10-million circuit achieved with a single microcontroller to scale a 10MHz frequency reference down to 1PPS. I’m using an ATTiny826 because that is what I have on hand, but these concepts apply to any microcontroller with a 16-bit timer.

ATTiny Breakout Board

Some AVRs come in DIP packages but their pin numbers may be different than the same chip in a SMT package. To facilitate prototyping using designs and code that will work identically across a breadboard prototype and a PCB containing SMT chips, I prefer to build DIP breakout boards using whatever SMT package I intend to include in my final builds. In this case it’s ATTint826 in a SOIC-20 package, and I can easily use this in a breadboard by soldering them onto SOIC-to-DIP breakout boards.

I assembled the breakout board by hand using a regular soldering iron. When working with small packages it helps so much to coat the pins with a little tack flux to facilitate wetting and prevent solder bridges. I’m presently using Chip Quik NC191. Even if flux is advertized as “no-clean”, it’s good practice and makes the boards look much nicer to remove remaining flux with acetone and Q-tips or brushes.

Circuit

Configuration Change Protection (CCP)

Traditional AVR microcontrollers used fuse bits to set the clock source, but modern series chips can change the clock source from within code. However, modifying the clock source requires temporarily disabling the configuration change protection (CCP) system.

Disabling the CCP only lasts four clock cycles, so the immediate next statement must be assignment of the new value. I use the following function to facilitate this action.

/* Write a value to a CCP-protected register */
void ccp_write(volatile register8_t* address, uint8_t value){
	CCP = CCP_IOREG_gc;
	*address = value;
}

// Use internal 20 MHz clock with CKOUT pin enabled
ccp_write(&CLKCTRL.MCLKCTRLA, CLKCTRL.MCLKCTRLA | CLKCTRL_CLKOUT_bm);

Do not use compound statements when writing to the CCP register. The code below fails to change clock as one may expect by looking at the code, presumably because the combined OR operation with the assignment exceeds four clock cycles. Instead of direct assignment, use the ccp_write function described above.

// WARNING: This code does not actually change the clock source
CCP = CCP_IOREG_gc;
CLKCTRL.MCLKCTRLA = CLKCTRL.MCLKCTRLA | CLKCTRL_CLKOUT_bm;

Configuring the Clock Source

Internal 10 MHz clock

This is the configuration I use to achieve a 10 CPU clock using the built-in 20 MHz oscillator.

void configure_clock_internal_10mhz(){
	ccp_write(&CLKCTRL.MCLKCTRLA, CLKCTRL.MCLKCTRLA | CLKCTRL_CLKOUT_bm); // 20 MHz internal clock, enable CKOUT
	ccp_write(&CLKCTRL.MCLKCTRLB, CLKCTRL_PEN_bm); // enable divide-by-2 clock prescaler
}

External 10 MHz clock

This is the configuration I use to clock the CPU from an external 10 MHz clock source applied to the EXTCLK pin.

void configure_clock_external(){
	ccp_write(&CLKCTRL.MCLKCTRLA, CLKCTRL_CLKSEL_EXTCLK_gc | CLKCTRL_CLKOUT_bm); // external clock, enable CKOUT
	ccp_write(&CLKCTRL.MCLKCTRLB, 0); // disable prescaler
}

Configuring the 16-Bit Timer

This is how I configured my ATTiny826’s TCA0 16-bit timer to fire an interrupt every 200 ms.

void configure_1pps(){
	// 10 MHz system clock with div64 prescaler is 156,250 Hz.
	// Setting a 16-bit timer's top to 31,250 means 5 overflows per second.
	TCA0.SINGLE.INTCTRL = TCA_SINGLE_OVF_bm; // overflow interrupt
	TCA0.SINGLE.CTRLB = TCA_SINGLE_WGMODE_NORMAL_gc; // normal mode
	TCA0.SINGLE.PER = 31249UL; // control timer period by setting timer top
	TCA0.SINGLE.CTRLA |= TCA_SINGLE_CLKSEL_DIV64_gc; // set clock source
	TCA0.SINGLE.CTRLA |= TCA_SINGLE_ENABLE_bm; // start timer
}

Alternatively, multiple timers could be cascaded to achieve a similar effect. Modern AVR series microcontrollers have sections in their datasheet describing considerations for cascading two 16-bit timers to create a single 32-bit timer. Using this strategy one could set the top of the counter to 5 million and arrange an interrupt to toggle an LED, resulting in a 1Hz signal with 50% duty.

Configuring the Interrupt System

This method is called whenever the timer’s overflow interrupt is triggered. Since it’s called 5 times per second, I just need a global counter to count the number of times it was called, and set an output pin to high on every 5th invocation.

uint8_t overflow_count;

ISR(TCA0_OVF_vect)
{
	overflow_count++;
	if (overflow_count == 5){
		overflow_count = 0;
		PORTB.OUT = PIN1_bm;
    } else {
		PORTB.OUT = 0;
	}
    
	TCA0.SINGLE.INTFLAGS = TCA_SINGLE_OVF_bm; // indicate interrupt was handled
}

Do not forget to enable global interrupts in your start-up sequence! This is an easy mistake to make, and without calling this function the overflow function will never be invoked.

sei(); // enable global interrupts

Results

We have achieved a light that blinks exactly once per second with roughly the same precision as the 10 MHz frequency reference used to clock the microcontroller. This output signal is ready to use for precision measurement purposes, such as toggling the gate of a discrete frequency counter.

Resources

  Feedback   Edit

Programming Modern AVR Microcontrollers

Blink a LED on a modern series AVR using Atmel-ICS or MPLAB Snap UPDI programmers.
December 9, 2022
circuit microcontroller

This page describes how to program Microchip’s newest series of AVR microcontrollers using official programming gear and software. I spent many years programming the traditional series of Atmel chips, but now several years after Microchip acquired Atmel I am interested in exploring the capabilities of the latest series of AVR microcontrollers (especially the new AVR DD family). Currently the global chip shortage makes it difficult to source traditional ATMega and STM32 chips, but the newest series of AVR microcontrollers feature an impressive set of peripherals for the price and are available from all the major vendors. pressive set of peripherals for the price and are available from all the major vendors.

TLDR

Blinking a LED is the “Hello, World” of microcontroller programming. Let’s take a look at the code necessary to blink a LED on pin 2 of an ATTiny286. It is compiled and programmed onto the chip using Microchip Studio.

#define F_CPU 3333333UL
#include <avr/io.h>
#include <util/delay.h>

int main(void)
{
	PORTA.DIR = 0xFF;
	while (1)
	{
		PORTA.OUT = 255;
		_delay_ms(500);
		PORTA.OUT = 0;
		_delay_ms(500);
	}
}

ATTiny826 Pinout

From page 14 of the ATTiny826 datasheet

SMT ATTiny Breakout Board

Many of the newer AVR series microcontrollers are not available in breadboard-friendly DIP packages. I find SOIC-to-DIP breakout boards (available on Amazon and eBay) to be useful for experimenting with chips in SOIC packages. Here I added extra power and PA4 (pin 2) LEDs and 10 kΩ current limiting resistors.

I power the device from the 3.3V or 5V pins on a FT232 USB breakout board. Although the topic is out of scope for this article, I find it convenient to use FTDI chips to exchange small amounts of data or debug messages between a microcontroller and a modern PC over USB without requiring special drivers.

Why is programming modern AVRs so difficult?

I’m surprised how little information there is online about how to reliably program modern AVR series microcontrollers. In late 2022 there is a surprisingly large amount of “advice” on this topic which leads to dead ends and broken or abandoned projects. After looking into it for a while, here is my opinionated assessment. Mouser and Digikey have links to expensive programmers, and Amazon has links to similar items but reviews are littered with comments like “arrived bricked” and “can not program AVR series chips”. DIY options typically involve abandoned (or soon-to-be abandoned?) GitHub repositories, or instructions for Arduino-related programming. I seek to consolidate and distill the most useful information onto this page, and I hope others will find it useful.

Atmel-ICE: Expensive but Effective

After using $5 ICSP programmers for the last decade I almost fell out of my chair when I saw Microchip’s recommended entry-level programmer is over $180! Digikey sells a “basic” version without cables for $130, but that still seems crazy to me. Also, $50 for a ribbon cable?

I found a kit on Amazon that sells the programmer with a cable for $126. It was hard for me to press that buy button, but I figured the time I would save by having access to modern and exotic chips during the present global chip shortage would make it worth it. After a couple days of free Prime shipping, it arrived. It was smaller than I thought it would be from the product photos.

The cable that came with the device seemed a bit hacky at first, but I’m happy to have it. The female 2.54 mm socket is easy to insert breadboard jumpers into.

I’m glad this thing is idiot proof. The very first thing I did after unboxing this programmer was hook it up to my power supply rails using reverse polarity. I misread the pin diagram and confused the socket with the connector (they are mirror images of one another). This is an easy mistake to make though, so here’s a picture of the correct orientation. Note the location of the tab on the side of the connector.

Atmel ICE Pinout Programming Connection

The AVR Ice was easy to use with Microchip Studio. My programmer was detected immediately, a window popped-up and walked me through updating the firmware, and my LED was blinking in no time.

MPLAB Snap: Cheap and Convoluted

Did I really need to spend $126 for an AVR programmer? Amazon carries the MPLAB Snap for $34, but lots of reviews say it doesn’t work. After easily getting the Atmel-ICE programmer up and running I thought it would be a similarly easy experience setting-up the MPLAB Snap for AVR UPDI programming, but boy was I wrong. Now that I know the steps to get this thing working it’s not so bad, but the information here was only gathered after hours of frustration.

Here are the steps you can take to program modern AVR microcontrollers with UPDI using a MPLAB Snap:

Step 1: Setup MPLAB

Step 2: Re-Flash the Programmer

Step 3: Acknowledge Your Programmer is Defective

Defective may be a strong word, but let’s just say the hardware was not designed to enable programming AVR chips using UPDI. Microchip Studio will detect the programmer but if you try to program an AVR you’ll get a pop-up error message that provides surprisingly little useful information.

Atmel Studio was unable to start your debug session. Please verify device selection, interface settings, target power and connections to the target device. Look in the details section for more information. StatusCode: 131107 ModuleName: TCF (TCF command: Processes:launch failed.) An illegal configuration parameter was used. Debugger command Activate physical failed.

Step 4: Fix Your Programmer

The reason MPLAB Snap cannot program AVR microcontrollers is because the UPDI pin should be pulled high, but the MPLAB Snap comes from the factory with its UPDI pin pulled low with a 4.7 kΩ resistor to ground. You can try to overpower this resistor by adding a low value pull-up resistor to Vcc (1 kΩ worked for me on a breadboard), but the actual fix is to fire-up the soldering iron and remove that tiny surface-mount pull-down resistor labeled R48.

Have your glasses? R48 is here:

These photos were taken after I removed the resistor. I didn’t use hot air. I just touched it a for a few seconds with a soldering iron and wiped it off then threw it away.

You don’t need a microscope, but I’m glad I had one.

Step 5: Reflect

You can now program AVR microcontrollers using UPDI with your MPLAB Snap! Blink, LED, blink.

Can you believe this is the officially recommended action? According to the official Microchip Engineering Technical Note ETN #36: MPLAB Snap AVR Interface Modification

I feel genuinely sorry for the Amazon sellers who are getting poor reviews because they sell this product. It really isn’t their fault. I hope Google directs people here so that they can get their boards working and leave positive reviews that point more people to this issue.

UPDI Programming with a Serial Adapter

There is no official support for UPDI programming using a serial adapter, but it seems some people have figured out how to do it in some capacity. There was a promising pyupdi project, but it is now deprecated. At the time of writing the leading project aiming to enable UPDI programming without official hardware is pymcuprog, but its repository has a single commit dated two months ago and no activity since. Interestingly, that commit was made by buildmaster@microchip.com (an unverified email address), so it may not be fair to refer to it as an “unofficial” solution. The long term support of the pymcuprog project remains uncertain, but regardless let’s take a closer look at how it works.

To build a programmer you just need a TTL USB serial adapter and a 1kΩ resistor. These are the steps I used to program a LED blink program using this strategy:

Ping the Microcontroller

This command returns the device ID (1E9328 for my ATtiny826) indicating the UPDI connection is working successfully

pymcuprog ping -d attiny826 -t uart -u com12

Connecting to SerialUPDI
Pinging device...
Ping response: 1E9328

Write a Hex File

I used Microchip Studio to compile my C code and generate the hex file. Now I can use pymcuprog to load those hex files onto the chip. It’s slower to program and inconvenient to drop to a terminal whenever I want to program a chip, but it works.

pymcuprog write -f LedBlink.hex -d attiny826 -t uart -u com12

Connecting to SerialUPDI
Pinging device...
Ping response: 1E9328
Writing from hex file...
Writing flash...

Conclusions

Resources

  Feedback   Edit