The personal website of Scott W Harden

Signal Filtering in Python

Over a decade ago I posted code demonstrating how to filter data in Python, but there have been many improvements since then. My original posts (1, 2, 3, 4) required creating discrete filtering functions, but modern approaches can leverage Numpy and Scipy to do this more easily and efficiently. In this article we will use scipy.signal.filtfilt to apply low-pass, high-pass, and band-pass filters to reduce noise in an ECG signal (stored in ecg.wav (created as part of my Sound Card ECG project).

Moving-window filtering methods often result in a filtered signal that lags behind the original data (a phase shift). By filtering the signal twice in opposite directions filtfilt cancels-out this phase shift to produce a filtered signal which is nicely aligned with the input data.

import scipy.io.wavfile
import scipy.signal
import numpy as np
import matplotlib.pyplot as plt

# read ECG data from the WAV file
sampleRate, data = scipy.io.wavfile.read('ecg.wav')
times = np.arange(len(data))/sampleRate

# apply a 3-pole lowpass filter at 0.1x Nyquist frequency
b, a = scipy.signal.butter(3, 0.1)
filtered = scipy.signal.filtfilt(b, a, data)

# plot the original data next to the filtered data

plt.figure(figsize=(10, 4))

plt.subplot(121)
plt.plot(times, data)
plt.title("ECG Signal with Noise")
plt.margins(0, .05)

plt.subplot(122)
plt.plot(times, filtered)
plt.title("Filtered ECG Signal")
plt.margins(0, .05)

plt.tight_layout()
plt.show()

Cutoff Frequency

The second argument passed into the butter method customizes the cut-off frequency of the Butterworth filter. This value (Wn) is a number between 0 and 1 representing the fraction of the Nyquist frequency to use for the filter. Note that Nyquist frequency is half of the sample rate. As this fraction increases, the cutoff frequency increases. You can get fancy and express this value as 2 * Hz / sample rate.

plt.plot(data, '.-', alpha=.5, label="data")

for cutoff in [.03, .05, .1]:
    b, a = scipy.signal.butter(3, cutoff)
    filtered = scipy.signal.filtfilt(b, a, data)
    label = f"{int(cutoff*100):d}%"
    plt.plot(filtered, label=label)

plt.legend()
plt.axis([350, 500, None, None])
plt.title("Effect of Different Cutoff Values")
plt.show()

Improve Edges with Gustafsson’s Method

Something weird happens at the edges. There's not enough data "off the page" to know how to smooth those points, so what should be done?

Padding is the default behavior, where edges are padded with with duplicates of the edge data points and smooth the trace as if those data points existed. The drawback of this is that one stray data point at the edge will greatly affect the shape of your smoothed data.

Gustafsson’s Method may be superior to padding. The advantage of this method is that stray points at the edges do not greatly influence the smoothed curve at the edges. This technique is described in a 1994 paper by Fredrik Gustafsson. "Initial conditions are chosen for the forward and backward passes so that the forward-backward filter gives the same result as the backward-forward filter." Interestingly this paper demonstrates the method by filtering noise out of an EKG recording.

# A small portion of data will be inspected for demonstration
segment = data[350:400]

filtered = scipy.signal.filtfilt(b, a, segment)
filteredGust = scipy.signal.filtfilt(b, a, segment, method="gust")

plt.plot(segment, '.-', alpha=.5, label="data")
plt.plot(filtered, 'k--', label="padded")
plt.plot(filteredGust, 'k', label="Gustafsson")
plt.legend()
plt.title("Padded Data vs. Gustafsson’s Method")
plt.show()

Band-Pass Filter

Low-pass and high-pass filters can be selected simply by customizing the third argument passed into the filter. The second argument indicates frequency (as fraction of Nyquist frequency, half the sample rate). Passing a list of two values in for the second argument allows for band-pass filtering of a signal.

b, a = scipy.signal.butter(3, 0.05, 'lowpass')
filteredLowPass = scipy.signal.filtfilt(b, a, data)

b, a = scipy.signal.butter(3, 0.05, 'highpass')
filteredHighPass = scipy.signal.filtfilt(b, a, data)

b, a = scipy.signal.butter(3, [.01, .05], 'band')
filteredBandPass = scipy.signal.lfilter(b, a, data)

Filter using Convolution

Another way to low-pass a signal is to use convolution. In this method you create a window (typically a bell-shaped curve) and convolve the window with the signal. The wider the window is the smoother the output signal will be. Also, the window must be normalized so its sum is 1 to preserve the amplitude of the input signal.

There are different ways to handle what happens to data points at the edges (see numpy.convolve for details), but setting mode to valid delete these points to produce an output signal slightly smaller than the input signal.

# create a normalized Hanning window
windowSize = 40
window = np.hanning(windowSize)
window = window / window.sum()

# filter the data using convolution
filtered = np.convolve(window, data, mode='valid')

plt.subplot(131)
plt.plot(kernel)
plt.title("Window")

plt.subplot(132)
plt.plot(data)
plt.title("Data")

plt.subplot(133)
plt.plot(filtered)
plt.title("Filtered")

Different window functions filter the signal in different ways. Hanning windows are typically preferred because they have a mostly Gaussian shape but touch zero at the edges. For a discussion of the pros and cons of different window functions for spectral analysis using the FFT, see my notes on FftSharp.

Resources

Test React Apps in Azure Pipelines

Azure Pipelines makes it easy to run tests in the cloud, but I found that a new React projects made with create-react-app fail to properly test in the cloud using the simple npm test command. Attempting this would display No tests found related to files changed since last commit but hang forever.

I solved this problem and got my React app to test properly in the cloud by adding -- --watchAll=false after npm test. This is my final azure-pipelines.yml file:

trigger:
  - master

pool:
  vmImage: "ubuntu-latest"

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: "10.x"
    displayName: "Install Node.js"

  - script: npm install
    displayName: "Install NPM"

  - script: npm run build
    displayName: "Build"

  - script: npm test -- --watchAll=false
    displayName: "Test"

A working React app that tests properly with Azure Pipelines is GitHub.com/swharden/AliCalc

Deleted from The Wayback Machine

This week my website was removed from the Wayback Machine. The Wayback Machine is an impressive website that lets you view what a website looked like years ago. As part of Archive.org Internet Archive, this website is truly impressive and holds entertainingly-old versions of most webpages. Just look at Amazon.com in the year 2000 for a good laugh.

I started this blog as a child twenty years ago and after seeing what the Wayback Machine pulled-up I realized that it may be best that the thoughts I had as a child stay in the past. I have personal copies of all my old blog posts, but with the wisdom of age and hindsight I'd much prefer that that material stay off the internet. Luckily I was able to get my website removed from the Wayback Machine, and this post documents how I did it.

For those of you wanting to do the same, this is how I did it: I sent an email to info@archive.org stating the following:

Please remove my website [MY URL] from the Wayback Machine. [MY URL]/robots.txt has been updated to indicate I do not wish this website to be archived.

https://lookup.icann.org/ shows that [MY URL] points to [HOSTING COMPANY] nameservers, and I have attached a recent invoice from [HOSTING COMPANY] as evidence that I own this domain.

If additional evidence or action is required (e.g., DMCA takedown notice) please let me know.

Thank you!
Scott

I'm not sure if editing robots.txt was necessary, but I felt it gave credence to the fact that I had control over the content of this domain. That file contains the following text. In the past I read this was all it took to get your website de-listed from the wayback machine, but I added this same file to another domain name of mine and it has not been de-listed.

User-agent: archive.org_bot
Disallow: /

User-agent: ia_archiver
Disallow: /

I attached an invoice from the present year showing a credit card payment to my hosting company for the domain as a PDF. Interestingly I did not have to show a history of domain ownership. I downloaded the invoice from my hosting company's billing page that day, and it displays my home address but not my email address.

Six days later, my site was removed. This is the email I received:

FROM: Office Manager (Internet Archive)

Hello,

The following has now been submitted for exclusion from the Wayback Machine at web.archive.org: [MY SITE]

Please allow up to a day for the automated portions of the process to run their course and for the changes to take effect.

– The Internet Archive Team

I reviewed a lot of websites before reaching my strategy. I was surprised to see some people using issuing DMCA takedown notices notices to Archive.org, and was happy to find this was not required in my case. Here are some of the resources I found helpful:

⚠️ WARNING: This may not be permanent. I'm not sure what will happen if I lose my domain name (and robots.txt file) in the future. It is possible that my site is still being archived, while not being available on the wayback machine, and that some time in the future my site will be re-listed.

If you have updated information send me an email so I can update this page! In the mean time, I hope this information will be useful for others interested in curating their historical online presence.

Leaving WordPress

After fifteen years using WordPress, I'm leaving it for a simpler alternative: flat markdown files. There were several reasons behind why I made the change. First, I was disappointed with how frequently I had to update WordPress (and upgrade my database) to stay current with security updates. Second, I didn't like how abstract post content was. The text of posts was stored in SQL tables and references to image URLs weren't easily accessible (posts point to content IDs, the URLs of which were stored in another table), and images and media were scattered all over the filesystem because the default image placement changed several times over the years. Finally I found that logging in to a web front-end just to write a post was a bit of a barrier that prevented me from writing more frequently.

I have been very active on GitHub over the last few years and used their platform to share my code instead of this website. Lots of code and notes belong in repositories there, yes, but sometimes I create neat things which would be better represented as one-off posts on my personal website. Some of my repositories have collected notes like these, so I look forward to migrating a lot of that content here. My hope is that the new system I put together will make it easier to share content by writing it in Markdown using the editors I'm already working in every day.

Dynamic Markdown Parsing with PHP

The system I'm using now is pretty simple. Every post is a folder, and each folder contains a markdown file along with all of the images and files that post references. At the top of the markdown file is a little header which has information like title, date, and categories (tags). I use a PHP script route HTTP requests and if a requested folder lacks index.html but has index.md, I serve that using Parsedown to convert it to HTML. I also add a few tweaks to do things like convert YouTube links to embedded videos and add syntax highlighting to code blocks. Backups are easy (I just zip the folder), and the website could be committed to source control. I'm leaning away from this because it's about 1GB (lots of images), but I'll consider it. Also, the URL is just the path to the folder.

There's a clear path toward generating a static site. If a folder lacks index.html, index.md is parsed and served. Switching to and from a static site can be achieved just by pre-converting all the markdown files to html and deleting them. I'll probably keep working on refining the PHP script until the conversions are reliably processing like I desire, then convert most of the old pages to static files. The cool thing about this method is that it lets me serve some posts statically but others dynamically.

Wordpress (slow) Markdown (fast)

Performing the Conversion

The conversion from WordPress to Markdown was semi-automated, but still labor-intensive.

  • I first dumped the database to a SQL file, parsed-out the content and metadata (url, title, date, and privacy status), then created the filesystem and markdown files.

  • I then had to manually inspect every markdown file and reformat it, converting inline HTML to markdown (mostly images, galleries, and divs for alignment formatting). In many cases code formatting was damaged over the years, so lots of my old code was run through an autoformatter.

  • I also had to hunt-down the media (images, MP3s, ZIP files, etc.) for every post, copy it to the same folder, and update the URLs to be relative. This was especially hard for galleries which only point to meta content IDs (stored in a separate database table), and my database had gotten damaged somewhere along the way over the years so I really struggled to find the right content sometimes.

  • I also added tags to indicate categories, carefully reviewing content and code and marking posts as "old" if they contained out of date examples (lots of Python 2 code) or code that I deemed today to be of very poor quality. Part of me wanted to delete (hide) old posts with bad code, but I decided to leave them up. It's a reminder of how long I've worked in improving my craft, and my revulsion to code I wrote in the past is an indication of how much I've learned since.

  • This process took me about 10 hours a day for 3 days in a row.

Along the way I had a few laughs at the ridiculousness of some of my old content. I think it's probably a good thing to encourage teenagers to have personal websites, but I also encourage professionals and employers not to give too much credence to ramblings written by a person decades ago that Google happens to remember. I didn't delete any content, but I marked most of the posts I made as a teenager as private and only exposed the ones that discuss this website.

History of this Blog

After reviewing all of my posts I now have a really good understanding of the evolution of the technologies I used to serve my website over the years. Here's a summary of the major events:

  • It started as a blog on GeoCities, with the oldest surviving post dating to June 2001. Back then adding content meant editing HTML files and using FTP to upload changes.

  • In 2002 I started hosting my website from a server at my house. Initially it was served with Windows/IIS using ASP for comments pages. On October 19, 2002 I switched to FreeBSD/Apache using PHP for comments pages.

  • I started using the Movable Type (a flat-file PHP-based CMS) on Aug 25, 2003.

  • I migrated to WordPress (a CMS that stored posts in a database) in 2005.

  • In 2020 I converted all my posts to Markdown using PHP to dynamically generate HTML (with an avenue to generate flat-file output).

USB Frequency Counter

I built a frequency counter with a USB interface based around a 74LV8154 32-bit counter, FTDI FT230XS (USB serial adapter), and an ATMega328 microcontroller. I've used this same counter IC in some old projects (1, 2, 3, 4, 5) this time I decided to I design the circuit a little more carefully, make a PCB, and use all surface-mount technology (SMT).

The micro USB port provides power and PC connectivity, and when running the device sends frequency to the computer every second. All the parameters can be customized in software, and source code is on the USB-Counter GitHub page.

I also added support for a 7-segment LED display. The counter works fine without the screen attached, but using the screen lets this device serve as a frequency counter without requiring a computer. This display is a MAX7219-driven display module which currently runs for $2 each on Amazon when ordered in packs of 5.

Precision Frequency Measurement

One advantage of this counter is that it is never reset. Since this circuit uses 32-bit counter IC, and every gate cycle transmits the current count to the computer over USB. Because every input cycle is measured high precision measurements of frequency over long periods of time are possible. For example, 1000 repeated measurements with a 1Hz gate allows frequency measurement to a precision of 0.01 Hz.

Internal or External Gating

An optional external 1PPS gate can be used for precise timing. The microcontroller is capable of generating gate cycles in software. Precision is limited to that of the TCXO used to clock the microcontroller (2.5 PPM). For higher-precision gating a resistor may be lifted and an external gate applied (e.g., 1PPS GPS signal).

TCXO Driving a Gate in Software

By clocking the microcontroller at 14.7456 MHz with a temperature-compensated crystal oscillator (TCXO) I'm able to communicate with the PC easily at 115200 baud, and with some clever timer settings and interrupts I'm able to toggle an output pin every 14,745,600 cycles to produce a fairly accurate 1PPS signal.

Maximum Counting Frequency

According to the SN74LV8154 datasheet the minimum expected maximum input frequency (fMAX) is 40 MHz. To count higher frequencies, a high-speed prescaler could be added to the input to divide-down the input signal to a frequency this counter can range. This was [discussed]() in the original issue that kicked-off this project, and Onno Hoekstra (PA2OHH) recommended the SAB6456 divide-by-64/divide-by-256 prescaler which supports up to 1 GHz input frequency. However, present availability seems to be limited. A similar chip, or even a pair of octal flip-flops that work in the GHZ range could achieve this functionality.

Design

By populating one of two input paths with components this device can serve as a sensitive frequency counter (with a small-signal amplifier front-end) or a pulse counter (with a simple 50 ohm load at the front-end).

Optional RF Amplifier Front-End

An optional amplifier front-end has been added to turn weak input into strong square waves suitable for driving the TTL counter IC. It is designed for continuously running input, and will likely self-oscillate if it is not actively driven.

This simulation shows a small 1 MHz signal fed into a high impedance front-end being amplified to easily satisfy TTL levels. The 1k resistor (R3) could be swapped-out for a 50 Ohm resistor for a more traditional input impedance if desired. LTSpice source files are in the GitHub repository in case you want to refine the simulation.

Components

PCB

Changes from rev 1.0

  • improved RF amplifier design
  • alternate path to bypass RF amplifier
  • corrected counter pin connections
  • free a MCU pin by making the status LED the gate
  • add a header for the serial LED display
  • run MCU at 14.7456 MHz
    • allows much faster serial transmission
    • no longer accepts external 10MHz reference
    • may still accept external gate
  • added screw holes
    • they're floating (not grounded) - should they be grounded?
  • switched to micro USB (from mini USB)

Build Notes

  • didnt have 27 ohm resistors. used 22 ohm.
  • I used a 500mW rated R15
  • I used a 500mW rated R13
  • make hand solder version of usb
  • make oscillator fit less awkwardly
  • Add a 7805 so 12V can be applied or USB. Use a 78L33 (not the reg on the FTDI chip) to power everything else.
  • This device doesn't work when plugged into a wall USB cord (power only, no data). It seems an active USB connection is required to cause the 3.3V regulator (built into the FTDI chip) to deliver power. The next revision should use a discrete 3.3V regulator.
  • For a standalone (LED) device no USB connection is needed. Make a version that accepts 12V and displays the result on the LED. Make the optional external gate easy to access. Break-out the TX pin so PC logging is still very easy.

Resources

page 1, page 2, page 3, page 4, page 5, page 6, page 7, page 8, page 9, page 10, page 11, page 12, page 13, page 14, page 15, page 16, page 17, page 18, page 19, page 20, page 21, page 22, page 23, page 24, page 25, page 26, page 27, page 28, page 29, page 30, page 31, page 32, page 33, page 34, page 35, page 36, page 37, page 38, page 39, page 40, page 41, page 42, page 43, page 44, page 45, page 46, page 47, page 48, page 49
All Blog Posts