The personal website of Scott W Harden
July 14th, 2010

# High Altitude Balloon Transmitter

SUMMARY: A small group of high school students taking an AP class for college credit launched a high-altitude weather balloon with a small payload. In addition to a video transmitter and GPS transmitter, they decided to include a simple transmitter built from scratch. This is the story of the project, with emphasis on the simple transmitter's design, construction, implementation, and reception (which surprised me, being detected ~200 miles away and lasting the entire duration of the flight!) [sample.ogg]

# 6/16/2010 - TRACKING

I'm impressed how well the transmitter/receiver worked! For only a few milliwatts, I was able to track that thing all the way from takeoff to landing in Gainesville, FL a few hundred miles away.

ANALYSIS: the text on the image describes most if it, but one of the most interesting features is the "multipathing" during the final moments of the descent, where the single carrier signal splits into two. I believe this is due to two Doppler shifts: (1) as the distance between the falling transmitter and the receiver is decreasing, producing a slight in increase in frequency, and (2) a signal reflected off of a layer of the atmosphere above the craft (the ionosphere?) before it gets to the receiver, the distance of which is increasing as the craft falls, producing a decrease in frequency. I'll bet I can mathematically work backwards and determine how high the craft was, how fast it was falling, and/or how high the layer of the reflecting material is - but that's more work than this dental student is prepared to do before his morning coffee!

HERE IS SOME AUDIO of some of the strongest signals I received. Pretty good for a few milliwatts a hundred miles away! [beeps.ogg]

# 6/16/2010 - THE FLIGHT

The design team:

Walking the balloon to its launch destination at NASA with an awesome rocket (Saturn 1B - identified by Lee, KU4OS) in the background.

The team again, getting ready for launch. I've been informed that the reason their hands are up is to prevent the balloon from tilting over too much. I'd imagine that a brush with a grass blade could be bad news for the project!

Last minute checks - you can see the transmitter and battery holders for it taped to the Styrofoam.

The transmitter in its final position:

Note the coil of yellow wire. That serves as a rudimentary "ground" for the antenna's signal to push off of. I wasn't very clear on my instructions on how to make it. I meant that it should be a huge coil wrapped around the entire payload (as large as it can be), which would have probably produced a better signal, but since I was able to capture the signal during the whole flight it turned out to be a non-issue.

The antenna can be seen dropping down as a yellow wire beneath the payload. (arrow)

Launch! Look how fast that balloon is rising!

It's out of our hands now. When I got the text message that it launched, I held my breath. I was skeptical that the transmitter would even work!

One of the students listening to my transmitter with QRSS VD software (score!)

Video capture from an on-board camera was also attempted (900MHz), but from what I hear it didn't function well for very long.

# 6/15/2010 - IMPROVED BUILD

Here you can see me (center arrow) showing the students how to receive the Morse code signal sent from the small transmitter (left arrow) using a laptop running QRSS VD (my software) analyzing audio from and an Icom706 mkII radio receiver attached to a dipole (right arrow).

I amped-up the output of the oscillator using an octal buffer chip (74HC240) with some decent results. I'm pleased! It's not perfect (it's noisy as heck) but it should be functional for a 2 hour flight.

Closeup of the transmitter showing the oscillator at 29.4912 MHz, the Atmel ATTiny44a AVR microcontroller (left chip), octal buffer 74HC240 (right chip), and some status lights which blink as the code is executed.

This is my desk where I work from home. Note the styrofoam box in the background - that's where my low-power transmitter lives (the one that's spotted around the world). All I needed to build this device was a soldering iron.

Although I had a radio, it is not capable of receiving 29MHz so I was unable to test the transmitter from home. I had to take it to the university to assess its transmitting capabilities.

I connected the leads to the output of the transmitter, shorted by a 39ohm resistor. By measuring the peak-to-peak voltage of the signal going into a resistor, we can measure its power.

Here's the test setup. The transmitter is on the blue pad on the right, and the waveform can be seen on the oscilloscope on the upper left.

With the amplifier off, the output power is just that of the oscillator. Although the wave should look like a sine wave, it's noisy, and simply does not. While this is unacceptable if our goal is a clean radio signal with maximum efficiency, this is good enough to be heard at our target frequency. The PPV (peak-to-peak voltage) as seen on the screen is about 100mV. Since I'm using a x10 probe, this value should be multiplied by 10 = 1V. 1V PPV into 39 ohms is about 3 milliwatts! ((1/(2*2^.5))^2/39*1000=3.2). For the math, see this post

With the amplifier, the output is much more powerful. At 600mV peak-to-peak with a 10x probe (actually 6V peak-to-peak, expected because that's the voltage of the 4xAAA battery supply we're using) into 39 ohms we get 115 millivolts! (6/(2*2^.5))^2/39*1000=115.38.

Notes about power: First of all, the actual power output isn't 115mW. The reason is that the math equations I used work only for pure sine waves. Since our transmitter has multiple waves in it, less than that power is going to produce our primary signal. It's possible that only 50mW are going to our 29MHz signal, so the power output assessment is somewhat qualitative. Something significant however is the difference between the measured power with and without the amplifier. The 6x increase in peak-to-peak voltage results in a 36x (6^2) increase in power, which is very beneficial. I'm glad I added this amplifier! A 36 times increase in power will certainly help.

# 6/14/2010 - THE BUILD

Last week I spoke with a student in the UF aerospace engineering department who told me he was working with a group of high school students to add a payload to a high-altitude balloon being launched at (and tracked by) NASA. We tossed around a few ideas about what to put on it, and we decided it was worth a try to add a transmitter. I'll slowly add to this post as the project unfolds, but with only 2 days to prepare (wow!) I picked a simplistic design which should be extremely easy to understand by everyone. Here's the schematic:

The code is as simple as it gets. It sends some Morse code ("go gators"), then a long tone (about 15 seconds) which I hope can be measured QRSS style. I commented virtually every line so it should be easy to understand how the program works.

``````#include <avr/io.h>
#include <util/delay.h>

char call[] = {2, 2, 1, 0, 2, 2, 2, 0, 0, 2, 2, 1, 0, 1, 2, 0, 2, 0, 2, 2, 2, 0, 1, 2, 1, 0, 1, 1, 1, 0, 0};
// 0 for space, 1 for dit, 2 for dah

void sleep()
{
_delay_ms(100);      // sleep for a while
PORTA ^= (1 << PA1); // "flip" the state of the TICK light
}

void ON()
{
PORTB = 255;          // turn on transmitter
PORTA |= (1 << PA3);  // turn on the ON light
PORTA &= ~(1 << PA2); // turn off the ON light
}

void OFF()
{
PORTB = 0;            // turn off transmitter
PORTA |= (1 << PA2);  // turn on the OFF light
PORTA &= ~(1 << PA3); // turn off the OFF light
}

void ID()
{
for (char i = 0; i < sizeof(call); i++)
{
if (call[i] == 0)
{
OFF();
} // space
if (call[i] == 1)
{
ON();
} // dot
if (call[i] == 2)
{
ON();
sleep();
sleep();
} // dash
sleep();
OFF();
sleep();
sleep(); // between letters
}
}

void tone()
{
ON(); // turn on the transmitter
for (char i = 0; i < 200; i++)
{ // do this a lot of times
sleep();
}
OFF();
sleep();
sleep();
sleep(); // a little pause
}

int main(void) // PROGRAM STARTS HERE
{
DDRB = 255; // set all of port B to output
DDRA = 255; // set all of port A to output
PORTA = 1;  // turn on POWER light

while (1)
{           // loop forever
ID();   // send morse code ID
tone(); // send a long beep
}
}``````

I'm now wondering if I should further amplify this signal's output power. Perhaps a 74HC240 can handle 9V? ... or maybe it would be better to use 4 AAA batteries in series to give me about 6V. [ponders] this is the schematic I'm thinking of building.

UPDATE: This story was featured on Hack-A-Day! Way to go everyone!

```---
title: High Altitude Balloon Transmitter
date: 2010-07-14 08:05:46
---

# High Altitude Balloon Transmitter

<blockquote class="wp-block-quote"><p><b>SUMMARY:</b> A small group of high school students taking an AP class for college credit launched a high-altitude weather balloon with a small payload. In addition to a video transmitter and GPS transmitter, they decided to include a simple transmitter built from scratch. This is the story of the project, with emphasis on the simple transmitter's design, construction, implementation, and reception (which surprised me, being detected ~200 miles away and lasting the entire duration of the flight!) [<a href="http://www.SWHarden.com/blog/images/beeps.ogg">sample.ogg</a>]</p></blockquote>

# 6/16/2010 - TRACKING

__I'm impressed __ how well the transmitter/receiver worked! For only a few milliwatts, I was able to track that thing all the way from takeoff to landing in Gainesville, FL a few hundred miles away.

<div class="text-center img-border">

[![](balloon_track_thumb.jpg)](balloon_track.jpg)

</div>

__ANALYSIS:__ the text on the image describes most if it, but one of the most interesting features is the "multipathing" during the final moments of the descent, where the single carrier signal splits into two. I believe this is due to two Doppler shifts: (1) as the distance between the falling transmitter and the receiver is decreasing, producing a slight in increase in frequency, and (2) a signal reflected off of a layer of the atmosphere above the craft (the ionosphere?) before it gets to the receiver, the distance of which is increasing as the craft falls, producing a decrease in frequency. I'll bet I can mathematically work backwards and determine how high the craft was, how fast it was falling, and/or how high the layer of the reflecting material is - but that's more work than this dental student is prepared to do before his morning coffee!

__HERE IS SOME AUDIO__ of some of the strongest signals I received. Pretty good for a few milliwatts a hundred miles away! \[[beeps.ogg](http://www.SWHarden.com/blog/images/beeps.ogg)\]

# 6/16/2010 - THE FLIGHT

The design team:

<div class="text-center img-border">

[![](DSC_7127_thumb.jpg)](DSC_7127.jpg)

</div>

<strong>Walking the balloon</strong> to its launch destination at NASA with an awesome rocket (Saturn 1B - identified by Lee, KU4OS) in the background.

<div class="text-center img-border">

[![](DSC_7210_thumb.jpg)](DSC_7210.jpg)

</div>

<strong>The team</strong> again, getting ready for launch. I've been informed that the reason their hands are up is to prevent the balloon from tilting over too much. I'd imagine that a brush with a grass blade could be bad news for the project!

<div class="text-center img-border">

[![](DSC_7232_thumb.jpg)](DSC_7232.jpg)

</div>

<strong>Last minute checks</strong> - you can see the transmitter and battery holders for it taped to the Styrofoam.

<div class="text-center img-border">

[![](DSC_7248_thumb.jpg)](DSC_7248.jpg)

</div>

The transmitter in its final position:

<div class="text-center img-border">

[![](DSC_7250_thumb.jpg)](DSC_7250.jpg)

</div>

Note the coil of yellow wire. That serves as a rudimentary "ground" for the antenna's signal to push off of. I wasn't very clear on my instructions on how to make it. I meant that it should be a huge coil wrapped around the entire payload (as large as it can be), which would have probably produced a better signal, but since I was able to capture the signal during the whole flight it turned out to be a non-issue.

<strong>The antenna</strong> can be seen dropping down as a yellow wire beneath the payload. (arrow)

<div class="text-center img-border">

[![](DSC_7253_thumb.jpg)](DSC_7253.jpg)

[![](DSC_7279_thumb.jpg)](DSC_7279.jpg)

</div>

<strong>Launch!</strong> Look how fast that balloon is rising!

<div class="text-center img-border">

[![](DSC_7294_thumb.jpg)](DSC_7294.jpg)

</div>

<strong>It's out of our hands</strong> now. When I got the text message that it launched, I held my breath. I was skeptical that the transmitter would even work!

<div class="text-center img-border">

[![](DSC_7297_thumb.jpg)](DSC_7297.jpg)

</div>

<strong>One of the students</strong> listening to my transmitter with QRSS VD software (score!)

<div class="text-center img-border">

[![](DSC_7365_thumb.jpg)](DSC_7365.jpg)

</div>

<strong>Video capture</strong> from an on-board camera was also attempted (900MHz), but from what I hear it didn't function well for very long.

<div class="text-center img-border">

[![](DSC_7334_thumb.jpg)](DSC_7334.jpg)

</div>

# 6/15/2010 - IMPROVED BUILD

__Here you can see me__ (center arrow) showing the students how to receive the Morse code signal sent from the small transmitter (left arrow) using a laptop running [QRSS VD (my software)](http://www.swharden.com/blog/qrss_vd/) analyzing audio from and an Icom706 mkII radio receiver attached to a dipole (right arrow).

<div class="text-center img-border">

[![](DSC_7082_thumb.jpg)](DSC_7082.jpg)
[![](72hc240_qrp_amplifier_thumb.jpg)](72hc240_qrp_amplifier.jpg)

</div>

__I amped-up the output of the oscillator__ using an octal buffer chip (74HC240) with some decent results. I'm pleased! It's not perfect (it's noisy as heck) but it should be functional for a 2 hour flight.

<div class="text-center img-border">

[![](01_closeup_thumb.jpg)](01_closeup.jpg)

</div>

Closeup of the transmitter showing the oscillator at 29.4912 MHz, the Atmel ATTiny44a AVR microcontroller (left chip), octal buffer 74HC240 (right chip), and some status lights which blink as the code is executed.

<div class="text-center img-border">

[![](02_workstation_thumb.jpg)](02_workstation.jpg)

</div>

__This is my desk__ where I work from home. Note the styrofoam box in the background - that's where my low-power transmitter lives (the one that's spotted around the world). All I needed to build this device was a soldering iron.

<div class="text-center img-border">

[![](03_room_thumb.jpg)](03_room.jpg)

</div>

__Although I had a radio,__ it is not capable of receiving 29MHz so I was unable to test the transmitter from home. I had to take it to the university to assess its transmitting capabilities.

__I connected the leads to the output of the transmitter, shorted by a 39ohm resistor.__ By measuring the peak-to-peak voltage of the signal going into a resistor, we can measure its power.

__Here's the test setup.__ The transmitter is on the blue pad on the right, and the waveform can be seen on the oscilloscope on the upper left.

<div class="text-center img-border">

[![](06_scope_thumb.jpg)](06_scope.jpg)

</div>

__With the amplifier off__, the output power is just that of the oscillator. Although the wave should look like a sine wave, it's noisy, and simply does not. While this is unacceptable if our goal is a clean radio signal with maximum efficiency, this is good enough to be heard at our target frequency. The PPV (peak-to-peak voltage) as seen on the screen is about 100mV. Since I'm using a x10 probe, this value should be multiplied by 10 = 1V. 1V PPV into 39 ohms is about __3 milliwatts!__ ((1/(2\*2^.5))^2/39\*1000=3.2). For the math, see [this post](http://www.swharden.com/blog/2010-05-28-measuring-qrp-radio-output-power-with-an-oscilliscope/)

__With the amplifier,__ the output is much more powerful. At 600mV peak-to-peak with a 10x probe (actually 6V peak-to-peak, expected because that's the voltage of the 4xAAA battery supply we're using) into 39 ohms we get __115 millivolts!__ (6/(2\*2^.5))^2/39\*1000=115.38.

__Notes about power:__ First of all, the actual power output isn't 115mW. The reason is that the math equations I used work only for pure sine waves. Since our transmitter has multiple waves in it, less than that power is going to produce our primary signal. It's possible that only 50mW are going to our 29MHz signal, so the power output assessment is somewhat qualitative. Something significant however is the difference between the measured power with and without the amplifier. The 6x increase in peak-to-peak voltage results in a 36x (6^2) increase in power, which is very beneficial. I'm glad I added this amplifier! A 36 times increase in power will certainly help.

<div class="text-center img-border">

[![](balloon_transmitter_final_thumb.jpg)](balloon_transmitter_final.png)

</div>

# 6/14/2010 - THE BUILD

__Last week I spoke with a student in the UF aerospace engineering department who told me he was working with a group of high school students to add a payload to a high-altitude balloon being launched at (and tracked by) NASA.__ We tossed around a few ideas about what to put on it, and we decided it was worth a try to add a transmitter. I'll slowly add to this post as the project unfolds, but with only 2 days to prepare (wow!) I picked a simplistic design which should be extremely easy to understand by everyone. Here's the schematic:

<div class="text-center img-border">

[![](balloon_transmitter_thumb.jpg)](balloon_transmitter.png)

</div>

__The code is as simple as it gets.__ It sends some Morse code ("go gators"), then a long tone (about 15 seconds) which I hope can be measured QRSS style. I commented virtually every line so it should be easy to understand how the program works.

```c
#include <avr/io.h>
#include <util/delay.h>

char call[] = {2, 2, 1, 0, 2, 2, 2, 0, 0, 2, 2, 1, 0, 1, 2, 0, 2, 0, 2, 2, 2, 0, 1, 2, 1, 0, 1, 1, 1, 0, 0};
// 0 for space, 1 for dit, 2 for dah

void sleep()
{
_delay_ms(100);      // sleep for a while
PORTA ^= (1 << PA1); // "flip" the state of the TICK light
}

void ON()
{
PORTB = 255;          // turn on transmitter
PORTA |= (1 << PA3);  // turn on the ON light
PORTA &= ~(1 << PA2); // turn off the ON light
}

void OFF()
{
PORTB = 0;            // turn off transmitter
PORTA |= (1 << PA2);  // turn on the OFF light
PORTA &= ~(1 << PA3); // turn off the OFF light
}

void ID()
{
for (char i = 0; i < sizeof(call); i++)
{
if (call[i] == 0)
{
OFF();
} // space
if (call[i] == 1)
{
ON();
} // dot
if (call[i] == 2)
{
ON();
sleep();
sleep();
} // dash
sleep();
OFF();
sleep();
sleep(); // between letters
}
}

void tone()
{
ON(); // turn on the transmitter
for (char i = 0; i < 200; i++)
{ // do this a lot of times
sleep();
}
OFF();
sleep();
sleep();
sleep(); // a little pause
}

int main(void) // PROGRAM STARTS HERE
{
DDRB = 255; // set all of port B to output
DDRA = 255; // set all of port A to output
PORTA = 1;  // turn on POWER light

while (1)
{           // loop forever
ID();   // send morse code ID
tone(); // send a long beep
}
}
```

__I'm now wondering if I should further amplify this signal's output power.__ Perhaps a 74HC240 can handle 9V? ... or maybe it would be better to use 4 AAA batteries in series to give me about 6V. \[ponders\] [this](http://www.SWHarden.com/blog/images/balloon_transmitter_amplified.png) is the schematic I'm thinking of building.

**UPDATE:** This story was featured on [Hack-A-Day](http://hackaday.com/2010/07/27/200-mile-rf-transmitter-and-high-altitude-balloon/)! Way to go everyone!

<div class="text-center img-border">

</div>

```
June 28th, 2010

# Getting GTK, Glade, and Python to Work in Windows

These screenshots show me running the Py2EXE-compiled script I wrote last weekend on a Windows 7 machine. Additionally there is a screenshot of the "Add/Remove Programs" window demonstrating which versions of which libraries were required.

```---
title: Getting GTK, Glade, and Python to Work in Windows
date: 2010-06-28 09:46:26
tags: python, old
---

# Getting GTK, Glade, and Python to Work in Windows

__These screenshots__ show me running the Py2EXE-compiled script I wrote last weekend on a Windows 7 machine. Additionally there is a screenshot of the "Add/Remove Programs" window demonstrating which versions of which libraries were required.

<div class="text-center img-border">

[![](needToInstall_thumb.jpg)](needToInstall.png)

</div>

```
June 24th, 2010

# Fast TK Pixelmap generation from 2D Numpy Arrays in Python

I had TKinter all wrong! While my initial tests with PyGame's rapid ability to render Numpy arrays in the form of pixel maps proved impressive, it was only because I was comparing it to poor TK code. I don't know what I was doing wrong, but when I decided to give TKinter one more shot I was blown away -- it's as smooth or smoother as PyGame. Forget PyGame! I'm rendering everything in raw TK from now on. This utilizes the Python Imaging Library (PIL) so it's EXTREMELY flexible (supports fancy operations, alpha channels, etc).

The screenshot shows me running the script (below) generating random noise and "scrolling" it horizontally (like my spectrograph software does) at a fast rate smoothly (almost 90 FPS!). Basically, it launches a window, creates a canvas widget (which I'm told is faster to update than a label and reduces flickering that's often associated with rapid redraws because it uses double-buffering). Also, it uses threading to handle the calculations/redraws without lagging the GUI. The code speaks for itself.

``````import Tkinter
from PIL import Image, ImageTk
import numpy
import time

class mainWindow():
times = 1
timestart = time.clock()
data = numpy.array(numpy.random.random((400, 500))*100, dtype=int)

def __init__(self):
self.root = Tkinter.Tk()
self.frame = Tkinter.Frame(self.root, width=500, height=400)
self.frame.pack()
self.canvas = Tkinter.Canvas(self.frame, width=500, height=400)
self.canvas.place(x=-2, y=-2)
self.root.after(0, self.start)  # INCREASE THE 0 TO SLOW IT DOWN
self.root.mainloop()

def start(self):
global data
self.im = Image.fromstring('L', (self.data.shape[1],
self.data.shape[0]), self.data.astype('b').tostring())
self.photo = ImageTk.PhotoImage(image=self.im)
self.canvas.create_image(0, 0, image=self.photo, anchor=Tkinter.NW)
self.root.update()
self.times += 1
if self.times % 33 == 0:
print "%.02f FPS" % (self.times/(time.clock()-self.timestart))
self.root.after(10, self.start)
self.data = numpy.roll(self.data, -1, 1)

if __name__ == '__main__':
x = mainWindow()``````
```---
title: Fast TK Pixelmap generation from 2D Numpy Arrays in Python
date: 2010-06-24 21:29:29
tags: python, old
---

# Fast TK Pixelmap generation from 2D Numpy Arrays in Python

__I had TKinter all wrong!__ While my initial tests with PyGame's rapid ability to render Numpy arrays in the form of pixel maps proved impressive, it was only because I was comparing it to poor TK code. I don't know what I was doing wrong, but when I decided to give TKinter one more shot I was blown away -- it's as smooth or smoother as PyGame. Forget PyGame! I'm rendering everything in raw TK from now on. This utilizes the Python Imaging Library (PIL) so it's EXTREMELY flexible (supports fancy operations, alpha channels, etc).

<div class="text-center img-border">

</div>

__The screenshot shows__ me running the script (below) generating random noise and "scrolling" it horizontally (like my spectrograph software does) at a fast rate smoothly (almost 90 FPS!). Basically, it launches a window, creates a canvas widget (which I'm told is faster to update than a label and reduces flickering that's often associated with rapid redraws because it uses double-buffering). Also, it uses threading to handle the calculations/redraws without lagging the GUI. The code speaks for itself.

```python
import Tkinter
from PIL import Image, ImageTk
import numpy
import time

class mainWindow():
times = 1
timestart = time.clock()
data = numpy.array(numpy.random.random((400, 500))*100, dtype=int)

def __init__(self):
self.root = Tkinter.Tk()
self.frame = Tkinter.Frame(self.root, width=500, height=400)
self.frame.pack()
self.canvas = Tkinter.Canvas(self.frame, width=500, height=400)
self.canvas.place(x=-2, y=-2)
self.root.after(0, self.start)  # INCREASE THE 0 TO SLOW IT DOWN
self.root.mainloop()

def start(self):
global data
self.im = Image.fromstring('L', (self.data.shape[1],
self.data.shape[0]), self.data.astype('b').tostring())
self.photo = ImageTk.PhotoImage(image=self.im)
self.canvas.create_image(0, 0, image=self.photo, anchor=Tkinter.NW)
self.root.update()
self.times += 1
if self.times % 33 == 0:
print "%.02f FPS" % (self.times/(time.clock()-self.timestart))
self.root.after(10, self.start)
self.data = numpy.roll(self.data, -1, 1)

if __name__ == '__main__':
x = mainWindow()
```

```
June 24th, 2010

# Detrending Data in Python with Numpy

⚠️ SEE UPDATED POST: Signal Filtering in Python

While continuing my quest into the world of linear data analysis and signal processing, I came to a point where I wanted to emphasize variations in FFT traces. While I am keeping my original data for scientific reference, visually I want to represent it emphasizing variations rather than concentrating on trends. I wrote a detrending function which I'm sure will be useful for many applications:

``````def detrend(data,degree=10):
detrended=[None]*degree
for i in range(degree,len(data)-degree):
chunk=data[i-degree:i+degree]
chunk=sum(chunk)/len(chunk)
detrended.append(data[i]-chunk)
return detrended+[None]*degree``````

However, this method is extremely slow. I need to think of a way to accomplish this same thing much faster. [ponders]

UPDATE: It looks like I've once again re-invented the wheel. All of this has been done already, and FAR more efficiently I might add. For more see scipy.signal.detrend.html

``````import scipy.signal
ffty=scipy.signal.detrend(ffty)``````
```---
title: Detrending Data in Python with Numpy
date: 2010-06-24 08:38:52
tags: python, old
---

# Detrending Data in Python with Numpy

> **⚠️ SEE UPDATED POST:** [**Signal Filtering in Python**](https://swharden.com/blog/2020-09-23-signal-filtering-in-python/)

__While continuing my quest__ into the world of linear data analysis and signal processing, I came to a point where I wanted to emphasize variations in FFT traces. While I am keeping my original data for scientific reference, visually I want to represent it emphasizing variations rather than concentrating on trends. I wrote a detrending function which I'm sure will be useful for many applications:

```python
def detrend(data,degree=10):
detrended=[None]*degree
for i in range(degree,len(data)-degree):
chunk=data[i-degree:i+degree]
chunk=sum(chunk)/len(chunk)
detrended.append(data[i]-chunk)
return detrended+[None]*degree
```

<div class="text-center">

[![](detrend_fft_thumb.jpg)](detrend_fft.png)

</div>

However, this method is extremely slow. I need to think of a way to accomplish this same thing much faster. \[ponders\]

__UPDATE:__ It looks like I've once again re-invented the wheel. All of this has been done already, and FAR more efficiently I might add. For more see [scipy.signal.detrend.html](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.detrend.html)

```python
import scipy.signal
ffty=scipy.signal.detrend(ffty)
```

```
June 23rd, 2010

# Insights Into FFTs, Imaginary Numbers, and Accurate Spectrographs

I'm attempting to thoroughly re-write the data assessment portions of my QRSS VD software, and rather than rushing to code it (like I did last time) I'm working hard on every step trying to optimize the code. I came across some notes I made about Fast Fourier Transformations from the first time I coded the software, and though I'd post some code I found helpful. Of particular satisfaction is an email I received from Alberto, I2PHD, the creator of Argo (the "gold standard" QRSS spectrograph software for Windows). In it he notes:

I think that [it is a mistake to] throw away the imaginary part of the FFT. What I do in Argo, in Spectran, in Winrad, in SDRadio and in all of my other programs is compute the magnitude of the [FFT] signal, then compute the logarithm of it, and only then I do a mapping of the colors on the screen with the result of this last computation.

Alberto, I2PHD (the creator of Argo)

UPDATE IN SEPTEMBER, 2020 (10 years later): I now understand that `magnitude = sqrt(real^2 + imag^2)` and this post is a bit embarrassing to read! Check out my .NET FFT library FftSharp for a more advanced discussion on this topic.

These concepts are simple to visualize when graphed. Here I've written a short Python script to listen to the microphone (which is being fed a 2kHz sine wave), perform the FFT, and graph the real FFT component, imaginary FFT component, and their sum. The output is:

Of particular interest to me is the beautiful complementary of the two curves. It makes me wonder what types of data can be extracted by the individual curves (or perhaps their difference?) down the road. I wonder if phase measurements would be useful in extracting weak carries from beneath the noise floor?

Here's the code I used to generate the image above. Note that my microphone device was set to listen to my stereo output, and I generated a 2kHz sine wave using the command `speaker-test -t sine -f 2000` on a PC running Linux. I hope you find it useful!

``````import numpy
import pyaudio
import pylab
import numpy

### RECORD AUDIO FROM MICROPHONE ###
rate = 44100
soundcard = 1  # CUSTOMIZE THIS!!!
p = pyaudio.PyAudio()
strm = p.open(format=pyaudio.paInt16, channels=1, rate=rate,
input_device_index=soundcard, input=True)
strm.read(1024)  # prime the sound card this way

### DO THE FFT ANALYSIS ###
fft = numpy.fft.fft(pcm)
fftr = 10*numpy.log10(abs(fft.real))[:len(pcm)/2]
ffti = 10*numpy.log10(abs(fft.imag))[:len(pcm)/2]
fftb = 10*numpy.log10(numpy.sqrt(fft.imag**2+fft.real**2))[:len(pcm)/2]
freq = numpy.fft.fftfreq(numpy.arange(len(pcm)).shape[-1])[:len(pcm)/2]
freq = freq*rate/1000  # make the frequency scale

### GRAPH THIS STUFF ###
pylab.subplot(411)
pylab.title("Original Data")
pylab.grid()
pylab.plot(numpy.arange(len(pcm))/float(rate)*1000, pcm, 'r-', alpha=1)
pylab.xlabel("Time (milliseconds)")
pylab.ylabel("Amplitude")
pylab.subplot(412)
pylab.title("Real FFT")
pylab.xlabel("Frequency (kHz)")
pylab.ylabel("Power")
pylab.grid()
pylab.plot(freq, fftr, 'b-', alpha=1)
pylab.subplot(413)
pylab.title("Imaginary FFT")
pylab.xlabel("Frequency (kHz)")
pylab.ylabel("Power")
pylab.grid()
pylab.plot(freq, ffti, 'g-', alpha=1)
pylab.subplot(414)
pylab.title("Real+Imaginary FFT")
pylab.xlabel("Frequency (kHz)")
pylab.ylabel("Power")
pylab.grid()
pylab.plot(freq, fftb, 'k-', alpha=1)
pylab.show()``````

After fighting for a while long with a "shifty baseline" of the FFT, I came to another understanding. Let me first address the problem. Taking the FFT of different regions of the 2kHz wave I got traces with the peak in the identical location, but the "baselines" completely different.

Like many things, I re-invented the wheel. Since I knew the PCM values weren't changing, the only variable was the starting/stopping point of the linear sample. "Hard edges", I imagined, must be the problem. I then wrote the following function to shape the PCM audio like a triangle, silencing the edges and sweeping the volume up toward the middle of the sample:

``````def shapeTriangle(data):
triangle=numpy.array(range(len(data)/2)+range(len(data)/2)[::-1])+1
return data*triangle``````

After shaping the data BEFORE I applied the FFT, I made the subsequent traces MUCH more acceptable. Observe:

Now that I've done all this experimentation/thinking, I remembered that this is nothing new! Everyone talks about shaping the wave to minimize hard edges before taking the FFT. They call it windowing. Another case of me re-inventing the wheel because I'm too lazy to read others' work. However, in my defense, I learned a lot by trying all this stuff -- far more than I would have learned simply by copying someone else's code into my script. Experimentation is the key to discovery!

```---
title: Insights Into FFTs, Imaginary Numbers, and Accurate Spectrographs
date: 2010-06-23 22:21:00
---

# Insights Into FFTs, Imaginary Numbers, and Accurate Spectrographs

__I'm attempting to thoroughly re-write the data assessment__ portions of my QRSS VD software, and rather than rushing to code it (like I did last time) I'm working hard on every step trying to optimize the code. I came across some notes I made about Fast Fourier Transformations from the first time I coded the software, and though I'd post some code I found helpful. Of particular satisfaction is an email I received from Alberto, I2PHD, the creator of Argo (the "gold standard" QRSS spectrograph software for Windows). In it he notes:

<blockquote class="wp-block-quote"><p>I think that [it is a mistake to] throw away the imaginary part of the FFT. What I do in Argo, in Spectran, in Winrad, in SDRadio and in all of my other programs is compute the magnitude of the [FFT] signal, then compute the logarithm of it, and only then I do a mapping of the colors on the screen with the result of this last computation.</p><cite> Alberto, I2PHD (the creator of Argo)</cite></blockquote>

> __UPDATE IN SEPTEMBER, 2020 (10 years later):__ I now understand that `magnitude = sqrt(real^2 + imag^2)` and this post is a bit embarrassing to read! Check out my .NET FFT library [FftSharp](https://github.com/swharden/FftSharp) for a more advanced discussion on this topic.

__These concepts are simple__ to visualize when graphed. Here I've written a short Python script to listen to the microphone (which is being fed a 2kHz sine wave), perform the FFT, and graph the real FFT component, imaginary FFT component, and their sum. The output is:

<div class="text-center">

[![](real_imaginary_fft_pcm_thumb.jpg)](real_imaginary_fft_pcm.png)

</div>

__Of particular interest__ to me is the beautiful complementary of the two curves. It makes me wonder what types of data can be extracted by the individual curves (or perhaps their difference?) down the road. I wonder if phase measurements would be useful in extracting weak carries from beneath the noise floor?

<div class="text-center">

[![](fft_base2_thumb.jpg)](fft_base2.png)

</div>

__Here's the code I used to generate the image above.__ Note that my microphone device was set to listen to my stereo output, and I generated a 2kHz sine wave using the command `` speaker-test -t sine -f 2000 `` on a PC running Linux. I hope you find it useful!

```python
import numpy
import pyaudio
import pylab
import numpy

### RECORD AUDIO FROM MICROPHONE ###
rate = 44100
soundcard = 1  # CUSTOMIZE THIS!!!
p = pyaudio.PyAudio()
strm = p.open(format=pyaudio.paInt16, channels=1, rate=rate,
input_device_index=soundcard, input=True)
strm.read(1024)  # prime the sound card this way

### DO THE FFT ANALYSIS ###
fft = numpy.fft.fft(pcm)
fftr = 10*numpy.log10(abs(fft.real))[:len(pcm)/2]
ffti = 10*numpy.log10(abs(fft.imag))[:len(pcm)/2]
fftb = 10*numpy.log10(numpy.sqrt(fft.imag**2+fft.real**2))[:len(pcm)/2]
freq = numpy.fft.fftfreq(numpy.arange(len(pcm)).shape[-1])[:len(pcm)/2]
freq = freq*rate/1000  # make the frequency scale

### GRAPH THIS STUFF ###
pylab.subplot(411)
pylab.title("Original Data")
pylab.grid()
pylab.plot(numpy.arange(len(pcm))/float(rate)*1000, pcm, 'r-', alpha=1)
pylab.xlabel("Time (milliseconds)")
pylab.ylabel("Amplitude")
pylab.subplot(412)
pylab.title("Real FFT")
pylab.xlabel("Frequency (kHz)")
pylab.ylabel("Power")
pylab.grid()
pylab.plot(freq, fftr, 'b-', alpha=1)
pylab.subplot(413)
pylab.title("Imaginary FFT")
pylab.xlabel("Frequency (kHz)")
pylab.ylabel("Power")
pylab.grid()
pylab.plot(freq, ffti, 'g-', alpha=1)
pylab.subplot(414)
pylab.title("Real+Imaginary FFT")
pylab.xlabel("Frequency (kHz)")
pylab.ylabel("Power")
pylab.grid()
pylab.plot(freq, fftb, 'k-', alpha=1)
pylab.show()
```

__After fighting for a while long with__ a "shifty baseline" of the FFT, I came to another understanding. Let me first address the problem. Taking the FFT of different regions of the 2kHz wave I got traces with the peak in the identical location, but the "baselines" completely different.

<div class="text-center">

[![](fft_base3_thumb.jpg)](fft_base3.png)

</div>

__Like many things, I re-invented the wheel.__ Since I knew the PCM values weren't changing, the only variable was the starting/stopping point of the linear sample. "Hard edges", I imagined, must be the problem. I then wrote the following function to shape the PCM audio like a triangle, silencing the edges and sweeping the volume up toward the middle of the sample:

```python
def shapeTriangle(data):
triangle=numpy.array(range(len(data)/2)+range(len(data)/2)[::-1])+1
return data*triangle
```

__After shaping the data BEFORE I applied the FFT,__ I made the subsequent traces MUCH more acceptable. Observe:

__Now that I've done all this experimentation/thinking,__ I remembered that this is nothing new! Everyone talks about shaping the wave to minimize hard edges before taking the FFT. They call it _windowing._ Another case of me re-inventing the wheel because I'm too lazy to read others' work. However, in my defense, I learned a lot by trying all this stuff -- far more than I would have learned simply by copying someone else's code into my script. Experimentation is the key to discovery!
```
Pages