The personal website of Scott W Harden

Graphing Computer Usage

I enjoy writing Python scripts to analyze and display linear data. One of my favorite blog entries is Linear Data Smoothing with Python, developed for my homemade electrocardiogram project. I installed a program called TimeTrack.exe on my work computer. It basically logs whenever you open or close a program. The data output looks like this:

"Firefox","Prototype of a Digital Biopsy Device - Mozilla Firefox","05/19/2009  9:45a","05/19/2009  9:45a","766ms","0.0"
"Firefox","Dual-Channel Mobile Surface Electromyograph - Mozilla Firefox","05/19/2009  9:46a","05/19/2009  9:46a","797ms","0.0"
"Windows Explorer","","03/24/2008  9:30a","05/19/2009  9:48a","49d 6h 9m","20.7"
"Windows Explorer","09_04_07_RA_SA_AV","05/19/2009  8:48a","05/19/2009  8:48a","1.0s","0.0"
"Windows Explorer","Image003.jpg - Windows Picture and Fax Viewer","05/18/2009  4:03p","05/18/2009  4:03p","1.2s","0.0"

I have a 13 MB file containing lines like this which I parse, condense, analyze, and display with Python. The script finds the first and last entry time and creates a dictionary where keys are the hours between the 1st and last log lines, parses the log, determines which time block each entry belongs to, and increments the integer (value of the dictionary) for its respective key. Something similar is repeated, but with respect to days rather than hours. The result is:

The code I used to generate this graph is:

# This script analyzes data exported from "TimeTrack" (a free computer usage
# monitoring program for windows) and graphs the data visually.

import time, pylab, datetime, numpy

# This is my computer usage data.  Generate yours however you want.
allHours = ['2008_10_29 0', '2009_03_11 5', '2009_04_09 5', '2008_07_04 10',
'2008_12_18 9', '2009_01_30 12', '2008_09_04 7', '2008_05_17 1',
'2008_05_11 5', '2008_11_03 3', '2008_05_21 3', '2009_02_19 11',
'2008_08_15 13', '2008_04_02 4', '2008_07_16 5', '2008_09_16 8',
'2008_04_10 5', '2009_05_10 1', '2008_12_30 4', '2008_06_07 2',
'2008_11_23 0', '2008_08_03 0', '2008_04_30 4', '2008_07_28 9',
'2008_05_19 0', '2009_03_30 7', '2008_06_19 3', '2009_01_24 3',
'2008_08_23 6', '2008_12_01 0', '2009_02_23 6', '2008_11_27 0',
'2008_05_02 5', '2008_10_20 13', '2008_03_27 5', '2009_04_02 9',
'2009_02_21 0', '2008_09_13 1', '2008_12_13 0', '2009_04_14 11',
'2009_01_31 7', '2008_11_04 10', '2008_07_09 6', '2008_10_24 10',
'2009_02_22 0', '2008_09_25 12', '2008_12_25 0', '2008_05_26 4',
'2009_05_01 10', '2009_04_26 11', '2008_08_10 8', '2008_11_08 6',
'2008_07_21 12', '2009_04_21 3', '2009_05_13 8', '2009_02_02 8',
'2008_10_07 2', '2008_06_10 6', '2008_09_21 0', '2009_03_17 9',
'2008_08_30 7', '2008_11_28 4', '2009_02_14 0', '2009_01_22 6',
'2008_10_11 0', '2008_06_22 8', '2008_12_04 0', '2008_03_28 0',
'2009_04_07 2', '2008_09_10 0', '2008_05_15 5', '2008_08_18 12',
'2008_10_31 5', '2009_03_09 7', '2009_02_25 8', '2008_07_02 4',
'2008_12_16 7', '2008_09_06 2', '2009_01_26 5', '2009_04_19 0',
'2008_07_14 13', '2008_11_01 5', '2009_01_18 0', '2009_05_04 0',
'2008_08_13 10', '2009_02_27 3', '2009_01_16 12', '2008_09_18 8',
'2009_02_03 7', '2008_06_01 0', '2008_12_28 0', '2008_07_26 0',
'2008_11_21 1', '2008_08_01 8', '2008_04_28 3', '2009_05_16 0',
'2008_06_13 5', '2008_10_02 11', '2009_03_28 6', '2008_08_21 7',
'2009_01_13 6', '2008_11_25 4', '2008_06_25 1', '2008_10_22 11',
'2008_03_25 6', '2009_02_07 6', '2008_12_11 4', '2009_01_01 4',
'2008_09_15 2', '2009_02_05 12', '2008_07_07 9', '2009_04_12 0',
'2008_04_11 5', '2008_10_26 4', '2008_05_28 3', '2008_09_27 14',
'2009_05_03 0', '2008_12_23 5', '2009_05_12 10', '2008_11_14 3',
'2008_07_19 0', '2009_04_24 8', '2008_04_07 1', '2008_08_08 11',
'2008_06_04 0', '2009_05_15 12', '2009_03_23 13', '2009_02_01 10',
'2008_09_23 11', '2009_02_08 3', '2008_08_28 4', '2008_11_18 9',
'2008_07_31 7', '2008_10_13 0', '2008_06_16 9', '2009_03_27 6',
'2008_12_02 0', '2008_05_01 7', '2009_04_05 1', '2008_08_16 9',
'2009_03_15 0', '2008_04_16 6', '2008_10_17 4', '2008_06_28 5',
'2009_01_28 10', '2008_04_18 0', '2008_12_14 0', '2008_11_07 6',
'2009_04_17 7', '2008_04_14 7', '2008_07_12 0', '2009_01_15 7',
'2009_05_06 8', '2008_12_26 0', '2008_06_03 7', '2008_09_28 0',
'2008_05_25 4', '2008_08_07 8', '2008_04_26 7', '2008_07_24 1',
'2008_04_20 0', '2008_11_11 4', '2009_04_29 0', '2008_10_04 0',
'2009_05_18 9', '2009_03_18 4', '2008_06_15 8', '2009_02_13 6',
'2008_05_04 5', '2009_03_04 2', '2009_03_06 3', '2008_05_06 0',
'2008_08_27 11', '2008_04_22 0', '2009_03_26 6', '2008_03_31 9',
'2008_06_27 5', '2008_10_08 4', '2008_09_09 4', '2008_12_09 3',
'2008_05_10 0', '2008_05_14 5', '2009_04_10 0', '2009_01_11 0',
'2008_07_05 8', '2009_01_05 7', '2008_10_28 0', '2009_02_18 11',
'2009_03_10 7', '2008_05_30 3', '2008_09_05 7', '2008_12_21 6',
'2009_03_02 6', '2008_08_14 5', '2008_11_12 5', '2008_07_17 8',
'2008_04_05 6', '2009_04_22 11', '2009_05_09 0', '2008_06_06 0',
'2009_01_03 0', '2008_09_17 6', '2009_03_21 3', '2009_02_10 7',
'2008_05_08 4', '2008_08_02 0', '2008_11_16 0', '2008_07_29 12',
'2008_10_15 5', '2008_06_18 5', '2009_03_25 2', '2009_01_10 0',
'2009_04_03 5', '2008_08_22 7', '2009_03_13 11', '2008_10_19 0',
'2008_06_30 8', '2008_09_02 9', '2008_05_23 4', '2008_12_12 7',
'2008_07_10 11', '2008_11_05 8', '2008_04_12 4', '2009_04_15 7',
'2008_12_24 1', '2008_09_30 0', '2008_05_27 2', '2008_08_05 10',
'2008_04_24 6', '2009_04_27 6', '2008_07_22 3', '2008_11_09 1',
'2008_06_09 6', '2008_10_06 14', '2009_03_16 7', '2008_05_22 5',
'2009_01_29 12', '2008_11_29 4', '2008_04_09 7', '2008_08_25 12',
'2009_02_15 0', '2008_03_29 7', '2008_06_21 7', '2008_10_10 9',
'2008_05_12 6', '2009_02_16 10', '2008_09_11 11', '2008_12_07 0',
'2008_07_03 6', '2009_04_08 3', '2009_01_23 7', '2009_01_27 5',
'2008_10_30 0', '2009_03_08 0', '2009_01_21 8', '2008_12_19 0',
'2008_05_16 2', '2009_01_25 1', '2009_02_26 5', '2008_09_07 2',
'2008_04_03 1', '2008_08_12 6', '2008_04_13 10', '2008_11_02 0',
'2008_07_15 0', '2009_04_20 3', '2009_02_24 10', '2009_05_11 8',
'2008_12_31 8', '2008_04_15 7', '2008_09_19 10', '2009_01_19 0',
'2008_11_22 3', '2008_07_27 2', '2009_02_04 7', '2009_03_31 1',
'2008_05_24 3', '2008_10_01 8', '2008_06_12 6', '2009_01_12 11',
'2008_11_26 8', '2009_04_01 10', '2009_02_28 0', '2008_08_20 6',
'2008_10_21 10', '2008_06_24 4', '2008_03_26 4', '2008_12_10 0',
'2008_09_12 0', '2008_05_09 7', '2009_02_17 7', '2008_07_08 6',
'2008_10_25 5', '2009_04_13 9', '2009_05_02 0', '2008_12_22 8',
'2008_09_24 9', '2009_01_20 5', '2008_11_15 6', '2009_04_25 10',
'2008_08_11 9', '2008_04_06 8', '2008_07_20 1', '2009_03_22 3',
'2008_06_11 6', '2008_09_20 3', '2009_05_14 10', '2008_11_19 0',
'2008_08_31 2', '2009_02_09 8', '2008_10_12 0', '2008_04_25 5',
'2008_06_23 4', '2009_01_07 8', '2008_08_19 0', '2008_12_05 2',
'2008_07_01 8', '2008_10_16 6', '2009_04_06 3', '2009_03_14 5',
'2008_09_01 2', '2008_12_17 14', '2008_05_18 7', '2008_04_01 2',
'2009_04_18 0', '2008_04_17 0', '2008_07_13 0', '2008_06_02 10',
'2008_09_29 6', '2008_12_29 0', '2009_05_05 8', '2008_04_19 0',
'2009_04_30 8', '2008_08_06 4', '2008_11_20 0', '2008_07_25 6',
'2009_02_06 6', '2009_03_29 3', '2009_05_17 0', '2009_03_19 7',
'2008_10_03 1', '2008_06_14 3', '2008_05_07 5', '2008_08_26 3',
'2008_11_24 9', '2008_04_21 8', '2008_04_23 4', '2008_10_23 11',
'2008_06_26 4', '2008_03_24 8', '2008_12_08 5', '2008_09_14 2',
'2009_01_02 6', '2008_04_08 0', '2008_10_27 6', '2009_04_11 0',
'2008_07_06 0', '2008_12_20 3', '2009_04_23 6', '2008_09_26 9',
'2008_05_31 0', '2008_07_18 4', '2008_11_13 6', '2008_08_09 2',
'2008_04_04 0', '2009_03_20 5', '2008_09_22 7', '2009_05_08 9',
'2008_06_05 7', '2008_07_30 7', '2008_11_17 10', '2008_05_03 0',
'2008_08_29 3', '2009_02_11 12', '2009_01_08 8', '2008_06_17 0',
'2008_10_14 7', '2009_03_24 11', '2008_08_17 6', '2008_12_03 0',
'2009_01_09 4', '2008_05_29 5', '2008_06_29 9', '2008_10_18 5',
'2009_04_04 0', '2008_12_15 10', '2009_03_12 0', '2009_03_05 7',
'2008_05_20 4', '2008_09_03 7', '2009_03_07 8', '2009_01_14 6',
'2008_05_05 5', '2008_11_06 7', '2008_07_11 6', '2009_04_16 9',
'2009_02_20 0', '2008_12_27 0', '2009_01_17 0', '2009_05_07 7',
'2008_11_10 5', '2008_07_23 11', '2009_04_28 0', '2008_04_27 2',
'2008_08_04 0', '2009_03_01 11', '2008_10_05 0', '2008_06_08 8',
'2009_05_19 5', '2008_04_29 4', '2008_11_30 0', '2009_01_06 8',
'2009_02_12 3', '2008_08_24 2', '2009_03_03 10', '2008_10_09 6',
'2008_06_20 2', '2008_05_13 10', '2008_12_06 0', '2008_03_30 7']

def genTimes():
    ## opens  exported timetrack data (CSV) and re-saves a compressed version.
    print "ANALYZING..."
    f=open('timetrack.txt')
    raw=f.readlines()
    f.close()
    times=["05/15/2009 12:00am"] #start time
    for line in raw[1:]:
        if not line.count('","') == 5: continue
        test = line.strip("n")[1:-1].split('","')[-3].replace("  "," ")+"m"
        test = test.replace(" 0:"," 12:")
        times.append(test) #end time
        test = line.strip("n")[1:-1].split('","')[-4].replace("  "," ")+"m"
        test = test.replace(" 0:"," 12:")
        times.append(test) #start time

    times.sort()
    print "WRITING..."
    f=open('times.txt','w')
    f.write(str(times))
    f.close()

def loadTimes():
    ## loads the times from the compressed file.
    f=open("times.txt")
    times = eval(f.read())
    newtimes=[]
    f.close()
    for i in range(len(times)):
        if "s" in times[i]: print times[i]
        newtimes.append(datetime.datetime(*time.strptime(times[i],
                                        "%m/%d/%Y %I:%M%p")[0:5]))
        #if i>1000: break #for debugging
    newtimes.sort()
    return newtimes

def linearize(times):
    ## does all the big math to calculate hours per day.
    for i in range(len(times)):
        times[i]=times[i]-datetime.timedelta(minutes=times[i].minute,
                                             seconds=times[i].second)
    hr = datetime.timedelta(hours=1)
    pos = times[0]-hr
    counts = {}
    days = {}
    lasthr=pos
    lastday=None
    while pos1:counts[pos]=1 #flatten
        if not daypos in days: days[daypos]=0
        if not lasthr == pos:
            if counts[pos]>0:
                days[daypos]=days[daypos]+1
                lasthr=pos
        pos+=hr
    return days #[counts,days]

def genHours(days):
    ## outputs the hours per day as a file.
    out=""
    for day in days:
        print day
        out+="%s %in"%(day.strftime("%Y_%m_%d"),days[day])
    f=open('hours.txt','w')
    f.write(out)
    f.close()
    return

def smoothListGaussian(list,degree=7):
    ## (from an article I wrote) - Google "linear data smoothing with python".
    firstlen=len(list)
    window=degree*2-1
    weight=numpy.array([1.0]*window)
    weightGauss=[]
    for i in range(window):
     i=i-degree+1
     frac=i/float(window)
     gauss=1/(numpy.exp((4*(frac))**2))
     weightGauss.append(gauss)
    weight=numpy.array(weightGauss)*weight
    smoothed=[0.0]*(len(list)-window)
    for i in range(len(smoothed)):
     smoothed[i]=sum(numpy.array(list[i:i+window])*weight)/sum(weight)
    pad_before = [smoothed[0]]*((firstlen-len(smoothed))/2)
    pad_after  = [smoothed[-1]]*((firstlen-len(smoothed))/2+1)
    return pad_before+smoothed+pad_after

### IF YOU USE MY DATA, YOU ONLY USE THE FOLLOWING CODE ###

def graphIt():
    ## Graph the data!
    #f=open('hours.txt')
    #data=f.readlines()
    data=allHours
    data.sort()
    f.close()
    days,hours=[],[]
    for i in range(len(data)):
        day = data[i].split(" ")
        if int(day[1])<4: continue
        days.append(datetime.datetime.strptime(day[0], "%Y_%m_%d"))
        hours.append(int(day[1]))
    fig=pylab.figure(figsize=(14,5))
    pylab.plot(days,smoothListGaussian(hours,1),'.',color='.5',label="single day")
    pylab.plot(days,smoothListGaussian(hours,1),'-',color='.8')
    pylab.plot(days,smoothListGaussian(hours,7),color='b',label="7-day gausian average")
    pylab.axhline(8,color='k',ls=":")
    pylab.title("Computer Usage at Work")
    pylab.ylabel("hours (rounded)")
    pylab.legend()
    pylab.show()
    return

#times = genTimes()
#genHours(linearize(loadTimes()))
graphIt()
Markdown source code last modified on January 18th, 2021
---
title: Graphing Computer Usage
date: 2009-05-20 08:44:57
tags: python, old
---

# Graphing Computer Usage

__I enjoy writing Python scripts to analyze and display linear data.__ One of my favorite blog entries is [Linear Data Smoothing with Python](http://www.swharden.com/blog/2008-11-17-linear-data-smoothing-in-python/), developed for my [homemade electrocardiogram](http://www.swharden.com/blog/category/diy-ecg-home-made-electrocardiogram/) project. I installed a program called TimeTrack.exe on my work computer. It basically logs whenever you open or close a program. The data output looks like this:

```
"Firefox","Prototype of a Digital Biopsy Device - Mozilla Firefox","05/19/2009  9:45a","05/19/2009  9:45a","766ms","0.0"
"Firefox","Dual-Channel Mobile Surface Electromyograph - Mozilla Firefox","05/19/2009  9:46a","05/19/2009  9:46a","797ms","0.0"
"Windows Explorer","","03/24/2008  9:30a","05/19/2009  9:48a","49d 6h 9m","20.7"
"Windows Explorer","09_04_07_RA_SA_AV","05/19/2009  8:48a","05/19/2009  8:48a","1.0s","0.0"
"Windows Explorer","Image003.jpg - Windows Picture and Fax Viewer","05/18/2009  4:03p","05/18/2009  4:03p","1.2s","0.0"
```

__I have a 13 MB file containing lines like this__ which I parse, condense, analyze, and display with Python. The script finds the first and last entry time and creates a dictionary where keys are the hours between the 1st and last log lines, parses the log, determines which time block each entry belongs to, and increments the integer (value of the dictionary) for its respective key. Something similar is repeated, but with respect to days rather than hours. The result is:

<div class="text-center">

[![](compusage_white_thumb.jpg)](compusage_white.png)

</div>

The code I used to generate this graph is:

```python
# This script analyzes data exported from "TimeTrack" (a free computer usage
# monitoring program for windows) and graphs the data visually.

import time, pylab, datetime, numpy

# This is my computer usage data.  Generate yours however you want.
allHours = ['2008_10_29 0', '2009_03_11 5', '2009_04_09 5', '2008_07_04 10',
'2008_12_18 9', '2009_01_30 12', '2008_09_04 7', '2008_05_17 1',
'2008_05_11 5', '2008_11_03 3', '2008_05_21 3', '2009_02_19 11',
'2008_08_15 13', '2008_04_02 4', '2008_07_16 5', '2008_09_16 8',
'2008_04_10 5', '2009_05_10 1', '2008_12_30 4', '2008_06_07 2',
'2008_11_23 0', '2008_08_03 0', '2008_04_30 4', '2008_07_28 9',
'2008_05_19 0', '2009_03_30 7', '2008_06_19 3', '2009_01_24 3',
'2008_08_23 6', '2008_12_01 0', '2009_02_23 6', '2008_11_27 0',
'2008_05_02 5', '2008_10_20 13', '2008_03_27 5', '2009_04_02 9',
'2009_02_21 0', '2008_09_13 1', '2008_12_13 0', '2009_04_14 11',
'2009_01_31 7', '2008_11_04 10', '2008_07_09 6', '2008_10_24 10',
'2009_02_22 0', '2008_09_25 12', '2008_12_25 0', '2008_05_26 4',
'2009_05_01 10', '2009_04_26 11', '2008_08_10 8', '2008_11_08 6',
'2008_07_21 12', '2009_04_21 3', '2009_05_13 8', '2009_02_02 8',
'2008_10_07 2', '2008_06_10 6', '2008_09_21 0', '2009_03_17 9',
'2008_08_30 7', '2008_11_28 4', '2009_02_14 0', '2009_01_22 6',
'2008_10_11 0', '2008_06_22 8', '2008_12_04 0', '2008_03_28 0',
'2009_04_07 2', '2008_09_10 0', '2008_05_15 5', '2008_08_18 12',
'2008_10_31 5', '2009_03_09 7', '2009_02_25 8', '2008_07_02 4',
'2008_12_16 7', '2008_09_06 2', '2009_01_26 5', '2009_04_19 0',
'2008_07_14 13', '2008_11_01 5', '2009_01_18 0', '2009_05_04 0',
'2008_08_13 10', '2009_02_27 3', '2009_01_16 12', '2008_09_18 8',
'2009_02_03 7', '2008_06_01 0', '2008_12_28 0', '2008_07_26 0',
'2008_11_21 1', '2008_08_01 8', '2008_04_28 3', '2009_05_16 0',
'2008_06_13 5', '2008_10_02 11', '2009_03_28 6', '2008_08_21 7',
'2009_01_13 6', '2008_11_25 4', '2008_06_25 1', '2008_10_22 11',
'2008_03_25 6', '2009_02_07 6', '2008_12_11 4', '2009_01_01 4',
'2008_09_15 2', '2009_02_05 12', '2008_07_07 9', '2009_04_12 0',
'2008_04_11 5', '2008_10_26 4', '2008_05_28 3', '2008_09_27 14',
'2009_05_03 0', '2008_12_23 5', '2009_05_12 10', '2008_11_14 3',
'2008_07_19 0', '2009_04_24 8', '2008_04_07 1', '2008_08_08 11',
'2008_06_04 0', '2009_05_15 12', '2009_03_23 13', '2009_02_01 10',
'2008_09_23 11', '2009_02_08 3', '2008_08_28 4', '2008_11_18 9',
'2008_07_31 7', '2008_10_13 0', '2008_06_16 9', '2009_03_27 6',
'2008_12_02 0', '2008_05_01 7', '2009_04_05 1', '2008_08_16 9',
'2009_03_15 0', '2008_04_16 6', '2008_10_17 4', '2008_06_28 5',
'2009_01_28 10', '2008_04_18 0', '2008_12_14 0', '2008_11_07 6',
'2009_04_17 7', '2008_04_14 7', '2008_07_12 0', '2009_01_15 7',
'2009_05_06 8', '2008_12_26 0', '2008_06_03 7', '2008_09_28 0',
'2008_05_25 4', '2008_08_07 8', '2008_04_26 7', '2008_07_24 1',
'2008_04_20 0', '2008_11_11 4', '2009_04_29 0', '2008_10_04 0',
'2009_05_18 9', '2009_03_18 4', '2008_06_15 8', '2009_02_13 6',
'2008_05_04 5', '2009_03_04 2', '2009_03_06 3', '2008_05_06 0',
'2008_08_27 11', '2008_04_22 0', '2009_03_26 6', '2008_03_31 9',
'2008_06_27 5', '2008_10_08 4', '2008_09_09 4', '2008_12_09 3',
'2008_05_10 0', '2008_05_14 5', '2009_04_10 0', '2009_01_11 0',
'2008_07_05 8', '2009_01_05 7', '2008_10_28 0', '2009_02_18 11',
'2009_03_10 7', '2008_05_30 3', '2008_09_05 7', '2008_12_21 6',
'2009_03_02 6', '2008_08_14 5', '2008_11_12 5', '2008_07_17 8',
'2008_04_05 6', '2009_04_22 11', '2009_05_09 0', '2008_06_06 0',
'2009_01_03 0', '2008_09_17 6', '2009_03_21 3', '2009_02_10 7',
'2008_05_08 4', '2008_08_02 0', '2008_11_16 0', '2008_07_29 12',
'2008_10_15 5', '2008_06_18 5', '2009_03_25 2', '2009_01_10 0',
'2009_04_03 5', '2008_08_22 7', '2009_03_13 11', '2008_10_19 0',
'2008_06_30 8', '2008_09_02 9', '2008_05_23 4', '2008_12_12 7',
'2008_07_10 11', '2008_11_05 8', '2008_04_12 4', '2009_04_15 7',
'2008_12_24 1', '2008_09_30 0', '2008_05_27 2', '2008_08_05 10',
'2008_04_24 6', '2009_04_27 6', '2008_07_22 3', '2008_11_09 1',
'2008_06_09 6', '2008_10_06 14', '2009_03_16 7', '2008_05_22 5',
'2009_01_29 12', '2008_11_29 4', '2008_04_09 7', '2008_08_25 12',
'2009_02_15 0', '2008_03_29 7', '2008_06_21 7', '2008_10_10 9',
'2008_05_12 6', '2009_02_16 10', '2008_09_11 11', '2008_12_07 0',
'2008_07_03 6', '2009_04_08 3', '2009_01_23 7', '2009_01_27 5',
'2008_10_30 0', '2009_03_08 0', '2009_01_21 8', '2008_12_19 0',
'2008_05_16 2', '2009_01_25 1', '2009_02_26 5', '2008_09_07 2',
'2008_04_03 1', '2008_08_12 6', '2008_04_13 10', '2008_11_02 0',
'2008_07_15 0', '2009_04_20 3', '2009_02_24 10', '2009_05_11 8',
'2008_12_31 8', '2008_04_15 7', '2008_09_19 10', '2009_01_19 0',
'2008_11_22 3', '2008_07_27 2', '2009_02_04 7', '2009_03_31 1',
'2008_05_24 3', '2008_10_01 8', '2008_06_12 6', '2009_01_12 11',
'2008_11_26 8', '2009_04_01 10', '2009_02_28 0', '2008_08_20 6',
'2008_10_21 10', '2008_06_24 4', '2008_03_26 4', '2008_12_10 0',
'2008_09_12 0', '2008_05_09 7', '2009_02_17 7', '2008_07_08 6',
'2008_10_25 5', '2009_04_13 9', '2009_05_02 0', '2008_12_22 8',
'2008_09_24 9', '2009_01_20 5', '2008_11_15 6', '2009_04_25 10',
'2008_08_11 9', '2008_04_06 8', '2008_07_20 1', '2009_03_22 3',
'2008_06_11 6', '2008_09_20 3', '2009_05_14 10', '2008_11_19 0',
'2008_08_31 2', '2009_02_09 8', '2008_10_12 0', '2008_04_25 5',
'2008_06_23 4', '2009_01_07 8', '2008_08_19 0', '2008_12_05 2',
'2008_07_01 8', '2008_10_16 6', '2009_04_06 3', '2009_03_14 5',
'2008_09_01 2', '2008_12_17 14', '2008_05_18 7', '2008_04_01 2',
'2009_04_18 0', '2008_04_17 0', '2008_07_13 0', '2008_06_02 10',
'2008_09_29 6', '2008_12_29 0', '2009_05_05 8', '2008_04_19 0',
'2009_04_30 8', '2008_08_06 4', '2008_11_20 0', '2008_07_25 6',
'2009_02_06 6', '2009_03_29 3', '2009_05_17 0', '2009_03_19 7',
'2008_10_03 1', '2008_06_14 3', '2008_05_07 5', '2008_08_26 3',
'2008_11_24 9', '2008_04_21 8', '2008_04_23 4', '2008_10_23 11',
'2008_06_26 4', '2008_03_24 8', '2008_12_08 5', '2008_09_14 2',
'2009_01_02 6', '2008_04_08 0', '2008_10_27 6', '2009_04_11 0',
'2008_07_06 0', '2008_12_20 3', '2009_04_23 6', '2008_09_26 9',
'2008_05_31 0', '2008_07_18 4', '2008_11_13 6', '2008_08_09 2',
'2008_04_04 0', '2009_03_20 5', '2008_09_22 7', '2009_05_08 9',
'2008_06_05 7', '2008_07_30 7', '2008_11_17 10', '2008_05_03 0',
'2008_08_29 3', '2009_02_11 12', '2009_01_08 8', '2008_06_17 0',
'2008_10_14 7', '2009_03_24 11', '2008_08_17 6', '2008_12_03 0',
'2009_01_09 4', '2008_05_29 5', '2008_06_29 9', '2008_10_18 5',
'2009_04_04 0', '2008_12_15 10', '2009_03_12 0', '2009_03_05 7',
'2008_05_20 4', '2008_09_03 7', '2009_03_07 8', '2009_01_14 6',
'2008_05_05 5', '2008_11_06 7', '2008_07_11 6', '2009_04_16 9',
'2009_02_20 0', '2008_12_27 0', '2009_01_17 0', '2009_05_07 7',
'2008_11_10 5', '2008_07_23 11', '2009_04_28 0', '2008_04_27 2',
'2008_08_04 0', '2009_03_01 11', '2008_10_05 0', '2008_06_08 8',
'2009_05_19 5', '2008_04_29 4', '2008_11_30 0', '2009_01_06 8',
'2009_02_12 3', '2008_08_24 2', '2009_03_03 10', '2008_10_09 6',
'2008_06_20 2', '2008_05_13 10', '2008_12_06 0', '2008_03_30 7']

def genTimes():
    ## opens  exported timetrack data (CSV) and re-saves a compressed version.
    print "ANALYZING..."
    f=open('timetrack.txt')
    raw=f.readlines()
    f.close()
    times=["05/15/2009 12:00am"] #start time
    for line in raw[1:]:
        if not line.count('","') == 5: continue
        test = line.strip("n")[1:-1].split('","')[-3].replace("  "," ")+"m"
        test = test.replace(" 0:"," 12:")
        times.append(test) #end time
        test = line.strip("n")[1:-1].split('","')[-4].replace("  "," ")+"m"
        test = test.replace(" 0:"," 12:")
        times.append(test) #start time

    times.sort()
    print "WRITING..."
    f=open('times.txt','w')
    f.write(str(times))
    f.close()

def loadTimes():
    ## loads the times from the compressed file.
    f=open("times.txt")
    times = eval(f.read())
    newtimes=[]
    f.close()
    for i in range(len(times)):
        if "s" in times[i]: print times[i]
        newtimes.append(datetime.datetime(*time.strptime(times[i],
                                        "%m/%d/%Y %I:%M%p")[0:5]))
        #if i&gt;1000: break #for debugging
    newtimes.sort()
    return newtimes

def linearize(times):
    ## does all the big math to calculate hours per day.
    for i in range(len(times)):
        times[i]=times[i]-datetime.timedelta(minutes=times[i].minute,
                                             seconds=times[i].second)
    hr = datetime.timedelta(hours=1)
    pos = times[0]-hr
    counts = {}
    days = {}
    lasthr=pos
    lastday=None
    while pos1:counts[pos]=1 #flatten
        if not daypos in days: days[daypos]=0
        if not lasthr == pos:
            if counts[pos]&gt;0:
                days[daypos]=days[daypos]+1
                lasthr=pos
        pos+=hr
    return days #[counts,days]

def genHours(days):
    ## outputs the hours per day as a file.
    out=""
    for day in days:
        print day
        out+="%s %in"%(day.strftime("%Y_%m_%d"),days[day])
    f=open('hours.txt','w')
    f.write(out)
    f.close()
    return

def smoothListGaussian(list,degree=7):
    ## (from an article I wrote) - Google "linear data smoothing with python".
    firstlen=len(list)
    window=degree*2-1
    weight=numpy.array([1.0]*window)
    weightGauss=[]
    for i in range(window):
     i=i-degree+1
     frac=i/float(window)
     gauss=1/(numpy.exp((4*(frac))**2))
     weightGauss.append(gauss)
    weight=numpy.array(weightGauss)*weight
    smoothed=[0.0]*(len(list)-window)
    for i in range(len(smoothed)):
     smoothed[i]=sum(numpy.array(list[i:i+window])*weight)/sum(weight)
    pad_before = [smoothed[0]]*((firstlen-len(smoothed))/2)
    pad_after  = [smoothed[-1]]*((firstlen-len(smoothed))/2+1)
    return pad_before+smoothed+pad_after

### IF YOU USE MY DATA, YOU ONLY USE THE FOLLOWING CODE ###

def graphIt():
    ## Graph the data!
    #f=open('hours.txt')
    #data=f.readlines()
    data=allHours
    data.sort()
    f.close()
    days,hours=[],[]
    for i in range(len(data)):
        day = data[i].split(" ")
        if int(day[1])&lt;4: continue
        days.append(datetime.datetime.strptime(day[0], "%Y_%m_%d"))
        hours.append(int(day[1]))
    fig=pylab.figure(figsize=(14,5))
    pylab.plot(days,smoothListGaussian(hours,1),'.',color='.5',label="single day")
    pylab.plot(days,smoothListGaussian(hours,1),'-',color='.8')
    pylab.plot(days,smoothListGaussian(hours,7),color='b',label="7-day gausian average")
    pylab.axhline(8,color='k',ls=":")
    pylab.title("Computer Usage at Work")
    pylab.ylabel("hours (rounded)")
    pylab.legend()
    pylab.show()
    return

#times = genTimes()
#genHours(linearize(loadTimes()))
graphIt()
```

Audio and Video on an ATMega88 Microcontroller!

At least 99% of my blog entries contain original content I created. However, I was so impressed by something I stumbled upon tonight that I absolutely cannot resist sharing it. It's a project which aims to output audio and video simultaneously from a single microcontroller. The author's site has all the details, but if you watch the video below you'll be amazed. (Just get through the 30 second narration at the beginning) Apparently the guy rendered video in horizontal lines from the software and outputted audio signals between horizontal lines! Amazing!

On second thought, this was no big deal in the 80's, so why am I so impressed by it now? The 8-bit microcontrollers this guy is programming is likely on par with the PCs of the 80's. I guess that in the 70's this would have been amazing because it was cutting edge. In the 80's this would have been boring because it was commonplace. In the 2000's, this is amazing because no one in my generation is old enough to remember how amazing this was in the 70's and 80's!

After researching some similar projects I realized I'm becoming fascinated with chiptune synthesizing code, hardware, and music. It's basically a type of code to tell a synthesizer how to synthesize the music, rather than having it play pre-recorded music. It's like a merge of a programming language and MIDI. For non-technical people, it's like giving a microchip the sheet music for all the different instruments of an orchestra and having the microchip play everything from the sheet music, rather than giving it a CD to play. Here's some video of a PC-based front-end to the audio creation process. Notice how each line represents a time, and how certain codes in certain channels represent notes (hence the MIDI aspect). Numbers on the far right represent the location of the memory, and notice also how it goes back and forth, replaying certain areas when necessary (to safe space, hence the coding aspect).

I can't describe my emotional state right now. It's like I have an extreme nostalgia for an era I never even lived in. This chip level audio synthesis stuff sounds amazingly fun to me. (I've already bookmarked nolife-radio.com and 8bitFM.com) I feel drawn toward it... but I'm scared to get sucked in. I wish I were a college student in the 80's studying electrical engineering. Here I am, having just gotten a master's in molecular biology and microbiology, and I feel like I wasted six years of my life in the process. I'm about to start dental school in August. Hopefully when I look back from the future I won't feel like I wasted another four years doing that.

Either way, I have an endless supply of possible projects to do this summer (not even going into the small list of projects I'm trying/expected to complete in the cardioneurophysiology lab I work in). I feel like I'm running out of time. I start dental school in August, and I dread the date. Not necessarily because I expect it to be difficult, but because I feel [however illogical, irrational, or ridiculous] that I'm actually doing something significant, working with my hands and working with my brain to do things that [almost] no one has done before, and doing things in a way that no one has ever thought of doing them. I feel like when I start resume classes, it's another four years of people telling me how I should do things so I can be exactly like everyone else. How can you exercise creativity as a dental student? I'm sure there are ways, but it's certainly leagues away from the projects engineering college students work on. As far as the career goes, if you're an engineer the best case scenario is that you do something no one has ever done before. If you're a dentist, the best case scenario is to do things exactly as everyone else does them. Maybe I'll go crazy and change the wallpaper in my office every few months.

Markdown source code last modified on January 18th, 2021
---
title: Audio and Video on an ATMega88 Microcontroller
date: 2009-05-19 20:49:40
tags: microcontroller
---

# Audio and Video on an ATMega88 Microcontroller!

__At least 99% of my blog entries contain original content I created.__  However, I was so impressed by something I stumbled upon tonight that I absolutely cannot resist sharing it.  It's a project which aims to output audio and video simultaneously from a single microcontroller.  The [author's site](http://www.linusakesson.net/scene/craft/index.php) has all the details, but if you watch the video below you'll be amazed.  (Just get through the 30 second narration at the beginning)  Apparently the guy rendered video in horizontal lines from the software and outputted audio signals between horizontal lines!  Amazing!

![](https://www.youtube.com/embed/sNCqrylNY-0)

__On second thought,__ this was no big deal in the 80's, so why am I so impressed by it now?  The 8-bit microcontrollers this guy is programming is likely on par with the PCs of the 80's.  I guess that in the 70's this would have been amazing because it was cutting edge.  In the 80's this would have been boring because it was commonplace.  In the 2000's, this is amazing because no one in my generation is old enough to remember how amazing this was in the 70's and 80's!

__After researching some similar projects__ I realized I'm becoming fascinated with [chiptune](http://en.wikipedia.org/wiki/Chiptune) synthesizing code, hardware, and music.  It's basically a type of code to tell a synthesizer how to synthesize the music, rather than having it play pre-recorded music.  It's like a merge of a programming language and MIDI.  For non-technical people, it's like giving a microchip the sheet music for all the different instruments of an orchestra and having the microchip play everything from the sheet music, rather than giving it a CD to play.  Here's some video of a PC-based front-end to the audio creation process. Notice how each line represents a time, and how certain codes in certain channels represent notes (hence the MIDI aspect).  Numbers on the far right represent the location of the memory, and notice also how it goes back and forth, replaying certain areas when necessary (to safe space, hence the coding aspect).

![](https://www.youtube.com/embed/17FmzHk_v0g)

__I can't describe my emotional state right now. It's like I have an extreme nostalgia for an era I never even lived in.__  This chip level audio synthesis stuff sounds amazingly fun to me.  (I've already bookmarked [nolife-radio.com](http://nolife-radio.com/) and [8bitFM.com](http://www.8bitfm.com/)) I feel drawn toward it... but I'm scared to get sucked in. I wish I were a college student in the 80's studying electrical engineering. Here I am, having just gotten a master's in molecular biology and microbiology, and I feel like I wasted six years of my life in the process. I'm about to start dental school in August. Hopefully when I look back from the future I won't feel like I wasted another four years doing _that_.

__Either way, I have an endless supply__ of possible projects to do this summer (not even going into the small list of projects I'm trying/expected to complete in the cardioneurophysiology lab I work in).  I feel like I'm running out of time.  I start dental school in August, and I dread the date.  Not necessarily because I expect it to be difficult, but because I feel [however illogical, irrational, or ridiculous] that I'm actually doing something _significant_, working with my hands and working with my brain to do things that [almost] no one has done before, and doing things in a way that no one has ever thought of doing them.  I feel like when I start resume classes, it's another four years of people telling me how I should do things so I can be exactly like everyone else.  How can you exercise creativity as a dental student? I'm sure there are ways, but it's certainly leagues away from the projects engineering college students work on. As far as the career goes, if you're an engineer the best case scenario is that you do something no one has ever done before.  If you're a dentist, the best case scenario is to do things exactly as everyone else does them.  Maybe I'll go crazy and change the wallpaper in my office every few months.

ATTiny2313 Controlling a HD44780 LCD with AVR-GCC

After a day of tinkering I finally figured out how to control a HD44780 display from an ATTiny2313 microcontroller. There are a lot of websites out there claiming to show you how to do this on similar AVRs. I tried about 10 of them and, intriguingly, only one of them worked! I think the problem is that many of those websites show code for an ATMega8 and I'm using an ATTiny2313. Since it took me so long to get this right I decided to share it on the internet for anyone else having a similar struggle.

You might recognize this LCD panel from some PC parallel port / LCD interface projects I worked on about 5 years ago. It's a 20-column, 2-row, 8-bit parallel character LCD. This means that rather than telling each little square to light up to form individual letters, you can just output text to the microcontroller embedded in the display and it can draw the letters, move the cursor, or clear the screen. These are the connections I made were:

  • LCD1 -> GND
  • LCD2 -> +5V
  • LCD3 (contrast) -> GND
  • LCD4 (RS) -> AVR D0 (pin2)
  • LCD5 (R/W) -> AVR D1 (pin3)
  • LCD6 (ES) -> AVR D2 (pin6)
  • LCD 11-14 (data) -> AVR B0-B3 (pins 12-15)

The code to control this LCD from the ATTiny2313 was found on Martin Thomas' page (dead link removed in 2019). I included the .h and .c files and successfully ran the following program on my AVR. I used the internal RC clock.

// ATTiny2313 / HD44780 LCD INTERFACE
#include <stdlib.h>;
#include <avr/io.h>;
#include <util/delay.h>;
#include "lcd.h"
#include "lcd.c"

int main(void)
{
    int i=0;
    lcd_init(LCD_DISP_ON);
    lcd_clrscr();
    lcd_puts("ATTiny 2313 LCD Demo");
    lcd_puts("  www.SWHarden.com  ");
    _delay_ms(1000);
    lcd_clrscr();
    for (;;) {
        lcd_putc(i);
        i++;
        _delay_ms(50);
    }
}
// modified the top of "lcd.h"
#define LCD_PORT         PORTB        /**&lt; port for the LCD lines   */
#define LCD_DATA0_PORT   LCD_PORT     /**&lt; port for 4bit data bit 0 */
#define LCD_DATA1_PORT   LCD_PORT     /**&lt; port for 4bit data bit 1 */
#define LCD_DATA2_PORT   LCD_PORT     /**&lt; port for 4bit data bit 2 */
#define LCD_DATA3_PORT   LCD_PORT     /**&lt; port for 4bit data bit 3 */
#define LCD_DATA0_PIN    0            /**&lt; pin for 4bit data bit 0  */
#define LCD_DATA1_PIN    1            /**&lt; pin for 4bit data bit 1  */
#define LCD_DATA2_PIN    2            /**&lt; pin for 4bit data bit 2  */
#define LCD_DATA3_PIN    3            /**&lt; pin for 4bit data bit 3  */
#define LCD_RS_PORT      PORTD     /**&lt; port for RS line         */
#define LCD_RS_PIN       0            /**&lt; pin  for RS line         */
#define LCD_RW_PORT      PORTD     /**&lt; port for RW line         */
#define LCD_RW_PIN       1            /**&lt; pin  for RW line         */
#define LCD_E_PORT       PORTD     /**&lt; port for Enable line     */
#define LCD_E_PIN        2            /**&lt; pin  for Enable line     */

// AND A LITTLE LOWER, I CHANGED THIS LINE TO 4-BIT MODE
#define LCD_FUNCTION_8BIT     0      /*   DB4: set 8BIT mode (0-&gt;4BIT mode) */

Here is video of the output. Notice how this display can show English (lowercase/uppercase/numbers) as well as the Japanese character set!

Note from Future Scott (ten years later, August, 2019):

The link to the downloadable source code from Martin Thomas' page is no longer functional. These links do work:

A more recent project which uses these displays is:

Markdown source code last modified on January 18th, 2021
---
title: ATTiny2313 Controlling a HD44780 LCD with AVR-GCC
date: 2009-05-17 20:27:44
---

# ATTiny2313 Controlling a HD44780 LCD with AVR-GCC

__After a day of tinkering I finally figured out how to control a HD44780 display from an ATTiny2313 microcontroller.__ There are a _lot_ of websites out there claiming to show you how to do this on similar AVRs. I tried about 10 of them and, intriguingly, only one of them worked! I think the problem is that many of those websites show code for an ATMega8 and I'm using an ATTiny2313. Since it took me so long to get this right I decided to share it on the internet for anyone else having a similar struggle.

<div class="text-center img-border">

[![](attiny_2313_lcd_hd44780_thumb.jpg)](attiny_2313_lcd_hd44780.jpg)
[![](attiny_2313_lcd_hd44780_2_thumb.jpg)](attiny_2313_lcd_hd44780_2.jpg)

</div>

__You might recognize this LCD panel__ from some [PC parallel port / LCD interface projects](http://www.swharden.com/blog/old-stuff-of-interest/#lcd) I worked on about 5 years ago. It's a 20-column, 2-row, 8-bit parallel character LCD. This means that rather than telling each little square to light up to form individual letters, you can just output text to the microcontroller embedded in the display and it can draw the letters, move the cursor, or clear the screen. These are the connections I made were:

*   LCD1 -&gt; GND
*   LCD2 -&gt; +5V
*   LCD3 (contrast) -&gt; GND
*   LCD4 (RS) -&gt; AVR D0 (pin2)
*   LCD5 (R/W) -&gt; AVR D1 (pin3)
*   LCD6 (ES) -&gt; AVR D2 (pin6)
*   LCD 11-14 (data) -&gt; AVR B0-B3 (pins 12-15)

__The code to control this LCD from the ATTiny2313__ was found on Martin Thomas' page (dead link removed in 2019). I included the .h and .c files and successfully ran the following program on my AVR. I used the internal RC clock.

```c
// ATTiny2313 / HD44780 LCD INTERFACE
#include <stdlib.h>;
#include <avr/io.h>;
#include <util/delay.h>;
#include "lcd.h"
#include "lcd.c"

int main(void)
{
    int i=0;
    lcd_init(LCD_DISP_ON);
    lcd_clrscr();
    lcd_puts("ATTiny 2313 LCD Demo");
    lcd_puts("  www.SWHarden.com  ");
    _delay_ms(1000);
    lcd_clrscr();
    for (;;) {
        lcd_putc(i);
        i++;
        _delay_ms(50);
    }
}
```

```c
// modified the top of "lcd.h"
#define LCD_PORT         PORTB        /**&lt; port for the LCD lines   */
#define LCD_DATA0_PORT   LCD_PORT     /**&lt; port for 4bit data bit 0 */
#define LCD_DATA1_PORT   LCD_PORT     /**&lt; port for 4bit data bit 1 */
#define LCD_DATA2_PORT   LCD_PORT     /**&lt; port for 4bit data bit 2 */
#define LCD_DATA3_PORT   LCD_PORT     /**&lt; port for 4bit data bit 3 */
#define LCD_DATA0_PIN    0            /**&lt; pin for 4bit data bit 0  */
#define LCD_DATA1_PIN    1            /**&lt; pin for 4bit data bit 1  */
#define LCD_DATA2_PIN    2            /**&lt; pin for 4bit data bit 2  */
#define LCD_DATA3_PIN    3            /**&lt; pin for 4bit data bit 3  */
#define LCD_RS_PORT      PORTD     /**&lt; port for RS line         */
#define LCD_RS_PIN       0            /**&lt; pin  for RS line         */
#define LCD_RW_PORT      PORTD     /**&lt; port for RW line         */
#define LCD_RW_PIN       1            /**&lt; pin  for RW line         */
#define LCD_E_PORT       PORTD     /**&lt; port for Enable line     */
#define LCD_E_PIN        2            /**&lt; pin  for Enable line     */

// AND A LITTLE LOWER, I CHANGED THIS LINE TO 4-BIT MODE
#define LCD_FUNCTION_8BIT     0      /*   DB4: set 8BIT mode (0-&gt;4BIT mode) */
```

__Here is video of the output.__ Notice how this display can show English (lowercase/uppercase/numbers) as well as the Japanese character set!

![](https://www.youtube.com/embed/mMEwFSkr1Ko)

<blockquote>

**Note from Future Scott (ten years later, August, 2019):**

The link to the downloadable source code from Martin Thomas' page is no longer functional. These links do work:

* https://senzor.robotika.sk/sensorwiki/index.php/AVR_lcd.c 

* https://senzor.robotika.sk/sensorwiki/index.php/AVR_lcd.h 

A more recent project which uses these displays is:

* https://www.swharden.com/wp/2017-04-29-precision-pressure-meter-project/ 

</blockquote>

Removing Textile Markup From Wordpress Entries

I realized that the C code from yesterday wasn't showing-up properly because of textile, a rapid, inline, tag-based formatting system. While it's fun and convenient to use, it's not always practical. The problem I was having was that in C code, variable names (such as delay) were becoming irrevocably italicized, and nothing I did could prevent textile from ignoring code while styling text. The kicker is that I couldn't disable it easily, because I've been writing in this style for over four years! I decided that the time was now to put my mad Python skills to the test and write code to handle the conversion from textile-format to raw HTML. I accomplished this feat in a number of steps. Yeah, I could have done hours of research to find a "faster way", but it simply wouldn't have been as creative. In a nutshell, I backed-up the SQL database using PHPMyAdmin to a single "x.sql" file. I then wrote a pythons script to parse this [massive] file and output "o.sql", the same data but with all of the textile tags I commonly used replaced by their HTML equivalent. It's not 100% perfect, but it's 99.999% perfect. I'll accept that. The output? You're viewing it! Here's the code I used to do it:

## This Python script removes *SOME* textile formatting from Wordpress
## backups in plain text SQL format (dumped from PHP MyAdmin). Specifically,
## it corrects bold and itallic fonts and corrects links. It should be easy
## to expand if you need to do something else with it.

infile = 'x.sql'

replacements=   ["r"," "],["n"," n "],["*:","* :"],["_:","_ :"],
                ["n","&lt;br&gt;n"],["&gt;*","&gt; *"],["*&lt; ","* &lt;"],
                ["&gt;_","&gt; _"],["_&lt; ","_ &lt;"],
                [" *"," &lt;b&gt;"],["* "," "],[" _"," &lt;i&gt;"],["_ ","&lt;/i&gt; "]
                #These are the easy replacements

def fixLinks(line):
    ## replace ["links":URL] with [&lt;a href="URL"&gt;links&lt;/a&gt;]. ##
    words = line.split(" ")
    for i in range(len(words)):
        word = words[i]
        if '":' in word:
            upto=1
            while (word.count('"')&amp;lt;2):
                word = words[i-upto]+" "+word
                upto+=1
            word_orig = word
            extra=""
            word = word.split('":')
            word[0]=word[0][1:]
            for char in ".),'":
                if word[1][-1]==char: extra=char
            if len(extra)&gt;0: word[1]=word[1][:-1]
            word_new='&lt;a href="%s"&gt;%s&lt;/a&gt;'%(word[1],word[0])+extra
            line=line.replace(word_orig,word_new)
    return line

def stripTextile(orig):
    ## Handle the replacements and link fixing for each line. ##
    if not orig.count("', '") == 13: return orig #non-normal post
    line=orig
    temp = line.split
    line = line.split("', '",5)[2]
    if len(line)&amp;lt;10:return orig #non-normal post
    origline = line
    line = " "+line
    for replacement in replacements:
        line = line.replace(replacement[0],replacement[1])
    line=fixLinks(line)
    line = orig.replace(origline,line)
    return line

f=open(infile)
raw=f.readlines()
f.close
posts=0
for raw_i in range(len(raw)):
    if raw[raw_i][:11]=="INSERT INTO":
        if "wp_posts" in raw[raw_i]: #if it's a post, handle it!
            posts+=1
            print "on post",posts
            raw[raw_i]=stripTextile(raw[raw_i])

print "WRITING..."
out = ""
for line in raw:
    out+=line
f=open('o.sql','w')
f.write(out)
f.close()

I certainly held my breath while the thing ran. As I previously mentioned, this thing modified SQL tables. Therefore, when I uploaded the "corrected" versions, I kept breaking the site until I got all the bugs worked out. Here's an image from earlier today when my site was totally dead (0 blog posts)

Markdown source code last modified on January 18th, 2021
---
title: Removing Textile Markup From Wordpress Entries
date: 2009-05-15 17:56:32
tags: old
---

# Removing Textile Markup From Wordpress Entries

__I realized that the C code from yesterday wasn't showing-up properly__ because of [textile](http://wordpress.org/tags/textile), a rapid, inline, tag-based formatting system.  While it's fun and convenient to use, it's not always practical.  The problem I was having was that in C code, variable names (such as _delay_) were becoming irrevocably italicized, and nothing I did could prevent textile from ignoring code while styling text.  The kicker is that I couldn't disable it easily, because I've been writing in this style for __over four years!__  I decided that the time was now to put my mad Python skills to the test and write code to handle the conversion from textile-format to raw HTML.
__I accomplished this feat__ in a number of steps.  Yeah, I could have done hours of research to find a "faster way", but it simply wouldn't have been as creative.  In a nutshell, I backed-up the SQL database using [PHPMyAdmin](http://en.wikipedia.org/wiki/PhpMyAdmin) to a single "x.sql" file.  I then wrote a pythons script to parse this [massive] file and output "o.sql", the same data but with all of the textile tags I commonly used replaced by their HTML equivalent.  It's not 100% perfect, but it's 99.999% perfect.  I'll accept that.  The output?  You're viewing it!  Here's the code I used to do it:

```python
## This Python script removes *SOME* textile formatting from Wordpress
## backups in plain text SQL format (dumped from PHP MyAdmin). Specifically,
## it corrects bold and itallic fonts and corrects links. It should be easy
## to expand if you need to do something else with it.

infile = 'x.sql'

replacements=   ["r"," "],["n"," n "],["*:","* :"],["_:","_ :"],
                ["n","&lt;br&gt;n"],["&gt;*","&gt; *"],["*&lt; ","* &lt;"],
                ["&gt;_","&gt; _"],["_&lt; ","_ &lt;"],
                [" *"," &lt;b&gt;"],["* "," "],[" _"," &lt;i&gt;"],["_ ","&lt;/i&gt; "]
                #These are the easy replacements

def fixLinks(line):
    ## replace ["links":URL] with [&lt;a href="URL"&gt;links&lt;/a&gt;]. ##
    words = line.split(" ")
    for i in range(len(words)):
        word = words[i]
        if '":' in word:
            upto=1
            while (word.count('"')&amp;lt;2):
                word = words[i-upto]+" "+word
                upto+=1
            word_orig = word
            extra=""
            word = word.split('":')
            word[0]=word[0][1:]
            for char in ".),'":
                if word[1][-1]==char: extra=char
            if len(extra)&gt;0: word[1]=word[1][:-1]
            word_new='&lt;a href="%s"&gt;%s&lt;/a&gt;'%(word[1],word[0])+extra
            line=line.replace(word_orig,word_new)
    return line

def stripTextile(orig):
    ## Handle the replacements and link fixing for each line. ##
    if not orig.count("', '") == 13: return orig #non-normal post
    line=orig
    temp = line.split
    line = line.split("', '",5)[2]
    if len(line)&amp;lt;10:return orig #non-normal post
    origline = line
    line = " "+line
    for replacement in replacements:
        line = line.replace(replacement[0],replacement[1])
    line=fixLinks(line)
    line = orig.replace(origline,line)
    return line

f=open(infile)
raw=f.readlines()
f.close
posts=0
for raw_i in range(len(raw)):
    if raw[raw_i][:11]=="INSERT INTO":
        if "wp_posts" in raw[raw_i]: #if it's a post, handle it!
            posts+=1
            print "on post",posts
            raw[raw_i]=stripTextile(raw[raw_i])

print "WRITING..."
out = ""
for line in raw:
    out+=line
f=open('o.sql','w')
f.write(out)
f.close()

```

__I certainly held my breath while the thing ran.__  As I previously mentioned, this thing modified SQL tables.  Therefore, when I uploaded the "corrected" versions, I kept breaking the site until  I got all the bugs worked out.  Here's an image from earlier today when my site was totally dead (0 blog posts)

<div class="text-center img-border">

[![](hostingwork_thumb.jpg)](hostingwork.jpg)

</div>

Simple Case AVR/PC Serial Communication via MAX232

I recently had the desire to be able to see data from an ATMEL AVR microcontroller (the ATTiny2313) for development and debugging purposes. I wanted an easy way to have my microcontroller talk to my PC (and vise versa) with a minimum number of parts. The easiest way to do this was to utilize the UART capabilities of the ATTiny2313 to talk to my PC through the serial port. One problem is that the ATTiny2313(as with most microcontrollers) puts out 5V for "high" (on) and 0V for "low" (off). The RS-232 standard (which PC serial ports use) required -15V for high and +15v for low! Obviously the microcontroller needs some help to achieve this. The easiest way was to use the MAX232 serial level converter which costs about 3 bucks at DigiKey. Note that it requires a few 10uF capacitors to function properly.

Here's a more general schematic:

I connected my ATTiny2313 to the MAX232 in a very standard way. (photo) MAX232 pins 13 and 14 go to the serial port, and the ATTiny2313 pins 2 and 3 go to the MAX232 pins 12 and 11 respectively. I will note that they used a oscillator value (3.6864MHz) different than mine (9.216MHz).

Determining the speed of serial communication is important. This is dependent on your oscillator frequency! I said I used a 9.216Mhz oscillator. First, a crystal or ceramic oscillator is required over the internal RC oscillator because the internal RC oscillator is not accurate enough for serial communication. The oscillator you select should be a perfect multiple of 1.8432MHz. Mine is 5x this value. Many people use 2x this value (3.6864Mhz) and that's okay! You just have to make sure your microchip knows (1) to use the external oscillator (google around for how to burn the fuses on your chip to do this) and (2) what the frequency of your oscillator is and how fast it should be sending data. This is done by setting the UBRRL value. The formula to do this is here:

The datasheet of your microcontroller may list a lot of common crystal frequencies, bandwidths, and their appropriate UBRR values. However my datasheet lacked an entry for a 9.216MHz crystal, so I had to do the math myself. I Googled around and no "table" is available! Why not make one? (picture, below). Anyway, for my case I determined that if I set the UBRR value to 239, I could transmit data at 2800 baud (bits/second). This is slow enough to ensure accuracy, but fast enough to quickly dump a large amount of text to a PC terminal.

AVR Baud Calculator

This will make your life easier. The page wormfood.net/avrbaudcalc.php has a chart of common crystals and the baud rates they work best with! Try to pick a combination that provides the least error possible...

This is the bare-minimum code to test out my setup. Just load the code (written in C, compiled with avr-gcc) onto your chip and it's ready to go. Be sure you set your fuses to use an external oscillator and that you set your UBRRL value correctly.

#include <avr/io.h>  
#include <avr/interrupt.h>  
#include <util/delay.h>  

int main (void)  
{  
  unsigned char data=0;  
  UBRRL = 239;  
  UCSRB = (1 < < RXEN) | (1 << TXEN);  
  UCSRC = (1 < < UCSZ1) | (1 << UCSZ0);  

  for (;;)  
  {  
    if (data>'Z'||data< 'A')  
    {  
      UDR = 10; UDR = 13; data='A';_delay_ms(100);  
    }  

    UDR = data;  
    data += 1;  
    _delay_ms(100);  
  }  
}  

Once you load it, it's ready to roll! It continuously dumps letters to the serial port. To receive them, open HyperTerminal (on windows, under accessories) or minicom (on Linux, look it up!). Set your baud rate to 2800 (or whatever you selected) and you're in business. This (picture below) is the output of the microcontroller to HyperTerminal on my PC. Forgive the image quality, I photographed the LCD screen instead of taking a screenshot.

This is the circuit which generates the output of the previous image. I have a few extra components. I have an LED which I used for debugging purposes, and also a switch (labeled "R"). The switch (when pressed) grounds pin 1 of the ATTiny2313 which resets it. If I want to program the chip, I hold "R" down and the PC can program it with the inline programmer "parallel port, straight-through, DAPA style). One cable going into the circuit is for the parallel port programmer, one cable is for the serial port (data transfer), and one is for power (5v which I stole from a USB port).

I hope you found this information useful. Feel free to contact me with any questions you may have, but realize that I'm no expert, and I'm merely documenting my successes chronologically on this website.

Markdown source code last modified on January 18th, 2021
---
title: Simple Case AVR/PC Serial Communication via MAX232
date: 2009-05-14 11:00:19
tags: microcontroller, circuit, old
---

# Simple Case AVR/PC Serial Communication via MAX232

__I recently had the desire__ to be able to see data from an ATMEL AVR microcontroller (the [ATTiny2313](http://www.SWHarden.com/blog/images/attiny-2313.gif)) for development and debugging purposes.  I wanted an easy way to have my microcontroller talk to my PC (and vise versa) with a minimum number of parts.  The easiest way to do this was to utilize the [UART](http://en.wikipedia.org/wiki/UART) capabilities of the [ATTiny2313](http://www.SWHarden.com/blog/images/attiny-2313.gif) to talk to my PC through the serial port. One problem is that the [ATTiny2313](http://www.SWHarden.com/blog/images/attiny-2313.gif)(as with most microcontrollers) puts out 5V for "high" (on) and 0V for "low" (off).  The RS-232 standard (which PC serial ports use) required -15V for high and +15v for low!  Obviously the microcontroller needs some help to achieve this.  The easiest way was to use the [MAX232 serial level converter](http://en.wikipedia.org/wiki/MAX232) which [costs about 3 bucks at DigiKey](http://search.digikey.com/scripts/DkSearch/dksus.dll?Detail&amp;name=MAX232CPE%2B-ND). Note that it requires a few 10uF capacitors to function properly.

<div class="text-center img-border">

[![](serialcircuit_thumb.jpg)](serialcircuit.png)

</div>

__Here's a more general schematic:__


<div class="text-center">

![](max232_serial_microcontroller.gif)

</div>

__I connected my ATTiny2313 to the MAX232__ in a very standard way. (photo)   MAX232 pins 13 and 14 go to the serial port, and the ATTiny2313 pins 2 and 3 go to the MAX232 pins 12 and 11 respectively.  I will note that they used a oscillator value (3.6864MHz) different than mine (9.216MHz).

__Determining the speed of serial communication__ is important.  This is dependent on your oscillator frequency!  I said I used a 9.216Mhz oscillator.  First, a crystal or ceramic oscillator is required over the internal RC oscillator because the internal RC oscillator is not accurate enough for serial communication.  The oscillator you select should be a perfect multiple of 1.8432MHz. Mine is 5x this value.  Many people use 2x this value (3.6864Mhz) and that's okay!  You just have to make sure your microchip knows (1) to use the external oscillator (google around for how to burn the fuses on your chip to do this) and (2) what the frequency of your oscillator is and how fast it should be sending data.  This is done by setting the UBRRL value.  The formula to do this is here:

<div class="text-center">

![](ubrrformula.gif)

</div>

__The datasheet of your microcontroller__ may list a lot of common crystal frequencies, bandwidths, and their appropriate UBRR values.  However my datasheet lacked an entry for a 9.216MHz crystal, so I had to do the math myself.  I Googled around and no "table" is available!  Why not make one? (picture, below).  Anyway, for my case I determined that if I set the UBRR value to 239, I could transmit data at 2800 baud (bits/second).  This is slow enough to ensure accuracy, but fast enough to quickly dump a large amount of text to a PC terminal.

<div class="text-center">

![](ubrr-table.gif)

</div>

## AVR Baud Calculator

This will make your life easier. The page <a href="http://www.wormfood.net/avrbaudcalc.php">wormfood.net/avrbaudcalc.php</a> has a chart of common crystals and the baud rates they work best with! Try to pick a combination that provides the least error possible...

__This is the bare-minimum code__ to test out my setup. Just load the code (written in C, compiled with avr-gcc) onto your chip and it's ready to go.  Be sure you set your fuses to use an external oscillator and that you set your UBRRL value correctly.

```c
#include <avr/io.h>  
#include <avr/interrupt.h>  
#include <util/delay.h>  

int main (void)  
{  
  unsigned char data=0;  
  UBRRL = 239;  
  UCSRB = (1 < < RXEN) | (1 << TXEN);  
  UCSRC = (1 < < UCSZ1) | (1 << UCSZ0);  

  for (;;)  
  {  
    if (data>'Z'||data< 'A')  
    {  
      UDR = 10; UDR = 13; data='A';_delay_ms(100);  
    }  
    
    UDR = data;  
    data += 1;  
    _delay_ms(100);  
  }  
}  
```

__Once you load it, it's ready to roll!__  It continuously dumps letters to the serial port.  To receive them, open HyperTerminal (on windows, under accessories) or minicom (on Linux, look it up!).  Set your baud rate to 2800 (or whatever you selected) and you're in business.  This (picture below) is the output of the microcontroller to HyperTerminal on my PC.  Forgive the image quality, I photographed the LCD screen instead of taking a screenshot.

<div class="text-center img-border">

[![](avr_serial_console_thumb.jpg)](avr_serial_console.jpg)

</div>

__This is the circuit__ which generates the output of the previous image.  I have a few extra components.  I have an LED which I used for debugging purposes, and also a switch (labeled "R").  The switch (when pressed) grounds pin 1 of the ATTiny2313 which resets it.  If I want to program the chip, I hold "R" down and the PC can program it with the inline programmer ["parallel port, straight-through, DAPA style](https://wikis.mit.edu/confluence/download/attachments/20512/dapa.png)).  One cable going into the circuit is for the parallel port programmer, one cable is for the serial port (data transfer), and one is for power (5v which I stole from a USB port).

<div class="text-center img-border">

[![](avr_max232_thumb.jpg)](avr_max232.jpg)

</div>

__I hope you found this__ information useful.  Feel free to [contact me](http://www.swharden.com/blog/send-scott-a-message/) with any questions you may have, but realize that I'm no expert, and I'm merely documenting my successes chronologically on this website.
Pages