Managing Credentials with Python

I enjoy contributing to open-source, but I’d prefer to keep my passwords to myself! Python is a great glue language for automating tasks, and recently I’ve been using it to log in to my web server using SFTP and automate log analysis, file management, and software updates. The Python scripts I’m working on need to know my login information, but I want to commit them to source control and share them on GitHub so I have to be careful to use a strategy which minimizes risk of inadvertently leaking these secrets onto the internet.

This post explores various options for managing credentials in Python scripts in public repositories. There are many different ways to manage credentials with Python, and I was surprised to learn of some new ones as I was researching this topic. This post reviews the most common options, starting with the most insecure and working its way up to the most highly regarded methods for managing secrets.

Plain-Text Passwords in Code

⚠️☠️ DANGER: Never do this

You could put a password or API key directly in your python script, but even if you intend to remove it later there’s always a chance you’ll accidentally commit it to source control without realizing it, posing a security risk forever. This method is to be avoided at all costs!

username = "myUsername"
password = "S3CR3T_P455W0RD"
logIn(username, password)

Obfuscated Passwords in Code

⚠️☠️ DANGER: Never do this

A slightly less terrible idea is to obfuscate plain-text passwords by storing them as base 64 strings. You won’t know the password just by seeing it, but anyone who has the string can easily decode it. Websites like https://www.base64decode.org are useful for this.

"""Demonstrate conversion to/from base 64"""

import base64

def obfuscate(plainText):
    plainBytes = plainText.encode('ascii')
    encodedBytes = base64.b64encode(plainBytes)
    encodedText = encodedBytes.decode('ascii')
    return encodedText


def deobfuscate(obfuscatedText):
    obfuscatedBytes = obfuscatedText.encode('ascii')
    decodedBytes = base64.b64decode(obfuscatedBytes)
    decodedText = decodedBytes.decode('ascii')
    return decodedText

original = "S3CR3T_P455W0RD"
obfuscated = obfuscate(original)
deobfuscated = deobfuscate(obfuscated)

print("original: " + original)
print("obfuscated: " + obfuscated)
print("deobfuscated: " + deobfuscated)

original: S3CR3T_P455W0RD
obfuscated: UzNDUjNUX1A0NTVXMFJE
deobfuscated: S3CR3T_P455W0RD

Passwords in Plain Text Files

⚠️ WARNING: This method is prone to mistakes. Ensure the text file is never committed to source control.

You could store username/password on the first two lines of a plain text file, then use python to read it when you need it.

with open("secrets.txt") as f:
    lines = f.readlines()
    username = lines[0].strip()
    password = lines[1].strip()
    print(f"USERNAME={username}, PASSWORD={password}")

If the text file is in the repository directory you should modify .gitignore to ensure it’s not tracked by source source control. There is a risk that you may forget to do this, exposing your credentials online! A better idea may be to place the secrets file outside your repository folder entirely.

💡 There are libraries which make this easier. One example is Python Decouple which implements a lot of this logic gracefully and can even combine settings from multiple files (e.g., .ini vs .env files) for environments that can benefit from more advanced configuration options. See the notes below about helper libraries that environment variables and .env files

Passwords in Python Modules

⚠️ WARNING: This method is prone to mistakes. Ensure the secrets module is never committed to source control.

Similar to a plain text file not tracked by source control (ideally outside the repository folder entirely), you could store passwords as variables in a Python module then import it.

from mySecrets import username, password
print(f"USERNAME={username}, PASSWORD={password}")

If your secrets file is in an obscure folder, you will have to add it to your path so the module can be found when importing.

import sys
sys.path.append("C:/path/to/secrets/folder")

from mySecrets import username, password
print(f"USERNAME={username}, PASSWORD={password}")

Don’t name your module secrets because the secrets module is part of the standard library and that will likely be imported in stead.

Passwords as Program Arguments

⚠️ WARNING: This method may store plain text passwords in your command history.

This isn’t a great idea because passwords are seen in plain text in the console and also may be stored in the command history. However, you’re unlikely to accidentally commit passwords to source control.

import sys
username = sys.argv[1]
password = sys.argv[2]
print(f"USERNAME={username}, PASSWORD={password}")

python test.py myUsername S3CR3T_P455W0RD

Type Passwords in the Console

You could request the user to type their password in the console, but the characters would be visible as they’re typed.

# ⚠️ This code displays the typed password
password = input("Password: ")

Python has a getpass module in its standard library made for prompting the user for passwords as console input. Unlike input(), characters are not visible as the password is typed.

# 👍 This code hides the typed password
import getpass
password = getpass.getpass('Password: ')

Extract Passwords from the Clipboard

This is an interesting method. It’s fast and simple, but a bit quirky. Downsides are (1) it requires the password to be in the clipboard which may expose it to other programs, (2) it requires installation of a nonstandard library, and (3) it won’t work easily in server environments.

Note that I trust pyperclip more than clipboard (which is just another developer wrapping pyperclip)

pip install pyperclip

Run after copying a password to the clipboard:

import pyperclip
password = pyperclip.paste()

Request Credentials with Tk

The Tk graphics library is a cross-platform graphical widget toolkit that comes with Python. A login window that collects username and password can be created programmatically and wrapped in a function for easily inclusion in scripts that otherwise don’t have a GUI.

I find this technique particularly useful when the username and password are stored in a password manager.

def getCredentials(defaultUser):
    """Request login credentials using a GUI."""
    import tkinter
    root = tkinter.Tk()
    root.eval('tk::PlaceWindow . center')
    root.title('Login')
    uv = tkinter.StringVar(root, value=defaultUser)
    pv = tkinter.StringVar(root, value='')
    userEntry = tkinter.Entry(root, bd=3, width=35, textvariable=uv)
    passEntry = tkinter.Entry(root, bd=3, width=35, show="*", textvariable=pv)
    btnClose = tkinter.Button(root, text="OK", command=root.destroy)
    userEntry.pack(padx=10, pady=5)
    passEntry.pack(padx=10, pady=5)
    btnClose.pack(padx=10, pady=5, side=tkinter.TOP, anchor=tkinter.NE)
    root.mainloop()
    return [uv.get(), pv.get()]

username, password = getCredentials("user@site.com")

Manage Passwords with a Keyring

The keyring package provides an easy way to access the system’s keyring service from python. On MacOS it uses Keychain, on Windows it uses the Windows Credential Locker, and on Linux it can use KDE’s KWallet or GNOME’s Secret Service.

Downsides of keyrings are (1) it requires a nonstandard library, (2) implementation may be OS-specific, (3) it may not function easily in cloud environments.

pip install keyring

# store the password once
import keyring
keyring.set_password("system", "myUsername", "S3CR3T_P455W0RD")

# recall the password at any time
import keyring
password = keyring.get_password("system", "myUsername")

Passwords in Environment Variables

Environment variables are one of the better ways of managing credentials with Python. There are many articles on this topic, including Twilio’s How To Set Environment Variables and Working with Environment Variables in Python. Environment variables are one of the preferred methods of credential management when working with cloud providers.

Be sure to restart your console session after editing environment variables before attempting to read them from within python.

import os
password = os.getenv('demoPassword')

There are many helper libraries such as python-dotenv and Python Decouple which can use local .env files to dynamically set environment variables as your program runs. As noted in previous sections, when storing passwords in plain-text in the file structure of your repository be extremely careful not to commit these files to source control!

Example .env file:

demoPassword2=superSecret

The dotenv package can load .env variables as environment variables when a Python script runs:

import dotenv
dotenv.load_dotenv()
password2 = os.getenv('demoPassword2')
print(password2)

Additional Resources

• Using .env Files for Environment Variables in Python Applications
• Environment Variables vs. Secrets In Python
• How To Set Environment Variables
• Working with Environment Variables in Python
• How to Set and Get Environment Variables in Python
• python-dotenv reads key-value pairs from a .env file and can set them as environment variables. It helps in the development of applications following the 12-factor principles.
• Python Decouple helps you to organize your settings so that you can change parameters without having to redeploy your app.

How do you manage credentials in Python? If you wish to share feedback or a creative method you use that I haven’t discussed above, send me an email and I can include your suggestions in this document.

WSPR Code Generator

WSPR (Weak Signal Propagation Reporter) is a protocol for weak-signal radio communication. Transmissions encode a station’s callsign, location (grid square), and transmitter power into a frequency-shift-keyed (FSK) signal that hops between 4 frequencies to send 162 tones in just under two minutes. Signals can be decoded with S/N as low as −34 dB! WSJT-X, the most commonly-used decoding software, reports decoded signals to wsprnet.org so propagation can be assessed all over the world.

WsprSharp is a WSPR encoding library implemented in C#. I created this library learn more about the WSPR protocol. I also made a web application (using Blazor WebAssembly) to make it easy to generate WSPR transmissions online: WSPR Code Generator

Try it online! https://swharden.com/software/wspr-code-generator/

WSPR Transmission Information

• One packet is 110.6 sec continuous wave
• Frequency shifts between 4 tones
• Each tone keys for 0.683 sec
• Tones are separated by 1.4648 Hz
• Total bandwidth is about 6 Hz
• 50 bits of information are packaged into a 162 bit message with FEC
• Transmissions always begin 1 sec after even minutes (UTC)

Resources

• WSPR Code Generator - encode WSPR messages from your browser
• WsprSharp - a .NET library for encoding and decoding WSPR transmissions using C#
• FSKview - spectrogram for viewing frequency-shift keyed (FSK) signals in real time
• Anatomy of a WSPR transmission
• WSPR on Wikipedia
• K1JT Program
• Grid square lookup
• WSJT on SourceForge
• WSPR Mode in WSJT-X

Reflection and XML Documentation in C#

In C#, you can document your code using XML directly before code blocks. This XML documentation is used by Visual Studio to display tooltips and provide autocomplete suggestions.

/// <summary>
///  This method performs an important function.
/// </summary>
public void MyMethod() {}

To enable automatic generation of XML documentation on every build, add the following to your csproj file:

<PropertyGroup>
  <DocumentationFile>MyProgram.xml</DocumentationFile>
</PropertyGroup>

However, XML documentation is not actually metadata, so it is not available in the compiled assembly. In this post I’ll show how you can use System.Reflection to gather information about methods and combine it with documentation from an XML file read with System.Xml.Linq.XDocument.

I find this useful for writing code which automatically generates documentation. Reflection alone cannot provide comments, and XML documentation alone cannot provide parameter names (just their types). By combining reflection with XML documentation, we can more completely describe methods in C# programs.

⚠️ WARNING: These code examples are intentionally simple. They only demonstrate how to read summary comments from XML documentation for methods found using reflection. Additional functionality can be added as needed, and my intent here is to provide a simple starting point rather than overwhelmingly complex examples that support all features and corner cases.

💡 Source code can be downloaded at the bottom of this article

✔️ The “hack” described on this page is aimed at overcoming limitations of partially-documented XML. A better solution is to fully document your code, in which case the XML document is self-sufficient. The primary goal of these efforts is to use XML documentation where it is present, but use Reflection to fill in the blanks about methods which are undocumented or only partially documented. Perhaps a better strategy would be to have a fully documented code base, with an XML file containing <summary>, <returns>, and <param> for every parameter. Tests can ensure this is and remains the case.

What does the XML Documentation look like?

There are some nuances here you might not expect, especially related to arrays, generics, and nullables. Let’s start with a demo class with documented summaries. Keep in mind that the goal of this project is to help use Reflection to fill in the blanks about undocumented or partially-documented code, so this example will only add a <summary> but no <param> descriptions.

DemoClass.cs

/// <summary>
/// Display a name
/// </summary>
public static void ShowName(string name)
{
    Console.WriteLine($"Hi {name}");
}

/// <summary>
/// Display a name a certain number of times
/// </summary>
public static void ShowName(string name, byte repeats)
{
    for (int i = 0; i < repeats; i++)
        Console.WriteLine($"Hi {name}");
}

/// <summary>
/// Display the type of the variable passed in
/// </summary>
public static void ShowGenericType<T>(T myVar)
{
    Console.WriteLine($"Generic type {myVar.GetType()}");
}

/// <summary>
/// Display the value of a nullable integer
/// </summary>
public static void ShowNullableInt(int? myInt)
{
    Console.WriteLine(myInt);
}

XML Documentation File

There are a few important points to notice here:

• Each method is a member with a name starting with M:
• Parameter types are in the member name, but not parameter names!
• Parameters might be listed in the XML, but they will be missing if only <summary> was added in code
• 💡 The key step required to connect a reflected method with its XML documentation is being able to determine the XML method name of that method. How to do this is discussed below…

<?xml version="1.0"?>
<doc>
    <assembly>
        <name>XmlDocDemo</name>
    </assembly>
    <members>
        <member name="M:XmlDocDemo.DemoClass.ShowName(System.String)">
            <summary>
            Display a name
            </summary>
        </member>
        <member name="M:XmlDocDemo.DemoClass.ShowName(System.String,System.Byte)">
            <summary>
            Display a name a certain number of times
            </summary>
        </member>
        <member name="M:XmlDocDemo.DemoClass.ShowGenericType``1(``0)">
            <summary>
            Display the type of the variable passed in
            </summary>
        </member>
        <member name="M:XmlDocDemo.DemoClass.ShowNullableInt(System.Nullable{System.Int32})">
            <summary>
            Display the value of a nullable integer
            </summary>
        </member>
    </members>
</doc>

XML Name Details

• Generics from types have a single ` character
• Generics from methods have double `` characters
• If the parameter is by “ref” then you need to pre-pend the @ character
• If the parameter is a pointer you need to pre-pend it with the * character
• If the parameter is an array, you need to add [] characters and the appropriate number of commas
• If the parameter is Nullable it will be wrapped in System.Nullable{}
• If the method is MethodInfo is a casing operator, then you need to pre-pend it with ~ character

Thanks Zachary Patten for sharing these details in an MSDN article and e-mail correspondence

Read XML Documentation File

This code reads the XML documentation file (using the modern XDocument) and stores method summaries in a Dictionary using the XML method name as a key. This dictionary will be accessed later to look-up documentation for methods found using Reflection.

private readonly Dictionary<string, string> MethodSummaries = new Dictionary<string, string>();

public XmlDoc(string xmlFile)
{
    XDocument doc = XDocument.Load(xmlFile);
    foreach (XElement element in doc.Element("doc").Element("members").Elements())
    {
        string xmlName = element.Attribute("name").Value;
        string xmlSummary = element.Element("summary").Value.Trim();
        MethodSummaries[xmlName] = xmlSummary;
    }
}

Determine XML Method Name for a Reflected Method

This example code returns the XML member name for a method found by reflection. This is the key step required to connect reflected methods with their descriptions in XML documentation files.

⚠️ Warning: This code sample may not support all corner-cases, but in practice I found it supports all of the ones I typically encounter in my code bases and it’s a pretty good balance between functionality and simplicity.

public static string GetXmlName(MethodInfo info)
{
	string declaringTypeName = info.DeclaringType.FullName;

	if (declaringTypeName is null)
		throw new NotImplementedException("inherited classes are not supported");

	string xmlName = "M:" + declaringTypeName + "." + info.Name;
	xmlName = string.Join("", xmlName.Split(']').Select(x => x.Split('[')[0]));
	xmlName = xmlName.Replace(",", "");

	if (info.IsGenericMethod)
		xmlName += "``#";

	int genericParameterCount = 0;
	List<string> paramNames = new List<string>();
	foreach (var parameter in info.GetParameters())
	{
		Type paramType = parameter.ParameterType;
		string paramName = GetXmlNameForMethodParameter(paramType);
		if (paramName.Contains("#"))
			paramName = paramName.Replace("#", (genericParameterCount++).ToString());
		paramNames.Add(paramName);
	}
	xmlName = xmlName.Replace("#", genericParameterCount.ToString());

	if (paramNames.Any())
		xmlName += "(" + string.Join(",", paramNames) + ")";

	return xmlName;
}

private static string GetXmlNameForMethodParameter(Type type)
{
	string xmlName = type.FullName ?? type.BaseType.FullName;
	bool isNullable = xmlName.StartsWith("System.Nullable");
	Type nullableType = isNullable ? type.GetGenericArguments()[0] : null;

	// special formatting for generics (also Func, Nullable, and ValueTulpe)
	if (type.IsGenericType)
	{
		var genericNames = type.GetGenericArguments().Select(x => GetXmlNameForMethodParameter(x));
		var typeName = type.FullName.Split('`')[0];
		xmlName = typeName + "{" + string.Join(",", genericNames) + "}";
	}

	// special case for generic nullables
	if (type.IsGenericType && isNullable && type.IsArray == false)
		xmlName = "System.Nullable{" + nullableType.FullName + "}";

	// special case for multidimensional arrays
	if (type.IsArray && (type.GetArrayRank() > 1))
	{
		string arrayName = type.FullName.Split('[')[0].Split('`')[0];
		if (isNullable)
			arrayName += "{" + nullableType.FullName + "}";
		string arrayContents = string.Join(",", Enumerable.Repeat("0:", type.GetArrayRank()));
		xmlName = arrayName + "[" + arrayContents + "]";
	}

	// special case for generic arrays
	if (type.IsArray && type.FullName is null)
		xmlName = "``#[]";

	// special case for value types
	if (xmlName.Contains("System.ValueType"))
		xmlName = "`#";

	return xmlName;
}

Get XML Documentation for a Reflected Method

Now that we have XmlName(), we can easily iterate through reflected methods and get their XML documentation.

// use Reflection to get info from custom methods
var infos = typeof(DemoClass).GetMethods()
                             .Where(x => x.DeclaringType.FullName != "System.Object")
                             .ToArray();

// display XML info about each reflected method
foreach (MethodInfo mi in infos)
{
    string xmlName = XmlName(mi);
    Console.WriteLine("");
    Console.WriteLine("Method: " + XmlDoc.MethodSignature(mi));
    Console.WriteLine("XML Name: " + xmlName);
    Console.WriteLine("XML Summary: " + MethodSummaries[xmlName]);
}

Output

Method: XmlDocDemo.DemoClass.ShowName(string name)
XML Name: M:XmlDocDemo.DemoClass.ShowName(System.String)
XML Summary: Display a name

Method: XmlDocDemo.DemoClass.ShowName(string name, byte repeats)
XML Name: M:XmlDocDemo.DemoClass.ShowName(System.String,System.Byte)
XML Summary: Display a name a certain number of times

Method: XmlDocDemo.DemoClass.ShowGenericType<T>(T myVar)
XML Name: M:XmlDocDemo.DemoClass.ShowGenericType``1(``0)
XML Summary: Display the type of the variable passed in

Method: XmlDocDemo.DemoClass.ShowNullableInt(int? myInt)
XML Name: M:XmlDocDemo.DemoClass.ShowNullableInt(System.Nullable{System.Int32})
XML Summary: Display the value of a nullable integer

Resources

Source Code

A simple-case working demo of these concepts can be downloaded here:

• XmlDocDemo.zip (4kb)

Documentation Generators

• DocFX - An extensible and scalable static documentation generator.

• Sandcastle - Sandcastle Help File Builder (SHFB). A standalone GUI, Visual Studio integration package, and MSBuild tasks providing full configuration and extensibility for building help files with the Sandcastle tools.

Zachary Patten’s Useful Article

There is an extensive article on this topic in the October 2019 issue of MSDN Magazine, Accessing XML Documentation via Reflection by Zachary Patten. The code examples there provide a lot of advanced features, but are technically incomplete and some critical components are only shown using pseudocode. The reader is told that full code is available as part of the author’s library Towel, but this library is extensive and provides many functions unrelated to reflection and XML documentation making it difficult to navigate. The method to convert a method to its XML documentation name is Towel/Meta.cs#L1026-L1092, but it’s coupled to other code which requires hash maps to be pre-formed in order to use it. My post here is intended to be self-contained simple reference for how to combine XML documentation with Reflection, but users interested in reading further on this topic are encouraged to read Zachary’s article.

Update: Potentially Useful Libraries

Update (Feb 21, 2021): I continued to do research on this topic. I thought I’d find a “golden bullet” library that could help me do this perfectly. The code above does a pretty good job, but I would feel more confident using something designed/tested specifically for this task. I looked and found some helpful libraries, but none of them met all me needs. For my projects, I decided just to use the code above.

DocXml

DocXml is a small .NET standard 2.0 library of helper classes and methods for compiler-generated XML documentation retrieval. Its API is very simple and easy to use with a predictive IDE. Out of the box though it was unable to properly identify the XML name of one of my functions. I think it got stuck on the generic method with a multi-dimensional generic array as an argument, but don’t recall for sure. For basic code bases, this looks like a fantastic library.

NuDoq

NuDoq (previously NuDoc?) is a standalone API to read and write .NET XML documentation files and optionally augment it with reflection information. According to the releases it was actively worked on around 2014, then rested quietly for a few years and new releases began in 2021. NuDoq looks quite extensive, but takes some studying before it can be used effectively. “Given the main API to traverse and act on the documentation elements is through the visitor pattern, the most important part of the API is knowing the types of nodes/elements in the visitable model.”

Towel

Towel is a .NET library intended to add core functionality and make advanced topics as clean and simple as possible. Towel has tools for working with data structures, algorithms, mathematics, metadata, extensions, console, and more. Towel favors newer coding practices over maintaining backwards compatibility, and at the time of writing Towel only supports .NET 5 and newer. One of Towel’s Meta module has methods to get XML names for reflected objects. It’s perfect, but requires .NET 5.0 or newer so I could not use it in my project.

Washcloth

I tried to create a version of Towel that isolated just the XML documentation reflection module so I could back-port it to .NET Standard. I created Washcloth which largely achieved this and wrapped Towel behind a simple API. This task proved extremely difficult to accomplish cleanly though, because most modules in the Towel code base are highly coupled to one another so it was very difficult to isolate the Meta module and I never achieved this goal to my satisfaction. I named the project Washcloth because a washcloth is really just a small towel with less functionality. Washcloth technically works (and can be used in projects back to .NET Core 2.0 and .NET Framework 4.6.1), but so much coupled code from the rest of Towel came along with it that I decided not to use this project for my application.

Final Thoughts

After a month poking around at this, here’s where I landed:

• Reading XML documentation is easy with System.Xml.Linq.XDocument

• Getting XML names for fields, properties, classes, constructors, and enumerations is easy

• Getting XML names for methods can be very hard

• If you’re creating something special, consider a custom solution like that shown above (~50 lines and you’re done).

• If you can target the latest .NET platform, consider the Meta module in Towel

• If you want a package that targets .NET Standard, consider DocXml (simple) or NuDoq (complex)

Get URL of a Directory with PHP

My website went down for a few hours today when my hosting company unexpectedly changed Apache’s root http folder to a new one containing a symbolic link. This change broke some of my PHP scripts because __DIR__ suddenly had a base path that was different than DOCUMENT_ROOT, and the str_replace method I was using to determine the directory URL assumed they would always be the same. If you Google “how to get the URL of the current directory with PHP”, you’ll probably find recommendations to use code like this:

// WARNING: DON'T USE THIS CODE!
$folderUrl = 'http://'.$_SERVER['HTTP_HOST'].str_replace($_SERVER['DOCUMENT_ROOT'], '', __DIR__); 

Let’s look at what these variables resolve to on my site:

Comparing __DIR__ with DOCUMENT_ROOT, notice that without real path resolution the base paths are different! This caused the string replace method to abruptly fail on my website, taking several sites down for a few hours. I’ll accept it as a rookie mistake on my part, and I’m sharing what I learned here in case it helps others in the future.

The Solution is to ensure all paths are converted to canonicalized absolute paths using realpath(). This will protect you from unexpected symbolic links. Notice this code also adds the appropriate HTTP or HTTPS prefix.

// This script displays the URL of the current folder
$realDocRoot = realpath($_SERVER['DOCUMENT_ROOT']);
$realDirPath = realpath(__DIR__);
$suffix = str_replace($realDocRoot, '', $realDirPath);
$prefix = isset($_SERVER['HTTPS']) ? 'https://' : 'http://';
$folderUrl = $prefix . $_SERVER['HTTP_HOST'] . $suffix;
echo $folderUrl;

LJP Theory and Correction

This page contains notes about the analytical methods LJPcalc uses to calculate LJP from ion tables as well as notes for experimenters about what causes LJP and how to compensate for it in electrophysiology experiments.

💡 Use LJPcalc to calculate Liquid Junction Potential (LJP) in your browser

LJP Calculation Notes

LJPcalc Calculation Method

LJPcalc calculates the liquid junction potential according to the stationary Nernst-Planck equation which is typically regarded as superior to the simpler Henderson equation used by most commercial LJP calculators. Both equations produce nearly identical LJPs, but the Henderson equation becomes inaccurate as ion concentrations increase, and also when calculating LJP for solutions containing polyvalent ions.

LJPcalc Ion Mobility Library

LJPcalc uses an extensive ion mobility library

The ion mobility table is stored in Markdown format. Not only does Markdown make it easy to display the table nicely in a browser, but it also makes the table easy to edit in any text editor. Users desiring to use their own ion mobilities or add new ions to the table can do so by editing the IonTable.md file adjacent to LJPcalc.exe as needed.

Influence of Ion Sequence on Calculated LJP

💡 LJPcalc automatically sorts the ion table into an ideal sequence prior to solving for LJP. Attention only needs to be paid to the ion sequence if automatic sorting is disabled.

When calculating LJP for a set of ions it is important to consider the sequence in which they are listed. Additional information can be found in Marino et al., 2014 which describes the exact computational methods employed by LJPcalc.

• The last ion’s c0 may be overridden to achieve electroneutrality on the c0 side. This will not occur if the sum of charge on the c0 side is zero.

• cL for most ions will be slightly adjusted to achieve electroneutrality on the cL side. The second-to-last ion’s cL (which cannot equal its c0) will remain fixed, while the last cL will be adjusted to achieve electroneutrality. During the solving process all cL values (but the second-from-last) will be slightly adjusted. The adjustments are likely negligible experimentally, but this is why cL values in the output table slightly differ from those given for inputs.

Effect of Temperature on LJP

The LJP is temperature dependent. There are two sources of temperature-dependent variation: the Einstein relation and the conductivity table. The former can be easily defined at calculation time, while the latter requires modifying conductances in the ion mobility table. These modifications typically have a small effect on the LJP, so standard temperature (25C) can be assumed for most applications.

The Einstein relation defines diffusion as D = µ * k * T where:

• D is the diffusion coefficient
• µ (mu) is ionic mobility
• k is the Boltzmann constant (1.380649e-23 J / K)
• T is temperature (K)

The ion conductivity table is temperature-specific. Ion conductivity was measured experimentally and varies with temperature. The ion conductivity table here assumes standard temperature (25C), but ion conductivity values can be found for many ions at nonstandard temperatures. LJPcalc users desiring to perform LJP calculations at nonstandard temperatures are encouraged to build their own temperature-specific ion tables.

Calculating Ionic Mobility from Charge and Conductivity

Ionic mobility is µ = Λ / (N * e² * |z|) where:

• µ (mu) is ionic mobility (m² / V / sec)
• Λ (Lambda) is molar conductivity (S * cm²/ mol)
• N is the Avogadro constant (6.02214076e23 particles / mol)
• e is the elementary charge (1.602176634e-19 Coulombs)
• z is the absolute value of the elementary charge of the ion

How to Correct for LJP in Electrophysiology Experiments

Patch-clamp electrophysiologists impale cells with glass microelectrodes to measure or clamp their voltage. Amplifier offset voltage is adjusted to achieve a reading of zero volts when the pipette is in open-tip configuration with the bath, but this voltage includes offset for a liquid junction potential (LJP) caused by the free exchange of ions with different mobilities between the pipette and bath solutions. Whole-cell patch-clamp experiments typically fill the pipette with large anions like gluconate, aspartate, or methanesulfonate, and their low mobility (relative to ions like K, Na, and Cl) causes them to accumulate in the pipette and produce a LJP (where the bath is more positive than then pipette). After establishment of whole-cell configuration, ions no longer freely move between pipette and bath solutions (they are separated by the cell membrane), so there is effectively no LJP but the offset voltage is still offsetting as if LJP were present. By knowing the LJP, the scientist can adjust offset voltage to compensate for it, resulting in more accurate measured and clamped voltages.

V_meter = V_cell + LJP

To correct for LJP, the electrophysiologist must calculate LJP mathematically (using software like LJPcalc) or estimate it experimentally (see the section on this topic below). Once the LJP is known it can be compensated for experimentally to improve accuracy of recorded and clamped voltages.

V_cell = V_meter - LJP

⚠️ This method assumes that the amplifier voltage was zeroed at the start of the experiment when the pipette was in open-tip configuration with the bath, and that concentration of chloride (if using Ag/AgCl electrodes) in the internal and bath solutions are stable throughout experiments.

Example Patch-Clamp LJP Calculation & Correction

This ion set came from in Figl et al., 2003 Page 8. They have been loaded into LJPcalc such that the pipette solution is c0 and the bath solution is cL. Note that the order of ions has been adjusted to place the most abundant two ions at the bottom. This is ideal for LJPcalc’s analytical method.

Loading this table into LJPcalc produces the following output:

Values for cL were adjusted to achieve electro-neutrality:

 Name               | Charge | Conductivity (E-4) | C0 (mM)      | CL (mM)      
--------------------|--------|--------------------|--------------|--------------
 K                  | +1     | 73.5               | 145          | 2.8098265   
 Na                 | +1     | 50.11              | 13           | 144.9794365 
 Mg                 | +2     | 53.06              | 1            | 1.9998212   
 Ca                 | +2     | 59.5               | 0            | 0.9999109   
 HEPES              | -1     | 22.05              | 5            | 4.9990023   
 Gluconate          | -1     | 24.255             | 145          | 0           
 Cl                 | -1     | 76.31              | 10           | 148.789725

Equations were solved in 88.91 ms
LJP at 20 C (293.15 K) = 16.052319631180264 mV

💡 Figl et al., 2003 Page 8 calculated a LJP of 15.6 mV for this ion set (720 µV lesser magnitude than our calculated LJP). As discussed above, differences in ion mobility table values and use of the Nernst-Planck vs. Henderson equation can cause commercial software to report values slightly different than LJPcalc. Experimentally these small differences are negligible, but values produced by LJPcalc are assumed to be more accurate. See Marino et al., 2014 for discussion.

If we have patch-clamp data that indicates a neuron rests at -48.13 mV, what is its true resting potential? Now that we know the LJP, we can subtract it from our measurement:

V_cell = V_measured - LJP

V_cell = -48.13 - 16.05 mV

V_cell = -64.18 mV

We now know our cell rests at -64.18 mV.

Zeroed Voltage = LJP + Two Electrode Half-Cell Potentials

The patch-clamp amplifier is typically zeroed at the start of every experiment when the patch pipette is in open-tip configuration with the bath solution. An offset voltage (V_offset) is applied such that the V_measured is zero. This process incorporates 3 potentials into the offset voltage:

• liquid junction potential (LJP) between the pipette solution and the bath solution (mostly from small mobile ions)
• half-cell potential (HCP) between the reference electrode and the bath solution (mostly from Cl)
• half-cell potential (HCP) between the recording electrode and the pipette solution (mostly from Cl)

When the amplifier is zeroed before to experiments, all 3 voltages are incorporated into the offset voltage. Since the LJP is the only one that changes after forming whole-cell configuration with a patched cell (it is eliminated), it is the only one that needs to be known and compensated for to achieve a true zero offset (the rest remain constant).

However, if the [Cl] of the internal or bath solutions change during the course of an experiment (most likely to occur when an Ag/AgCl pellet is immersed in a flowing bath solution), the half-cell potentials become significant and affect V_measured as they change. This is why agar bridge references are preferred over Ag/AgCl pellets. See Figl et al., 2003 for more information about LJPs as they relate to electrophysiological experiments.

Measuring LJP Experimentally

It is possible to measure LJP experimentally, but this technique is often discouraged because issues with KCl reference electrodes make it difficult to accurately measure (Barry and Diamond, 1970). However, experimental measurement may be the only option to calculate LJP for solutions containing ions with unknown mobilities.

To measure LJP Experimentally:
• Step 1: Zero the amplifier with intracellular solution in the bath and in your pipette
• Step 2: Replace the bath with extracellular solution
• Step 3: The measured voltage is the negative LJP (invert its sign to get LJP)

✔️ Confirm no drift is present by replacing the bath with intracellular solution after step 3 to verify the reading is 0. If it is not 0, some type of drift is occurring and the measured LJP is not accurate.

❓ Why invert the sign of the LJP? The LJP measured in step 2 is the LJP of the pipette relative to the bath, but in electrophysiology experiments convention is to refer to LJP as that of the bath relative to the pipette. LJPs for experiments using typical ACSF bath and physiological K-gluconate pipette solutions are usually near +15 mV.

⚠️ Do not measure LJP using a Ag/AgCl reference electrode! because mobility will be low when the bath is filled with intracellular solution (physiological intracellular solutions have low [Cl]). Use a 3M KCl reference electrode instead, allowing high [K] mobility in intracellular solution and high [Cl] mobility in extracellular solution.

References

• Marino et al. (2014) - describes a computational method to calculate LJP according to the stationary Nernst-Planck equation. The JAVA software described in this manuscript is open-source and now on GitHub (JLJP). Figure 1 directly compares LJP calculated by the Nernst-Planck vs. Henderson equation.

• Perram and Stiles (2006) - A review of several methods used to calculate liquid junction potential. This manuscript provides excellent context for the history of LJP calculations and describes the advantages and limitations of each.

• Shinagawa (1980) “Invalidity of the Henderson diffusion equation shown by the exact solution of the Nernst-Planck equations” - a manuscript which argues that the Henderson equation is inferior to solved Nernst-Planck-Poisson equations due to how it accounts for ion flux in the charged diffusion zone.

• Lin (2011) “The Poisson The Poisson-Nernst-Planck (PNP) system for ion transport (PNP) system for ion transport” - a PowerPoint presentation which reviews mathematical methods to calculate LJP with notes related to its application in measuring voltage across cell membranes.

• Nernst-Planck equation (Wikipedia)

• Goldman Equation (Wikipedia)

• EGTA charge and pH - Empirical determination of EGTA charge state distribution as a function of pH.

• LJPCalcWin - A Program for Calculating Liquid Junction Potentials

• LJP Corrections (Axon Instruments Application Note) describes how to calculate LJP using ClampEx and LJPCalcWin and also summarizes how to measure LJP experimentally

• LJP Corrections (Figl et al., AxoBits 39) summarizes LJP and discusses measurement and calculation with ClampEx