Connecting Two Unix Systems Using Point-To-Point Protocol (Quick Version)

I had (have?) a detailed discussion of how to use pppd (Point-To-Point Protocol Daemon) in the pipeline. However, Twitter friend Gravis posted a very nice article on how PPP the protocol works, and I felt compelled to convert my original detailed post into a more concise form!

The Point-To-Point Protocol (PPP) is a Layer 2 protocol that is meant to connect computers in a network, like Ethernet or Wifi. However, unlike Ethernet and Wifi, PPP doesn't specify how voltages should be sent from one computer to another. The Fundamentals section in Gravis' article provides great detail as to the rationale. An important consequence of PPP not specifying how voltages are sent is that lots of computing hardware can be used to send and receive PPP packets, such as RS-232 ports.

This article will focus on sending/receiving PPP packets over Universal Asynchronous Receiver Transmitter (UART). Due to their simplicitly, UARTs are extremely common hardware for communications and debugging on Systems-on-a-Chip (SoCs), FPGAs, Single-Board Computers (SBCs), and just many computers in general. Considering their proliferation, a UART becomes a powerful communications interface when combined with PPP and TCP/IP; you now have a means to connect computers without any other I/O interfaces to the rest of the TCP/IP-speaking Internet if you so choose!

The Point-To-Point Protocol Daemon

One common PPP implementation is the Point-To-Point Protocol Daemon for Unix systems, or pppd. pppd provides both a daemon (service) and kernel component to create a network interface that can send and receive TCP/IP packets (encapsulated by PPP) to and from a UART.

You can use pppd to set up two computers running Unix or a Unix-like OS to communicate to each other over their own local TCP/IP network. In fact that's a major characteristic of a Point To Point network! In such a setup, both computers can contact each other via applications like ssh, but are blissfully unaware of any other computers that exist in the world.

In the absence of any dedicated networking hardware, you can still use pppd to connect Unix systems to the Internet. This article will describe how to use pppd to connect to the Internet proper, using my current setup at home as an example. My setup should be applicable to most use cases, as long as one of the computers in your PPP links has another network interface besides a UART, such as an Ethernet port or Wifi adapter.

My Setup

I have two Single-Board Computers (SBCs) deployed at present- an ASUS TinkerBoard running a Debian variant, and a Raspberry Pi 1 Model A+ running NetBSD. The TinkerBoard SoC has 5 UARTs, an Ethernet port, and a Wifi adapter/antenna. 4 UARTs are exposed on GPIO headers. I use one of the additional UARTs exclusively for PPP. The Raspberry Pi has only a single UART. I will describe how to use a single UART for both a terminal and a PPP session later.

UARTs are often exposed as three or more pin headers on your computer's PCB- receive (RX), transmit (TX), and ground (GND) 1 . For PPP communication to work, at a minimum, you will need to connect GND to GND, RX to TX, and TX to RX between your two computers. The minimum setup works just fine and is what I use.

Your SBC vendor will typically list UART pins as part of their GPIO pinout on their website (NanoPI NEO, for example). Many SBCs, including TinkerBoard use a Raspberry Pi-compatible GPIO header, which will expose at least one UART. If the pinout is not publicly available, you may need to reverse engineer the pinout! Or, if a USB port is available, you can use USB to Serial Cable to create a new UART and pins- I can confirm pppd works fine with USB adapters on Linux and NetBSD.

Picture of my TinkerBoard and Raspberry Pi connected to each other via UART.

In the above image, my TinkerBoard (right) and Raspberry Pi (left) are connected to each other by three wires. Green connects TinkerBoard RX to Raspberry Pi TX, white connects TinkerBoard TX to Raspberry Pi RX, and black connects both boards' grounds to each other. The remaining wires implement soft reset switches and are not currently wired up properly.

The ribbon cables on both boards are Adafruit downgrade ribbon cables to connect the GPIO header to a breadboard out-of-shot. The hat on the TinkerBoard is a Perma-Proto hat to break out two UARTs. The hat on the on the Raspberry Pi is custom perfboard also breaking out the UART and reset pins; I only had one Perma-Proto hat :).

There is no Ethernet cable because I opt to use TinkerBoard's Wifi instead.

Other Setups

If you don't have a TinkerBoard and Raspberry Pi, the important part of my setup is that the TinkerBoard has multiple pieces of hardware for network interfaces- Wifi, Ethernet, and UART; my Raspberry Pi has only one- a UART. By routing UART traffic between two computers via PPP to/from other existing networking hardware, it is possible to connect the UART-only computer to the Internet.

In the next sections, I will talk about how to set up pppd for both my TinkerBoard and Raspberry Pi. If you don't have those particular pieces of hardware, the config options I provide should be good defaults.

TinkerBoard and Debian

As far as I understand, pppd nominally treats both peers in a connection as interchangable. Unlike some communications protocols like SPI or I2C, there is no concept of controller or peripheral. Therefore, it normally doesn't make much of a difference which peer you set up first to bring up a PPP link3 .

However, there are options you can pass to pppd to share settings (at least IP address- there might be more?) from one peer to another. Seeing as the TinkerBoard peer in my setup needs to route traffic to/from the Internet anyway, which already requires different set of config options from the Raspberry Pi, I use extra config options to treat the TinkerBoard as a controller for a Raspberry Pi peripheral.

/etc/ppp/options.ttyS1 Config

Besides the command line, pppd initializes itself using a configuration file containing options under the /etc/ppp directory. Options file names take the form of options.$DEVICE, where $DEVICE represents the device name of your serial port. You will need to know the names that your Unix system assigned your serial ports before setting up pppd. In my case, I have allocated UART2 on the TinkerBoard GPIO header specifically for the PPP link. This is called ttyS2 by Linux 2 .

I have posted my /etc/ppp/options.ttyS2 config file in full below, and I will briefly explain what each options entails afterwards. If you want more information, I encourage you to consult the pppd man page, which details all possible options that pppd understands on the command line or in a config file:

noauth
local
nodetach
nocrtscts
xonxoff
192.168.1.164:192.168.1.165
persist
proxyarp

Aside from setting up the above config file, you will probably want to enable IP forwarding, described in the next section. IP forwarding is not required to set up a Point-To-Point link by itself, but it is required so that both ends of the link, especially the UART-only side, can access the Internet.

IP Forwarding

Any traffic that's meant for the Raspberry Pi from one of the other interfaces needs to somehow reach the UART interface with the PPP link. With the above options, pppd itself will:

As you can see, pppd delegates some responsibilites to the OS.

pppd has a kernel component too, but these syscalls seem to not be pppd-component specific.

As you may have guessed, it is also kernel's responsibility to make sure that traffic from the TinkerBoard's Ethernet and Wifi interfaces somehow get sent over the UART wires. That somehow is called IP forwarding.

for you, and (as far as I understand) will not check other interfaces for packets meant for one of the two computers reachable on the UART interface. Instead, you must ask the OS to forward packets for you from one interface to the computers on another interface, effectively turning one of your computers with a PPP link into a router.4

Normally, Linux will not forward packets from one interface to another by default. You have a few options:

/etc/systemd/system/rpippp.service Unit File

[Unit]
Description=PPP
Wants=network-online.target
After=network-online.target
[Service]
Type=idle
ExecStart=/usr/sbin/pppd -d /dev/ttyS2 576000
Restart=always
TimeoutStartSec=120
[Install]
WantedBy=multi-user.target
Alias=ppp.service

Since my TinkerBoard is running a Debian variant, it also comes with systemd. So we create a unit file for our pppd service.

At this point, one half of our PPP link is fully set up. If running Linux, setting up pppd on the other half of the link is similar to the above procedure:

  1. Identify the *nix device name of the UART the second computer which connected to the first computer.
  2. Populate the pppd config file with the correct options (described below).
  3. Set up a corresponding system service for pppd to easily setup/tear down the PPP link (at boot, preferably- mention getty autodetects?). The command line options -d and baud rate should be identical to the first computer, and the device name should reflect the UART identified in step 1.

Since we don't need to forward traffic over the other computer's UART to anywhere else, the IP forwarding step is unnecessary on the other half of the link.

However, my Raspberry Pi runs NetBSD instead of Linux for various reasons, and thus I will describe setting up the PPP link on the Raspberry Pi side using NetBSD. The setup is a bit more involved, but it has one neat advantage: a single UART can be used to connect to the Raspberry Pi using either a PPP session or via the traditional login over serial port. Multiplexing a UART like this requires special logic in the getty program that monitors serial ports before login. The NetBSD getty has PPP detection logic, while I'm not aware of a Linux getty package that does.

No matter which *nix variant the other half of the link is running, you should still use a config file to the one I present below on the other computer.

Raspberry Pi Model A+ and NetBSD

Because my Raspberry Pi is an original Model A+, it doesn't have an Ethernet port. This means that short of using a USB Wifi adapter (which I find flaky), using the UART is one of the only ways to connect to Internet. However, there's only one UART on the Raspberry Pi5 ), and I need that UART occassionally for a serial console for debugging.

As it turns out, there's nothing preventing me from using the same UART for a serial console and a PPP connection a long as I don't try to use pppd and a serial console at the same time over the same UART.

NetBSD comes with pppd as part of the base set tarball, so there is no bootstrapping problem with needing to download pppd before using pppd. If pppd wasn't part of the base distribution, I would use a separate computer to download the pppd package for your distro and then transfer it via flash drive ("sneakernet") or use Kermit over the UART via a USB-to-serial cable connection.

NetBSD getty

The getty implementation provided with NetBSD is capable of multiplexing normal login via a serial console with a PPP session. A Linux getty with such capability may be coming soon.

/etc/gettytab

# PPP network link login
#
# these entries can be used by ISPs or others who want to be able
# to offer both a "shell" and a PPP login on the same port. Setting
# the "pp" attribute allows getty(8) to recognize a PPP link start
# negotiation, and invoke the program listed, in addition to normal
# login(1).
#
# N.B.: if PPP is recognized, this bypasses normal login/password
# exchange; the expectation is that you'll configure pppd (or whatever)
# to require a PAP or CHAP handshake for authentication after PPP is
# started up.
#
# It is also recommended that you use hardware (CTS/RTS) flow control
# on the port, and run the port as fast as possible, to allow modems
# extra time to do data compression, if enabled.
#
ppp:np:ce:ck:pp=/usr/sbin/pppd:
#
ppp.19200|PPP-19200:sp#19200:tc=ppp:
ppp.38400|PPP-38400:sp#38400:tc=ppp:
ppp.57600|PPP-57600:sp#57600:tc=ppp:
ppp.115200|PPP-115200:sp#115200:tc=ppp:
ppp.230400|PPP-230400:sp#230400:tc=ppp:
ppp.460800|PPP-460800:sp#460800:tc=ppp:
ppp.500000|PPP-500000:sp#500000:tc=ppp:
ppp.576000|PPP-576000:sp#576000:tc=ppp:

/etc/ttys

#       $NetBSD: ttys,v 1.8 2019/09/25 23:09:21 abs Exp $
#
#       from: @(#)ttys  5.1 (Berkeley) 4/17/89
#
# name  getty                           type    status          comments
#
console "/usr/libexec/getty ppp.576000" vt100   on secure
constty "/usr/libexec/getty default"    vt100   off secure
ttyE0   "/usr/libexec/getty Pc"         wsvt25  off secure
ttyE1   "/usr/libexec/getty Pc"         wsvt25  off secure
ttyE2   "/usr/libexec/getty Pc"         wsvt25  off secure
ttyE3   "/usr/libexec/getty Pc"         wsvt25  off secure
tty00   "/usr/libexec/getty default"    unknown off secure
tty01   "/usr/libexec/getty default"    unknown off secure
tty02   "/usr/libexec/getty default"    unknown off secure
tty03   "/usr/libexec/getty default"    unknown off secure
tty04   "/usr/libexec/getty default"    unknown off secure
tty05   "/usr/libexec/getty default"    unknown off secure
tty06   "/usr/libexec/getty default"    unknown off secure
tty07   "/usr/libexec/getty default"    unknown off secure

/etc/ppp/options.console Config

Now, we need to set up a config file on the Raspberry Pi side of things. It is perfectly fine to use console (as in /dev/console) as the serial device name. NetBSD will assign /dev/console to the UART exposed on the GPIO header, which is what we want.

ipcp-accept-local
noauth
local
xonxoff
defaultroute

Here are short explanations on new options that pppd needs for this scenario where the Raspberry Pi spawns pppd instances when it detects PPP activity from the TinkerBoard:

/etc/resolv.conf

# Generated by resolvconf
domain $DOMAIN
nameserver $NAMESERVER

Next Steps

Tuning Speed

After you've verified

Baud RateData CopiedSpeed
230400524288 bytes22.1 kB/s
460800524288 bytes41.2 kB/s
500000524288 bytes40.9 kB/s
576000524288 bytes44.3 kB/s
921600524288 bytes44.1 kB/s
1000000524288 bytes29.9 kB/s

UARTs are not designed for high speeds, emitting voltages over a single-ended interface suspectible to noise. However, the speeds you can get (tens of kBps) are acceptable for a number of background tasks and downloading software.

Enjoy Your New Internet Connection!

For better or worse, connecting a computer to the outside world is a prerequisite for making the computer useful over long periods of time. While there is a lot of work to get data from source to destination across the world, we can leverage the existing design to connect machines to the outside world that would normally be overlooked. We used pppd to connect an overlooked machine like the Raspberry Pi Model A to the outside world. By leveraging existing relatively simple hardware like a UART, an Internet connection will hopefully give the Model A and similar computers many more years of useful life.

Since you've made it this far, I imagine you want to actually see the PPP link working. Well, ask and you shall receive!

Example terminal session of connecting to the Raspberry Pi via from a Wifi-connected desktop computer on the same network.

The above GIF shows me scping a cross-compiled binary for ARM NetBSD from a desktop computer to my Raspberry Pi running NetBSD. From there, I ssh into the Raspberry Pi, and show via ifconfig that there is only a loopback and PPP interface and no USB adapters (hopefully this is convincing enough!). I finish the session by running the freshly transferred GNU Hello world program.

I used a desktop computer in my house instead of the TinkerBoard for this session to show that the Raspberry Pi is reachable from other computers besides the TinkerBoard. While both the desktop and TinkerBoard are using their wireless network interfaces (WiFi), a session similar to mine should be duplicable with any combination of Ethernet or Wifi adapters (or even more PPP links if you so dare :)).

I recorded my terminal session with asciinema, and converted the output hello.cast to a GIF using gifcast.

Troubleshooting

The following section briefly discusses problems I have personally run into when getting a PPP link between my TinkerBoard and Raspberry Pi to work.

TinkerBoard doesn't boot when connected to the Raspberry Pi.

I have to take some time to diagnose this; in my current setup, the TinkerBoard will refuse to boot if the Raspberry Pi has powered on and drives any pins on the TinkerBoard's I/O connector. To fix this, power off the Raspberry Pi until the TinkerBoard starts booting. I will need to experiment to find which GPIOs are the culprit.

TinkerBoard boots, then ceases all communications with the Raspberry Pi.

Make sure your pppd options file contains local on both the TinkerBoard and Raspberry Pi. It is possible that without this option, pppd on the TinkerBoard will wait for hardware flow control signals that will never arrive.

I can't ssh into the Raspberry Pi from a remote computer besides the TinkerBoard.

Make sure IP forwarding is enabled on the TinkerBoard, and make sure that pppd on the TinkerBoard is configured with the proxyarp option. See footnote #4, as there may be other failure modes due to not including those options that I've since forgotten.

I can't access the Internet via domain names on the Raspberry Pi.

Make sure your nameserver in resolv.conf is set up properly. If using DHCP, this is taken care of for you. However, pppd setup falls under "static" IP, and while it handles most static IP configuration steps via its config file, setting up your nameserver is not one of those steps.

I think this is a NetBSD issue; NetBSD uses the break condition (abnormally long low voltage) to start the kernel debugger. Any noise on the line during TinkerBoard and/or NetBSD boot could create that condition, causing the Raspberry Pi to stop responding. And my own experience is that pppd doesn't tolerate the boot messages sent from the Raspberry Pi well, seeing those messages as garbage. Therefore I recommend not starting the pppd link at boot via systemctl enable. Rather, wait until you need to access the other end of the UART after boot has completed on both ends of the link, and then run systemctl start rpippp.service.

Thanks to the multiplexed getty and pppd on the NetBSD side, one good debugging technique on the Pi side is to stop the PPP link and access the Pi via serial console using a program such as minicom. If you press "Enter" and see db>, the kernel debugger has been invoked. Type c, (for continue) then "Enter" to exit the debugger. If you see login: or similar without a login banner, the NetBSD getty has stopped listening for PPP packets. You will need to log in, and then log back out to restart the PPP detection FSM.

After a recent power outage, I had to manually restart both the TinkerBoard (because of the GPIO problem above) as well as the Raspberry Pi. Even after waiting until after both the TinkerBoard and Raspberry Pi finished boot, and debugging using the above procedure, the Raspberry Pi wasn't responding to any activity over the serial console, even in minicom; it seemed to have crashed completely. The On a hunch, when I rebooted the Pi, I commented out the entirity of my /etc/ppp/ip-up script. The PPP link came up successfully. At this point, I stopped the link, used minicom to restore the /etc/ppp/ip-up script, and I was able to restart the link successfully too. I wonder if there's a bug with ntpdate that makes it refuse to run until after the link has been brought up once since the last reboot? I will need to investigate.

Footnotes

1 The power pin (Vcc) may also be exposed be exposed as a pin. Some USB UART adapters, such as FTDI cables, also provide a Vcc output pin to supply power to target boards with a UART. In our case, I assume the board is _not_ being powered by a USB adapter, and so Vcc pins can be ignored.

2 The RockChip that powers the TinkerBoard has 5 UARTs. From reading the manual and comparing pinouts, UART0 is not exposed on the GPIO header. UART1 is exposed on the GPIO header at the same pins as on a Raspberry Pi, so I leave that alone for compatibility. This leaves UART2 as the first free available UART- and thankfully, it has a device node!

3 When doing my first PPP experiments back in 2017, I debugged various link bringup scenarios using USB UARTs; I was able to get a working link no matter which order I brought up both peers- including Internet.

4 If you enable Proxy ARP, chances are you want to enable IP forwarding as well. I seem to vaguely recall during my early PPP debugging sessions that enabling Proxy ARP but not IP forwarding, or vice-versa, would result in different errors. In 2020 however, I can't seem to duplicate different failure modes. If I don't have both proxy ARP and IP forwarding, the Raspberry Pi is inaccessible from all other computers, but the TinkerBoard's ppp interface is accessible just fine.

5 I don't know at present how to use the mini UART.

Last Updated: 2020-07-02