Tortoise or Hare? Nvidia Jetson TK1

I’m quite pleased with the Intel NUC boards that came with my surplus Rabbit cloud, but I can’t say the same about the Nvidia Jetson TK1 boards. They’re weird, flaky, limited, and haven’t proved useful even in simple applications. Frankly, I’m not sure what I’m going to do with the 15 Jetsons sitting in my office!

Read my entire Rabbit Cloud series

• Introducing Rabbit: I Bought a Cloud!
• Running Rabbits: More About My Cloud NUCs
• Tortoise or Hare? Nvidia Jetson TK1
• Powering Rabbits: The Mean Well LRS-350-12 Power Supply

Meet George Jetson

The Jetson was one of the first high-profile AI prototyping boards released, and was quite impressive back in 2014. This was before the Raspberry Pi 2 was released, so it would have been compared to the original Pi, which had a 700 MHz single-core Broadcom BCM2835 CPU. The TK1 sported a quad-core Tegra K1 SoC (which used the Cortex A15 microarchitecture and actually included a fifth low-power “companion” core) running at up to 2.3 GHz, blowing the doors off the Pi and actually quite competitive with low-end PCs of the day.

But this performance was overshadowed by the inclusion of a then-modern “Kepler” GPU with 192 CUDA cores. The Jetson TK1 became the darling of the AI set, with robots and autonomous driving experiments built around it. No doubt this is also why the original Rabbit streaming service specified five Jetson TK1 boards as companions to the ten NUCs on each tray. It would have been an impressive-sounding setup for the time.

But times change.

The Jetson TK1 is a surprisingly limited board today, since it’s so proprietary and integrated:

• 2GB of memory might have sounded fine in 2014, but since it’s soldered to the board and not socketed like the NUC it is a major limitation today
• The USB 3.0 port is nice, but there’s only one and it’s flaky, needing special firmware and causing issues sometimes; you can get an adapter to use the micro-USB port as a second port, but a USB 3.0 hub is easier to find
• The Gigabit Ethernet port is just slow, only able to hit 68% of the throughput of the equivalent port on the NUC
• It can boot Linux for Tegra from the 16GB integrated eMMC storage, but this was never updated past Ubuntu 14 and it’s fiddly to get anything else to boot
• The Kepler GPU is not supported by the latest software, nor is it all that fast by today’s standards

On the plus side, there are some nice aspects to the board:

• The Tegra K1 CPU is reasonably powerful for an embedded board even today
• Integrated RAM and eMMC mean the salvage company delivered a bootable and usable board, unlike the NUC which came without RAM or storage
• The mini PCI-E slot is half-length but extends “off the board” without obstruction so it should be able to accommodate other peripherals
• It has a nicer SD slot than the NUC, with a spring-loaded ejector
• It has a SATA port, though a drive would require external power

Objectively weighing the plusses and minuses of the Jetson TK1, I decided that it would be useful in a supporting role, while the NUCs were the star of the show. I planned to use a few Jetson boards on each tray as a simple router, DHCP server, DNS server, NTP server, and perhaps a storage server or cluster.

Booting the Jetson TK1

The embedded eMMC storage is the logical boot medium for the Jetson TK1, so that was my focus from the start. It could theoretically also boot from SD card, USB drive, or SATA, but all of these require a functioning eMMC pre-boot environment.

The Jetson TK1 has a micro-USB port that allows the eMMC to be flashed with Linux for Tegra or another operating system. I found the flashing process to be simple once I got used to the quirks. Rather than rehashing the whole process, here’s my quick list of lessons learned to help anyone else:

• Linux for Tegra must be installed from an Intel x86 host running Ubuntu 14.04. It will not work from a Pi, a Jetson, or a different or more modern version of Linux. You have been warned!
• Download Linux for Tegra from Nvidia’s Tegra archive. I suggest grabbing the original Release 19.3 as well as the latest Release 21.8. Get the Driver Package and Sample Root Filesystem for your release of choice. That’s all you need.
• Connect the Jetson TK1 to your Linux server with a quality micro-USB to USB cable. Some random cables in my drawer did not work. If you don’t see “Nvidia” in lsusb on the host, try another cable.
• Enable eMMC flash mode by holding the “Recovery” button while tapping the “Reset” button once. No need to hold it longer or time anything.
• Carefully follow the Quick-Start Guide, including running the commands with sudo. Make sure you have plenty of high-speed storage available; it will not fit in a 32 GB card.
• Maximize the available eMMC space by flashing using “sudo ./flash.sh -S 14580MiB jetson-tk1 mmcblk0p1”
• Linux for Tegra includes a boat-load of useless software, so clean it up after installation by running a ton of “apt-get purge” commands (I scripted these and might share that later)

Follow my lead, and you’ll have your Jetson TK1 running Ubuntu 14.04 in no time. But then what? Ubuntu 14.04 reached the end of the LTS window on April 30, 2019 (even before L4T 21.8 was released) but still works fine. Just don’t expect many bug fixes, and be prepared to “fight” to get modern software installed. I’ve already been hunting down back-ported libraries (libssl, for one) and I’m not doing much with it. And don’t even think of exposing a Ubuntu 14 server to the open internet, since there’s no telling if there are un-patched exploits out there and they likely won’t be fixed.

Hare or Tortoise?

The Jetson seems pretty snappy when running, so I was feeling good about using mine as “Hares” to support the NUC “Rabbits” in my cloud cluster. But things have not worked out well so far.

The first thing I wanted to do was configure a Jetson as a simple router and DHCP server, allowing the cloud to exist independently of the network to which it is connected. Like most people, I use 192.168 behind my NAT, so I planned to create another 192.168 subnet for each cloud tray. This would give the NUCs plenty of IP addresses even if they need a few for various containers and services.

The ISC DHCP Server is easy to set up in Ubuntu 14 and seems to work fine with the Rabbit configuration.

Since the Jetson TK1 has a single Ethernet port, I planned to use a USB 3.0 Ethernet adapter as a second port for routing. The included NetGear switch is un-managed, which makes tagged VLANs a little more challenging. My old Cable Matters ASIX-based USB 3.0 adapter was recognized by Ubuntu with no special software to install.

Routing can be as simple as enabling net.ipv4.ip_forward in sysctl.conf, especially if a firewall or NAT is not needed. It was more challenging to enable a second subnet on my pfSense router, which needed both a gateway and static route added. I also needed to add outbound NAT rules on the pfSense router, and I still don’t have this working correctly.

I am actually considering using OpenVPN to set up an outbound tunnel from the Rabbit router to my pfSense router instead. This would allow me to set up the Rabbit tray anywhere (home or office) and have it come up correctly regardless. We’ll see when/if I get around to this!

But right now I’m having major issues with the TK1…

First, it appears that the Gigabit Ethernet port is connected to a slow I/O line somehow. Some have suggested that it’s on the USB interface (like the Pi) but it appears to be on the PCI bus in lspci. Regardless, it’s slow. I recorded between 600 and 640 Mbits/s using iPerf, and it appears to be about the same in both directions. For comparison, the NUCs are pushing about 940 Mbit/s in the same test.

Doing some research, I came across this popular blog post suggesting that the open source r8169 driver isn’t appropriate for the older Realtek 8111 ethernet controller, and that one should use the r8168-dkms driver instead. Sure enough, the Linux for Tegra install does use the R8169 driver.

Although the correct driver does appear in the Linux for Tegra universal repository, I was unable to get it to “take over” for the port. After un-commenting the “universal” lines from /etc/apt/sources.list and updating the apt repository, I installed the new driver with “sudo apt install r8168-dkms”. I then blacklisted the r8169 driver per step 3 in that blog post and rebooted. But no matter what I did, the port continues to use the r8169 driver.

Even worse, I’m having issues using the Jetson as a router. Even without iptables running, it’s slow and flaky. I’m getting time-outs on ssh sessions and if I can’t reliably log in through the Jetson, it probably won’t work well for regular use. I might try setting things up using VLANs direct to the pfSense router, or sourcing a Mini-PCIe Ethernet adapter, but this seems like over-kill. Maybe I’ll just use one of the NUCs as a router.

Stephen’s Stance: Jetson TK1 is Meh

All this is to say that I am not impressed by the Jetson TK1. The hardware is inflexible and limited, the operating system is out of date, and the networking is slow. I’m frankly not sure if I will be able to use the Jetson boards for anything productive.