quan m. nguyen

Introduction

The purpose of this page is to document a procedure through which an interested user can install an executable image of the RISC-V architectural port of the Linux kernel.

A project with a duration such as this requires adequate documentation to support future development and maintenance. This document is created with the hope of being useful; however, its accuracy is not guaranteed.

Table of Contents

Copyright Information

This document is currently under development. This document is copyrighted by Quan Nguyen, © 2014. All rights reserved. I do intend, however, to release this document to the public freely upon its satisfactory completion.

Meta-installation Notes

Running Shell Commands

Instructive text will appear as this paragraph does. Any instruction to execute in your terminal will look like this:

$ echo "execute this"

Optional shell commands that may be required for your particular system will be backed by a blue background, and marked "Optional":

Optional $ echo "call this, maybe"

When booted into the Linux/RISC-V kernel, and some command is to be run, it will appear as a root prompt against a black background:

# echo "run this in linux"

If you will need to replace a bit of code that applies specifically to your situation, it will be surrounded by [square brackets].

The Standard Build Unit

Installing the Toolchain (11.81 + ε SBU)

Let's start with the directory in which we will install our tools. Find a nice, big expanse of hard drive space, and let's call that $TOP. Change to the directory you want to install in, and then set the $TOP environment variable accordingly:

$ export TOP=$(pwd)

For the sake of example, my $TOP directory is on s141.millennium, at /scratch/quannguyen/noob, named so because I believe even a newbie at the command prompt should be able to boot Linux using this tutorial. Here's to you, n00bs!

Installing the RISC-V simulator (0.40 SBU)

If we are starting from a relatively fresh install of GNU/Linux, it will be necessary to install the RISC-V toolchain. The toolchain consists of the following components:

• riscv-gcc, a RISC-V cross-compiler
• riscv-fesvr, a "front-end" server that services calls between the host and target processors on the Host-Target InterFace (HTIF) (it also provides a virtualized console and disk device)
• riscv-isa-sim, the ISA simulator and "golden standard" of execution
• riscv-opcodes, the enumeration of all RISC-V opcodes executable by the simulator
• riscv-pk, a proxy kernel that services system calls generated by code built and linked with the RISC-V Newlib port (this does not apply to Linux, as it handles the system calls)
• riscv-tests, a set of assembly tests and benchmarks

In actuality, of this list, we will need to build only riscv-fesvr and riscv-isa-sim. These are the two components needed to simulate RISC-V binaries on the host machine. We will also need to build riscv-linux-gcc, but this involves a little modification of the build procedure for riscv-gcc.

First, clone the tools from the ucb-bar GitHub repository:

$ git clone git@github.com:ucb-bar/riscv-tools.git

This command will bring in only references to the repositories that we will need. We rely on Git's submodule system to take care of resolving the references. Enter the newly-created riscv-tools directory and instruct Git to update its submodules. If you want to use ssh-add to prevent yourself from typing in your private key password many times, do so now by running

Optional $ exec ssh-agent /bin/bash
$ ssh-add ~/.ssh/id_rsa

Then, cd into the directory and then initialize and update the submodules.

$ cd $TOP/riscv-tools
$ git submodule update --init

To build GCC, we will need several other packages, including flex, bison, autotools, libmpc, libmpfr, and libgmp. Ubuntu distribution installations will require this command to be run. If you have not installed these things yet, then run this:

Optional $ sudo apt-get install autoconf automake autotools-dev libmpc-dev libmpfr-dev libgmp-dev gawk build-essential bison flex texinfo gperf

Before we start installation, we need to set the $RISCV environment variable. The variable is used throughout the build script process to identify where to install the new tools. (This value is used as the argument to the --prefix configuration switch.)

$ export RISCV=$TOP/riscv

If your $PATH variable does not contain the directory specified by $RISCV, add it to the $PATH environment variable now:

$ export PATH=$PATH:$RISCV/bin

One more thing: If your machine doesn't have the capacity to handle 16 make jobs (or conversely, it can handle more), edit build.common to change the number specified by JOBS.

Optional $ sed -i 's/JOBS=16/JOBS=[number]/' build.common

Building riscv-linux-gcc (11.41 SBU)

riscv-linux-gcc is the name of the cross-compiler used to build binaries linked to the GNU C Library (glibc) instead of the Newlib library. You can build Linux with riscv-gcc, but you will need riscv-linux-gcc to cross-compile applications, so we will build that instead.

The SYSROOT Concept

When installing these toolchains, the make system often generates a wide variety of libraries and other files. In particular, building Glibc involves building the run-time dynamic linker and the C standard library (ld.so.1 and libc.so.6, in this case). These, together with header files like stdio.h, comprise the system root, an often-necessary set of files for a fully operational system.

When we built riscv-tools, there was no need for specifying where to install these files, because we assumed we would always be running on the host machine through a simulator; all of the libraries are on the host system. Now that we're running our binaries from within an operating system, we will have to provide these libraries and headers if we want to run dynamically-linked binaries and compile programs natively.

We now must instruct the riscv-linux-gcc build process to place our system root files in a place we can get to them. We call this directory SYSROOT. Let's set our $SYSROOT environment variable for easy access throughout the build process. Ensure that this directory is not inside our $RISCV variable.

$ cd $TOP
$ mkdir sysroot
$ export SYSROOT=$TOP/sysroot

The Linux Headers

In an apparent case of circular dependence (but not really), we have to give the riscv-linux-gcc the location of the Linux headers. The Linux headers provide the details that Glibc needs to function properly (a prominent example is include/asm/bitsperlong.h, which sets Glibc to be 64-bit or 32-bit). There's a copy of the headers in the riscv-gcc repository, so make a usr/ directory in $SYSROOT and copy the contents of riscv-gcc/linux-headers into the newly created directory.

$ mkdir $SYSROOT/usr
$ cp -r $TOP/riscv-tools/riscv-gcc/linux-headers/* $SYSROOT/usr

Take note that we supply both the variables INSTALL_DIR and SYSROOT. Even though we are diverting some files to the $SYSROOT directory, we still have to place the cross-compiler somewhere. That's where INSTALL_DIR comes into play.

When we originally built riscv-gcc, we built it using the "newlib" makefile target (in riscv-tools/riscv-gcc/Makefile). This "linux" target builds riscv-linux-gcc with glibc (and the Linux kernel headers). Because we now have to build glibc, it will take much more time. If you don't have the power of a 16 core machine with you, maybe it's time to get a cup of coffee.

Building the Linux Kernel (0.40 + ε SBU)

Obtaining and Patching the Kernel Sources

We are finally poised to bring in the Linux kernel sources. Change out of the riscv-tools/riscv-gcc directory and clone the riscv-linux Git repository into this directory: linux-3.4.xx, where xx represents the current minor revision (which, as mid-July 2013, is "53").

$ cd $TOP
$ git clone git@github.com:ucb-bar/riscv-linux.git linux-3.4.53

Building BusyBox (0.26 SBU)

Creating a Root Disk Image

Once you've mounted the disk image, you can edit the files inside. There are a few directories that you should have:

• /bin
• /dev
• /etc
• /lib
• /proc
• /sbin
• /tmp
• /usr
• /bin
• /lib
• /sbin

So create them:

$ cd mnt
$ mkdir -p bin etc dev lib proc sbin tmp usr usr/bin usr/lib usr/sbin

Then, place the BusyBox executable we just compiled in /bin.

$ cp $TOP/busybox-1.21.1/busybox bin

Line 1 mounts the procfs filesystem onto /proc. Line 2 does similarly for tmpfs. Line 3 mounts the HTIF-virtualized block device (htifbd) onto root. Line 4 installs the various BusyBox applet symbolic links in /bin and elsewhere to make it more convenient to run them. Finally, line 5 opens up an ash shell on the HTIF-virtualized TTY (ttyHTIF) for a connection.

Download a copy of the example inittab using this command:

$ curl http://www.ocf.berkeley.edu/~qmn/linux/linux-inittab > etc/inittab

If you would like to use getty instead, change line 5 to invoke that:

Once you've booted Linux and created the symlinks with line 4, they will persist between boots of the Linux kernel. This will cause a bunch of unsightly errors in every subsequent boot of the kernel. At the next boot, comment out line 4.

Also, we will need to create a symbolic link to /bin/busybox for init to work.

$ ln -s ../bin/busybox sbin/init

Add your final touches and binaries to your root disk image, and then unmount the disk image.

$ cd ..
$ sudo umount mnt

"Help! It doesn't work!"

I know, I've been there too. Good luck!

Optional Commands

Depending on your system, you may have to execute a few more shell commands or execute them differently. It's not too useful if you've arrived here after reading the main text of the document; it's best that you're referred here instead.

Building the Full Toolchain (7.62 SBU)

If you want to build riscv-gcc (as distinct from riscv-linux-gcc), riscv-pk, and riscv-tests, then simply run the full build script rather than the abbreviated one I provided.

Optional $ ./build.sh

Installing a Fresh Copy of the Linux Headers

If you (or someone you know) has changed the Linux headers, you'll need to install a new version to your system root before you build riscv-linux-gcc to make sure the kernel and the C library agree on their interfaces. (Note that you'll need to pull in the Linux kernel sources before you perform these steps. If you haven't, do so now.)

First, go to the Linux directory and perform a headers check:

Optional $ cd $TOP/linux-3.4.53
$ make ARCH=riscv headers_check

Once the headers have been checked, install them.

Optional $ make ARCH=riscv headers_install INSTALL_HDR_PATH=$SYSROOT/usr

(Substitute the path specified by INSTALL_HDR_PATH if so desired.)

Installing riscv-linux-gcc to a Different Directory than riscv-gcc

It may be desirable to install riscv-linux-gcc to a different directory. If that is the case, then run these commands instead of the ones prescribed at the end of the section:

Optional $ export RISCV_LINUX_GCC=[/path/to/different/directory]
$ export PATH=$PATH:$RISCV_LINUX_GCC/bin
$ make linux INSTALL_DIR=$RISCV_LINUX_GCC SYSROOT=$SYSROOT

First, set the environment variable $RISCV_LINUX_GCC to the directory in which you want the new tools to be installed. Then, add $RISCV_LINUX_GCC/bin to your $PATH environment variable. Finally, invoke make, specifying $RISCV_LINUX_GCC as the target installation directory.

Using Filesystem in Userspace (FUSE) to Create a Disk Image

If you are unable (or unwilling) to use mount to mount the newly-created disk image for modification, and you also have Filesystem in Userspace (FUSE), you can use these commands to modify your disk image.

First, create a folder as your mount point.

Optional$ mkdir mnt

Then, mount the disk image with FUSE. The -o +rw option is considered experimental by FUSE developers, and may corrupt your disk image. If you experience strange behaviors in your disk image, you might want to delete your image and make a new one. Continuing, mount the disk:

Optional $ fuseext2 -o rw+ root.bin mnt

Modify the disk image as described, but remember to unmount the disk using FUSE, not umount:

Optional $ fusermount -u mnt

Building BusyBox as a Dynamically-Linked Executable

If you want to conserve space on your root disk, or you want to support dynamically-linked binaries, you will want to build BusyBox as a dynamically-linked executable. You'll need to have these libraries:

• libc.so.6, the C library
• ld.so.1, the run-time dynamic linker

If BusyBox calls for additional libraries (e.g. libm), you will need to include those as well.

These were built when we compiled riscv-linux-gcc and were placed in $SYSROOT. So, mount your root disk (if not mounted already), cd into it, and copy the libraries into lib:

Optional $ cp $SYSROOT/lib/libc.so.6 lib/
$ cp $SYSROOT/lib/ld.so.1 lib/

That's it for the libraries. Go back to the BusyBox configuration and set BusyBox to be built as a dynamically-linked binary by unchecking the CONFIG_STATIC box in the menuconfig interface.

• CONFIG_STATIC=n, listed as "Build BusyBox as a static binary (no shared libs)" in BusyBox Settings → Build Options

To make things a little faster, I've used a bit of sed magic instead.

Optional $ cd $TOP/busybox-1.21.1
$ sed -i 's/CONFIG_STATIC=y/# CONFIG_STATIC is not set/' .config

Then, rebuild and reinstall BusyBox into mnt/bin.

Optional $ make -j
$ cd $TOP/linux-3.4.53/mnt
$ cp $TOP/busybox-1.21.1/busybox bin

Appendices

Building the Native Compiler (3.19 SBU)

GCC is a complicated beast, to say the least. As of this writing, the configuration to build a version of RISC-V GCC that runs on Linux/RISC-V is still located on the "native" branch of the riscv-gcc repository. If you desire, read on to build the native compiler...

First, we'll need to make a bigger disk image. A size of 256 MiB seems to be plenty enough. Then, fill it with all of the goodies from the example root image.

$ cd $TOP/linux-3.4.53
$ dd if=/dev/zero of=root-gcc.img bs=1M count=256
$ mkfs.ext2 -F root-gcc.img
$ mkdir mnt-gcc
$ sudo mount -o loop root-gcc.img mnt-gcc
$ cd mnt-gcc
$ mkdir -p mkdir -p bin etc dev lib proc sbin tmp usr usr/bin usr/lib usr/sbin
$ curl http://www.ocf.berkeley.edu/~qmn/linux/linux-inittab > etc/inittab

If you want a better text editor than piping echo to a file, now is the time to rebuild BusyBox with vi. These instructions will assume you've got the original BusyBox binary. Either way, copy it into bin, and set up the init symlink.

$ cp $TOP/busybox-1.21.1/busybox bin/
$ ln -s ../bin/busybox sbin/init

Now, let's build the native compiler. Change into the riscv-gcc repository, clean up the other things, and then check out the "native" branch.

$ cd $TOP/riscv-tools/riscv-gcc
$ make clean
$ git checkout native

We'll need to apply a bothersome and hackish patch to GCC so that SSIZE_MAX is defined. I really didn't want to do this, but it will suffice for now.

$ patch -p1 < native-patches/native-host-linux.patch

In case if you're wondering, native-patches/ also contains augmented configuration files for the MPC, GMP, and MPFR libraries so that it will recognize riscv as a machine. Someday, we hope that we won't have to patch it.

Now, we are ready to build the native compiler. Run the configure script to generate a fresh Makefile, and then invoke make with this command:

$ ./configure
$ make native SYSROOT=$SYSROOT

Note that we've only supplied $SYSROOT, because this compiler only runs within Linux/RISC-V. Therefore, we should place it along with our other files in $SYSROOT.

During the build process, your machine will automatically fetch the sources for GMP, MPC, and MPFR (the GNU Multi Precision Arithmetic Library, the GNU Multi Precision C Library, and the GNU Multiple Precision Floating-Point Reliably Library). It will patch the sources as previously described, and then it will automatically build them, too. (They'll be removed at the end of the native build once it's complete because they will interfere with building the "newlib" or "linux" Makefile targets.)

Once the build is complete, your binaries will be located in $SYSROOT/usr/bin. It would be easy enough to copy out gcc, but there are a bunch of other files that are needed for it to run properly. We'll copy those now.

$ cd $TOP/linux-3.4.53/mnt-gcc
$ cp $SYSROOT/usr/bin/gcc usr/bin
$ cp $SYSROOT/usr/bin/{as,ld,readelf} usr/bin
$ cp $SYSROOT/usr/lib/crt{1,i,n}.o usr/lib
$ cp $SYSROOT/usr/lib/libc.a usr/lib
$ mkdir usr/libexec
$ cp -r $SYSROOT/usr/libexec/gcc usr/libexec
$ cp -r $SYSROOT/usr/lib/gcc usr/lib
$ cp $SYSROOT/lib/ld.so.1 lib
$ cp $SYSROOT/lib/libc.so.6 lib
$ cp $SYSROOT/lib/libdl.so.2 lib
$ cp $SYSROOT/usr/lib/libgcc_s.so* lib
$ cp -r $SYSROOT/usr/include usr/

I could tell you what the importance of these libraries are, but you can learn it yourself: try excluding any of the libraries and see if your program works at all.

Change out of the directory, unmount the root disk image, and then boot the kernel.

$ cd ..
$ sudo umount mnt-gcc
$ spike +disk=root-gcc.img vmlinux

In a short moment, the Linux kernel will boot. If you haven't done anything special to your shell or your user configuration, you'll soon be at a root prompt:

#

Use echo, hopefully built-in to your ash applet, to write out a short "Hello world!" program:

# echo -e '#include <stdio.h>\n int main(void) { printf("Hello world!\\n"); return 0; }' > /tmp/hello.c

Invoke gcc.

# gcc -o /tmp/hello /tmp/hello.c

Wait for a dreadfully long time, and then run your hello world program on Linux/RISC-V.

# /tmp/hello
Hello world!

Congratulations! You've got a native gcc working on your system! If you get this far, think about it. You've used an x86 GCC compiler to compile riscv-linux-gcc, an x86 to RISC-V cross-compiler. Then, you used riscv-linux-gcc to compile the native version of GCC, which runs on Linux/RISC-V to compile RISC-V binaries. If your head isn't spinning from all that meta, you may consider reading Hofstadter.

"Hello world!" compiled and run on Linux/RISC-V and readelf output.

References