8/18/2004
Eric Low

Exactly one year ago today, I started setting up a "Remote login to DOSEMU" setup, so people could log in through SSH2 and securely work in DOS. That exact same box is still up and being used in production by several people each and every day. It has worked almost flawlessly.

However, as it turned out, there became a need for different users to simultaneously log onto the network through unique NIC's... or unique MAC addresses, at least. At first, we only needed two, so I simply popped another network card into the computer. As soon as I ran ipx_interface and added another interface to the second network card (eth1), and proceeded to plug that NIC into the network switch, I immediately started getting all sorts of messages about network collisions streaming down the screen. I quickly unplugged the network cable to eth1, and was suddenly swamped with work trying to get our network back up. Craploads of collisions on your main switch can quickly bring everything to a screeching halt. :( In case you're wondering, it's an old Netgear 24-port unmanaged switch. The old Netgear switches used the same chips as more expensive switches, which was my main factor in purchasing it (way back when even a cheap 24-port switch like this was about $700, and the expensive brands were more like $2000-$3000). So I'm kind of surprised that it didn't handle massive collisions, such as this, much better.

Instead of figuring out what the problem was with using a second interface with the ipx utilities, I decided that there must be a better way. First thing I looked at was tun/tap. I downloaded and installed the rpm of tun/tap, tun-1.1-6.rh72.i386.rpm, from http://vtun.sourceforge.net/tun/, only discover that it was merely the module and did not include any of the user space programs. I looked around and discovered that the program I needed, tunctl, was part of the User Mode Linux (UML) package. I downloaded this from http://umlbuilder.sourceforge.net/walkthrough.shtml and proceeded to read the HOWTO, simply because at first glance, UML seemed extremely invasive.

In laymen's talk, it seemed that UML let you run Linux on top of Linux. Much like an emulator, but less complete and with more setup. Anyhow, installing it involves patching/recompiling your kernel, and then installing a set of user space utilities. For a number of reasons, I decided to set up a new Suse installation, running kernel 2.6. First I did a bare installation, then got ahold of the latest, greatest kernel version (2.6.7) and customized it for my purposes. I installed that new kernel and made sure everything booted ok.

Next, I made a copy of that kernel source directory that I could patch with UML and not screw anything up. I downloaded the matching kernel patch, uml-patch-2.6.7-1.bz2, and copied it to the base kernel source directory (in my case, this was /usr/src/linux-2.6.7-uml/). I then applied it by typing bzcat uml-patch-2.6.7-1.bz2 | patch -p1, which went off without a hitch. Next, I needed to configure the UML kernel, so I typed make menuconfig ARCH=um.

There were now a LOT of UML options on the menus. Some important ones that I made sure were checked included:

It was critical, obviously, to do networking with the outside world using virtual network adapters (the whole point in using UML in the first place. Here is what my UML Network Devices options looked like:

I made sure that TUN/TAP transport was on, but all other transports were turned off. I also made sure that Ethernet network tap was checked under Networking support.

I also turned on Networking support and Honeypot ProcFS on, and Host filesystem OFF under UML-specific options. The latter two, obviously, were for security. Under Character Devices, I turned on pty channel support and tty channel support and DISABLED Legacy (BSD) PTY support. I'm pretty sure (but not positive) that I've only dealt with Legacy PTY support before, so I prepared for a challenge and a little bit of education.

Under Block Devices, I turned off Virtual block device for security reasons. I left most of the other kernel options the same as they were before the uml patch.

Now that I was done, I saved the config, then typed make linux ARCH=um. For a minute it looked like everything would compile ok... but NO! It crashed out with the following error:

As it turned out, this problem was a real bitch to figure out. Most clues pointed to a problem with, or abscence of, kernel headers. I spent a lot of time fooling with glibc-common, glibc-headers, and glibc-kernheaders trying to fix this, and as it turned out, this was way off base.

To fix this problem, first back up your .config file, then run make mrproper from your kernel source directory. This will delete your .config file, so be sure to copy it back, and then run make linux ARCH=um. This problem is only present if you already compiled from this kernel source dir for a normal i386 installation. You MUST run make mrproper in between this ordinary compile and your uml compile.

After I did this, I got much further in the compile, but eventually ran into this new error:

I tried turning on "Do an extra kallsyms pass" under General setup --> Configure standard kernel features (for small systems) (which obviously means that I had Load all symbols for debugging/kksymoops on as well), but this seemed to do nothing.

I took a look inside arch/um/kernel/ksyms.c, and discovered that there were two separate EXPORT_SYMBOL(os_ioctl_generic); lines (sitting near the top, about 10 lines apart). I commented the second one out, and tried compiling again. This time I got plenty of warnings, but it kept going and really looked like it was going to compile. And as it turned out, it did so successfully. There now appeared an executable called linux in the kernel source directory from which I had compiled. Note that this is about a 5MB file, but most of that is simply symbolic information - the actual uml kernel (the part loaded into memory) is actually relatively normal in size. I created a directory called /umlfs, under which I would organize all my uml stuff. After creating this directory, I copied my executable uml kernel there by typing cp /usr/src/linux-2.6.8.1/linux /umlfs.

Next, I extracted the uml_utilities and tried compiling them by typing make all. Here is the error that I got:

I had readline installed, but that didn't seem to have any readline.h or history.h. Some clues pointed to needing the gdb package installed, but after I installed it, I still got the same error.

I looked around, and it appeared that I needed the "GNU readline library." I downloaded the tar.gz, extracted it, and ran ./configure, followed by make and then make install. All of that completed without any problems, so I went back to the uml_utilities directory and typed make all again. This time, everything compiled without a problem!

Now, before I could use UML, I had to either create or download a filesystem that I could boot into. Because I couldn't find a SuSE filesystem or a junior-sized RedHat one, I decided to try a small one I found for slack. The file was about 15MB, but uncompressed to 210MB. Perfectly-sized for my purposes. I downloaded this file and copied it to my /umlfs directory.

Before I could use this filesystem, it had to be mounted. First, I created the directory that this filesystem could be mounted to, by typing mkdir /umlfs/mnt/remote1. If you try mounting without first creating the mount location, mount will give you an error message and die. Now that the destination location was there, I mounted that filesystem by typing mount -o loop -t auto root_fs_slack8.1 /umlfs/mnt/remote1. Notice how I used a type of "auto." I had to do this because I did not know the filesystem type, and luckily it auto-detected it for me. I typed mount -l to list what was currently mounted, and saw that the filesystem type was, in fact, ext2.

This is wrong:

Now that it was mounted, it was time for the big moment. I was going to try loading uml. I tried typing /umlfs/linux udb0=root_fs_slack8.1. For a second it looked like it would mount, but then it died with the following error: VFS: Cannot open root device "98:0" or unknown-block(98,0) Please append a correct "root=" boot option Kernel panic: VFS: Unable to mount root fs on unknown-block(98,0) This was most likely because I did not have an entry for the virtual disk under /dev (/dev/udb0). Rather than adding that entry, I thought it might be easier just to specify the root on the kernel's command line. I tried loading it again by typing /umlfs/linux udb0=/umlfs/root_fs_slack8.1 root=/umlfs/root_fs_slack8.1. It looked like it was going to work, except it decided to go to sleep with the following error: Initializing Cryptographic API sleeping process 3517 got unexpected signal : 11 Since that didn't work, I tried manually adding the proper entry to /dev. I did this by typing the following commands: cd /dev mknod --mode=660 udb0 b 98 0 mknod --mode=660 udb1 b 98 16 chown root:disk udb0 chown root:disk udb1 I tried loading the kernel once again, this time by typing /umlfs/linux udb0=/umlfs/root_fs_slack8.1 root=/dev/udb0. It failed with the following error: nbd: registered device at major 43 Kernel panic: read of switch_pipe failed, errno = 9 sleeping process 4226 got unexpected signal : 11 Since I had just added the proper /dev entries, I tried unmounting and remounting the filesystem before loading linux once again. I did this with the following commands: umount /umlfs/mnt/remote1 mount -o loop -t ext2 /umlfs/root_fs_slack8.1 /umlfs/mnt/remote1 /umlfs/linux udb0=/umlfs/root_fs_slack8.1 I did get a little bit further this time, namely the abscence of the kernel panic error. This is how it failed: nbd: registered device at major 43 sleeping process 5990 got unexpected signal : 11

Now, notice what I did wrong. This is almost too embarassing to put on here, but I don't want other people to waste time on the same mistake. All that was because of a simple spelling error! I typed udb0, when in fact it should have been ubd0!!!

So... First I went back and removed by /dev/udb0 and /dev/udb1 entries that did not belong there.

After a fresh reboot to make sure no files were locked or open, I tried starting uml with the basic command line, /umlfs/linux ubd0=/umlfs/root_fs_slack8.1 (note that this was before I tried mounting anything). I got the following error:

Well now, apparently I've been missing a step! Apparently my filesystem needed to be assigned to a device (/dev/ubd/0), which I hadn't been doing. I asked Ben, my friendly Linux expert, how I could accomplish this. He told me to look into mkfs. However, after looking at the man file, it appeared that you could only use mkfs when creating a brand new filesystem. That was not what I wanted to do. I explained my problem to him further, and he suggested using /dev/loop0 if I needed to see it as a device. I had been doing that earlier when using mount, so I went back and examined this possibility.

As I was going back to examine this possibility, I had a big argument with Ben about whether or not uml did, in fact, assign this filesystem to the /dev/ubd/0 device on its own. According to the HOWTO, it did.. but the HOWTO also said that uml used /dev/ubd0 rather than /dev/ubd/0. I had already tried the prior, with no avail. There was obviously a discrepancy in the HOWTO at some point! For shits and giggles, I tried creating the /dev/ubd/0 through /dev/ubd/1 devices manually and running uml again. I tried with the same command line, /umlfs/linux ubd0=/umlfs/root_fs_slack8.1, but got the exact same error message. At any rate, they should really fix the HOWTO. I deleted the /dev/ubd directory and moved on.

As it turned out, I was messing with all this /dev/ubd crap in the wrong place! What you need to do is create this directory under the /dev directory of the filesystem you wish uml to use. Here are the commands I used:

Once that was done, I simply loaded uml by typing ./linux ubd0=/umlfs/root_fs_slack8.1. Voila! Now fix those damned udb names in the HOWTO!

Next, I had to configure the networking. The uml_utilities package that I had compiled above included a little utility called tunctl to create tun/tap virtual network interfaces. I was interested in tap devices, since they support all ethernet frames, rather than tun devices, which only support IP frames.

Before I went on, I knew that once I set up a tap interface, I would need a way to let it see the outside world. To do this requires the bridge utilities, so I found these at, and downloaded them from, http://bridge.sourceforge.net/download.html. After I extracted the archive, I entered the directory and typed ./configure, followed by make and then make install. I now had the bridge utilities installed.

Note that I already had bridging enabled in the kernel (Device Drivers -> Networking Support -> Networking Options -> 802.1d Ethernet Bridging). I actually compiled it in, but you can also enable it as a module and it will be automatically loaded the first time you run brctl.

Now that I had the proper tools, I typed the following commands to put the network (both real and virtual) up:

Note that if you wanted to assign the tap interface to a specific user, you would type tunctl -u 100, where 100 was the uid of that user. In this case, I also didn't specify an IP address for tap0 because I was only going to use IPX from within these uml sessions. If you do this, you MUST manually assign a MAC address when loading UML! This is because UML normally assigns the MAC based on the IP address, using fe:fd:nn:nn:nn:nn where nn.nn.nn.nn is the device IP address. And since every UML session/virtual interface must have a unique MAC, it will cause problems if they are all FE:FD:00:00:00:00! Note that if you manually assign it, you should probably make sure that the first byte is EVEN, otherwise it will be interpreted as a broadcast address. Also, when you set up a bridge like that, make sure you set up your gateway configuration once again, like I did on that last line.

Once I had run those commands, I could see eth0, tap0, and remote1 when I typed ifconfig. Neither eth0 nor tap0 had an assigned IP, but remote1 did, and therefore I could still access the network from the local machine using IP. I saw that tap0 was raking up dropped TX packets, so at least it was doing something! I decided to try running uml, except this time specifying the interface that would appear inside, as well as the corresponding MAC address. To do this, I typed /umlfs/linux ubd0=/umlfs/root_fs_slack8.1 mem=64m eth0=tuntap,tap0,fe:fd:aa:aa:aa:01.

Once inside uml, I typed ifconfig eth0 up to put the interface online. I could then see it, and guess what? It was receiving packets! :) In Slackware, the network card configuration is stored in the /etc/rc.d/rc.inet1 script file. I looked at the script, and it appeared that the network cards were never actually put online unless the IP protocol was used. If you look at the default sections that are supposed to put up eth0 or eth1, they will be completely skipped unless you either set it to use DHCP or change the IP address from the default 127.0.0.1 or even a blank. So, unless you use IP, the network will never come online! So, I simply added the line /sbin/ifconfig eth0 up to the script right after the section where it sets up the loopback interface. I shut down that window, and restarted uml. This time, eth0 came up as it was supposed to, and was once again receiving packets.

Now, I had to remember how to do IPX. I downloaded and installed the latest ipxutils package.

I needed to install ipxutils inside of uml as well, but Slackware does not use rpm, of which type the package type I had was (Isn't passive great?!). Instead, I typed rpm2tgz ipxutils-2.2.3-1.i386.rpm to convert it to a .tar.gz file, then installed it by typing installpkg ipxutils-2.2.3-1.i386.tgz. That all went off without a hitch, and now I had ipxutils installed. To start the network, I created a script called ipx, copied it to the /etc/rc.d directory, and added a line which ran this script to the /etc/rc.d/rc.inet1 script. My ipx script contained the following lines:

This first adds 802.2 IPX to the eth0 interface, then manually adds a route to our Netware server. The latter is required unless you run ipx_route as root. When I tried running ipx_interface, it said I needed glibc-2.3, which apparently Slackware didn't have installed. So, I downloaded that in .rpm format, intending to convert it to .tgz by typing rpm2tgz glibc-2.3.3-98.i586.rpm. However, when I tried this, it always failed with the "gzip: stdin: unexpected end of file" message. So, I went to the slackware site and downloaded the actualy 2.3.2-7 glibc tgz file that was meant for Slackware. That one took a while with installpkg, but installed without a hitch.

Now that glibc was in place, I ran my ipx script. No problems there. I ran ifconfig, and presto! 802.2 was there. Now I needed DosEMU. I downloaded the latest developer release and untarred it while in uml. I edited compiletime-settings and changed the following lines:

I then tried running ./default-configure, but it quickly failed because apparently no C compiler was installed. I downloaded gcc and installed it by typing installpkg gcc-3.3.4-i486-1.tgz. Once again, ./default-configure failed, but this time it looked like I ran out of room on my filesystem image! I backed up the filesystem, then ran the following commands to resize it:

I tried starting UML, but this time it failed with the following error:

I wasn't sure if my resizing process was bad, or if I had corrupted something when I fixed the file with e2fsck beforehand. I decided to run uml from my backup once again. Guess what? It halted with the same error! I ran e2fsck again, but this time did not fix the following error:

I ran uml again, and this time it ran just fine. I exited out by running halt, which I had not been doing before. After exiting, I ran e2fsck again, and got no errors. Guess just killing the window to get out of uml wasn't doing my filesystem any good <grin>. Now that the filesystem was ok, I ran dd and resize2fs once more. Presto! My filesystem was now 300MB.

I ran uml once again and tried compiling dosemu. It failed once again, and when I checked config.log, I found the following error: gcc: installation problem, cannot exec `as': No such file or directory. I looked at the man page for as and it looked like it was part of the binutils package. I downloaded this and installed it by typing installpkg binutils-2.15.90.0.3-i486-1.tgz. I ran ./default-configure once again, and this time got the following error:

After looking around on the web, it appeared I needed gcc-g++ installed as well. So I downloaded and installed that, then tried again. Still the same sanity error. I also installed glibc-g77 and glibc-solibs (The first one is the Fortran compiler, which wasn't really needed, but I figured it couldn't hurt), but those also did no good. I also upgraded to glibc-2.3.3, since I know 2.3.2 has some bugs. Still the same error!

A closer look at config.log revealed that the compiler was looking for a file called /usr/include/linux/limits.h. I did a search for this file, and it existed under the /usr/include directory. The /usr/include/linux directory did not exist at all! So, I simply typed ln -sv /usr/include /usr/include/linux to create a symbolic link (essentially a virtual linux directory under /usr/include/ that in fact pointed to /usr/include). No more error!! :) It did, however, yell at me for not having lex or flex installed. I downloaded and installed flex and tried again. Now it yelled at me for not having bison, so I downloaded that as well.

Now when I compiled, I got the following error:

Ok, so my symbolic link wasn't perfect <grin>. :) I decided to do it right this time... I removed my symbolic link by typing rm /usr/include/linux and installed the glibc-kernheaders package instead. This, of course, created and populated the /usr/include/linux directory as it should be. This time, everything worked! I did go back and install a few things that it complained about, such as perl, however. I knew I would need perl anyhow. After I ran ./default configure, I tried running make and discovered that I didn't have the make package installed. I installed it and tried again, but now got the following error:

I looked around on the web, and read somewhere that perhaps my version of bison was too new. I downgraded and ran make again, and this time everything went off without a hitch! So in a nutshell, here's how to compile/install DOSEmu:

Once that was installed, I tried running dosemu. On the first run, it asks you for the location of a bootable DOS, which I already had set up as /dosemu/freedos, and then it made me accept the license and all that. However, after that was all entered, it crashed with the following error:

I looked at my /etc/fstab file, and it did contain the line, "none /proc proc defaults 0 0" which is correct. I did a grep on the dosemu source, and it looked like my problem was most likely in src/base/init/config.c when it looks for the line starting with "cpu family :" in /proc/cpuinfo.

What I did first to try to trick dosemu, however, is simply hex edit the dosemu.bin file and change the /proc/cpuinfo to /eros/cpuinfo. I then created the directory, /eros, and copied and pasted the /proc/cpuinfo file into /eros/cpuinfo. I tried running dosemu, and got an error that read:

I searched around and found that, at least in older versions of DOSEmu, vm86plus enhancements could be disabled. I tried compiling DOSEmu that way by typing ./default-configure --enable-novm86plus and then running make. I then hex edited 2.3.1.0/bin/dosemu.bin and changed the /proc/cpuinfo text once again. To put the new executables in place, I ran make install. Unfortunately, it appeared that this option no longer worked. :( Same error! I also tried simply running ./configure --enable-novm86plus followed by make and make install, which appeared to work ok and accept the flag in dosemu-1.3.1, but still got the same error.

Well, after spending all this time messing with User-Mode-Linux, I was seriously thinking that it wasn't the answer. It is incomplete, as is shown by its failure to implement real mode (vm86). They say that UML is great for testing purposes, but I find that hard to believe since it is missing such a basic part of the kernel. Granted, it's mostly in there for DOSEmu, but it has been integrated for quite some time!

It was then that I began to consider other options, such as Plex86.

./default-configure --target=athlon-pc-linux-gnu

I went to run slist to list out the available servers and make sure it could see them, but apparently that utility wasn't part of the ipxutils suite. So I downloaded ncpfs and installed that as well.

If you build your own kernel, and want to boot it from one of the filesystems distributed from this site, then, in nearly all cases, devfs must be compiled into the kernel and mounted at boot time. The exception is the SuSE filesystem. For this, devfs must either not be in the kernel at all, or "devfs=nomount" must be on the kernel command line. Any disagreement between the kernel and the filesystem being booted about whether devfs is being used will result in the boot getting no further than single-user mode.

I had to look around for quite a while for this one! I read somewhere that /lib/modules/2.6.7/build should be a symlink to /usr/src/linux-2.6.7 (with the numbers matching your kernel, of course). I had, however, renamed that directory to /usr/src/linux-2.6.7, thereby breaking the link. Soooooo, I repaired it! Well, I still got the same error running make, but I'm sure that little repair didn't hurt anything <grin>. I had a feeling this had to do with my not having kernel-headers installed.. but if you've ever tried to find a matching kernel-headers rpm, you know how fucking impossible they are to find! It's like they don't want you to find them!!!

Well, it turns out that they match the kernel headers up with glibc and not the kernel itself. So, I installed the following packages:

What I needed was in eather glibc-kernheaders or glibc-headers, but I installed them both just for good measure.

I was still running the 2.4.21 kernel on this machine, so I downloaded the matching kernel patch, uml-patch-2.4.21-1.bz2. I then made a backup of my existing kernel tree by typing mkdir /usr/src/linux-2.4.21-uml and then cp -r -p --preserve=all /usr/src/linux-2.4.21/* /usr/scr/linux-2.4.21-uml. Once that copied, I applied the patch by copying the patch to that directory and from there, typing bzcat uml-patch-2.4.21-1.bz2 | patch -p1. That went off without a hitch, so I ran make menuconfig ARCH=um, which brought me into the kernel config but without most of the usual kernel options being displayed. Of course, there were plenty of new options too, and most of them looked pretty fucking sweet. I was immediately geeked about UML and what it must be capable of. I turned off a bunch of stuff that I didn't need, but made sure to turn on COW device because I thought I might need it and Universal TUN/TAP device driver support for obvious reasons. After I saved the new config, I ran make linux ARCH=um

downloaded uml_utilities_20040406.tar.bz2

Note: to use TUN/TAP, the kernel options required are "802.1d Ethernet Bridging" (in "Networking options") and "Universal TUN/TAP device driver support" (in "Network device support").

More to come...