title: Debugging the boot filesystem environment seen by syslinux?
body: <p>Hope it's OK to jot this down, even if I cannot accept answer immediately (and hope it's OK for SO - as there is a C patch below):</p>
  
  <p>It seems I screwed up the hard disk on my desktop PC (<code>{DRDY err}</code>). So I wanted to run a bootable media to run <code>fsck</code>, but the CD on this desktop is broken, so I can only use USB flash. I have a couple of USB thumbdrives with Ubuntu and Suse - these start booting on the desktop; but during boot, udev tries to detect hard drives, and since the hard disk is screwed, it just loops there, and the respective OS never finishes booting.</p>
  
  <p>So I tried to download <a href="http://www.sysresccd.org/Download" rel="nofollow noreferrer">SystemRescueCd</a>; I have this USB thumbdrive, on which I tried to install SystemRescueCD:</p>
  
  <pre><code># lsusb with sudo, to retrieve all info
  $ sudo lsusb -v -d 058f:6387 | grep -i 'id\|iManufacturer\|iProduct\|iSerial\|bInterface'
  Bus 001 Device 043: ID 058f:6387 Alcor Micro Corp. Transcend JetFlash Flash Drive
    idVendor           0x058f Alcor Micro Corp.
    idProduct          0x6387 Transcend JetFlash Flash Drive
    iManufacturer           1 takeMS
    iProduct                2 Mem-drive Mini
    iSerial                 3 C5E7F0CC
        bInterfaceNumber        0
        bInterfaceClass         8 Mass Storage
        bInterfaceSubClass      6 SCSI
        bInterfaceProtocol     80 Bulk (Zip)
  
  # search by serial:
  $ find /dev/disk/by-id/ -name '*C5E7F0CC*'
  /dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0:0-part1
  /dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0:0
  
  # list and get device node
  $ ls -la /dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0:0
  lrwxrwxrwx 1 root root 9 2013-03-25 20:37 /dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0:0 -&gt; ../../sdc
  $ ls -la /dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0\:0-part1
  lrwxrwxrwx 1 root root 10 2013-03-25 20:37 /dev/disk/by-id/usb-takeMS_Mem-drive_Mini_C5E7F0CC-0:0-part1 -&gt; ../../sdc1
  
  # it is /dev/sdc - list disk info
  $ sudo fdisk -l /dev/sdc
  
  Disk /dev/sdc: 2108 MB, 2108686336 bytes
  94 heads, 29 sectors/track, 1510 cylinders
  Units = cylinders of 2726 * 512 = 1395712 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disk identifier: 0x0003e405
  
     Device Boot      Start         End      Blocks   Id  System
  /dev/sdc1   *           1        1511     2059263+   c  W95 FAT32 (LBA)
  </code></pre>
  
  <p>I tried to use my Ubuntu 11.04 Natty netbook to image the thumbdrive - and I used both</p>
  
  <ul>
  <li>the recommended <a href="http://www.sysresccd.org/Sysresccd-manual-en_How_to_install_SystemRescueCd_on_an_USB-stick#Execute_the_installer" rel="nofollow noreferrer">usb_inst.sh installer</a>; and</li>
  <li>I tried to use <code>unetbootin</code> (via <code>sudo apt-get install unetbootin</code>);</li>
  </ul>
  
  <p>in both of these cases, when I try to boot the USB thumbdrive on the desktop, the boot procedure fails with:</p>
  
  <pre><code>SYSLINUX 4.02 debian-20101016 CHS Copyright (C) 1993-2010 H. Peter Anvin et al
  ERROR: No configuration file found
  No DEFAULT or UI configuration directive found!
  boot:
  </code></pre>
  
  <p>.... with prompt at boot. (In fact, <code>unetbootin</code> fails at "Verifying DMI Pool Data", before entering <code>syslinux</code> - probably because it is much older than the .iso I'm trying to image).</p>
  
  <p>First I checked the md5 as mentioned in <a href="http://www.sysresccd.org/forums/viewtopic.php?f=14&amp;t=3055" rel="nofollow noreferrer">No Default or UI Configuration Found!</a></p>
  
  <pre><code>$ md5sum ./systemrescuecd-x86-3.5.0.iso
  48552b9e905872bd5061eb112b73ea20  ./systemrescuecd-x86-3.5.0.iso
  </code></pre>
  
  <p>... but it seems OK, as per <a href="http://www.sysresccd.org/Sysresccd-versions" rel="nofollow noreferrer">Sysresccd-versions</a>.</p>
  
  <p>Then I tried to reformat the drive to FAT16 (via <code>sudo gparted /dev/sdc</code>); and repeated both <code>usb_inst.sh</code> and <code>unetbootin</code> methods - again no dice. Funny enough, in all of these cases, if I try to run the flash USB thumbdrive in the QEMU emulator:</p>
  
  <pre><code># sudo apt-get install qemu
  sudo qemu -hda /dev/sdc
  </code></pre>
  
  <p>... it boots fine - showing the syslinux menu and so on; however, boot always fails on the desktop.</p>
  
  <p>Here I should mention, that I could write down the following from the boot screen of the problematic desktop PC:</p>
  
  <pre><code>Award Modular BIOS v6.00PG
  AMDRS740 BIOS
  </code></pre>
  
  <p>It has a boot menu accessed via <kbd>F12</kbd>, and in the boot menu, among other options, these are for USB:</p>
  
  <pre><code>...
  USB-FDD
  USB-ZIP
  USB-CDROM
  USB-HDD
  ...
  </code></pre>
  
  <p>Typically, I choose USB-HDD - but I've tried the others; either the procedure freezes before even entering <code>syslinux</code> - or the boot fails as described above.</p>
  
  <p>There is advice to rename directories/files manually from <code>isolinux</code> to <code>syslinux</code> (<a href="https://askubuntu.com/a/128168">Trying to boot from usb - Ask Ubuntu</a>) - when I used <code>usb_inst.sh</code>, only <code>syslinux/isolinux.bin</code> would have to be renamed. There is also advice to copy <code>syslinux.cfg</code> to the root of the USB flash thumbdrive (<a href="https://superuser.com/a/182064/39752">Cannot boot Live USB, Linux - Super User</a>). But still no improvements - <code>syslinux</code> is still complaining that it is missing the configuration file - which apparently is the <code>syslinux.cfg</code>.</p>
  
  <p>Then I tried to look if it is possible to somehow "debug" syslinux; found <a href="http://reboot.pro/topic/15205-log-tracingdebuggingtrouble-shooting-in-syslinux/" rel="nofollow noreferrer">log tracing/debugging/trouble shooting in syslinux - The Syslinux Project - reboot.pro</a>:</p>
  
  <blockquote>
    <p>> Do we have specific commands to trace or log syslinux?<br>
    Being open source, one is able to compile Syslinux and enable extra debugging output.</p>
  </blockquote>
  
  <p>also <a href="http://ubuntuforums.org/archive/index.php/t-2020351.html" rel="nofollow noreferrer">[SOLVED] Stuck on boot: Syslinux Problem [Archive] - Ubuntu Forums</a>: "_
  Debugging syslinux is described at <a href="http://www.syslinux.org/wiki/index.php/Development/Debugging" rel="nofollow noreferrer">http://www.syslinux.org/wiki/index.php/Development/Debugging</a> , but effective debugging (if I recall correctly) requires recompiling it to add the debug hooks._". However, <a href="http://www.syslinux.org/wiki/index.php/Development/Debugging" rel="nofollow noreferrer">Development/Debugging - Syslinux Wiki</a> talks about something called <code>bochs</code>; and I suspect that is to debug <code>syslinux</code> itself - not necessarily to "debug" (or query) the <em>environment</em> it is in.</p>
  
  <p>Anyways, at last, I could see no way out but to get <code>syslinux</code> from source; basically, this was needed so it builds:</p>
  
  <pre><code>sudo apt-get install nasm
  sudo apt-get install uuid-dev
  git clone git://git.kernel.org/pub/scm/boot/syslinux/syslinux.git syslinux-git
  cd syslinux-git/
  make OPTFLAGS+=-DDEBUG=1
  </code></pre>
  
  <p>Turns out, it isn't really clear how to enable such debugging, that will show what <code>syslinux</code> "sees" when plugged in a given computer; given that I do load into <code>syslinux</code> at boot, the problem is what does it see as a filesystem. I tried to enable the DEBUG environment variable as shown above (after adding <code>override OPTFLAGS :=</code> to the <code>Makefile</code>) - but that, in itself, generated no new messages during boot failure. I have used the following command to "burn" the USB thumbdrive (after unmounting it from the Gnome applet):</p>
  
  <pre><code>sudo ./linux/syslinux --stupid --directory /syslinux --install /dev/sdc1
  </code></pre>
  
  <p>... and I've tried both with <code>stupid</code> and without (and both for the source-built version, and the one from the Ubuntu package repositories for Natty).</p>
  
  <p>Grepping through the source, I realized there is something called <code>rosh</code> (<a href="http://www.syslinux.org/wiki/index.php/Read-Only_SHell(rosh.c32)" rel="nofollow noreferrer">Read-Only SHell</a>) - however, that compiles as a <code>rosh.c32</code> - and one is supposed to have it as a boot kernel option in <code>syslinux.cfg</code> - which, as noted, I cannot load. So <code>rosh.c32</code> is unfortunately not much help for my problem.</p>
  
  <p>However, given that <code>rosh</code> implements the <code>ls</code> command, I tried to copy relevant portions into the code of <code>syslinux</code> - and trigger a <code>ls /</code> listing of the root when <code>syslinux</code> scans for the configuration file. With those changes, recorded in <a href="http://sdaaubckp.sourceforge.net/patch/syslinux-e40ba60-rosh-ls.patch" rel="nofollow noreferrer">syslinux-e40ba60-rosh-ls.patch</a>; now I get the following when I boot:</p>
  
  <pre><code>SYSLINUX 4.06 CHS 5-ge40ba60* Copyright (C) 1993-2010 H. Peter Anvin et al
  Listing: "/"
  rosh_ls_arg_dir 0 files found
  Listing: "/syslinux"
  Listing: ""
  CurrentDirName: "/syslinux/"
  confignamebuf: /syslinux/extlinux.conf; realpath -1
  confignamebuf: /syslinux/syslinux.cfg; realpath -1
  confignamebuf: /boot/syslinux/extlinux.conf; realpath -1
  confignamebuf: /boot/syslinux/syslinux.cfg; realpath -1
  confignamebuf: /syslinux/extlinux.conf; realpath -1
  confignamebuf: /syslinux/syslinux.cfg; realpath -1
  confignamebuf: /extlinux.conf; realpath -1
  confignamebuf: /syslinux.cfg; realpath -1
  ERROR: No configuration file found
  No DEFAULT or UI configuration directive found!
  </code></pre>
  
  <p>Interestingly; for the root <code>/</code>, the <code>_ls</code> function at least returns "0 files"; the others ("/syslinux", and the empty string "") already fail at the <code>opendir</code> call - and so the <code>_ls</code> function doesn't even get called!</p>
  
  <p>I would have thought that my slapstick copying of the <code>ls</code> function would not work as intended; but running the thumbdrive in <code>qemu</code> on netbook, does in fact provide a full listing of files - and given that at least for <code>/</code>, the function gets called and returns on the desktop - I'd suspect that it does indeed work.</p>
  
  <p>However, that still doesn't solve my problem - <em>why</em> does <code>syslinux</code>, after boot, see 0 files under the root <code>/</code>? What else could I do to debug this problem? I wouldn't mind patching some C code into <code>syslinux</code> - but I just don't know what I should be looking for, that would point me to correct preparation of the USB thumbdrive for booting on the desktop machine...</p>