Expanding an VirtualBox XP system disk

After spending an extraordinary amount of time trying to figure out why I couldn’t get NTFSResize to work, I finally found Seppe vanden Broucke’s log entry. For posterity, I’ll record what I did, here, following partly the instructions in the original post and one commenter’s modification:

  1. Get the SysRescCD ISO image. (Any live linux CD should work, including Knoppix.)
  2. Create a new (larger) VirtualBox disk image (*.vdi) from within VirtualBox.
  3. In the VirtualBox Details tab for your Windows instance, set your original VDI as the IDE Primary Master.
  4. Set the new (empty) VDI as the IDE Primary Slave.
  5. And set the linux ISO as the CD/DVD IDE Secondary Master.
  6. Ensure that the instance is set to boot from the CD/DVD first in System->Boot Order.
  7. Boot the instance.
  8. At the command line, run
    fdisk -l /dev/sda
    You should see a partition there of type 7, NTFS.
  9. Then run
    fdisk -l /dev/sdb
    You should see an error message that says there is no partition on that device.
  10. Then run the command:
    dd if=/dev/sda of=/dev/sdb
    This will take a long time. It means copy from the input file /dev/sda to the output file /dev/sdb.
  11. Shutdown the system with:
    shutdown -h now
  12. Change the Storage->IDE Secondary Master (CD/DVD) from the ISO image to your regular optical drive, or remove it completely so that it will boot into the original (small) Windows image.
  13. In Windows, select Start->Run… and type:
    diskpart.exe
  14. Type:
    select disk 1
    to select the (large) disk.
  15. Type:
    list disk
    to see that you’ve selected the right one with the right size.
  16. Type:
    select partition 1
    to select the 1st partition on that disk.
  17. Type:
    list partition
    to see that you’ve selected the right partition.
  18. Type:
    extend
    to extend the volume all the way to the end of the device.
  19. Turn off the virtual instance and select the new (larger) VDI as the Storage->IDE Primary Master.
  20. Boot the instance again and you should have a larger system (C:) disk.

98-all-your-atmel-belong-to-us

Foobarred is Normal

Recently I started using my linux laptop as my primary avr-usb development environment. When I upgraded the laptop to the current Ubuntu-LTS release (10.4 aka Lucid lynx) a bunch of stuff was broken including all of the wonderful udev rules provided by my linux savvy friends at dorkbotpdx.

I mean jeesh even lsusb was broken.

$ lsusb -vd "03eb:"|grep iM
cannot read device status, Operation not permitted (1)
iManufacturer           1

When I started looking at what changed and how to adjust, the web dead ended to a lot of threads like the one at http://ubuntuforums.org/showthread.php?t=1360412 ; where two people declared that having to escalate privileges to root in order to talk to a user device was “normal”. Even when the developer said it wasn’t normal the ubuntu folk redeclared it normal (apparently after Bush you just have to repeat something blatantly stupid for it to be true). As an administrator the last thing you want is everything and its dog requiring root privileges.

Hanging around the #ubuntu channel was a lot like having people repeat the searches on the web that provided me with the same dead ends that I joined the channel trying to resolve.

Cherchez la femme (look at the squeeze)

This was getting stupid. All I wanted to do was to have devices that I could plug in and program and then communicate with them using ruby or perl or some other haphazzardly thrown together scripts without having to be root. Then I realized once again that ubuntu is really focused on making the users life easier and that this leads to a lot of non technical help. So I asked my friends what the nick name was for the Debian release that was the basis for the Ubuntu release nick named “Lucid” and then re did all of my dead ended web queries replacing “Ubuntu Lucid” with “Debian Squeeze”.

The results were heavy on the technical detail and light on social skills.

More importantly I quickly found the solution that I needed in the middle of this link (http://git.zerfleddert.de/cgi-bin/gitweb.cgi/usb-driver?a=blob_plain&f=README&hb=HEAD) what I was missing was the difference between the new and the old udev rules.

“If you are running a newer version of udev (as in Debian Squeeze and
Ubuntu 9.10), you need to adapt the rules-file to the new udev-version:
sed -i -e ’s/TEMPNODE/tempnode/’ -e ’s/SYSFS/ATTRS/g’ -e ’s/BUS/SUBSYSTEMS/’ \
/etc/udev/rules.d/xusbdfwu.rules”

One line of sed was all I needed and it had taken me a week of asking about lucid when i should have been asking about squeeze. With this I also was able to find the changes to the lay out of the /dev and /proc trees and the new tools to monitor udev and diagnose issues.

All-Your-Atmel-Belong to Us

I really just wanted to change the permissions so I could use my devices. Adding the following udev rule to your system will do just that.  You will also need to restart the udev service. Both of these will require you to be root.

# cat >/etc/udev/rules.d/98-all-your-atmel-belong-to-us.rules<<EOF
#------------------------/etc/udev/rules.d/98-all-your-atmel-belong-to-us.rules
#
# Make atmel devices (dfu, LUFA, obdev) accessible in userland
# 
ACTION=="add", SUBSYSTEMS=="usb", ATTRS{idVendor}=="03eb", MODE="0666"
EOF
# service udev restart

To actually own all my atmel I could have also added ‘ ,USER=”don” ‘ after the mode part but I just wanted to talk my Atmel usb-avr devices and the open source firmware I was developing. And the above rule made things “just work”.

$ lsusb -vd "03eb:"|grep iM
iManufacturer           1 mycompany.com
$

This rule fixed all of my LUFA based devices including the my open source arduino programmer as well as the dfu programmer.used to code them. Hopefully it will be a while before I have to go through this again.

More Sinatra from the Ubuntu Folk.

I remember when the kids decided that they wanted to adapt the unnecessarily complicated system V init system and I kissed the simple days goodbye (sometimes I miss early bsd/freebsd and sometimes i miss it alot). Nowadays even the sysv-init isnt complicated enough. We gotta have upstart.

So after upgrading two systems to Lucid, I can tell you that most of the details have been well paid attention to, and a few problems we were having with 9.10 and hardy actually got solved. But there is still stuff that wasn’t broken that the kids just had to fix. So without further adeau I will tell you how to get gdm (and X) to go away after you have installed sysv-rc configured it to runlevel 3 (multiuser - no x) and its still there.

Setting the runtime.

If you have a working inittab the existing scripts will reference it so you can skip to how to get rid of gdm.

if you don’t you will find that the default runlevel is set to 2 (linux’s ‘user defined’ graphical runlevel since rc5 wasnt good enough for some reason) in /etc/init/rc-sysinit.conf. This should be changed to look like this.

# Default runlevel, this may be overriden on the kernel command-line
# or by faking an old /etc/inittab entry
env DEFAULT_RUNLEVEL=3

rebooting and typing “runlevel” should verify the change.

Why is gdm/X still running?

Beats the begebes out of me. But if you want it to stop you need to edit the /etc/init/gdm.conf and comment out the “start on” conditions and change them so that they correspond to the appropriate runlevel.

#start on (filesystem
#          and started dbus
#          and (graphics-device-added fb0 PRIMARY_DEVICE_FOR_DISPLAY=1
#               or drm-device-added card0 PRIMARY_DEVICE_FOR_DISPLAY=1
#               or stopped udevtrigger))
start on runlevel [24]
stop on runlevel [0136]

Grubby.

If you don’t want gdm/x you probably don’t want the splash junk either. While /boot/grub/menu.lst allows you to set default options like nosplash every kernel upgrade i have done in the past year explicitly asks for quiet and splash way out at the end of each “kernel” line so just plan on hand editing the darned thing after each kernel update. I am sure you can uninstall splash but I prefer configuring things to keeping track of what needs to be uninstalled for a system to run correctly.

Damned Kids.

Hope this saves you some of your precious hair. All and all lucid (server and desktop) seems pretty sane and stable but they are still fixing things whether or not they were broken.

Dont even get me started on last weeks apparmor configuration update busting our dns.

Tracing Multiscale Mechanisms of Drug Disposition in Normal and Diseased Livers

Our second ISL tracing paper has been published online as a JPET FastForward Article Tracing Multiscale Mechanisms of Drug Disposition in Normal and Diseased Livers.

I’ll repeat the brief blurb on my agent based modeling website.

This paper talks in relative detail about how hypothesis formulation and falsification (failed validation) can be done at a fine grain when only coarse grained validation data is available. Because our in silico liver (ISL) is an analog built in software, we can trace its internals. And because the internals of the analog were designed to map to the internal structure and dynamics of its referent (wet-lab liver perfusion experiments), traces of the ISL become detailed hypotheses about the internals of the liver. However, those detailed hypotheses are not falsifiable, except to the extent that they fail to reproduce the coarse validation data. Nothing can be done about that until we design wet-lab experiments to perform on real livers. In the meantime, though, we can alter the ISL mechanisms so that the coarse grained data matches that taken from wet-lab experiments under different conditions. In this case, we build 3 ISLs that generate the outflow profiles for drug and a sucrose marker for: 1) normal healthy livers, 2) alcohol damaged livers, and 3) carbon tetrachloride damaged livers. With the traces for each of the 3 ISLs, based upon the validated (i.e. not proven true, of course, but proven true enough) mechanisms of the ISL, we can formulate ‘proto-theories’ for the translation of an experimental liver from a healthy to a diseased (cirrhotic) state.

Note that the particulars of the ‘proto-theories’ suggested by these traces are not as sophisticated as those that might be generated by an expert hepatologist. In fact, these ‘proto-theories’ may even seem bizarre or patently false to such an expert (though I believe they don’t seem so to the experts). Indeed, as Box’s aphorism says,

… all models are wrong; the practical question is how wrong do they have to be to not be useful.

The point is not to build computer programs that attempt to compete with the hypothesis formulation of experts. This is not an AI project. The point is to build devices, with whatever tools are available including computers, that make the experts more efficient and effective. By formulating these ‘proto-theories’ about the translation of healthy livers to diseased livers (and vice versa), models like the ISLs provide a foil or sounding board to help sharpen the theories developed by the experts.

Reprogramming your avr-usb device using atmel’s built in bootloader

Getting code onto the MidiMonster or Benito device.

Midi Monster Button Locations.

Midi Monster Button Locations.
Benito 7g Switch Positions.

Benito 7g Switch Lcations
Benito 2010 Switch Locations

Benito 2010 Switch Locations
Benito Without Buttons.

Benito Without Buttons

All of the code on the Benito and MidiMonster devices is open source and references an open source library called the Lightweight Usb For Avr (lufa). Getting the code compiled and onto the device requires a few other open source tools.

AVR-GCC

The most current and stable release of the toochain for the AVR has untill recently been maintained by Eric Wedddington and released as WinAvr (http://sourceforge.net/projects/winavr/) Winaver integrates nicely into atmels avr studio http://www.atmel.com/dyn/Products/tools_card.asp?tool_id=2725 and I reccomend that you get both if you are running windows. Each Winavr Release is closely followed by objective developments CrossPack for avr http://www.obdev.at/products/crosspack/index.html and a script for building the current toolchain on linux which is hosted by AvrFreaks at present there is also a debian package that was put out last month http://www.avrfreaks.net/index.php?name=PNphpBB2&file=viewtopic&t=90172

HWB? DFU?

Most Atmel “atmega” devices have a pin dedicated to determining whether or not the device goes into the bootloader depending on the devices flag settings. This pin is labled HWB (for hardware boot) on most of the datasheets. The usb avr family has this mode set up by default. When you hold the hwb pin low and reset the device it goes into the bootloader. On these devices the built in bootloader uses a usb device class called the Device Firmware Uploader (DFU). Atmel provides a tool called flip for programming DFU devices on windows and linux. There is an open source programmer called dfu-programmer http://dfu-programmer.sourceforge.net/

That other guys stuff.

Those of you who have worked with the teensy boards from paul stoffregon will notice some subtle differences. Paul wrote his own (closed source) bootloader rather than use the bootloader tha comes installed on the chips. Then he uses a hardware trick similar to the auto reset hack to make a single button manipulate both the reset and the hwb pins to put his boards into the bootloader. I didnt feel that either the two button arrangement or the builtin bootloaders were broken so I like most people working with these chips dont fix them.

Using the dfu-programmer

All of the programs in Dean Cameras Lightweight Usb for Avr have a “dfu” target. Once you have the target into the DFU mode you can simply

$make dfu

This will cause the dfu-programmer to erase the flash reprogramm it with a new hex file and restart the chip. You can also do this manually with the following commands

$dfu-programmer atmega32u2 erase
$dfu-programmer atmega32u2 flash mycode.hex
$dfu-programmer atmega32u2 start

This will cause the dfu-programmer to erase the flash reprogramm it with a new hex file and restart the chip.

In Silico Liver port to MASON

I set this task aside awhile back. When I returned to it, we had moved the source repository to a different machine. I had lots of changes in my local sandbox. Several of the prerequisite libraries had been upgraded. Etc. After doing all the catch-up work and checking it in, however, it compiled and ran right out of the box! I’m always surprised when that sort of thing happens.

Way early ISL in MASON mock-up.

Windows Vista using a CUPS PDF printer hosted on Debian

So, every time I upgrade my system, it seems like my printers break.  I don’t know if it’s bad management of the cups debian packages, the peculiarity of windows, or my own incompetence.  The breaks have wildly varying symptoms.  So, I haven’t seen a pattern in them that allows me to fix things gracefully.  Sometimes Samba is to blame.  Sometimes CUPS (-PDF) is to blame.  And sometimes Windows is to blame.  Well, when I upgraded from Lenny to Squeeze awhile back, of course it broke my virtual PDF printer that I use to generate electronic invoices.  When I’d try to connect to the printer (as configured in Lenny), I got the following error:

Windows cannot connect to the printer. Operation
could not be completed (error 0×0000000d).

(No, it wasn’t in those colors or with that background… but I figured I’d try to find a way to make you, dear reader, feel as annoyed as I felt when I saw the error.) Anyway, with such a wonderfully informative error, I didn’t know quite where to turn. The logs (cups, samba, auth.log, syslog, etc.) on the server gave no indication that anyone had tried to connect. And I could browse the other shares from the vista machine nicely and those connections did show up in the logs. So, I had a strong indicator that the problem was NOT my upgrade from Lenny to Squeeze. But, being the stubborn idiot that I am, I started fiddling around with the printer and samba configs anyway…. which lead nowhere, of course. The upside is that I am just a smidgeon more familiar with cups and samba. [sigh]

When I finally typed the error into a search engine, I found the answer provided by some participants of Microsoft TechNet, which I repeat here for posterity:

From: AHarsent

I've seen this problem with a few printers now, not
consistantly[sic] between any particular makes or
models. However, I have found a consistant[sic]
fix to connect to any network printer from vista:

Run a command prompt and type the following:

   net use LPT2: \\servername\printer

This sets up a behind-the-scenes connection to
the printer.

Then go through the add printer wizard, choose to
add a local printer, and choose port LPT2… It will
probably ask for a printer driver, and you can just
choose whatever XP driver you might have been using
before. The printer is then installed without any errors,
and works just fine.

Of course, since I’m using CUPS-PDF, I don’t have a printer driver. I just use the Generic MS Publisher Imagesetter, which seems to work nicely. And now that I’ve made the right sacrifices to the right gods, I’m back to being able to generate PDFs from any program on the windows machine.

Acid Testing Your Time Machine Backups

Disks Die, Laptops break, Bags are Stolen.

It’s more or less a fact of life,

In the 4 year process of transitioning from a noisy sun E-250 in the garage to something that could actually be in the same room with you I arrived at the above network arrangement.

Most of the my data is hosted by a central server and when I had more systems all of the home directories were NFS mounted (which is a major PITA on OSX). In the last hear I have worked primarily on my MacBook, occasionally syncing my projects with the file server. After migrating to leopard I purchased a 1 Terabyte external disk and backed up both the file server and my laptop using “Time Machine”. As I worked more and more on the laptop I did less and less syncing of the work to the file server. The thing about time machine is that its thoughtless. Set it up and as long as you get your laptop home and leave it on for a few hours and it should keep your work safe for you.

Two weeks ago My Bag was stolen.

On top of the work I had actively been doing I lost two active sketchbooks and in the end this loss is probably the most painful. Since I couldn’t immediately restore the data using the process described below, I spent much of last week recreating the designs for this months workshops. Once the critical work was done I turned to see what I could recover. I had used time machine to restore a couple of files I had foobared but I had never had to restore all 270G of data and applications. My Intel Imac did not have enough disk nor did it have most of the software. So I bought a 1T disk from Old Town Computers and “CarbonCopyCloner“ed a bootable operating system onto it and put it into my Intel Imac (Ernesto).

WHERE’S MY BACKUP?!?

After googling around I found that I was supposed to restore my files using the “Migration Assistant” tool. I had used migration assistant to move users and applications to new systems but never to restore files. When I first tried the migration assistant I mounted the disk remotely but I couldn’t see the backup. So I moved the disk to the target machine. Even then I only saw the option of restoring the server and not my laptop.
Panic set in until I found a file named with my laptops name and a .sparsebundle extension. Right clicking on this lets me open it with the appropriate mounting utility. But…. Not until it had checked the file system. Walking away and doing something else for a few hours let me come back to the image mounted. When I ran the migration assistant again the option of restoring the laptop was there.

SUCCESS!

Starting the restore process and going to sleep I woke to find that it had successfully restored all of my data and applications! I was only out the extra time on the designs and the actual property.

FAIURE.

Now all of my active work was on a disk which was large enough to also contain the rest of my images/music/video and web content which had been kept on the file server. Since I could see the fileserver’s backup and I was so happy with my success I tried then to restore the file server data.

Mounting your home directories on external disks on OSX is a lot like mounting them remotely. It just plain sucks. Half the time if I made any changes in the disks connected to the file server it would mount them in a different place. Then when you logged in it would create a new mount point and empty directory and you could not remount the disk where it was supposed to be until you removed the new directory.

When I restored the file server user I had few options. I could restore the user (giving an estimated size of like 20 meg , not the  >400G I expected) and on the next screen I could restore the file system at the top level directory. This did not show me enough data to represent the data directory either. I could also have restored the entire system and then re-restored the Intel files over it but I really didn’t want to deal with the mess.

So I punted.

I just plugged the file server disk in and dragged my home folder to the new disk. I will probably test the restore process to the file server with the external disk attached to make sure the data is recoverable on that system. Once this is done I will reformat the time machine disk for future backups and send the file servers external disk to another state. In the mean time I will rest better knowing that my time machine backups actually work transparently while I work.

Dual coordinate plots in R

The default plot functions in R don’t seem to facilitate plotting two dependent variables using different scales on the same plot, against the same independent axis. But it’s not that difficult, as I learned from: Jim Lemon’s “Kickstarting R”. But even his example isn’t simple enough for me. So, I generated this:

par(mar=c(5,4,4,4))
plot(1:11, 1:11, ylab="", col="green", axes=F)
axis(1,1:11)
axis(2,1:11, col="green")
par(new=T)
plot(1:11, 5:-5, ylab="", axes=FALSE, pch=3, col="blue")
axis(4, -5:5, col="blue")
par(new=F)

which produces this:

Postgres + Ruby + OSX=Multi Architectural Hell

I was finally able to get my ruby/rails environment setup with my database of choice.

It wasnt easy as postgres will not build with more than one architecture at a time, I had gone through similar hell the last time I tried to build anything on OSX that had to compile against libraries made for different architectures. Fortunately the solution was relatively simple.

My database is running 64 bit and I use it for many other applications that are compiled against either multiple architectures including 64bit but also some which are 64 bit only.

Ruby on the other hand is compled for 32 bit which worked flawlessly on mysql and most of the other gems i installed when I got to the postgres it puked in weird ways and when I tried every third solution on the net it was always the same.

Bash-3.2# gem install pg
Building native extensions.  This could take a while...
ERROR:  Error installing pg:
ERROR: Failed to build gem native extension.

The bottom line from one of the only success stories that I found on the net was that you either built everything 64bit or nothing. Easy enough if you want to build all of your dependencies by yourself. The whole point of gems is to have the tools work for you. I wasnt really able to figure this out in the web full of people beating their heads on the same problem each harder than the other and each reaching the same it doesnt work conclusion but I had the fortune of getting completely away from the web long enough to realize that only the client and libraries needed to match the architecture and since all interaction between rails and the database went through a socket the architecture of the client didn’t have to be the same as the database.

so going back to my postgres source tree i did a

...postgresql-8.3.6# make distclean

and then reconfigured the database using the prefix /usr/local32

..postgresql-8.3.6# ARCHFLAGS='-arch i386' ./configure --prefix=/usr/local32/

Then I just had to tel ruby/gem to use the 32 bit libraries. Since ruby uses pg_config I made sure that the 32bit version is in the path first.

# PATH=/usr/local32/bin/:$PATH ARCHFLAGS='-arch i386' \
gem install pg  -- --with-pgsql-dir=/usr/local32/

And we are off but my god what at PITA.

p4_error: Could not gethostbyname for host

For awhile now, we’ve been unable to run using the “-nolocal” flag to the mpirun command in order to execute our simulation across the slave nodes without also treating the master as a slave. When I would run with:

$ mpirun -nolocal -np 7 ./simulation 

I would see an error like the following:

p0_10460: p4_error: Could not gethostbyname for host hostname; may be. invalid name

Being lazy, I spent quite a bit of time trawling the intertubes looking for a canned solution and found many dead-ends talking about the resolver, nsswitch.com, etc/hosts, etc. So, I finally sighed and reluctantly pulled out my programming hat and wrote the following:

#include <netdb.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <malloc.h>
int main(int argc, char *argv[])
{
  char *user_host = calloc(32, sizeof(char));
  int error = gethostname(user_host,32);
  if (error != 0) {
    printf(”gethostname failed with error %d\n”, error);
    return error;
  } else
    printf(”user_host = %s\n”, user_host);

  struct addrinfo *res0, hint;
  memset(&hint, 0, sizeof(hint));
  hint.ai_family = PF_UNSPEC;
  hint.ai_flags = AI_CANONNAME;
  error = getaddrinfo(user_host, NULL, &hint, &res0);
  if (error != 0) {
    printf(”getaddrinfo failed with error %d.\n”, error);
  }

  struct hostent *he;
  he = gethostbyname(user_host);
  if (!he)
    printf(”gethostbyname(gethostname()) failed\n”);
  else
    printf(”gethostbyname(gethostname()) = %s\n”, he->h_name);

  free(user_host);
  user_host = “hostname”;
  he = gethostbyname(user_host);
  if (!he)
    printf(”hostname(/*hardcoded*/ \”hostname\”) failed\n”);
  else
    printf(”hostname(/*hardcoded*/ \”hostname\”) = %s\n”, he->h_name);
  return 0;
}

And that showed me that the string returned by gethostname() was flawed in some way… some insidiously evil and occult way that doesn’t show up on the terminal screen. Redirecting the output to a file and editing that file with vi showed me that there was a SPACE on the end of the string returned by gethostname(). Ugh.

Other symptoms you might see if you have cruft in your /etc/hostname entry:

sudo: unable to resolve host
hostname => hostname
hostname -f => Unknown host
hostname -a => Unknown host

Virtualizing the UCSF Cluster.

The computing cluster we maintain at uscf is built on a stock ubuntu hardy (desktop) with software added to it either to match the previous configuration or as needed by the experiments which are being conducted on the system. This decision was made as a compromise between the user community which was familiar with Ubuntu and the need for a stable (LTS) platform from an administration perspective. Since the hardware on the cluster is aging and reinstallation of systems requiring on site administration is expensive. We are evaluating alternatives such as the Amazon Elastic Cloud. This is an overview of an attempt to create a virtual instance of the servers used in the cluster. (The process is actually done twice since the head node is built on intrepid (desktop) because the processor configuration was not compatible with hardy’s install kernel.)

Out of hundreds of prebuilt images avaliable for the “Elastic Cloud” there are two sets of ec2 AMIs that closely match the platform that we are running. One is from alestic http://alestic.com/2009/04/official-ubuntu-ec2 and the other is a relatively new bundle from Ubuntu http://www.ubuntu.com/products/whatisubuntu/serveredition/features/ec2.

My initial attempt to create a package used the alestic desktop image. On top of taking more than 7 hours to merge the packages installed on the reference platform the result was completely unusable. Some of this was due to my not catching packages which did not work on ec2 such as grub and the kernel which was installed on the reference platform. Unlike VMware’s virtualization the amazon cloud is pretty particular about a few things. For this reason I found it prudent to start with systems which were native to the cloud and modify them.

The Ubuntu AMI advertises itself as “server” which is bare bones to the point of uselessness. Comparing the installed packages on the reference platform to the server left over 500 packages to sort through.
Fortunately the instantiated version allows you to install a set of packages which includes the desktop. This left us a delta of around 100 packages. Eliminating the packages that were not likely to work on the EC2 such as the standard kernel and grub resulted in less than 90 new packages to install.

Once I set up the ec2 tools and environment in accordance with amazons documentation http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/. I created an instance of the Ubuntu AMI (ami-5d59be34)

$ ec2-run-instances ami-5d59be34 -k n01-keypair
$ ec2-describe-instance (...wait until its running)
$ ssh -i id_rsa-furm-keypair ubuntu@ec2-75-101-175-209.compute-1.amazonaws.com

With the basic instance installed you can  “sudo tasksel” and select Ubuntu desktop (in addition to the ssh server and any other appropriate packages). Once it completes the installation you can get a list of the installed packages and compare it to the packages on the reference platform and using aptitude install the additional packages. I use aptitude since it handles dependencies more ‘apt’ly than apt-get.

# dpkg --get-selections|grep install|cut -f1>ubuntu-intrepid-ec2.packages
# exit
$ exit
$ scp -i ~/.ssh/id_rsa-n01-keypair ubuntu@ec2-75-101-175-209.compute-1.amazonaws.com:ubuntu-intrepid-ec2.packages .
$ dpkg --get-selections|grep install |cut -f1 >reference.package
$ diff ubuntu-intrepid-ec2.packages reference.packages |grep \>|sed 's/> //'>packages2add
$ nano packages2add <<<<<< delete linux* grub and any kernel packages.>>>>>>>
$ cp -i ~/.ssh/id_rsa-n01-keypair packages2add  ubuntu@ec2-75-101-175-209.compute-1.amazonaws.com:.
$ ssh -i ~/.ssh/id_rsa-n01-keypair ubuntu@ec2-75-101-175-209.compute-1.amazonaws.com
$ sudo bash
# for p in `cat packages2add` ; do echo $p;  aptitude -y install $p; done

At this point you should back check the new manifest and reboot the instance to make sure that you didn’t break anything before using your modified instance to create a new bundle. The ubuntu AMIs come with two disks defined by default / and /mnt which are close to the same size. you use /mnt to create an image. The process is convoluted at best. There is a pretty good description at http://alestic.com/2009/06/ec2-ami-bundle.

$ scp -i ~/.ssh/id_rsa-n01-keypair  *.pem ubuntu@ec2-75-101-175-209.compute-1.amazonaws.com:.
cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem     100%  916     0.9KB/s   00:00
pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem       100%  922     0.9KB/s   00:00
$ ssh -i ~/.ssh/id_rsa-n01-keypair ubuntu@ec2-75-101-175-209.compute-1.amazonaws.com
$ sudo bash
# ec2-bundle-vol -r i386 -d/mnt -p ucsf-hardy-node \
-u XXXX-XXXX-XXXXX \
-k /home/ubuntu/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem \
-c /home/ubuntu/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem \
-s 10240 -e /mnt,/tmp,/root/.ssh,/home/ubuntu/.ssh
Copying / into the image file /mnt/ucsf-intrepid-head-node...
... <<<<<<< wait for a small eternity >>>>>>>
Bundling image file
... <<<<<<< wait for a nother small eternity >>>>>>>
Creating bundle manifest...
ec2-bundle-vol complete..
#ec2-upload-bundle  -b ucsfintrepid -m /mnt/ucsf-intrepid-head-node.manifest.xml   \
-a XXXXXXXXXXXXXXXX  -s XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Creating bucket...
Uploading bundled image parts to the S3 bucket ucsfintrepid ...
Uploaded ucsf-intrepid-head-node.part.000
...
Uploaded ucsf-intrepid-head-node.part.167
Uploading manifest ...
Uploaded manifest.
Bundle upload completed.

Once this is done you have to register the bundle this is done from the local machine (reference).here.

$ ec2-register ucsfintrepid/ucsf-intrepid-head-node.manifest.xml
IMAGE    ami-f339d89a

Then you can test load it.

$ ec2-run-instances ami-f339d89a -k n01-keypair
$ ec2-describe-instances
...INSTANCE    i-af261cc6    ami-f339d89a    ec2-67-202-25-96.compute-1.amazonaws.com    domU-12-31-39-00-65-C8.compute-1.internal    running    n01-keypair    0        m1.small    2009-07-20T03:41:11+0000    us-east-1c    aki-714daa18    ari-6a5bbc03        monitoring-disabled
$ ssh -i ~/.ssh/id_rsa-n01-keypair  ubuntu@ec2-67-202-25-96.compute-1.amazonaws.com

If this is successfull you can terminate your original instances.

$ for i in `ec2-describe-instances |grep INSTANCE |cut -f2` ; do ec2-terminate-instances $i; done
INSTANCE    i-cd4f75a4    running    shutting-down
INSTANCE    i-af261cc6    running    shutting-down.

(This is important since amazon charges a nominal amount for each instance’s runtime ).

Converting Android 1.5 “Camcorder” videos with SAMR audio codec on Ubuntu Jaunty with ffmpeg

So, T-Mobile pushed the Android 1.5 update to me last Sunday (I think).  I installed it and subsequently had to record a video with the new Camcorder application.  It works well; but produces videos with the audio track in AMR format.  The standard ffmpeg distributed with Ubuntu doesn’t have AMR compiled in because the AMR codec isn’t liberated/free.  I followed Chez’ instructions with a few modifications to get a tweaked ffmpeg that will convert videos recorded with my G1 phone.

  • Run the standard ffmpeg to get the configure options used to compile it.
  • apt-get install libmp3lame-dev libgsm1-dev libshroedinger-dev libspeex-dev libtheora-dev libvorbis-dev libdc1395-22-dev.
  • Download, unpack, configure [--prefix=/usr/local/amr], make && make install both libamr-nb and -wb.
  • svn checkout svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg-2009-06-02.
  • configure [*], make && make install ffmpeg.
  • export LD_LIBRARY_PATH=/usr/local/amr/lib:/usr/local/ffmpeg/lib:$LD_LIBRARY_PATH
  • /usr/local/ffmpeg/bin/ffmpeg -i video.3gp -ar 44100 vide.mpg

[*] I ended up using the following configure command for ffmpeg:
./configure –enable-gpl –enable-postproc –enable-x11grab –prefix=/usr/local/ffmpeg –enable-avfilter –enable-avfilter-lavf –enable-libgsm –enable-libschroedinger –enable-libspeex –enable-libtheora –enable-libvorbis –enable-pthreads –disable-stripping –enable-libdc1394 –disable-armv5te –disable-armv6 –disable-armv6t2 –disable-armvfp –disable-neon –disable-altivec –disable-vis –enable-shared –disable-static –enable-libamr-nb –enable-libamr-wb –enable-nonfree –extra-cflags=-I/usr/local/amr/include –extra-ldflags=-L/usr/local/amr/lib

Reinventing the Wheel (watcher)

Its not very often that you get to go back to old designs and make improvements. Recently I got to rework a design from last year and build on the collective knowlege of 2 design teams and 3 design cycles.

Does practice make perfect? We shall see.

The first run.

Last year I got to build the electronics for a stationary bike racing system. The system had a large ballfield style clock with four hands that were run by these monster stepper motors The initial design was for two bikes done by Percor: a company that makes excercize equipment. Their design used these industrial (expensive and very bulky) cherry hall effect sensors which they loaned us for a race at Lance Armstrong’s bike shop in Austin (http://www.mellowjohnnys.com/). My job for that cycle was to replace Percor’s pic based board with a 4 bike setup.

I broke the stepper driver section into four individual boards. For the processor I built a custom mega128 based board and did the programming in wiring.

Second iteration.

When the Austin race was over Percor needed their parts back so I got to rebuild the sensors. I found some allegra hall effect sensors which were a small fraction of the cost of the cherry’s (like (75c<$25.00)X4). As an attempt to get a zero indicator on the clock I experimented with a triangle based design

While I never did get this particular part of the clock to work correctly the triange stuck as a shape that I liked working with.

I also wanted to replace the cat5 wire used in the Percor design with with less expensive phone cord. (I got a panicked phone call  from Texas and had to remotely direct someone how to get some very expensive ethernet cable on a weekend: it was not cool). To do this I put a non inverting buffer on the sensors and since buffers come in 6s I put an indicator led on the sensors output. In the practice of setting this up for the next race, the blinking lights were invaluable.

Third Time Charm.

A few weeks ago I was asked about another race setup in SanFrancisco. They were going to use a system called open sprints for the display but the open sprints people were out of their hardware for the sensors. So I got a 3 week timeline to put together a sensor and hardware interface that would be compatible with open sprints.

Open sprints (www.opensprints.org) is an open source ruby based software which takes information from an arduino and presents the race data in a way that can be projected instead of having a physical race clock. The only thing that needed to be built was the input side.

So I went back to what I liked and didn’t like about my earlier design and compared and contrasted the opensprints, the percor and my designs to come up with a new system which I could then have fabbed by our local pcb fab (sunstone.com). Looking at the opensprints design I revisited the hall effect sensors and found a set of sensors to experiment with and began to lay out my boards.

I had intended to use the dart design with a through hole led and a throuch hole (sip) sensor and then have some laser cut transparent plastic which could then be “lit” up by the led on the board. A second piece of opaque plastic would be cut to insulate the bottom of the board.

After talking to some of my ee freinds I decided to replace the on board drivers with a single schmidt trigger on the other end of the sensor cable (the percor design used these as well) When selecting the parts I found myself looking at a few options for the rj11 connectors. I ordered a few of each as well as both sip and the origional smt sensor from last year.

Who needs plastic?

When I got the parts I started looking at these rj11 connectors and rethinking things. The connectors were surface mount and shielded. The shielding wasn’t needed but they matched the un-soldermasked boards in texture in color they were about 7 cents more expensive apiece but they looked futuristic.

I went back to surface mount only. If I got rid of the sips and used the surface mount rj11 mounts then the bottom of the board would be shield/ground no insulation needed. The metal from the surface of the board and cable connector would reflect the light from the led and if any attention were payed to the layout of the traces it would look cool!

Even in the hand rolled prototype!

Who needs an Arduino?

At this point given Paul Stoffregon’s (pjrc.com) “teensyduino” software and his 90usb based boards there is no reason to buy any other arduino or arduino clone. ( at least in cases where you need usb to serial solution — in cases where usb-serial is not needed the dorkboard rules :) My first thought was to do a carrier board for on of Paul’s boards and adapt the opensprints code for the inverted inputs. The opensprints code needs to be adapted to invert the input signal and to adjust the registers for the different processor but using the teensyquino codebase that should be simple enough.
In this application however, the board was so simple that it made more sense to just adopt the eagle design for my at90usb162 board (the benito).

Tomorrow I get the boards from sunstone and I should have the hardware built out by late friday. The customer will be in portland saturday.

Does practice make perfect? We will let you know by next week.

Matlab C API, engdemo.c, and JMatLink “Can’t start MATLAB engine.”

OK.  Since I stupidly spent too much time on this problem, I feel like I have to write about it here in the hopes that I can decrease the stupidity in the world.

I’ve written a GUI for a Matlab simulation.  On Linux, we use the JMatLink bridge between Java and Matlab.  At some point, when doing a dist-upgrade on my Ubuntu laptop, the JMatLink bridge stopped working, yielding the uninformative error:

Can't start MATLAB engine.

Well, I proceeded to hack JMatLink, turn on debugging for it, futzing with my LD_LIBRARY_PATH, and my PATH, changing JDKs (since there is a problem with the Sun JDKs and the X11 library libxcb (see here and here for more info on that — I finally downgraded to the gutsy libx11-6 and libx11-dev), etc.

In any case, my fog finally lifted and I decided to try the engdemo.c file provided by Mathworks as the canonical demonstration of calling Matlab from C. And, sure enough, I get the same uninformative error: Can’t start MATLAB engine.” So, I Ask.com’ed (no I don’t usually use the Eye of Sauron for searching) for URLs with this text in them and found: this post by “Mohammed Hasan”. Sure enough, I did an:

$ sudo apt-get install csh

and all is now well. [sigh]

Embedding Quartz Compositions in Cocoa Applications.

Yesterday I asked Greg Borenstein what he was using to do his really cool Arduino presentations to show the hardware alongside the code being used to program it. His answer suprised me. It was a simple Quartz Composition run using Composer. He used it to flip the views put out by his web cam so that they made sense from the users viewpoint. I have been seeing a lot of people using quartz composer to do testing and other cool graphic things.

An example I have been working with takes a camera that I had on a pole with a motor that moved it around like a periscope. The camera is sideways so the view has to be rotated. I did this by hand using Core Image this summer using filters and transforms but wanted to do it in Quartz Composer. The composition looks like this.

And the resulting image like this.

On the way down to UCSF last month I experimented with some code that embeds a quartz composition into an application.

You can download it from our svn repository ( https://ssl.tempusdictum.com/pubsvn/trunks/QuartzSnap/ add this url to your x-code scm repositories) and play with it. It builds into a standalone application.

Most all of the work is done by adding your composition to the xcode project and then placing a Quartz Composer View (QCView) from the objects library in your window using Interface builder. The only code I needed to make my application usable (to take pictures) was the action to save the view to a file.

——————————– CameraViewController.h —————————-


#import <Cocoa/Cocoa.h>
#import <Quartz/Quartz.h>

@interface CameraViewController : NSObject {

IBOutlet NSButton *SnapButton;
IBOutlet QCView *CameraView;
NSImage    *TheSnapShot;
NSData *theSnapData;

}
- (IBAction)DoSnapShot:(id)sender;
@end

——————————– CameraViewController.m —————————-


#import "CameraViewController.h"

@implementation CameraViewController

- (IBAction)DoSnapShot:(id)sender {
   theSnapData = [[CameraView snapshotImage] TIFFRepresentation];
   int runResult;
   NSSavePanel *sp = [NSSavePanel savePanel];
   [sp setCanSelectHiddenExtension:YES];
   [sp setRequiredFileType:@"tif"];
   runResult = [sp runModalForDirectory:nil file:nil];
   if (runResult == NSOKButton) {
     if (![theSnapData writeToFile:[sp filename] atomically:YES])
       NSLog(@”Well that didn’t work!”);
   }
}

@end

And that is pretty much it.

Compared to the several pages of code it took to get the same results using Core Image this was a snap.

Lost passwords on Mac OS X

So, I inherited TDI’s MacBook from Cosmo who acquired a new one. He was nice enough to set up my user account with no password. Well, of course, that was untenable. So, I immediately set my password. … Time passed because I was occupied with other things. And, of course, I forgot the password. (As usual, this episode allowed the Mac vs. PC argument to bubble to the surface again. I rejoice in such arguments because diversity is the key to agility and dissent is the primary marker for diversity.)

Most websites claim you need an OS X install CD to reset the password. But, the helpful “Dr. Michael” showed us the unix way to do it, with which I’m so much more comfortable. In case that post disappears, here is a modified version of what he said that worked for me:

  1. boot into single user mode (holding command s)
  2. execute the two recommended commands

    /sbin/fsck -fy <return>

    and

    /sbin/mount -wu / <return>
  3. type

    passwd <username>
  4. type the new password <return>
  5. confirm the new password <return>
  6. restart with

    reboot <return>

3 ways to 3 volts

I have a few devices that I need to interface to either the benito or another avr board that run at 3.3v. one is an ethernet interface, one is a bluetooth modem and then there are the x-bees.

One Way: Simple Voltage Divider.

When looking for interface examples for the xbee I found a site that started out with a circuit from maxstream and then once everything was working tried a resistor based voltage divider. This is how I got my first set of x-bees running via the ftdi chips.

One thing that this circuit didn’t do for me was to provide a decent pull-down for the reset. The 2 transistors on the right of this circuit are an attempt to do just that.

Another Way: Run everything at 3.3v

If you look at the datasheet for the atmega168 you will find the following diagram.

Looking at this you can see that 16mhz may or may not be in the operating range at 3.3v. But if you drop the crystal down to 8mhz you are good for sure. You may have to change a few things but it is doable.

The Ever Illusive Third: Buffers.

The idea of adding another layer at each interface between boards can have you asking for some Tzatziki and falafel to go with the PITA. There are buffers which are designed specifically for this. Many of the odd, many of them expensive. But there are also many families of buffer which are “tolerant” of a range of voltages outside of their supply. One of the buffers I checked out was the 74abt126 which is a tri state buffer. (I was looking at that because with the tristate buffers you can actually do a two way buffer by selecting the direction). I am not entirely certain that the 5v receive side is kosher with these.

The other buffers I am looking at with 5v tolerant i/o are the 74lvc125 and 74lvc126. These are 3.6v native parts with 5v “tolerant” io.

Since I ordered the wrong series of parts last group order I will have to wait a week to check this out.

See Also.

Swarm’s configure script: Can’t find threads

We recently settled on Ubuntu as a common operating system for the machines on which we run biological models. That meant I have to reinstall Swarm and we encountered the same error Jim cites here.  Our failure is because the configure script doesn’t test for the amd64 platform.  I added the following lines to the configure (after line 19,539):

elif test -d ${jdkdir}/jre/lib/amd64/native_threads; then
threads=native
JAVACMD="\${jdkdir}/jre/bin/java"

That seems to have fixed our problem, perhaps it will help others who encounter the same error.

Cellular Automata Video Synthesizer

Finished assembling my CAVS from Critter and Guitari.

Here it is in action:
CA Video Synthesizer in action

It’s more difficult than I expected to find a stable, interesting CA. But, although at first annoying, the audio helps find stable and interesting CAs.

Return top