Shnatsel's experiments

MiniWheatFS aka reliable ramdisks

2011-11-07T10:23:00.000-08:00

Filesystems stored in RAM seem to be around for years, but they still have a problem that plagues them: they are limited by the amount of free RAM + swap on the machine. It's like that by design, and at a glance it seems perfectly OK. Actually, it is not. Not many people actually make 100Gb swaps just to use a gigabyte of RAM filesystem. And while sometimes it's feasible for a server, for desktop systems it's definitely unacceptable.

A solution was proposed by Yves Trudeau in 2009. The idea is to use a union filesystem that writes files to the ramdisk while it has enough free space, and if there's not enough space in the ramdisk in a particular file, write it to an ordinary filesystem (hard disk). Yves named his idea MiniWheatFS and even created a launchpad project for it. Looks like he was too lazy too search for decent tools to implement the idea (or maybe Google banned him? who knows), but the implementation he proposed (based on UnionFS-FUSE) is not workable even in 2011.

Luckily, I'm not banned in Google search, so I've investigated some union filesystems that could make it work.

UnionFS-FUSE, as proposed by Yves, is still way underpowered for such things as of 2011. In our case it will always write to ramdisk, and when it runs out of space on it, returns write errors.

AuFS is a great kernel-space implementation of a union filesystem. It supports several rules of choosing the directory in which the file will be created. Unfortunately, it doesn't have a rule for our use case (I've tested all the rules myself and then asked the developers, it really doesn't). I hope we will see a rule for our use case in the future - this is the most promising option because it doesn't have the insane overhead that FUSE has.

mhddfs is a FUSE-based filesystem that was designed to unite several hard disks into a single writable filesystem. I've managed to implement the MiniWheatFS concept using it, and ran some tests. I've created a 100mb tmpfs and copied files there using cp command. All the files were cached before writing. The average file size is 3Mb. Hard disk write speed is 60Mb/s (cheap 1Tb barracuda). Here are the results from time command:

300mb data

HDD
real 0m5.620s
user 0m0.010s
sys 0m0.710s

mhddfs
real 0m5.111s
user 0m0.020s
sys 0m0.960s

100 mb data

HDD
real 0m1.073s
user 0m0.000s
sys 0m0.270s

mhddfs
real 0m0.864s
user 0m0.000s
sys 0m0.280s

tmpfs
real 0m0.137s
user 0m0.000s
sys 0m0.120s

As you can see, using mhddfs over a ramdisk provides performance comparable to writing data to the hard disk. The FUSE overhead is so insane that even with my blazingly fast Core2Quad CPU and slow disk I get very little performance boost even on large files before the ramdisk is filled up and a performance penalty when the ramdisk is full. I've ran some tests on lots of small files too, and it turned out that FUSE overhead by far defeats the HDD seek time, with mhddfs writes being 2 times slower than HDD writes.

As you can see, such setups are possible but not feasible in Linux as of 2011, unless you have an EXTREMELY slow disk.

Of course, I'd love to be proven wrong!

Web browser icon

2011-08-26T15:03:00.000-07:00

Just a concept I wanted to explore.

link

Initially made with Midori in mind, but it looks too technical for an elementary app.

Based on DanRabbit's elementary icons, GPLv3. Feel free to remix etc.
Update: check this page for the new Midori icon designed by DanRabbit himself!

We've Just Revolutionized Alpha Testing

2011-07-22T02:35:00.000-07:00

Think back to the last time you wanted to look at a newer, unstable release of some app. Was it easy to do? Or was it a compilation from source full of mystical errors, booting a newer OS release from a live CD while fighting black-screen-on-boot bugs, or a terribly slow and awkward virtual machine installation (which still didn’t make much sense because you couldn’t try it with your real data)? I’ll guess it was closer to the latter.

Now, imagine you can try the absolutely latest version of any app, on your real data, safely, side-by-side with the stable version, without manual compilation, rebooting, virtualization and whatever else, in a few clicks. Sounds good? Sounds awesome to me. And now, it’s real.

Meet Glimpse

Yes, this dream is finally real, and it’s called Glimpse. The only thing that you will need is an ISO of Ubuntu Oneiric, but it can download or update one automatically. Then, after a one-time setup, viola! You can test any unstable apps, fully integrated into your desktop, not interfering with the stable versions of themselves. Safely sandboxed, and working with your real data. Wait, WHAT?!

Are you kidding?

No. But it’s really funny to look at people who hear this for the first time! The trick is, the sandboxed apps are allowed to read your real data, but when they write to it or modify it in any other way, all the changes stay within their sandbox. Your real files on your system are left intact. And if you break some files in the sandbox, you can simply purge the modifications made in it and start over.

For example, imagine you’re testing an unstable version of a cool music player...

Without Glimpse: First, you spend lots of time on setting up the build environment (again?! how do those guys manage to change it so quickly?!), including unstable version of compiler that breaks all other builds in the system. Then, when you finally manage to try the player for 10 minutes, you realize that the “improved writing metadata to files” has just broken half of your music library. You’re out of luck or have to restore a manual backup (if one exists).

With Glimpse: You open Glimpse, click “Update sandbox”, and the latest version of the player arrives in a few minutes, not hours. And if you find that all the music files in the sandbox are broken, you think “thanks to Glimpse, my files are safe,” click the “Purge sandbox files” button, and your original music is available in the sandbox again.

How on Jupiter did you do that?!

You’ll laugh, but all the technology was already there, all I had to do is to “connect the dots.”

It’s enough to say that Glimpse is written entirely in BASH. For the real work it relies on:

zsync to download only those parts of ISO that have changed instead of pulling the whole 700Mb monster every time
chroot for creating an isolated environment (instead of virtualization)
Ubuntu Customization Kit and its awesome library to avoid reinventing the wheel
Launchpad Recipes to get the absolutely latest code built, packaged and gift-wrapped
AuFS to be able write to home directory and keep the real files intact
Gaffel and Zenity to have a simple GTK2/GTK3 user interface

Most of these technologies (except Gaffel, which was written just now with Glimpse in mind, and recipes, that left beta several months ago) have been available for a few years already. It’s strange that I’m inventing Glimpse, and nobody did this before me. The only explanation I can make up is “others got stuck in polishing chroot”, but it doesn’t sound very convincing.

Limitations

Everything has them, and you’d better know them before you meet them.

Glimpse is not yet accepted as an official elementary app because of its poor GUI. So you probably will not get any support from elementary developers.
Whatever you run in Glimpse, it uses the host’s (your system’s) kernel. By design.
Some system directories are not really sandboxed - they’re mounted as is. In most cases it doesn’t matter, but if you’re aiming for mass destruction, be warned that these directories are not sandboxed:

/dev/shm
/proc
/sys
/dev/pts
/var/run
/tmp

Exploits targeted to breaking chroot jail can break Glimpse sandboxing too. It will never happen on a desktop system, but if you’re going to test the cutting-edge server exploits, do it in a virtual machine.
An encrypted home folder can not be accessed from the sandbox. Any help on fixing this is appreciated.
The elementary project provides daily builds for most of its apps “out of the box,” but other apps will be at their versions from the current unstable Ubuntu. You’ll have to add a daily build PPA for each app you want to be at bleeding-edge version.
One has to write a separate profile for each new OS one wants to sandbox. Complexity may vary. Running Fedora or openSUSE would require quite some tweaks, but it’s not that hard when there are two working examples already.
Sandboxed apps try to use host’s theme settings, but they don’t have access to host’s theme files. Switch your host system’s theme to elementary in advance (unless you like that old Windows 95-style interface!).

Try it yourself

The user interface is rather ambiguous, but usable, and the underlying technology is rather stable by now (I use it all the time). Glimpse runs on Ubuntu Maverick and up. elementary OS Jupiter is also supported, so now you can finally get Marlin, BeatBox, Pantheon Shell and whatnot in Jupiter. At the moment of writing this the package containing Ubuntu profile hasn’t built yet due to a Launchpad lag, and Gaffel doesn’t show OS logos, but it should be fixed in a few hours.

Glimpse is not an official elementary project yet, which means that you probably won’t get any support from elementary developers. If you still wish to install Glimpse, run:

sudo add-apt-repository ppa:glimpse-hackers/stable

sudo apt-get update

sudo apt-get install glimpse glimpse-profile-elementary glimpse-profile-ubuntu

Then find it in Applications > System and enjoy!

Hacking

All code is stored in Launchpad: https://code.launchpad.net/glimpse

The core is very minimal, less than 50 lines of BASH. The real work is done by plugins. And, you know, having a Fedora plugin would be cool ;)

Please report any bugs you encounter.

Fixing existing bugs is a good idea too.

The current UI supports translations, but I won’t upload the translation files anywhere because it was deemed unsatisfactory and I don’t want people to waste time on useless work.

May the chroot be with you.
Happy testing!

Bonus section: screenshots

Sandbox chooser and app launcher
Sandbox first run dialog
Marlin with GTK3 theming and overlay scrollbars out-of-the-box