I've spent the last day and a half trying to get compositing working in ubuntu. This is actually an amalgamation of a bunch of guides I've found; I shouldn't be given full credit for this.
The first thing you need to do is install xcompmgr and transset. These are found in the hoary universe repositories, if you havent already done so, enable the universe repo in your /etc/apt/sources.list. Its not very hard to do this, search the forums for how to do it
sudo apt-get install xcompmgr transset
xcompmgr is the composite manager (the program/extension that makes things look pretty) and transset sets windows transparencies.
Now, the next thing you have to do is actually enable compositing. This is done via a simple edit of your /etc/X11/xorg.conf
sudo gedit /etc/X11/xorg.conf
Add the following section after the "module" section:
Section "Extensions" Option "Composite" "Enable" EndSection
This tells xorg to enable compositing. Now, just so you know, unless you have a good video card, compositing most likely slow down xorg. Apparantly nvidia cards fare better than ati cards, since you can enable acceleration (which we'll do later) which'll actually make x use your video card for rendering.
As I said earlier, if you have an nvidia card it'll run better. I'm assuming you've already installed the nvidia binary drivers and such, if you haven't then do so. Add the following lines to the "device" section:
Option "RenderAccel" "true" Option "AllowGLXWithComposite" "true"
"RenderAccel" is actually an option you should already have if you're running an nvidia card, but I've included it incase you haven't put it in yet. I'm pretty sure this only works on nvidia cards; can someone tell me if there's an equivelant for ati?
"AllowGLXWithComposite" is a command that allows you to use OpenGL while compositing is running. Apparantly it can be buggy, though, so if you have problems you might want to disable compositing in your xorg.conf whenever you want to use opengl (just comment out the compositing option with #)
Now, if you have an ati card, add these lines instead:
Option "backingstore" "true" Option "AllowGLXWithComposite" "true"
According to a cached google page, backingstore is "used to enable the server's support for backing store, a mechanism by which pixel data for occluded window regions is remembered by the server thereby alleviating the need to send expose events to X clients when the data needs to be redisplayed."
(Since I don't own an ati card, I apologize if any of my facts are incorrect. If someone with an ati card can correct me, I'll make sure to update this.)
OK, now for the really fun part. We're going to actually set it so compositing is enabled when you start GNOME. Go to system -> preference -> sessions. Go to startup programs, and click "add."
Now, before I continue, I'm going to explain a few of xcompmgr's options:
-c : enable shadows
-s : enable simple shadows
-fF : enable fadeins/fadeouts
You can mix-and-match those commands. For example, if you want just shadows, use xcompmgr -c. If you want simple shadows, use xcompmgr -s. If you want to enable shadows and fadeins, use xcompmgr -cfF. I wouldn't recommend using -cs; that'll probably break the program.
So, where it asks you for the command, enter xcompmgr and then the argument. For example, if you'd like to enable shadows and fadeins/fadeouts, you'd enter:
Finally, in the order, change it to 0. This'll make sure it's the first thing that GNOME runs. Apparantly things run much better this way.
Alright! Now you can restart X (ctrl+alt+backspace), log back in, and you should have compositing running!
This is just an extra little command. If you want to set certain windows as transparent, then run the command "transset" in the console. Your mouse will turn into a crosshair; simply click on the window you want to set as transparent. The transparency value can be anywhere from 0 (completely transparent) to 1 (opaque.) It defaults to .75, and back to 1 if the window is already transparent.
For example, if you want to make a window half-transparent: