Michael's Software Thoughts & Ramblings

SWT Table Images And Transparency

2010-04-16T17:05:00.007-06:00

I recently came across a Windows platform issue using images inside an SWT table. A little research turned up this long-standing open SWT bug.

The problem results when you change the background color of a row in a table. I like to alternate colors on the rows in my tables, so I create a LabelProvide like this:


private class PortfolioViewLabelProvider extends LabelProvider implements ITableLabelProvider, ITableColorProvider {

    @Override
    public Color getBackground(Object element, int columnIndex) {
      return null;
    }

    public Image getColumnImage(Object obj, int index) {
      Image retImage = null;
      return retImage;
    }

    public String getColumnText(Object obj, int index) {
      String retVal = "";
      return retVal;
    }

    @Override
    public Color getForeground(Object element, int columnIndex) {
      return null;
    }
  }

Of course, you would actually want to return some real values in those methods.

Using a JFace TableViewer allows the table to be decorated with the listener via the setLabelProvider method.

Under Linux, SWT uses GTK native widgets for table. An example of one of my tables looks like this:

Make sure your monitor is set up so you see each row is a different shade--the first row is a deep gray, and the second row is black.

The table looks fine and all of the images ( transparent png files ) are correctly displayed. Now look at the same application on Windows 7.

Notice the images have black backgrounds now which is the background of the table. The transparency mask in the png is ignored--annoying and ugly. Look closely at the circled area to see the problem.

Luckily, with a little help from old posts in that bug and some trial and error, I came up with an easy fix that works for Windows and Linux. Using the same label provider as above, we can get the row color during the SWT.EraseItem event and erase the area with the same color used for the row:


  topPortfoliosViewer.getTable().addListener(SWT.EraseItem, new Listener() {
      @Override
      public void handleEvent(Event event) {
        event.gc.setBackground(plp.getBackground(event.item.getData(), event.index));
        event.gc.fillRectangle(event.getBounds());
      }
    });

And the result on Windows 7:

The result is how I want it. The images blend in correctly. I'm not sure what the real problem is here--is it Windows or Eclipse? Either way, the fix works for me, and I can continue on with development.

The Future Of OpenSolaris...Still Unwritten and Uncertain

2010-04-16T16:09:00.005-06:00

I've been following the activities of the OpenSolaris project for some time with interest. With the acquisition of Sun by Oracle, it's future is still uncertain. Since my last entry on the subject, I have only played with it a few times inside a virtual machine.

I remember back when I was in college and OS/2 Warp came out. It was so cool and I was immediately converted. Despite the coolness factor, the conversion was short lived. I don't remember my reasoning for abandoning OS/2--maybe it was my introduction to Linux in a computer lab or the lack of OS/2's ability to satisfy my computer needs compared to Windows or Linux. Whatever the reason, I can't help but recall that experience as I now look at and ponder the fate of OpenSolaris.

I don't think OpenSolaris has a large enough community outside of full-time, paid developers to support the operating system's advancement. Sure, there are a few different distributions out there, but these deal primarily with packaging the kernel--not low-level kernel development. As with OS/2 warp, a few cool technologies found in no other operating system (i.e. DTrace and ZFS ) will not be enough to grow the community and keep the system alive. Without a large and fully open community, a code base the size of an operating system needs some major company-backed financing to grow.

Only time will tell the true fate of OpenSolaris. I think some years down the road I'll look back on my OpenSolaris experience and remember the brief experience with a smile and wonder what may have been.

GNOME Composting in OpenSolaris

2009-11-28T10:42:00.001-07:00

If your graphics card supports it, OpenSolaris will turn on the 3D effects when you login. It does this by enabling the compiz window manager which replaces the default GNOME window manager, metacity.

I only recently discovered that metacity also supports composting. It doesn't offer the feature list of compiz, but, it does provide nice drop shadows and task switcher.

To use composting in metacity, set Visual Effect to None in the System/Preferences/Appearance dialog box. Apply the changes and your desktop will be running metacity.

Next, run this command to enable composting in metacity:

gconftool-2 -s /apps/metacity/general/compositing_manager  -t bool  true

and you can run this command to turn it off:

gconftool-2 -s /apps/metacity/general/compositing_manager  -t bool  false

Here's a simple snapshot showing the shadows:

If you really like tweaking your desktop, compiz works very well and provides lots of features. If you're happy with drop shadows and a nice tab switcher, metacity+compositing will do the job nicely.

OpenSolaris Performance Issues Resolved

2009-10-11T20:43:00.002-06:00

I've been playing with some soon-to-be-released operating systems--notably Open Suse 11.2 and OpenSolaris 2010.02. I always really enjoy OpenSolaris when I run it in a virtual machine under Linux and I enjoy working on Solaris workstations at my employment. I've tried OpenSolaris on and off over the past several months on my home computers. When I've installed it directly onto a partition and run it directly on bare metal, I've noticed performance issues and I've gone back to Linux as the primary OS.

In my latest endeavors of running OpenSolaris, I noticed that the disk io seemed to be the performance killing culprit. Doing simple disk io tests under both Linux and OpenSolaris, Linux was just destroying the io numbers I was getting under OpenSolaris. My test was a simple "dd if=/dev/zero of=test bs=4048 count=100000 conv=fsync"--nothing fancy, but enough to demonstrate an issue. Linux numbers were around 60 MB/s while OpenSolaris numbers were in the 15-20 MB/s. No settings helped out the OpenSolaris numbers--dd block sizes, file system block sizes, zfs settings, etc. Interestingly enough, on an older computer, the disk io was just fine under OpenSolaris, so why was it so slow on newer/faster computer?

I was almost ready to give up and wipe the OpenSolaris partition once more, but I had one last area I had not tried. I remembered the SATA settings that are within the bios settings of my Intel motherboard. I changed the SATA mode from Legacy to AHCI. Sure enough, my OpenSolaris hard drive speeds were now on par with Linux. Inspection of the loaded modules showed that OpenSolaris had loaded the ahci driver. Additional online research shows that one should set the mode to AHCI for OpenSolaris.

I'm definitely happy with the results on all 4 home machines. 3 diverse desktop models and 1 laptop are running very smoothly. I'll plan on detailing some lessons learned and tutorials in the near future.

OpenSolaris 2009.06 Progressing Quickly

2009-07-01T17:26:00.003-06:00

Previously, I spoke of my OpenSolaris experiences. Over the past week, I've been on vacation, and I decided to try the latest release out to see how it's moving along. I installed version 2009.06 over my laptops Fedora installation. Installation was once again very simple--just a few questions, and it was off installing. I typically choose to use the entire drive as a single ZFS pool. OpenSolaris does not support disk encryption out of the box like Fedora or other Linux distributions, but work is underway in this area.

After the initial install, I immediately upgraded to the development repositories to bring in the more more bleeding-edge stuff( Fedora is pretty bleeding edge too ). An update and a reboot into the new boot environment, and I was up and running.

I discovered there are a few really nice new features and surprises in the latest builds.

First off is the integration of the new sound system--Boomer for OpenSolaris. This is now all integrated with the latest development builds. Sound playback worked wonderfully on my Laptop. In flash videos and playback in general under the previous sound system, sliding the volume control up and down caused serious pauses and stutter. The new sound system works much more smoothly and I noticed no more pauses or stutters while changing the volume. I have noticed that I get no headphone sound right now on my laptop, but overall, the new sound system is a nice improvement.

The next thing I noticed was the brightness control finally works on my laptop. The hotkeys don't work, but the Gnome brightness applet allows me to control the brightness. This was a major drawback of previous builds, since the only solution seemed to be a Linux live cd boot to fix the brightness.

Just a few days ago, I notice that my other major gripe has been fixed too--drivers for my two home computers wireless cards are now available! I'll be testing this out soon, since this means I have the option to put OpenSolaris on two home computers without hardware upgrades.

As I've moved around to different Wifi points, the Network Manager has had no problems connecting up to the different access points automatically using WPA2 and WEP encryption. I've had fewer issues with the automated OpenSolaris Network manager than the Fedora network manager.

As far as usability, my laptop experience has been far more pleasant than previous versions of OpenSolaris. With Eclipse Galileo builds supporting OpenSolaris ( 32 bit ) as a first class citizen, my development IDE needs are met.

Off the top of my head, there are three areas still lacking when comparing OpenSolaris to modern Linux distributions.

The first area is one I mentioned earlier--ZFS/filesystem encryption. I prefer, and many companies now require laptops to have encrypted hard drives. At the rate things are moving in the OpenSolaris development, I suspect this will no longer be an issue in the months to come.

The second area lacking is multimedia support. The issue here is a licensing issue where OpenSolaris can not ship with codecs to provide mp3 playback or encrypted dvd playback support. Attempting to play such media pops up an information box allowing one to get the mp3 from Fluendo for free or to pay for the suite of codecs ( I'm not even sure if this package will allow dvd playback). An alternative solution is to download the latest MPlayer source code and compile the code. There were a few issues getting MPlayer to compile, but drop me a line if you need help. Once MPlayer was compiled with Xv support, I was able to play back some dvd's though I did see an annoying video stutter happen every few seconds. I suspect one should also be able to compile the gstreamer plugins that provide the proper mp3 and video codecs too, though I've not gotten to try that yet.

The final area is performance. Linux has made great strides in the last year in performance. Boot time races between Ubuntu and Fedora have made a very noticeable difference for Linux users who run laptops or desktops and pay attention to the boot up time. OpenSolaris is not quite there on startup time. It still takes a bit longer to get to a login screen compared to Fedora 11 or Ubuntu 9.04, however, it's not terrible. The first OpenSolaris bootup after install is a lot longer than subsequent boot times, so don't give up right away if startup times are important to you. Shutdowns are fast enough to not be an issue for me. The desktop is very responsive, though I do see some waits when say trying to open a new terminal while un-tarring a large file. Linux has been tuned to provide a snappy desktop response in most distributions now, I suspect there are some kernel knobs in the OpenSolaris kernel or ZFS settings to help out here too. For the most part though, I find OpenSolaris as responsive now as I find Linux, but I think Linux still has a slight out-of-the-box advantage here.

OpenSolaris 2009.06+ is already much better that the 2008.11 version I tried earlier. Six months has brought in new drivers and new functionality for the desktop. As Oracle hopefully continues to support the effort, I think the future is holding a definite spot for OpenSolaris. Give it a shot and try it out.

Linux Kernel 2.6.30 Fixes Filesystem Latency Issues

2009-04-30T15:11:00.005-06:00

After reading the post Kernel-Log-What-s-coming-in-2-6-30-File-systems-New-and-revamped-file-systems--/news/113157, I saw improvements were made to problems with the Linux kernel regarding slowness under heavy disk I/O.

After running 2.6.30-rc3, my computer no longer grinds to a halt under large amounts of disk I/O.

My only legitimate reason to abandon Linux for another OS ( BSD, Solaris ) has been resolved.

Pixie Hollow Flash Problems With Linux Return

2009-02-27T16:21:00.003-07:00

It seems the folks at Disney's Pixie Hollow site now force everyone to use the Flash 10 player to hit the site. Since the combination of Flash 10 and Linux fail on this site ( see my earlier post here, my kids are out of luck until this is resolved.

This is the only site I have ever seen that fails to be cross-platform/Flash compliant. If I were real ambitious, I'd investigate it further. But for now, telling my kids to play on another site works just fine.

OpenSolaris Wireless Disappoints

2009-02-27T15:50:00.004-07:00

In yet another attempt to play with the latest OpenSolaris builds, I was sorely disappointed when I went to get the wireless card working.

A desktop had been giving me hard freezes with the latest Debian install, so I threw on OpenSolaris to see if that would help. Only after a complete install did I come to find out the Linksys WMP54G ( RT2561/RT61Chipset ) is not supported under OpenSolaris. This is a few common card I've picked up at my local Best Buy.

In one last attempt to fix the freezing system, I threw on Ubuntu. Ubuntu is a little nicer about discovering and automatically installing proprietary drivers, so it put on the Nvidia drivers. I noticed it was using a different version of the drivers.

A re-install of Debian Lenny with the legacy 173.14.xx Nvidia drivers and I have yet to have a freeze.

Linux wins again.

The KDE 4 to GNOME Switch

2009-01-25T18:30:00.005-07:00

I was happy to see the latest news that Linus switch to GNOME--a move I made and blogged about previously.

While testing out KDE4 and deciding on GNOME, I kept thinking, "...but if Linus prefers KDE, I must be missing something."

Anyway, the word on the street is that "KDE 4.2" is the "KDE 4" we've been waiting for. I'll definitely give it a spin and see.

I think KDE 4 visually looks great--even when in non-compiz/non-accelerated mode--the widgets and desktop look good.

In order for GNOME and KDE to compete with modern OS's, they need to look and behave like other modern desktops. The non-compiz GNOME desktop is starting to look dated to me, and I just don't trust a compiz-enabled GNOME desktop to be stable enough--based on my most recent Fedora 10 testing. Sure you can run compiz and change out panels and add cool applets, but that's a lot of work especially for the non-techie desktop user. Out-of-the-box, KDE 4 just looks modern, though it's been buggy and lacking in usability.

Flash, Linux, and PixieHollow

2009-01-25T18:00:00.002-07:00

So I decided to try a little experiment and move some household computers over to Linux. Going into it, I figured there wouldn't be too many problems. The games played by my kids are, for the most part, online games developed in Flash. With Flash 10 now readily available for Linux on 32 and 64 bit machines, Linux would be on par with Windows Flash functionality.

With the computers now happily booting into Debian, I had one of my children log in to test it out. Wouldn't you know it, the latest online Flash-based game Disney's Pixie Hollow failed to load. Everything looks fine until you get a strange error message about it losing the login information.

After trying several different combinations of Firefox and other browsers I found a solution.

Solution:Use a Flash 9 version of the Linux plugin--not the latest Flash 10 plugin. From there, everything has worked fine.

On a 64 bit Linux install, this means running a 32-bit version of Firefox with the Flash 9 plugin. Running nspluginwrapper didn't work either, so I stuck with the 32 bit Firefox / Flash 9 combination.

To date, all the online Flash games play great and are indistinguishable from their Window's counterparts on the outside. Of course, I sleep better knowing what's driving the code.

Locating older versions of Flash to download was little tricky, so here is the link.

So far, no major complaints from the family on the migration. I still have a Windows machine for iTunes and a few other Windows games floating around. My goal will be to move that functionality to Wine or VirtualBox in the very near future.

Linux 64 bit disk IO Performance

2009-01-07T19:11:00.003-07:00

You may be asking yourself why I've even bothered looking at other operating systems if Linux with an accompanying distro does everything I need. The answer is that Linux has two main problems that have been driving me crazy lately--no glitch free audio and terrible responsiveness when copying large files.

The audio problem is well known, and efforts have been underway to address the issue. I think the Linux audio playback is getting better now with PulseAudio, so this issue doesn't bother me as much any more.

The disk io responsiveness issue is less known. The issue appears when copying lots of or large amounts of data on a 64-bit kernel. The kernel load escalates to the point the system is unresponsive ( the load climbs up to 9+ on a beefy machine with a single copy command running). This is the primary reason I started looking at other operating systems.

A few days ago, I came across this linux kernel bug entered over 2 years ago. Interestingly enough, the load only escalates when I boot up using a 64 bit kernel--the 32 bit kernel is fine. I'm planning to jump onto this related kernel bug and see if I can offer testing or other support here shortly.

Follow up 1/14/2009: Looks like there's some progress on the performance bug. A Slashdot article here explains more.

FreeBSD 7.1 Laptop Experience

2009-01-07T17:30:00.007-07:00

FreeBSD 7.1 was release a few days ago. I gave it a run on my x86 laptop over the past few days, and I really liked everything I experienced. Here are a few quick bullets on what I focused on and encountered:

Stability -- No problems. The OS ran fine, and I never had to force a reboot. I did have to force a power off once or twice--GNOME hung on a shutdown menu command.

Audio -- The audio experience on FreeBSD was glitch free despite any other tasks I was running.

Flash 9 support -- I got flash 9 running in a native FreeBSD Firefox browser using nspluginwrapper. Flash audio worked fine too, and I didn't have any crashes or other stability issues.

Java Support -- I had a native JDK 6 running fine with Netbeans 6.5. I read that Glassfish would also work, but never installed it.

USB Yank Support -- This is a nit-pick for server usage, but I consider it a critical bug for desktop usage. If you yank out a mounted USB drive on FreeBSD and try to force an unmount, the kernel blows up and hangs. I did come across a post somewhere that this is now being addressed by the FreeBSD Foundation. Yay!!!

Misc. Stuff -- The only thing I really missed from Linux to FreeBSD on the laptop was the extra buttons support. OpenSUSE installed the necessary stuff to make the laptop's mute/volume/brightness buttons work out of the box. FreeBSD supplied the necessary module to control my laptop's brightness, but I would need to research on how to tie the buttons into brightness and volume control functionality. I also don't know if this is really a kernel issue or a user space program issue. My guess is I could probably make the buttons work by installing some user space programs and configuring GNOME.

FreeBSD would support all my computing needs, but there is still one major issue holding me back from full adoption--NVidia driver support on x64 architectures. It's unclear who exactly to blame for this. NVidia is waiting on FreeBSD full kernel support to handle the binary NVidia driver, and the kernel developers are not focused on making the extensive changes required to fully support the driver. While progress is being made, it's a slow process ( over 4 years now ). Just check out this nvnews thread for the ugly details.

One solution I have would be to run the 32 bit FreeBSD kernel. The problem I have here is the machine has >4GB of memory. Now, FreeBSD supports PAE to address all the memory, but FreeBSD does not support loadable kernel modules with PAE. I'm unable to convince myself to yank some memory out of the machine to make the graphics card work. I'd even be willing to sacrifice 3d support if I could get decent 2D dual headed DVI support in a 64 bit kernel. From reading the forums, this seems iffy too using either the nv or nouveau open source nvidia drivers.

It seems myself and many other "would-be" 64 bit NVidia FreeBSD users are out of luck for now.

OpenSolaris No More (again)

2008-12-27T11:59:00.003-07:00

After getting most everything up and running, I finally got a chance to "work" on the OpenSolaris machine. Unfortunately, I was less than impressed with the desktop performance--even with the zfs memory changes. The desktop became sluggish to me.

I can sleep well now knowing I gave OpenSolaris a good, fair comparison to Linux for what I needed. Linux ( OpenSuse 11.1 ) comes out on top as the winner for me in performance, routing/nat capabilities, device driver support and best overall server/workstation experience.

OpenSolaris Audio And Memory

2008-12-26T13:43:00.003-07:00

Sound is now working. I installed the latest OSS package. After a reboot, the update did not fix the sound issue--the card was still not recognized. I then tried source files, but had issues getting it to build. Un-installed the oss package, rebooted, and the audio worked. I can't explain what changed, but audio is now working on an Intel Corporation 82801JI (ICH10 Family) HD Audio Controller just fine.

OpenSolaris is getting a new sound engine--let's hope it works better than the various Linux offerings we've struggled with over the years.

I did have to reboot the OpenSolaris box this morning. For the second day in a row, all the memory was used up in the morning, and the machine became completely un-responsive. I came across the post Where is the memory gone?, and I applied the limits to the ZFS cache. Hopefully, that will fix the problem.

The machine has been running great other than the memory issue. I have NAT and Firewalls working now too.

OpenSolaris with Blender 3D, Fetchmail, Sendmail, Procmail , Dovecot, and the Gimp

2008-12-22T15:32:00.003-07:00

Today I got the OpenSolaris install a few steps closer to the replaced Linux install. I compiled Blender 3D, the Gimp, Procmail, and Dovecot--a few issues with Blender compiling, but then a Wiki explained how to build on OpenSolaris.

OpenSolaris already has fetchmail and sendmail packages in the main repo, so it was a matter of configuring sendmail to use the procmail mailer. The hardest part was figuring out how to make procmail use Maildir format. The answer was simple:

in /etc/procmailrc add this line:

DEFAULT=$HOME/Maildir/

The trick was the "trailing" slash to tell procmail to use maildir format. The man page for procmailrc clearly states so. I'm sometimes to quick to google something before RTFM, and what is really a simple solution becomes overly complex by endless online searches.

OpenSolaris 2008.11 Revisited

2008-12-21T16:42:00.005-07:00

Picture of OpenSolaris 2008.11 Desktop.

I last spoke about my encounters with OpenSuse, FreeBSD, and OpenSolaris. After writing that entry, for some reason, I kept thinking about my encounter with OpenSolaris. Maybe it was that I'm back on Solaris machines at work, or maybe it was the small OpenSolaris icon Firefox mistakenly replaced on one of my always visible bookmark links. Either way, I've decided to give it another go--this time diving in.

Initially, I ran it for a few days on my laptop. The primary purpose being to get more familiar with its capabilities as a desktop machine. Once I was satisfied and more familiar with it's capabilities, I wiped my "just-updated" OpenSuse 11.1 off my main home computer. My experience is still a work in progress, however, I am very happy with it so far. It does have a learning curve--even for an experience Linux user, however, I think it will be worth it.

Here are some high level thoughts in various areas:

Install Experience:
Very simple, if not easier than the most recent Linux Fedora/OpenSuse distros. The most complicated piece for me was partitioning--the install gui does not let do any advanced partitioniong, so you must bring up a command terminal and run the "format" command. While text-based, the entire format/fdisk capabilities are very easy for anyone with any fdisk or parted experience.

Startup Experience:
I _really_ like the whole service-based system as opposed to the rc.d/init.d scripts. Once you learn the svcadm and svcs commands, it's really cool. Services are managed very well and each has a dedicated log file. Failed services go into a maintenance mode where an admin can diagnose and restart. The initial boot time was slow, however, future boots are much faster. Linux users familiar with grub will be right at home. OpenSolaris uses grub with the additional Boot Environment capabilities to manage multiple "versions" of the installed software on the zfs file system--very nice for user updates and kernel updates.

Initial Desktop / Network Experience:
OpenSolaris ships with NVidia drivers, so it was a few mouse clicks to have the initial desktop up and running with Compiz on a dual-head display. The desktop network icons work fairly well to automagically attach to wireless or cabled networks. I had a few issues with wireless, so I had to come up to speed with ifconfig and dladm. The nice thing about both tools is they are very well documented in man pages and online docs.

32 / 64 bit differences:
This was hard to figure out--most Linux distros you just download the 32 or 64 bit installer, then you have that version installed. Of course 64 bit installs run 32 bit code. OpenSolaris is a little different. It's a unified OS, so the installed installs 32 and 64 libs. During the boot, the system figures out which kernel to boot.

Now, even though my kernel is 64 bit, when I launch firefox, it's the 32 bit version. Many programs have 32 and 64 bit version, so my guess is you can choose you desired architecture on many programs. Look for the bin/amd64 and lib/64 folders to get an idea--I'm still figuring this out.

Software and Updates:
The IPS ( network based package system ) does a decent job of updating packages. One of the criticisms of OpenSolaris is the available number of software packages. Even from the Linux crowd, you come across this argument from time to time ( shame on you ). It is noteworthy to see the available number of packages already available. I did feel that many of the packages on BlastWave and other repos were just too old. Any experienced Linux user would realize that the argument of "Distro X has 10k packages and OpenSolaris only has a fraction of that" is not really an argument. Linux was in the same boat just a few years back, and OpenSolaris ( IPS infrastructure in place now ) will catch up quickly. In addition, all one has to do ( just like we all did a few years ago on Linux ) is to download the source code to a package with its dependencies-- a few ./configure / make / make install commands later and it's there.

I did this on OpenSolaris and now have running dansguardian, openldap, mplayer, postgres, squid, kdelibs, kmymoney, and others. Some of these do have pre-compiled packages--I chose to get more recent versions the old fashioned way. Needless to say most compiled without incident--and using 64 bit flags or 32 bit flags where I desired.

Bottom line is don't let the current list of available pre-packaged software hold you back--if you know how to build src pacakges, you're set. Otherwise ask around or just give it time.

Device Driver Supprot
I have 2 issues with device driver support. The first is audio on the ICH10 hd intel chip. The second is printers. I believe the audio is being looked into now. OpenSolaris printing gives you 2 options ( the default lp service or CUPS). Switching between the two is simple ( print-service command ). Under lp, I printed fine to my laserjet, but not to the hp inkjet. Both printers were detected fine. Under CUPS, neither printer worked--I believe there is a known CUPS issue with USB printers.

I'm not too concerned here, as even the latest Linux distros suffer from audio issues or other device problems. Linux does support more devices right now. Given how far OpenSolaris has come in the past 6 months, I think 2009 will be a huge package/support device year for OpenSolaris.

Advanced Networking
I use a few advanced Linux networking features--namely virtual networks, bandwidth throttling, firewalls, and NAT routing. I have not got my OpenSolaris box doing this yet--I have to learn how to do this on OpenSolaris. I think firewalls and NAT are already there. Bandwidth control and virtual network cards are part of the OpenSolaris CrossBow project. Just recently, the capabilities were merged into the OpenSolaris kernel. I did the Linux equivalent of a kernel update on my laptop to learn how. My next big step will be to do it on this server. More to come in this department.

To summarize:

Is OpenSolaris ready for the desktop?
...YES. As ready as Linux is. You could set up an OpenSolaris box for any non-techie member of your family, and they could surf the web, get emails, write documents, etc. Linux and OpenSolaris still can not satisfy one's need for IPOD sync, camera photo management without a knowledgeable Unix go-to-guy to call.

Is there a Linux to OpenSolaris learning curve and initial struggle?
...YES. However, it's much less that say a Windows to Linux/BSD/Solaris conversion.

Is the struggle worth it?
...YES. For me, I'm very impressed and happy. I certainly don't want to say Linux is better or BSD is better or OpenSolaris is better. Choice is good and so is competition -- even in an open source world arena. I'm not sure why I like OpenSolaris. All I can say is it just feels good. I don't get too caught up in the licensing differences and arguments. If I can run it at home and at work and it's open source, I'm satisfied. It's also refreshing to have an excellent performing, stable OS capable of running my favorite open source programs with a company like Sun behind it. I sort of feel like the days when I switched from Windows to OS/2 Warp 3.0. Although Solaris has been around for a long time, to me, it feels like an evolutionary step forward. I still have lots to learn here too.

I'll be adding posts related to this migration experience as I learn more and become more familiar with OpenSolaris.

Fedora, OpenSuse, FreeBSD, OpenSolaris

2008-11-29T09:52:00.005-07:00

I've been running Fedora for some time at home and on several work computers--most recently the latest Fedora 10 development branches. I ventured out to try OpenSuse, and brought in Beta 5. I must admit, I was blown away. I had last run OpenSuse a year or so ago, and was not as impressed as I was with Ubuntu at the time. In case you're wondering my preferred Linux distro goes something like this ( this goes back a ways )

Slackware->RedHat->Gentoo->Ubuntu->Fedora->OpenSuse

Even given all the hype around the recent Fedora 10 release, I am still much more impressed with OpenSuse. I prefer zypper more than yum, and YaST2 either in console mode or graphics mode is much nicer than anything I've seen in Fedora or any other distro I've tried.

Earlier I posted about Fedora directory server and openldap. Well, I'm running OpenLDAP now and very happy with it. Fedora still has the nice gui admin tool. I stumbled on OpenSuse's User and Group Management tool which fully and graphically supports a back-end ldap configuration. So, it's easy to manage your ldap users with OpenSuse!

OpenSuse is well polished, looks great, easy to install ( again the best installer I've used ), the DVD even has graphical tools that analyze and help you fix a broken system.

Bottom line, if you haven't tried OpenSuse, I'd recommend trying release 11.1.

In my spare time the last few days, I also tried out FreeBSD. I ran it on my laptop for a few days and was impressed--getting to know the system and how it worked in comparison to Linux. I often try testing things out first in a virtual machine, but I don't think you really know an OS until you run it on bare hardware--especially with things like 3D acceleration support and hardware support.

I eventually decided I would really test it out, and cleared of my home server/workstation--could FreeBSD support everything I needed??? The first road block I hit was no NVidia 3d support on the AMD64 bit platform. Even though I checked NVidia's site, I somehow missed the i386 on the driver listed. Ok, I'll through on the i386 FreeBSD version. Well, this time I realized that virtualization products ( like VirtualBox I use ) have not been ported to any BSD. I do like running VirtualBox on my Linux system. We'll, as much as I wanted to really dig deep and run FreeBSD, I decided against it--I could have lived without VirtualBox, but knowing I had full 64 bit 3D support on Linux and VirtualBox, I decided Linux suited my needs better.

Next stop...OpenSolaris 2008.11. I'd run 2008.05, but had to ask myself what it really gave me over Linux outside of ZFS and DTrace which I can personally live without. I installed 2008.11 on my workstation. OpenSolaris has a great installer--very simple. Booting up caused me a little pain, because the installer did not overwrite my previous OpenSuse boot loader. I had to wipe out the MBR, reinstall, then it was fine.

OpenSolaris 2008.11 booted up into Gnome and it looks great. It immediately identified my printers ( both HPs ), but I had no full sound card support. I ran it for a bit, but was again asking myself, "do you really want to go through the trouble. Re-compiling from scratch all the packages you just get with a standard Linux distro. Installing add-on packages just to get features you get out of the box with Linux like firewalls, traffic shaping, network virtualization??" Not that those couldn't be done with OpenSolaris, but again I asked myself "why go through the pain?" when Linux already does it and does it very well. The answer was "no", and I am back with OpenSuse.

OpenSolaris and FreeBSD are great operating systems, however, Linux has come a long way and has so much to offer me today.

Encoding Video Using Mencoder for the Xbox 360 with AC3 5.1 Surround Sound

2008-11-27T10:35:00.003-07:00

Some time ago, I spent a lot of time learning how to encode video using the MPlayer encoder mencoder. I eventually found settings that worked, but the final video did not support AC3 ( 5.1 ) surround sound.

The question came up to me the other day about the Xbox's ability to play back 5.1, and I thought I would revisit the issue. I found an existing video on the net the said it was an AVI for the Xbox with 5.1 sound. Sure enough, I threw it on a usb drive, plugged it in, and had 5.1 surround sound.

I looked at the specs of the file, and after a few trial and error attempts, I had a dvd converted and playing back on my Xbox 360 in full glorious 5.1 surround sound.

The magic command I use is:

mencoder  dvd://1 -ovc lavc -oac copy -channels 6 -lavcopts vcodec=mpeg4:threads=2:vbitrate=4500:autoaspect:turbo:mbd=2:trell=1:vpass=1 -o test.avi -ffourcc xvid -mc 0 -noskip -vf scale=854:480,softskip,harddup -ofps 24000/1001

This command produces an avi file with a variable-rate mpeg4 video and a 5.1 AC3 audio channel.

A few interesting points on the command. I use two-pass encoding. So, you'd have to run the command 2x changing the vpass to 2 on the second run. Even though I told it to copy the stream, unless I specifically added the "-channels 6" command, mencoder only encoded 2 channels. Another issue is that you have to force the FOURCC to xvid. The final issue I had was the aspect ratio. Without forcing the scale to 854x480 ( 16/9 DVD ), the Xbox would not scale it to 16/9. mencoder allows the -aspect flag, however, as the mencoder man page states, it is only recognized by mplayer. So, scale the video while encoding it, and it plays back fine.

There are lots of other options to mencoder--it's got to be the most complex command I've ever seen. The documentation for mplayer is great, so check there if you have questions.

If anyone knows how to or can figure out if you can encode H264 videos using avi or mpeg format with a 5.1 audio channel, drop me a line.

Handbrake is another great tool for converting videos. They have an XBox 360 preset, but it looks like it only supports faac two channel audio with x264 video. Handbrake is great since it auto-crops the video. Using mplayer/mencoder, I have to play the video first to figure out if it needs cropping and to get the correct cropping parameters to pass into mencoder.

Now, if only I had a huge server farm to encode my DVD collection.

EXT3, EXT4, JFS, XFS Weekend Encounters

2008-10-12T22:00:00.003-06:00

Sometimes I'm a gluten for punishment when it comes to technology. My self-inflicted punishment this weekend was spending time formatting and copying data using xfs, jfs, and ext4 filesystems under Linux.

It started out several days ago when I got the itch to move from ext3 to a different filesystem to test the performance. I've been through this before, but it had been a while. I was tempted to try ext4 right off the bat, but decided to try xfs. I ran xfs for several days, and I really liked it. However, in an effort to fix my sound card not living up to Fedora's 10 promise of glitch-free audio ( this is another story in and of itself ), I had to hard-reboot my computer and ended up with a 0-length file on an opened file. This is not unusual for an unclean shutdown of xfs, so I wasn't mad, but decided to try jfs. A few reboots later, and a lot of time sitting around waiting to backup/restore a 90+GB root partition, jfs was up an running on the root partition. No problems, but I then came across the ext3 options for "xfs-like and jfs-like" behaviour by adding data=writeback,nobh,commit=30 mount options. So, back to ext3 with those options, and I was impressed. No official benchmarks to post, but it felt good and I was back on an ext filesystem.

After running ext3 for a while, I realized Fedora Rawhide had moved from ext4dev ( the pre-release ) to ext4. Why not, I already had the data backed up, so...reboot, rsync, mkfs.ext4, and a cp and I was back up running ext4 with the "data=writeback,nobh,commit=30" mount options.

While these mount options are a little more risky on hard reboots, it appears the worst that should happen is stale data--not a 0 byte file like xfs was so kind to produce. I can live with old data, but corrupt data is where I draw the line.

So far, the ext4 is blazing fast. It's running over a software raid 0 config. On the first reboot, the raid device forced a fsck. I was very impressed the 150GB partition fsck finished in less than a minute--compared to the several minute fsck the ext3 partition took on it's initial boot.

ext4 is the filesystem of choice for me now.

GNOME is Back for me

2008-09-24T18:40:00.002-06:00

I previously posted about KDE 4.1. While I enjoyed it's cool looks, I've found GNOME is currently just more productive for me. So, I find myself running GNOME at work and home. Choice is great, and I'm sure this won't be the last time I try out KDE.

KDE 4.1 Much Improved

2008-08-03T20:06:00.002-06:00

I installed KDE 4.1 today on a few of my machines. I'm quite happy with it now, and I will be making the switch from GNOME back to KDE. KDE 4.0 was too buggy to work with. After a few crashes of KDE 4.0 within the first hour of installation, I went right back to GNOME 2.22.

GNOME is fine for the most part, though there were often times I wanted to customize something that I just couldn't do ( dual headed background limitations, running a separate screensaver process per head slamming the cpus, and lack of any customization in the newly, revamped gdm architecture come to mine ). But I digress. For me, KDE 4.1 is stable, looks great, and so far is not lacking any customizations I need. It took a few minutes of figuring out where the team is headed with Desktop folder views. I think they're way cool, and they help one keep a nice, organized desktop.

If you gave up on KDE 4.0 like I did, jump back on! KDE 4.1 shows the dedication of the KDE team to answering issues and pioneering the way for the Linux desktop.

Flex BlazeDS XML Parsing

2008-07-07T11:08:00.002-06:00

I've received some great comments on my recent entry Simplified BlazeDS and JMS. A few people had some issues, and I wanted to clarify one thing I discovered early on, fixed, forgot, and then was reminded of as others encountered the same problem -- Flex BlazeDS xml parsing does not remove any spaces or linefeeds from the xml between the xml tags and the value. If, for instance, your xml formatter places line feeds between your xml tags and the containing text values, you will encounter problems.

Also I've set up an email at mmartinsoftware@comcast.net for those who wish to email me or want sample code emailed to them. I'll have to look for some hosting space to make it easier down the road.

OpenLDAP 2.4.9 Multi-Master Replication Stability Improved

2008-05-16T09:54:00.004-06:00

I pulled in OpenLDAP 2.4.9 to re-visit some multi-master testing I had done on the previous version. 2.4.9 was a better experience for me than 2.4.8. I re-ran some tests of running two replicated servers on the same machine. I would continuously load in several hundred or thousand user records on one server, then delete them.

The stability is improved from 2.4.8. I have yet to have a server core on me--even after abruptly shutting done one or both servers during a batch load or delete.

My only issue now is that when performing a bulk load, if you interrupt the client during the load ( Control-c ldapadd ) the servers get out of sync. The server you directly attach to in order to load the records contains more records than the "replicated" server. Under most circumstances, this would not be a huge issue, but it does show an easy way for a client to get the servers out of sync.

With the 2.4.9 release of OpenLDAP, I would trust the multi-master synchronization enough now for use on production systems.

Simplified BlazeDS and JMS

2008-05-08T19:29:00.004-06:00

I've been using Flex for some time now, and finally got around to trying out BlazeDS. What really peaked my interest was it's claim to publish and consume JMS messages from both topics and queues.

The one area in web development that I always struggle with is asynchronous messaging from the server to clients over HTTP. I'm pleased to announce that I got a simple application up and running.

The goal of this post is to share the example and lessons learned in order to save others some of the problems I ran into.

My project had a few main goals:

Get a simple pub/sub example up and running
Avoid using the "pre-configured" download. I wanted to get a project up and running "from scratch"
Use my existing ActiveMQ broker running in it's own JVM
Use my existing Tomcat install
Use mxml as much as possible and avoid writing Actionscript
Keep it simple to get a working example

Before I get into the example code, a few things I ran into:

The BlazeDS documentation is good, but far from perfect. The messaging examples switch from mxml to actionscript, so it makes it a little difficult to piece things together. Also, it's not clear in the examples what many of the values point to--i.e. is that the JMS Topic name or the destination name???? I also found a few syntax problems in the examples which caused some "cut and paste" problems for me.
A key piece I discovered after nothing was working at all, was the use of the "-services" flag to the mxmlc compiler. You have to compile your mxml file with the services file you plan to call.
Don't rely on client side faults. Prior to finding out about the "-services" flag, I would call the send method in the client, but nothing happened--the faultHandler was never called on the client, and nothing appeared in the server logs. Once I compiled with the "-services" flag, things began to click.
Make sure the JNDI names are correct. BlazeDS looks up the JMS ConnectionFactory and Topics via JNDI calls to Tomcat's JNDI. Make sure you specify the full JNDI name of the resource. i.e java:comp/env/jms/messageTopic

The example contains 5 main files:

A standard web.xml file with a BlazeDS servlet to handle the remote calls
A Tomcat context.xml file to add the JNDI values for the required JMS resources
A BlazeDS services-context.xml file to set up the services and channels
A BlazeDS messaging-config.xml that stores the messaging service(s).
A test.mxml file which is compiled into the Flex application.

The image to the right shows the file locations in the final WAR layout.

web.xml
The web.xml file is pretty basic. The main 2 things to point out are the HttpFlexSession listener and the MessageBrokerServlet. The servlet loads up you services, so it is passed the location of the services-config.xml file inside the WAR file.


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
<display-name>BlazeDS</display-name>
<description>BlazeDS Application</description>

<!-- Http Flex Session attribute and binding listener support -->
<listener>
<listener-class>flex.messaging.HttpFlexSession</listener-class>
</listener>

<!-- MessageBroker Servlet -->
<servlet>
<servlet-name>MessageBrokerServlet</servlet-name>
<display-name>MessageBrokerServlet</display-name>
<servlet-class>flex.messaging.MessageBrokerServlet</servlet-class>
<init-param>
<param-name>services.configuration.file</param-name>
<param-value>/WEB-INF/flex/services-config.xml</param-value>
</init-param>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>MessageBrokerServlet</servlet-name>
<url-pattern>/messagebroker/*</url-pattern>
</servlet-mapping>
<welcome-file-list>
<welcome-file>index.html</welcome-file>
<welcome-file>index.htm</welcome-file>
<welcome-file>index.jsp</welcome-file>
</welcome-file-list>
</web-app>

context.xml
The context.xml file is for Tomcat. You can read more about it in the Tomcat documentation. The key take away is the "name" attribute is the JNDI name that has to match the jndi name used in the messaging-config.xml file.


<?xml version="1.0" encoding="UTF-8"?>
<Context path="/BlazeTest">
<Resource name="jms/flex/TopicConnectionFactory"
        type="org.apache.activemq.ActiveMQConnectionFactory"
        description="JMS Connection Factory"
        factory="org.apache.activemq.jndi.JNDIReferenceFactory"
        brokerURL="tcp://localhost:61616"
        brokerName="myBroker"/>
<Resource name="jms/messageTopic"
        type="org.apache.activemq.command.ActiveMQTopic"
        description="a simple topic"
        factory="org.apache.activemq.jndi.JNDIReferenceFactory"
        physicalName="messageTopic"/>
</Context>

services-config.xml
The services-config.xml file is used by BlazeDS. It is first used when you go to compile the mxml file. When you deploy the war file to Tomcat, this file is also loaded up by the MessageBrokerServlet.

During compilation, you pass it in like this using the relative path to it from your mxml file.

~/java_libraries/flex3/bin/mxmlc -services=WEB-INF/flex/services-config.xml test.mxml

My current file turns on logging which can help during resolve messaging problems like jndi resources not being found. This file loads in the messaging-config.xml file which is described below.

The channels determine how the client will talk to the server. The endpoint has to match the servlet-mapping url-pattern contained in the web.xml. Since the servlet is grabbing /messagebroker/* and my application is deployed under BlazeTest, the endpoint becomes:

http://localhost:8080/BlazeTest/messagebroker/amf

The "amf" on the end of the URL can be called anything--it's what distinguishes one channel url from another channel url. Here, I am only using one channel.


<?xml version="1.0" encoding="UTF-8"?>
<services-config>
<services>
<service-include file-path="messaging-config.xml" />
</services>
<security/>
<channels>
<channel-definition id="my-amf" class="mx.messaging.channels.AMFChannel">
<endpoint url="http://localhost:8080/BlazeTest/messagebroker/amf" class="flex.messaging.endpoints.AMFEndpoint"/>
</channel-definition>
</channels>
<logging>
<target class="flex.messaging.log.ConsoleTarget" level="Debug">
<properties>
 <prefix>[BlazeDS] </prefix>
 <includeDate>false</includeDate>
 <includeTime>false</includeTime>
 <includeLevel>false</includeLevel>
 <includeCategory>false</includeCategory>
</properties>
<filters>
 <pattern>Endpoint.*</pattern>
 <pattern>Service.*</pattern>
 <pattern>Configuration</pattern>
</filters>
</target>
</logging>
</services-config>

messaging-config.xml
This file stores the JMS services and tells BlazeDS where to get the JMS ConnectionFactory and Topics from the JNDI lookup. The part here that caused be a little problem was making sure the connection-factory and destination-jndi-name values had the correct JNDI name Tomcat expected. Also, note the destination id value. When you set up the consumer or producer in the mxml file, you don't tell it what Topic or Queue to connect to, you tell it what destination to connect to. This makes sense, since the mx:Producer and mx:Consumer attribute is called "destination", but it did throw me for a little loop.


<?xml version="1.0" encoding="UTF-8"?>
<service id="message-service" class="flex.messaging.services.MessageService">
<adapters>
<adapter-definition id="actionscript" class="flex.messaging.services.messaging.adapters.ActionScriptAdapter" default="true" />
<adapter-definition id="jms" class="flex.messaging.services.messaging.adapters.JMSAdapter"/>
</adapters>
<default-channels>
<channel ref="my-amf"/>
</default-channels>
<destination id="message-destination">
<properties>
<jms>
  <destination-type>Topic</destination-type>
  <message-type>javax.jms.TextMessage</message-type>
  <connection-factory>java:comp/env/jms/flex/TopicConnectionFactory</connection-factory>
  <destination-jndi-name>java:comp/env/jms/messageTopic</destination-jndi-name>
  <delivery-mode>NON_PERSISTENT</delivery-mode>
  <message-priority>DEFAULT_PRIORITY</message-priority>
  <acknowledge-mode>AUTO_ACKNOWLEDGE</acknowledge-mode>
  <initial-context-environment>
    <property>
      <name>Context.INITIAL_CONTEXT_FACTORY</name>
      <value>org.apache.activemq.jndi.ActiveMQInitialContextFactory</value>
    </property>
    <property>
      <name>Context.PROVIDER_URL</name>
      <value>tcp://localhost:61616</value>
    </property>
  </initial-context-environment>
</jms>
</properties>
<channels>
<channel ref="my-amf"/>
</channels>
<adapter ref="jms"/>
</destination>
</service>

test.mxml
The test.mxml is the final file. It's the GUI code. The file gets compiled into a swf file as I showed up above. My version uses mx:Producer and mx:Consumer mxml tags, though you can also instantiate these using Actionscript if you'd like. I prefer to write as little script as possible. The main key in this file was that I had to subscribe the consumer to it's destination before it would consume messages. Everything else is pretty simple, and you can see how easy it is to get a client sending and receiving messages to the server-side JMS broker over HTTP. This is why I like Flex!


<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx='http://www.adobe.com/2006/mxml' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
backgroundGradientColors="[0xcfe4ff, 0xffffff]" paddingTop="5" paddingLeft="5"
paddingRight="5" paddingBottom="0" applicationComplete="logon()">
<mx:Script>
<![CDATA[
        import mx.rpc.events.FaultEvent;
        import mx.messaging.*;
        import mx.messaging.messages.*;
        import mx.messaging.events.*;
         import mx.controls.Alert;

        private function messageHandler(event:MessageEvent):void {
          receiveTextArea.text = event.message.body  + "\n" + receiveTextArea.text;
        }
        private function acknowledgeHandler(event:MessageAckEvent):void{
        }
        private function faultHandler(event:MessageFaultEvent):void {
          Alert.show('Fault!', 'Alert Box', mx.controls.Alert.OK);
        }


        private function logon():void {
            consumer.subscribe();
         }

        private function sendMessage():void {
            var message:AsyncMessage = new AsyncMessage();
            message.body = sendTextArea.text;
            producer.send(message);
            sendTextArea.text="";
        }


    ]]>
</mx:Script>
<mx:Producer id="producer" destination="message-destination" acknowledge="acknowledgeHandler(event);" fault=" faultHandler(event)"/>
<mx:Consumer id="consumer" destination="message-destination" message="messageHandler(event)" acknowledge="acknowledgeHandler(event);" fault=" faultHandler(event)"/>
<mx:HBox>
<mx:TextInput id="sendTextArea"/>
<mx:Button label="Send" click="sendMessage();"/>
</mx:HBox>
<mx:TextArea id="receiveTextArea" width="50%" height="200"/>
</mx:Application>

The client app looks something like this:

In Conclusion, BlazeDS is an extremely simple way to use JMS with remote Flex clients over HTTP connections. The back-end configuation is somewhat complex, but mostly boilerplate and not too bad once you piece it together.

I haven't "stress" tested the messaging yet, or done very complex messaging to be able to say how robust and scalable it is, but so far, It does live up to it's claims. Best of all, I can now do asynchronous messaging to remote clients without writing code.

Java Self-Signed Certificates and Firefox 3 beta 5

2008-04-23T09:06:00.004-06:00

A few weeks ago, I started having problems with Firefox 3 beta 5 and my self-signed certificates being used by some Tomcat servers. Though I could add the certificate as an exception, Firefox 3 beta 5 would not let me get in. I was using Java's keytool with the -genkeypair option to create the certificate.

I recently discovered a solution. keytool by default uses the DSA algorithm when generating the self-signed cert. Earlier versions of Firefox accepted these keys without problem. With Firefox 3 beta 5, using DSA doesn't work, but using RSA does. Passing "-keyalg RSA" when generating the self-signed certificate creates a cert the Firefox 3 beta 5 fully accepts.

I'm not sure if this is by Firefox design or not, but at least it's working for me.

More OpenLDAP

2008-02-27T17:48:00.005-07:00

I tested OpenLDAP multimaster replication some more today. I went back to the official relase version 2.4.8. I ran my test on an AMD 64 bit machine with both servers on the same box--different ports and different installation folders. Things ran fine for the most part. Synchronization was consistent in loading and removing 5000 entries like this:


dn: cn=Fred XX_0,dc=mgm,dc=com
objectClass: inetOrgPerson
objectClass: top
givenName: Fred
sn: XX
cn: Fred XX_0

I don't know much about the berkeley db, but my issues seemed to happen after abrupt shutdowns and what I think are database corruptions. slapd cored on each server in the same place at different times during the testing:

Server 1 Core:


#0  is_ad_subtype (sub=0x0, super=0x832430) at ad.c:489
#1  0x00000000004213a7 in attrs_find (a=0x936998, desc=0x832430) at attr.c:647
#2  0x00000000004352bf in test_ava_filter (op=0x41801640, e=0x914a18, ava=0x41800fb0, type=163) at filterentry.c:617
#3  0x0000000000435761 in test_filter (op=0x41801640, e=0x914a18, f=0x41800fd0) at filterentry.c:88
#4  0x0000000000480d9b in bdb_search (op=0x41801640, rs=0x41800ec0) at search.c:845
#5  0x0000000000470be2 in overlay_op_walk (op=0x41801640, rs=0x41800ec0, which=op_search, oi=0x87b2b0, on=0x0) at backover.c:653
#6  0x00000000004710d5 in over_op_func (op=0x41801640, rs=0x41800ec0, which=op_search) at backover.c:705
#7  0x000000000046a56e in syncrepl_entry (si=0x8a31b0, op=0x41801640, entry=0x90add8, modlist=0x418015a8, syncstate=1,
   syncUUID=, syncCSN=0x0) at syncrepl.c:1989
#8  0x000000000046c395 in do_syncrep2 (op=0x41801640, si=0x8a31b0) at syncrepl.c:844
#9  0x000000000046de8c in do_syncrepl (ctx=0x41801df0, arg=) at syncrepl.c:1226
#10 0x000000000041a692 in connection_read_thread (ctx=0x41801df0, argv=) at connection.c:1213
#11 0x00000000004fa6f4 in ldap_int_thread_pool_wrapper (xpool=0x83c890) at tpool.c:625
#12 0x00000035f1e06407 in start_thread () from /lib64/libpthread.so.0
#13 0x00000035f12d4b0d in clone () from /lib64/libc.so.6

After this happened, a restart of slapd would always hang on either server with this stack trace:


#0  0x00000035f1e076dd in pthread_join () from /lib64/libpthread.so.0
#1  0x00000000004e4501 in syncprov_db_open (be=0x8a27c0, cr=) at syncprov.c:2632
#2  0x0000000000470868 in over_db_func (be=0x8a27c0, cr=0x7fffad390610, which=) at backover.c:62
#3  0x0000000000427350 in backend_startup_one (be=0x8a27c0, cr=0x7fffad390610) at backend.c:224
#4  0x000000000042761a in backend_startup (be=0x8a27c0) at backend.c:316
#5  0x000000000040505a in main (argc=4, argv=0x7fffad3908d8) at main.c:932

I'm happy to see Gavin's interest in my blog. It shows his dedication--a tribute to the OpenSource community.

OpenLDAP MultiMaster Replication Redemption

2008-02-27T11:14:00.004-07:00

After some tweaking of my slapd.conf, I was able to get multimaster replication working a lot more reliably than my earlier attempts. I pulled in the latest source code - whatever was comitted to cvs after 2.4.8 release, and tested with this version. Here is my current slapd.conf file syncrepl settings:


#serverID       1  ldap://server1:9009
serverID       2

overlay syncprov

syncRepl rid=1
        provider=ldap://server1:9009
        binddn="cn=Manager,dc=mgm,dc=com"
        bindmethod=simple
        credentials=ldap
        searchbase="dc=mgm,dc=com"
        type=refreshAndPersist
        retry="5 + 5 +"
        interval=00:00:00:05

syncRepl rid=2
        provider=ldap://server2:9009
        binddn="cn=Manager,dc=mgm,dc=com"
        bindmethod=simple
        credentials=ldap
        searchbase="dc=mgm,dc=com"
        type=refreshAndPersist
        retry="5 + 5 +"
        interval=00:00:00:05


mirrormode      true
database monitor

It seems if you switch replication types from refreshAndPersist to refreshOnly, things get messed up. I prefer the refreshAndPersist. I left the interval option in my config, though it's not used in refreshAndPersist mode. Previously, my retry intervals were very short, so I think this is why I was getting un-predictable results in the number of entities replicated. The timeouts may have been hit and I wasn't waiting long enough.

I stress tested OpenLDAP much more than I have FDS. I would stress test by ctrl-c'ing a running slapd process while thousands of adds or deletes were being done to another server. Without hard stopping a server, replication seemed to work well. With hard stopping the server, the only issue I had was sometimes the stopped server would freeze and hang when coming back up and performing an ldapsearch at the same time--some sort of connection deadlock or something. I had to kill -9 slapd and start it again, then it would sync back up. Under most situations, servers being killed and started and killed and started again would not be a common occurrence.

I still saw a few SEGV's sometimes when bringing up a server after stopping it during bulk adds or deletes--like this one I just got. I deleted 5000 entries on server1, then as server2 was processing the deletes, I ctrl-c'd server2. Then bringing server2 back up. I got this:


#0  0x00ace375 in memmove () from /lib/libc.so.6
#1  0x0810a51c in bdb_dn2id_children (op=0x8d71028, txn=0x0, e=0x8be1cd4) at dn2id.c:351
#2  0x0810606f in bdb_cache_children (op=0x8d71028, txn=0x0, e=0x8be1cd4) at cache.c:1008
#3  0x080e4cac in bdb_hasSubordinates (op=0x8d71028, e=0x8be1cd4, hasSubordinates=0xa15caa0c) at operational.c:54
#4  0x080e4e09 in bdb_operational (op=0x8d71028, rs=0xa168c168) at operational.c:101
#5  0x080d3001 in overlay_op_walk (op=0x8d71028, rs=0xa168c168, which=op_aux_operational, oi=0x8b74868, on=0x8b74e38) at backover.c:653
#6  0x080d351d in over_op_func (op=0x8d71028, rs=0xa168c168, which=op_aux_operational) at backover.c:705
#7  0x0808066b in fe_aux_operational (op=0x8d71028, rs=0xa168c168) at backend.c:1868
#8  0x080802b9 in backend_operational (op=0x8d71028, rs=0xa168c168) at backend.c:1885
#9  0x08084531 in slap_send_search_entry (op=0x8d71028, rs=0xa168c168) at result.c:764
#10 0x080e7953 in bdb_search (op=0x8d71028, rs=0xa168c168) at search.c:869
#11 0x080d3001 in overlay_op_walk (op=0x8d71028, rs=0xa168c168, which=op_search, oi=0x8b74868, on=0x8b74e38) at backover.c:653
#12 0x080d351d in over_op_func (op=0x8d71028, rs=0xa168c168, which=op_search) at backover.c:705
#13 0x08076256 in fe_op_search (op=0x8d71028, rs=0xa168c168) at search.c:368
#14 0x08076a47 in do_search (op=0x8d71028, rs=0xa168c168) at search.c:217
#15 0x0807416c in connection_operation (ctx=0xa168c238, arg_v=0x8d71028) at connection.c:1084
#16 0x080748e0 in connection_read_thread (ctx=0xa168c238, argv=0x10) at connection.c:1211
#17 0x0816c9a4 in ldap_int_thread_pool_wrapper (xpool=0x8b4eea0) at tpool.c:663
#18 0x00d0650b in start_thread () from /lib/libpthread.so.0
#19 0x00b30b2e in clone () from /lib/libc.so.6

Running slapd again and it came up just fine and synchronized started again.

OpenLDAP multimaster replication is working a lot better than I first experienced. The multimaster replication is not bullet proof, but it's probably adequate now for many situations.

OpenDS Looks Promising

2008-02-24T09:31:00.002-07:00

Sun's OpenDS project -- https://opends.dev.java.net/ -- looks to be a very promising LDAP implementation. I haven't gotten into it much, but as I installed it this morning, I was pleasantly surprised.

The install was the easiest of FDS or OpenLDAP. A nice gui steps you through the initial install. Replication setup was simple, as the gui prompts you to identify another server already participating in the replication. OpenDS, by default, supports multi-master replication. I believe this is, in fact, the only replication it supports. I think it would be useful to have the ability to force read-only replicated servers, but I didn't see if this was possible.

I easily set up 3 servers on my machine ( a Dual-core Opteron 185 with 2GB of memory running Fedora 8 64bit ). Using OpenDS, I generated the example ldif of 10k users, and loaded it up. Replication started immediately. OpenDS provides a nice, simple gui for simple monitoring, so it was easy to see the updates going to the other 2 servers. participating in the replicated cluster.

The ldif additions were slow--it took several minutes to load the 10k users. My machine load went up past 7, and with running several servers, my computer was having to swap memory quite a bit. During the load, I shut down one server, brought it up for a minute or two, then down and up again. I wanted to see how this server would handle the synchronization when it was not up.

When the load finished, the replicated server that stayed up the entire time, had the same number of entries as the server I loaded the ldif into--10,003. The server I shut down, however, was about 50+ entries short with some errors in the replication log:

[24/Feb/2008:09:22:26 -0700] category=SYNC severity=MILD_ERROR msgID=14876739 msg=Could not replay operation AddOperation(connID=-1, opID=47, dn=uid=user.1333,ou=People,dc=example,dc=com) with ChangeNumber 000001184c39fe467f4300000537 error Canceled

Apart from that, OpenDS is off to an excellent start--especially for it's age. It's by far the easiest server to get up and running. I'll be watching as it matures to see how it performs and stabilizes.

OpenLDAP vs Fedora Directory Server (cont)

2008-02-22T17:02:00.002-07:00

Today I tried to get OpenLDAP going one last time. I erased the database on each machine and recompiled the code from scratch. My simple 10-person adds and deletes were working and synchronizing ok, so I was starting to think it was working. I bumped up the ldif to have 5000 names. Synchronization started up, but then things went crazy and eventually one of the servers seg-faulted. That was enough OpenLDAP for me.

I went ahead and shut down OpenLDAP and fired up FDS. In no time at all, I had a simple Master-slave replication set up and adding and removing 5000 names ( using ldapadd ) was working perfectly. I still need to get it going in a multi-master setup--looks easy enough.

My recommendation: If you're jumping into LDAP, start up with FDS.

OpenLDAP vs Fedora Directory Server

2008-02-21T19:17:00.002-07:00

I was recently coming up to speed on LDAP. Eager and ready, I got OpenLDAP version 2.4.7 and came up to speed with LDAP in general and had a server up and running fairly quickly.

While working with OpenLDAP, and editing and loading ldifs, I was quickly hoping some tool existed to manage the basic ldap tasks. I installed phpLdapAdmin which seemed to do the job. I enabled the ppolicy module and found out that trying to clear a users password inside phpLdapAdmin ( setting the text box to empty and then committing) caused OpenLDAP to exit with an assertion error. Ouch!! Determined that this was a pretty obvious bug, I found the latest source code had already fixed the issue. I installed the patch, and no longer did slapd exit when setting an empty password.

Today, I noticed 2.4.8 was released which also had the password fix, so I pulled that in and upgraded.

The next OpenLDAP task was to get multi-master replication up and going. After getting two servers set up I was able to add a single user and remove it from either server. Everything looked good. I decided to try refreshOnly syncing instead of refreshAndPersist. However, after changing the sync method on both servers, as soon as I restarted the servers and both servers connected, one would seg fault. I changed both back to refreshAndPersist, tested the single add and delete, and went to the next step--bulk loads.

I added 10 users to an ldiff. When I loaded them up, all ten would load up fine into the local server, but only one or two users from the list would get replicated to the other server. After deleting and adding several times, I could never get all 10 to replicate. I thought the computers not being ntp synced was an issue, but getting them synced up did not fix the issue.

I realize N-Way multi master has only been around since October or so. It would appear to me, it's not yet ready for production use if you are planning to do multi-master replication.

While working with OpenLDAP, I learned about the existence of Fedora Directory Server, and running Fedora myself, I got that up and going too. The experience has been completely different. The initial setup was simple ( RPMS -- no compiles ). The web-based java management tool is tons more functional than phpLdapAdmin, the documention is incredible, and it has yet to crash on me. FDS now manages my simple home network user accounts, and is now my LDAP server of choice.

Tomorrow, I will test FDS mult-master replication and report back on my findings. The multi-master replication is more mature than OpenLDAPs, so I have high expectations.

LVM Disk Striping vs Raid0

2008-01-01T16:38:00.001-07:00

With the recent migration to Fedora 8, on my desktop I decided to put the root device onto a manually created striped lvm during Fedora's install with the default stripe size across the group of 2 partitions--one on each disk.

Today, I ran hdparm -tT on the striped volume, but was discouraged by the Timing buffered disk reads--they maxed out about 60MB/sec -- pretty much the same as a normal partition without any raid or lvm striping. I backed up the install, removed the lvms and stuck a Raid0 back on. After restoring the backup, I'm back to some nice speed reads:

/dev/md0:
Timing cached reads: 1222 MB in 2.00 seconds = 611.27 MB/sec
Timing buffered disk reads: 344 MB in 3.01 seconds = 114.25 MB/sec

It would seem that LVM striping is no substitute for software Raid0.

Cyrus mail server and Maildir conversion woes

2007-12-31T20:33:00.000-07:00

Over the holiday break, I decided to switch from Ubuntu to Fedora Linux distribution. In part, because I'll be using RedHat at work now, and also it seems Fedora is a little more bleeding edge--more on that in a future post.

Anyway, Fedora uses Cyrus as the default imap server, so I installed it and read up to see what it offers over other mail servers. Seems popular enough and feature rich. The initial install was fine, but I then needed to convert my existing Maildir saved emails to Cyrus. While the format of the files is the same, the file names themselves are different.

After several attempts at various imap conversion tools to no avail--and I did not want to spend time making perl modules work and other various hacks--I tried my own little copy/rename script. This worked, however, when Thunderbird went to pull in the 2000+ emails, it completely jacked up the cpu load and continuously spawned multiple imapd processes every minute or so. I'm not sure if that was Thunderbird,Cyrus,or a residual effect of me doing raw copies of existing email files over. Several hours later and almost reverting to a Maildir compatible server ( dovecot is bundled with Fedora 8 too ), I found a simple solution.

I fired up Evolution and to my surprise, it supports direct Maildir folders in addition to imap/pop/etc. After registering a Maildir server and my Cyrus imap server, it was just a matter of drag and drop from Maildir to the Cyrus server--no other cludges or complex imap scripts needed.

I suppose the imap convert scripts ( mailutil, isync, etc ) serve a purpose, but it seems they are not without problems and several other hack scripts seem necessary to get them to work with a standalone Maildir folder.

Jumping On

2007-11-13T22:56:00.000-07:00

Here I go. Jumping onto the blogger bandwagon.