I’d like to take a different approach with this review, one that descends below the GUI to highlight aspects that server administrators in particular would be more interested in.

OpenSolaris is just a few months old now, with its first version being 2008.5, released two months ago. Since then, Sun engineers and community members alike have been making bi-weekly updates to its various components (currently at build 93) in preparation for the next full release, 2008.11. One who has a OpenSolaris 2008.5 installation may track these intervening builds as they become available using the Image Packaging System’s update features. Haters of SVR4 package management rejoice.

Before I begin, I’m going to assume that you, the reader, have already heard plenty about the big, oft-quoted features such as ZFS, DTrace, and SMF. If you haven’t, get on over to those links and read up as you’re missing out on some really good stuff. If you have, I’ll show off some compelling new sub-features of those systems, as well as other, unrelated ones. So sit back and read along and try this stuff out when you’re done!

OpenSolaris Is A Storage Multi-Tool

This preview focuses on the new storage-related components of OpenSolaris. This is a category that has received quite a bit of attention over the past two years with many new components being integrated into the OS.

• ZFS. Prior to the availability of ZFS, Solaris was pretty much on-par with the built-in storage management features of peer OSes, with advantages in some areas of storage management (multipathing and fibre channel) and deficiencies in others (UFS and LVM were getting tough to manage in ever-growing, multi-TB environments.) The inception and inclusion of ZFS drastically improved the situation and brought the concept of storage management to a level of capability and availability (free!) that the industry had yet to see.But don’t think that the designers of ZFS took a break from the action.

  ZFS as it exists in OpenSolaris sports major performance improvements and several additional features of note:

  • GZIP Compression. GZIP compression may be applied to a ZFS filesystem of ZVOL in addition to the original LZJB scheme. GZIP compression offers better compression ratios when compared to LZJB, but at a higher cost in terms of CPU power needed. If the best compression possible is required and CPU capacity is not an issue, this new compression method will make you happy.
  • Case Insensitivity. One may set a ZFS filesystem to operate with no regard to the case of file names, a la Windows NTFS/FAT and default Mac OS HFS filesystems. This feature was added in conjunction with the new CIFS server (see below).
  • Autoreplace. This is a new property of a zpool, which defaults to “off”. What it allows you to do when set to “on” is if a drive in a zpool dies and is subsequently pulled and replaced, ZFS will detect this and automatically bring it back into the pool and do what it needs to do with it to restore the pool to an optimal state. When left in the default state, manual intervention after the drive replacement is required.
• Easy iSCSI. One of the first big feature adds after ZFS went GA is the ability to easily create LUNs for export via iscsitgtd. Now, remember that iscsitgtd serves out block devices, not filesystems, so one must create and use raw ZVOLs that get their own device entries in /dev/zvol/… so that they can be treated like any raw disk device. After creating a ZVOL, exporting it out via iSCSI is as simple as:
  zfs create -V 256G pool/my/zvol
zfs set shareiscsi=on pool/my/zvol
- or do it in one command -
zfs create -o shareiscsi=on -V 256G pool/my/zvol
  You can also create LUNs that are hosted on UFS, or any other supported filesystem for that matter. Management of the iSCSI LUNs presented by iscsitgtd is accomplished via the iscsitadm(1M) utility so you can set up things such your custom IQNs, ACLs, CHAP or RADIUS auth, iSNS properties, and so on.
• Served Via CIFS. No, this is not a re-packaged Samba with Solaris-specific tweaks. This is the real deal - a native, fully integrated CIFS server that implements the CIFS/SMB LM 0.12 protocol and MSRPC services. It can run in the simple Workgroup mode, or as a member of a Windows AD domain with the full ability to use a domain controller for conferring access and other rights, including the mapping of AD users to UNIX users (so this means that the ZFS or UFS filesystem that comprises a CIFS share can also be exported via NFS in dissimilar environments.) This makes OpenSolaris a truly viable alternative to Windows Server for high-performance, integrated CIFS share serving. Combined with the filesystem management of ZFS, this new CIFS server feature is very compelling. Have a ZFS filesystem that you need exported to some Windows (or Mac, or Linux) boxes? Just like it was with iSCSI, it’s this simple:
  zfs set sharesmb=on pool/my/fs
  You can set additional share parameters, such as its advertised name by replacing “=on” with other arguments. See the section for the set option in the sharemgr(1m) man page. Management of LM users, groups and server mode are accomplished with the separate smbadm(1M) command.
• NDMP Backups. That’s right, a new NDMP service is now present for all your enterprise backup needs. Have some of those expensive Legato Networker NDMP licenses to burn, or want back up a NetApp or other NDMP-capable device to a Sun Fire X4540 so you can pitch it and its support contract out the window? Fire up the NDMP daemon and go to town with the ndmpadm(1M) command. This new service in OpenSolaris supports NDMP versions 2, 3, and 4. There is also a nifty way to get statistics on your NDMP sessions with the ndmpstat(1M) command.
• COMSTAR. Or Common Multiprotocol SCSI Target, is quickly becoming the in-kernel nexus of sorts for exporting a generic “block device” outside the system over an array of protocols and transports. What this subsystem allows you to do is take a ZVOL and export it over things such as Fibre Channel and FCoE. That’s right, if you have a system with Qlogic QLA/QLE24xx cards, you can turn them into a target rather than an initiator and serve LUNs over FC on your SAN. Your OpenSolaris box is looking like a classic storage array now, but with far more features and flexibility. Fiber Channel over Ethernet (FCoE) is also an option. Future plans include bringing the aforementioned iSCSI server under COMSTAR’s domain as well. Earlier Qlogic HBAs such as the 2Gb models (QLC23xx) are not supported as targets as those cards lack features required in their firmware to put them in such a mode.
• Replicate With AVS. Sun has offered a filesystem-agnostic block-level replication software suite for some time called StorageTek Availability Suite, or AVS. This was a pay-for product, but Sun has graciously donated it in full to OpenSolaris, so now it is free to use. AVS allows you to configure synchronous or asynchronous replication over the network to a remote sever, with additional capabilities such as shadow images (termed as “Instant Image” in AVS). AVS lives in-kernel, and situates itself between the filesystem (ZFS, UFS, etc) and the disk devices, copying blocks off to their configured destination. Here are some demos by lead developer Jim Dunham that demonstrate the use and capabilities of AVS.
• SAM-QFS. As with AVS, SAM-QFS was formerly an unbundled, pay-for product from Sun, but has been open sourced and provided as a integrated part of OpenSolaris. SAM-QFS has two major components - QFS, which is a SAN-based multi-writer and reader filesystem, and SAM, a hierarchal storage management (HSM) system which sits on top of QFS.QFS in particular is found in both the HPC and service provider data center role where multiple nodes require concurrent read/write access to the same file system over fibre channel or iSCSI. It can be utilized in a single-writer setup, where one node can write but all others are read-only, or a multi-writer setup, where > 1 nodes require write access. In the latter case, additional infrastructure is required in the form of a metadata server, the role of which is to manage and coordinate locks and write access amongst the involved nodes.

  Along with QFS, there is the SAM component. This allows one to age data off to cheaper bulk storage (cheap SATA arrays) and/or tape-based long term storage.

That’s a snapshot of the major additions, but there are still a lot of smaller projects that have been integrated, or are on the verge of being made available such as pNFS, MMS, ADM, and Honeycomb. As you can see, OpenSolaris offers quite a bit for those looking for highly flexible ways to store, manage, and export data to and from other systems… and it’s all built in and fully functional. No licenses or hidden costs.

The OpenSolaris Storage Community is a great place to keep track of what’s new, and provides a very nice visual representation of the various storage layers and components in OpenSolaris.

Hopefully this will work well enough to allow Solaris users to read and write to HFS or HFS+ formatted disks and disk images. This includes iPods that were initialized on a Mac. Please check out the page I made for it and lend a hand if you’re interested!

Technorati Tags: solaris, opensolaris, sun, mac, macos, apple

Mercury is a Dell PowerEdge 860 with a Intel Xeon X3220 (quad core, 2.4Ghz) and 4GB 8GB of 667Mhz DDR2 RAM. Unlike lithium, mercury’s storage is entirely internal in the form of two mirrored 500GB SATA drives. This is to keep the entire package in 1 rack unit of space to keep colocation costs down.

What really excites me about this new server is that it is running Solaris 10 8/07 (lithium was running a very patched Solaris 8 FCS!). Solaris installed without a hitch and the 860’s onboard BCM5721 NICs are recognized by the bge driver, as are its IPMI baseboard controller by the bmc driver. The chipset on this system is the Intel ICH7 and unfortunately the Solaris ahci driver supports only the ICH6 at the moment, so the drives are running just fine in IDE compatibility mode.

This upgrade wasn’t just a mere update of hardware and OS. I also completely changed how the mail storage works and also make use of ZFS file systems for each user home directory and virtual web site:

1. Out with uw-imap, in with Cyrus. All mail is delivered to Cyrus, so there are no more maildir-style spools sitting in each person’s home directory.
2. To take advantage of Cyrus’s features, elemental.org is now operating its own Kerberos realm, ELEMENTAL.ORG. This is my first time running my own Keberos KDC, and I love it. Cyrus and Sendmail, via SASL, now offer GSSAPI authentication. Using Solaris’s pam_krb5_migrate.so.1 PAM module, as people log in with their UNIX passwords, a Kerberos principle is made for them and they are granted tickets. Pine is configured to connect to Cyrus and authenticate with GSSAPI, so shell users don’t have to type in or save their password when accessing their email!
3. As I mentioned, all user data is now stored on a mirrored ZFS pool. Each user and virtual website gets their own ZFS file system and this will allow me to keep tabs on disk usage (and easily delete a user or site if the need should arise.) The zpool’s net size is 442GB.
4. All incoming email is goes through greylist, ClamAV, and finally SpamAssassin milters.
5. I’m more at ease and familiar with Solaris’s SMF facility now, having made a point to write SMF manifests for the services I’m running rather than plain old init scripts.

In addition, I’m now monitoring several aspects and services on the new system using Cacti.

Here’s to another 8 years of hopefully trouble-free operation!

Technorati Tags: Sun, Dell, Solaris, ZFS

1. Log in to the Actiontec’s web interface (typically by going to http://192.168.1.1/)
2. Select Network, click on “Ethernet (Broadband)” and its edit icon. Down the page, you’ll see a button labled “Release”. It’s important to release the IP address VZ’s network has given the Actiontec, or it’ll refuse to allot one to your Airport Extreme once you bring that up in its place.
3. Immediatly turn off the Actiontec. Remove the “WAN” ethernet cable from it, and plug it into the “WAN” port of your Airport base station. Turn the Airport on.
4. The Airport base station should boot up and request an IP from VZ’s DHCP server. Speaking of which, the “Internet Connection” setting in the Airport should be “DHCP” and not “PPPoE”. VZ no longer uses PPPoE on its FiOS lines.
5. Configure your Aiport wireless network as you see fit and you’re done. No more Actiontec.

Step 2 is muy importante. If you don’t do that, your Airport base station will be sitting there with a blinking amber light because the VZ network is refusing to give it an IP, simply because it still thinks that your (no longer operating) Actiontec has it.

Technorati Tags: apple, airport, verizon, fios

I was lucky enough to present a slide show on how we at UMBC have been combining OpenAFS with new ZFS and Zones features of Solaris 10 to obtain a more resilient AFS server infrastructure. You can view a PDF of my presentation here.

Technorati Tags: afs, solaris, zfs, zones, opensolaris

6510364 “War Mode” in ALOM-CMT is required by the US NAVY which is currently missing

Awesome. I love knowing that my T1000s have a “war mode” now. Perhaps Sun should call it the SkyNet T1000 ;) Hopefully it’s the feature and not the US Navy that was “currently missing.”

Technorati Tags: sun, coolthreads, sparc, ultrasparc, solaris, navy, war

In particular, I take issue with this comment from Mr. Zaitcev:

“This is the problem OpenSolaris is facing today in the nutshell: it has no breadth. It has a very limited number of excellent technologies, such as ZFS.”

No breadth? Care to, well, add some breadth to that statement, Mr. Zaitcev? Making a comment like that doesn’t mean you can toss out just one perceived example and end your argument at that.

It would appear that all of Mr. Zaitcev’s experience with Solaris/OpenSolaris comes from reading 3rd party accounts of the big new features in Solaris. This is exactly what I referred to in my opening paragraph… all these anti-Solaris pundits more than likely have zero hands-on expeience with the stuff they’re harshing on. People like Mr. Zaitcev read anecdotes and stories, come up with their own idea as to how things are based on those stories, and produce comically uninformed jabs posts such as the one linked above.

No breadth? Just what is the breadth that Mr. Zaitcev thinks is missing? Is breadth in this case even quantifiable? Is his supposition based solely on the age old (and aged) driver count argument? Does Mr. Zaitcev think that all Solaris is, is an ancient kernel which happened to have a few new concepts tacked on top of it?

I would bet that if Mr. Zaitcev sat down and tried to use Solaris in a real-world environment, he’d soon learn that Solaris has everything one needs in a data center environment… he just hasn’t discovered them (or read about them, natch) yet for himself. Who knows, perhaps he’d even appreciate them.

Technorati Tags: linux, solaris, opensolaris, fud, sun

It appears that DaveM (Linux networking and SPARC port guru) has gotten seriously wound up in response to a blog post by Jeff Bonwick (Sun’s storage and kernel guru.)

As one can see in Jeff’s post, the suject he wrote about was within the greater context of using Solaris as a storage appliance OS (something I have an interest in) and why Solaris/OpenSolaris can and would excel when it comes to being the kernel of a storage OS.

I’m a storage guy. In the course of my work I have to not only work with Solaris hosts on my SAN, but also Windows and Linux (and soon, AIX). So I have a front-row seat when it comes to witnessing and dealing with how these various OSes deal with storage, from the filesystem to multipathing, to the HBA… and let me tell you, Linux is quite not the joy in this specific area as most people think it is on a general, all-encompassing level.

And that’s what Jeff’s angle was.

Now on to Dave’s rebuttle.

“The implication is that Linux is not rock-solid and that it does break and corrupt people’s data. Whereas on the other hand Solaris, unlike the rest of the software in this world, is without any bugs and therefore won’t ever break or corrupt your data.”

No OS comes without fault, but some OSes have faults that are more glaring than others in their analogous areas. Staying within the storage context of this discussion, I have to say, again, Linux is no shining star here.

ReiserFS is arguably the most advanced fs in terms of features when it comes to the portfolio of Linux file systems, but its issues with stability are such that you’re really walking on eggshells whenever you employ it. I have been personally told too many first-hand accounts and read plenty more on the Internets regarding its tendency to be fine and then fail spectacularly. It has been likened to a time-delayed /dev/null of sorts, and the future of it is in doubt with the legal troubles of its designer and Namesys limbo. Is any version of ReiserFS a viable Linux storage technology for a production environment? I say No. That’s sad because I dare say at one point ReiserFS had some promise.

EXT2 and 3… tried and true. Very stable and moderately fast for most tasks. But it’s an “old guard” file system. As such, it’s not very flexible, and any flexibility it gets comes from using a volume manager underneath of it. In the days where the notion of handing a server a 1TB LUN is nothing to blink at, this inflexibility can be suffocating in a dynamic environment. These “old guard” file systems (yes, Solaris’s UFS is one of them, too) are more like mere utility file systems than practical ones for today’s mass storage needs. It’s good for holding a machine’s OS and that’s about it.

XFS… Of all the file systems in the Linux file system portfolio, this one gets the gold star. Stable, fast, and decently scalable with the large amount of data you can stuff in it… but it still suffers the same problems EXT[23] and other “old guard” file systems do in terms of flexibility. In other words, it’s just a file system. Keep in mind that this critique is coming from a guy who worked with XFS on IRIX often and absolutely loved XLV… back in its the day.

As it stands now, the mainline Linux kernel doesn’t offer anything which embodies the file system triple play: being stable && fast && flexible. Solaris’s ZFS has this. I’ve so far entrusted 30TB of spinning rust to it, and it has yet to let me down. Sure, there are projects here and there that have the eventual goal endowing Linux with a ZFS analog, but as of right now they’re nothing production quality and are definitely not something a admin can call RedHat to get support for.

There are plenty of other aspects to the storage context… the fibre channel stack, for one, and other things such as multipath IO implementations and volume manager and management layers (which Linux has a host of… not necessarily a good thing… LVM, LVM2, MPIO, RDAC… it makes your head spin.)

But as far as this storage-oriented discussion goes, file systems are indeed the make or break aspect. This is why Jeff said what he said. Linux has no ZFS. Windows has no ZFS. It is not that Linux or Windows need ZFS itself in order to compete, it’s that they need to develop and employ the concepts that ZFS implements and do so as clearly and concisely as ZFS has.

Anyway, enough about storage. Now, why is it that the Linux community (let alone a prominent member of it) has to react so violently to any questioning of its perceived superiority? Is it misplaced or excess pride? Have they not tried things other than Linux recently and they’re just flying with blinders on? Is it just the social culture which prevails within it? What ever it is, seeing posts like Dave’s makes my toes curl with embarrassed amazement.

A friendly message to Dave: Chill the ad hominems, mkay? Crying “FUD!” at the mere sight of someone who you perceive as poo-poo’ing an aspect of your interest doen’t typically translate into a well thought-out rebuttle. You took the low road and tried to convey Jeff as being some instrument of some nefarious, Mr. Burns-like person at Sun. Is vilifying instead of cool-headed technical discourse really your desired style? Has anyone come biting at you saying “oh, he’s a Linux kernel developer, so he has an agenda”?

Technorati Tags: linux, solaris, opensolaris, storage, fud, zfs, fuzzy bunnies

Well, I want to write some more about this with an industry-wide perspective because as of this past Thursday, Apple is now playing a similar game regarding their Xserve RAID systems. With the release of RAID Admin Tools 1.5.1 and associated firmware, Apple has removed LUN Masking as a feature of the Xserve RAID. Yep. Removed it. In a minor version release of the software, no less. Absolutely astonishing.

So, with the Sun StorageTek 6140 and its crippled features (unless you fork over $10+ mega bucks for a Storage Domains license pack of adquate seat count) and Apple rather brashly removing LUN Masking for no real stated reason and, to top it off, without warning, where does this leave us? And what of the (otherwise reputable) mid-range storage vendors who are left (HP? IBM?); who’s to say they won’t pull a similar stunt down the line?

Well, I know IBM is out of the picture for me as they OEM the same LSI Engenio system that Sun uses for the 6140. Yep, both IBM and Sun sell the exact same system, only IBM calls it the DS4700 Express and Sun calls their version the StorageTek 6140. Their only appreciable difference is one comes in IBM Black and the other in Sun Silver. You also have to buy the IBM equivalent of the 6140’s Storage Domains, which IBM calls “Partitions”. Talk about a screwed up sense of storage terminology.

Anyway, that pretty much leaves HP, and I’m petty unfamiliar with their product line or prices. I don’t even know if I can even get HP kit since I’m not aware of any current State of Maryland purchasing contract with them for this sort of stuff.

So what’s with this apparent vendor hate of LUN Masking in mid-range systems, anyway? One either has to pay out the nose to have it (regarding Sun and IBM) or it’s there but disappears into the night (Apple). Crikey. Whoever does product planning at Engenio, Sun, IBM, and Apple needs a serious reality check. For us people where mid-range is high-end, this behavior matters quite a bit. It just seems like feature sets are imploding rather than expanding, removing a distinct competitive advantage from these products.

Technorati Tags: sun, ibm, engenio, lsi, apple, solaris, storage

Sun hasn’t had a Intel CPU in its product line since Sun discontinued its Pentium 4-based V60z server several years ago when the company was teething its new x86 product line. This product line eventually developed into the exclusively AMD Opteron-based servers we have today. Intel/Xeon was out, AMD/Opteron was in. With today’s announcement, both Intel and AMD will now share Sun’s x86 product portfolio.

I’ve noticed that reaction to this news has been mixed, with some saying it’s good, and others saying “WTF, mate?”. Sun’s Opteron-based Galaxy servers are top-notch, so this has lots of people utterly surprised… like having a great night out with someone and then being dumped on the doorstep.

Well, it’s surprising news to say the least. As I thought about it more, though, it isn’t bad at all for Sun, and really isn’t all that forboding for AMD. In exchange for a Intel-based product line, Intel will seriously push Solaris for Sun. That is exquisitely good news. Sun now has a product line which can serve both AMD and Intel customer preferences.

Think of that “iPod Halo Effect”. Sun doesn’t have to turn away customers who want Intel CPUs now, and with Intel pushing Solaris, hopefully more applications and thus more Solaris installations will be in customers’ data centers. I dare say that those customer will like Solaris, and perhaps look to buy (more) Sun servers.

Besides, Sun isn’t the only company to straddle the divide between AMD and Intel. Dell, a traditionally staunch Intel ally, added Opteron servers to its product line last year, as did IBM. HP has offered systems with CPUs from both companies for at least 1.5 years, if not longer.

It’s a move that makes sense, especially for Sun, and that in itself something we should applaud… moves that make sense (duh!)

Technorati Tags: sun, intel, amd, solaris