Its hardware is a quad core Xeon-equipped Dell PowerEdge 860, a small 1u server. Its pair of internal drives are Seagate SATA2, and were connected to the on-board Intel ICH7-based SATA controller. But there was something fishy about this in that the Solaris ahci SATA driver never attached to it and instead the drives ran in IDE mode. Despite my best efforts, I couldn’t change this. I eventually found out the reason - Dell crippled the SATA controller in the system BIOS to allow only IDE mode!

So this server was sold with “SATA drives”, which would imply a fully functioning SATA controller to drive them… but not quite. IDE mode means there were no benefits of SATA NCA and other niceties.

To fix this, I got a LSI SAS3041E-R controller - a 4x PCIe card that uses the LSISAS1064E chipset and offers 4 SATA ports. In Solaris land, this card would be driven by the mpt driver, a proven driver as the LSI SAS 1064 and 1068 chipsets are used to drive the on-board hard drives in pretty much every current Sun x86 and Niagara-based SPARC systems.

I installed this card in the single 8x PCIe slot in the PE860, and ran a 24″ SATA cable from it to HDD1, and used the existing Dell cable that connected the on-board controller to HDD1 to connect HDD0 to the card. After some fiddling in /boot/solaris/bootenv.rc to tell the kernel the new device path to its boot drive, the mpt driver attached and I was good to go.

I kicked off a SVM mirror resync as a basic test of sequential IO, and I hit 75MB/s reading from one drive and writing to the other. Not bad. A zpool scrub of my mirrored ZFS pool of 66.5GB of data (pool is 444GB in size) took just over an hour.

So if you’re thinking about a 4 or 8 port SAS/SATA card, consider the LSI SAS3041 or SAS3080/3081 cards, respectively. Both come in PCI-X and PCIe flavors and are supported by Solaris (and OpenSolaris) just fine.

/usr/X11/bin/scanpci output:
pci bus 0x0001 cardnum 0x00 function 0x00: vendor 0x1000 device 0x0056
LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS

Kernel boot messages:
scsi: /pci@0,0/pci8086,2779@1/pci1000,3090@0 (mpt0): Rev. 8 LSI, Inc. 1064E found.
scsi: /pci@0,0/pci8086,2779@1/pci1000,3090@0 (mpt0): mpt0 supports power management.
pcplusmp: pciex1000,56 (mpt) instance 0 vector 0x38 ioapic 0xff intin 0xff is bound to cpu2
scsi: /pci@0,0/pci8086,2779@1/pci1000,3090@0 (mpt0): mpt0 Firmware version v1.17.2.0 (IR)
scsi: /pci@0,0/pci8086,2779@1/pci1000,3090@0 (mpt0): mpt0: IOC Operational.
scsi: /pci@0,0/pci8086,2779@1/pci1000,3090@0 (mpt0): mpt0: Initiator WWNs: 0x500605b0000fa840-0x500605b0000fa843
pcie_pci: PCIE-device: pci1000,3090@0, mpt0
genunix: mpt0 is /pci@0,0/pci8086,2779@1/pci1000,3090@0
scsi: sd4 at mpt0: target 4 lun 0
genunix: sd4 is /pci@0,0/pci8086,2779@1/pci1000,3090@0/sd@4,0
genunix: /pci@0,0/pci8086,2779@1/pci1000,3090@0/sd@4,0 (sd4) online
scsi: sd3 at mpt0: target 5 lun 0
genunix: sd3 is /pci@0,0/pci8086,2779@1/pci1000,3090@0/sd@5,0
genunix: /pci@0,0/pci8086,2779@1/pci1000,3090@0/sd@5,0 (sd3) online

I’d like to take a different approach with this review, one that descends below the GUI to highlight aspects that server administrators in particular would be more interested in.

OpenSolaris is just a few months old now, with its first version being 2008.5, released two months ago. Since then, Sun engineers and community members alike have been making bi-weekly updates to its various components (currently at build 93) in preparation for the next full release, 2008.11. One who has a OpenSolaris 2008.5 installation may track these intervening builds as they become available using the Image Packaging System’s update features. Haters of SVR4 package management rejoice.

Before I begin, I’m going to assume that you, the reader, have already heard plenty about the big, oft-quoted features such as ZFS, DTrace, and SMF. If you haven’t, get on over to those links and read up as you’re missing out on some really good stuff. If you have, I’ll show off some compelling new sub-features of those systems, as well as other, unrelated ones. So sit back and read along and try this stuff out when you’re done!

OpenSolaris Is A Storage Multi-Tool

This preview focuses on the new storage-related components of OpenSolaris. This is a category that has received quite a bit of attention over the past two years with many new components being integrated into the OS.

• ZFS. Prior to the availability of ZFS, Solaris was pretty much on-par with the built-in storage management features of peer OSes, with advantages in some areas of storage management (multipathing and fibre channel) and deficiencies in others (UFS and LVM were getting tough to manage in ever-growing, multi-TB environments.) The inception and inclusion of ZFS drastically improved the situation and brought the concept of storage management to a level of capability and availability (free!) that the industry had yet to see.But don’t think that the designers of ZFS took a break from the action.

  ZFS as it exists in OpenSolaris sports major performance improvements and several additional features of note:

  • GZIP Compression. GZIP compression may be applied to a ZFS filesystem of ZVOL in addition to the original LZJB scheme. GZIP compression offers better compression ratios when compared to LZJB, but at a higher cost in terms of CPU power needed. If the best compression possible is required and CPU capacity is not an issue, this new compression method will make you happy.
  • Case Insensitivity. One may set a ZFS filesystem to operate with no regard to the case of file names, a la Windows NTFS/FAT and default Mac OS HFS filesystems. This feature was added in conjunction with the new CIFS server (see below).
  • Autoreplace. This is a new property of a zpool, which defaults to “off”. What it allows you to do when set to “on” is if a drive in a zpool dies and is subsequently pulled and replaced, ZFS will detect this and automatically bring it back into the pool and do what it needs to do with it to restore the pool to an optimal state. When left in the default state, manual intervention after the drive replacement is required.
• Easy iSCSI. One of the first big feature adds after ZFS went GA is the ability to easily create LUNs for export via iscsitgtd. Now, remember that iscsitgtd serves out block devices, not filesystems, so one must create and use raw ZVOLs that get their own device entries in /dev/zvol/… so that they can be treated like any raw disk device. After creating a ZVOL, exporting it out via iSCSI is as simple as:
  zfs create -V 256G pool/my/zvol
zfs set shareiscsi=on pool/my/zvol
- or do it in one command -
zfs create -o shareiscsi=on -V 256G pool/my/zvol
  You can also create LUNs that are hosted on UFS, or any other supported filesystem for that matter. Management of the iSCSI LUNs presented by iscsitgtd is accomplished via the iscsitadm(1M) utility so you can set up things such your custom IQNs, ACLs, CHAP or RADIUS auth, iSNS properties, and so on.
• Served Via CIFS. No, this is not a re-packaged Samba with Solaris-specific tweaks. This is the real deal - a native, fully integrated CIFS server that implements the CIFS/SMB LM 0.12 protocol and MSRPC services. It can run in the simple Workgroup mode, or as a member of a Windows AD domain with the full ability to use a domain controller for conferring access and other rights, including the mapping of AD users to UNIX users (so this means that the ZFS or UFS filesystem that comprises a CIFS share can also be exported via NFS in dissimilar environments.) This makes OpenSolaris a truly viable alternative to Windows Server for high-performance, integrated CIFS share serving. Combined with the filesystem management of ZFS, this new CIFS server feature is very compelling. Have a ZFS filesystem that you need exported to some Windows (or Mac, or Linux) boxes? Just like it was with iSCSI, it’s this simple:
  zfs set sharesmb=on pool/my/fs
  You can set additional share parameters, such as its advertised name by replacing “=on” with other arguments. See the section for the set option in the sharemgr(1m) man page. Management of LM users, groups and server mode are accomplished with the separate smbadm(1M) command.
• NDMP Backups. That’s right, a new NDMP service is now present for all your enterprise backup needs. Have some of those expensive Legato Networker NDMP licenses to burn, or want back up a NetApp or other NDMP-capable device to a Sun Fire X4540 so you can pitch it and its support contract out the window? Fire up the NDMP daemon and go to town with the ndmpadm(1M) command. This new service in OpenSolaris supports NDMP versions 2, 3, and 4. There is also a nifty way to get statistics on your NDMP sessions with the ndmpstat(1M) command.
• COMSTAR. Or Common Multiprotocol SCSI Target, is quickly becoming the in-kernel nexus of sorts for exporting a generic “block device” outside the system over an array of protocols and transports. What this subsystem allows you to do is take a ZVOL and export it over things such as Fibre Channel and FCoE. That’s right, if you have a system with Qlogic QLA/QLE24xx cards, you can turn them into a target rather than an initiator and serve LUNs over FC on your SAN. Your OpenSolaris box is looking like a classic storage array now, but with far more features and flexibility. Fiber Channel over Ethernet (FCoE) is also an option. Future plans include bringing the aforementioned iSCSI server under COMSTAR’s domain as well. Earlier Qlogic HBAs such as the 2Gb models (QLC23xx) are not supported as targets as those cards lack features required in their firmware to put them in such a mode.
• Replicate With AVS. Sun has offered a filesystem-agnostic block-level replication software suite for some time called StorageTek Availability Suite, or AVS. This was a pay-for product, but Sun has graciously donated it in full to OpenSolaris, so now it is free to use. AVS allows you to configure synchronous or asynchronous replication over the network to a remote sever, with additional capabilities such as shadow images (termed as “Instant Image” in AVS). AVS lives in-kernel, and situates itself between the filesystem (ZFS, UFS, etc) and the disk devices, copying blocks off to their configured destination. Here are some demos by lead developer Jim Dunham that demonstrate the use and capabilities of AVS.
• SAM-QFS. As with AVS, SAM-QFS was formerly an unbundled, pay-for product from Sun, but has been open sourced and provided as a integrated part of OpenSolaris. SAM-QFS has two major components - QFS, which is a SAN-based multi-writer and reader filesystem, and SAM, a hierarchal storage management (HSM) system which sits on top of QFS.QFS in particular is found in both the HPC and service provider data center role where multiple nodes require concurrent read/write access to the same file system over fibre channel or iSCSI. It can be utilized in a single-writer setup, where one node can write but all others are read-only, or a multi-writer setup, where > 1 nodes require write access. In the latter case, additional infrastructure is required in the form of a metadata server, the role of which is to manage and coordinate locks and write access amongst the involved nodes.

  Along with QFS, there is the SAM component. This allows one to age data off to cheaper bulk storage (cheap SATA arrays) and/or tape-based long term storage.

That’s a snapshot of the major additions, but there are still a lot of smaller projects that have been integrated, or are on the verge of being made available such as pNFS, MMS, ADM, and Honeycomb. As you can see, OpenSolaris offers quite a bit for those looking for highly flexible ways to store, manage, and export data to and from other systems… and it’s all built in and fully functional. No licenses or hidden costs.

The OpenSolaris Storage Community is a great place to keep track of what’s new, and provides a very nice visual representation of the various storage layers and components in OpenSolaris.

Hopefully this will work well enough to allow Solaris users to read and write to HFS or HFS+ formatted disks and disk images. This includes iPods that were initialized on a Mac. Please check out the page I made for it and lend a hand if you’re interested!

Technorati Tags: solaris, opensolaris, sun, mac, macos, apple

Mercury is a Dell PowerEdge 860 with a Intel Xeon X3220 (quad core, 2.4Ghz) and 4GB 8GB of 667Mhz DDR2 RAM. Unlike lithium, mercury’s storage is entirely internal in the form of two mirrored 500GB SATA drives. This is to keep the entire package in 1 rack unit of space to keep colocation costs down.

What really excites me about this new server is that it is running Solaris 10 8/07 (lithium was running a very patched Solaris 8 FCS!). Solaris installed without a hitch and the 860’s onboard BCM5721 NICs are recognized by the bge driver, as are its IPMI baseboard controller by the bmc driver. The chipset on this system is the Intel ICH7 and unfortunately the Solaris ahci driver supports only the ICH6 at the moment, so the drives are running just fine in IDE compatibility mode.

This upgrade wasn’t just a mere update of hardware and OS. I also completely changed how the mail storage works and also make use of ZFS file systems for each user home directory and virtual web site:

1. Out with uw-imap, in with Cyrus. All mail is delivered to Cyrus, so there are no more maildir-style spools sitting in each person’s home directory.
2. To take advantage of Cyrus’s features, elemental.org is now operating its own Kerberos realm, ELEMENTAL.ORG. This is my first time running my own Keberos KDC, and I love it. Cyrus and Sendmail, via SASL, now offer GSSAPI authentication. Using Solaris’s pam_krb5_migrate.so.1 PAM module, as people log in with their UNIX passwords, a Kerberos principle is made for them and they are granted tickets. Pine is configured to connect to Cyrus and authenticate with GSSAPI, so shell users don’t have to type in or save their password when accessing their email!
3. As I mentioned, all user data is now stored on a mirrored ZFS pool. Each user and virtual website gets their own ZFS file system and this will allow me to keep tabs on disk usage (and easily delete a user or site if the need should arise.) The zpool’s net size is 442GB.
4. All incoming email is goes through greylist, ClamAV, and finally SpamAssassin milters.
5. I’m more at ease and familiar with Solaris’s SMF facility now, having made a point to write SMF manifests for the services I’m running rather than plain old init scripts.

In addition, I’m now monitoring several aspects and services on the new system using Cacti.

Here’s to another 8 years of hopefully trouble-free operation!

Technorati Tags: Sun, Dell, Solaris, ZFS

http://docs.sun.com/app/docs/doc/817-0547/getjd?l=en&a=view

Technorati Tags: sun, solaris

I was lucky enough to present a slide show on how we at UMBC have been combining OpenAFS with new ZFS and Zones features of Solaris 10 to obtain a more resilient AFS server infrastructure. You can view a PDF of my presentation here.

Technorati Tags: afs, solaris, zfs, zones, opensolaris

6510364 “War Mode” in ALOM-CMT is required by the US NAVY which is currently missing

Awesome. I love knowing that my T1000s have a “war mode” now. Perhaps Sun should call it the SkyNet T1000 ;) Hopefully it’s the feature and not the US Navy that was “currently missing.”

Technorati Tags: sun, coolthreads, sparc, ultrasparc, solaris, navy, war

In particular, I take issue with this comment from Mr. Zaitcev:

“This is the problem OpenSolaris is facing today in the nutshell: it has no breadth. It has a very limited number of excellent technologies, such as ZFS.”

No breadth? Care to, well, add some breadth to that statement, Mr. Zaitcev? Making a comment like that doesn’t mean you can toss out just one perceived example and end your argument at that.

It would appear that all of Mr. Zaitcev’s experience with Solaris/OpenSolaris comes from reading 3rd party accounts of the big new features in Solaris. This is exactly what I referred to in my opening paragraph… all these anti-Solaris pundits more than likely have zero hands-on expeience with the stuff they’re harshing on. People like Mr. Zaitcev read anecdotes and stories, come up with their own idea as to how things are based on those stories, and produce comically uninformed jabs posts such as the one linked above.

No breadth? Just what is the breadth that Mr. Zaitcev thinks is missing? Is breadth in this case even quantifiable? Is his supposition based solely on the age old (and aged) driver count argument? Does Mr. Zaitcev think that all Solaris is, is an ancient kernel which happened to have a few new concepts tacked on top of it?

I would bet that if Mr. Zaitcev sat down and tried to use Solaris in a real-world environment, he’d soon learn that Solaris has everything one needs in a data center environment… he just hasn’t discovered them (or read about them, natch) yet for himself. Who knows, perhaps he’d even appreciate them.

Technorati Tags: linux, solaris, opensolaris, fud, sun

It appears that DaveM (Linux networking and SPARC port guru) has gotten seriously wound up in response to a blog post by Jeff Bonwick (Sun’s storage and kernel guru.)

As one can see in Jeff’s post, the suject he wrote about was within the greater context of using Solaris as a storage appliance OS (something I have an interest in) and why Solaris/OpenSolaris can and would excel when it comes to being the kernel of a storage OS.

I’m a storage guy. In the course of my work I have to not only work with Solaris hosts on my SAN, but also Windows and Linux (and soon, AIX). So I have a front-row seat when it comes to witnessing and dealing with how these various OSes deal with storage, from the filesystem to multipathing, to the HBA… and let me tell you, Linux is quite not the joy in this specific area as most people think it is on a general, all-encompassing level.

And that’s what Jeff’s angle was.

Now on to Dave’s rebuttle.

“The implication is that Linux is not rock-solid and that it does break and corrupt people’s data. Whereas on the other hand Solaris, unlike the rest of the software in this world, is without any bugs and therefore won’t ever break or corrupt your data.”

No OS comes without fault, but some OSes have faults that are more glaring than others in their analogous areas. Staying within the storage context of this discussion, I have to say, again, Linux is no shining star here.

ReiserFS is arguably the most advanced fs in terms of features when it comes to the portfolio of Linux file systems, but its issues with stability are such that you’re really walking on eggshells whenever you employ it. I have been personally told too many first-hand accounts and read plenty more on the Internets regarding its tendency to be fine and then fail spectacularly. It has been likened to a time-delayed /dev/null of sorts, and the future of it is in doubt with the legal troubles of its designer and Namesys limbo. Is any version of ReiserFS a viable Linux storage technology for a production environment? I say No. That’s sad because I dare say at one point ReiserFS had some promise.

EXT2 and 3… tried and true. Very stable and moderately fast for most tasks. But it’s an “old guard” file system. As such, it’s not very flexible, and any flexibility it gets comes from using a volume manager underneath of it. In the days where the notion of handing a server a 1TB LUN is nothing to blink at, this inflexibility can be suffocating in a dynamic environment. These “old guard” file systems (yes, Solaris’s UFS is one of them, too) are more like mere utility file systems than practical ones for today’s mass storage needs. It’s good for holding a machine’s OS and that’s about it.

XFS… Of all the file systems in the Linux file system portfolio, this one gets the gold star. Stable, fast, and decently scalable with the large amount of data you can stuff in it… but it still suffers the same problems EXT[23] and other “old guard” file systems do in terms of flexibility. In other words, it’s just a file system. Keep in mind that this critique is coming from a guy who worked with XFS on IRIX often and absolutely loved XLV… back in its the day.

As it stands now, the mainline Linux kernel doesn’t offer anything which embodies the file system triple play: being stable && fast && flexible. Solaris’s ZFS has this. I’ve so far entrusted 30TB of spinning rust to it, and it has yet to let me down. Sure, there are projects here and there that have the eventual goal endowing Linux with a ZFS analog, but as of right now they’re nothing production quality and are definitely not something a admin can call RedHat to get support for.

There are plenty of other aspects to the storage context… the fibre channel stack, for one, and other things such as multipath IO implementations and volume manager and management layers (which Linux has a host of… not necessarily a good thing… LVM, LVM2, MPIO, RDAC… it makes your head spin.)

But as far as this storage-oriented discussion goes, file systems are indeed the make or break aspect. This is why Jeff said what he said. Linux has no ZFS. Windows has no ZFS. It is not that Linux or Windows need ZFS itself in order to compete, it’s that they need to develop and employ the concepts that ZFS implements and do so as clearly and concisely as ZFS has.

Anyway, enough about storage. Now, why is it that the Linux community (let alone a prominent member of it) has to react so violently to any questioning of its perceived superiority? Is it misplaced or excess pride? Have they not tried things other than Linux recently and they’re just flying with blinders on? Is it just the social culture which prevails within it? What ever it is, seeing posts like Dave’s makes my toes curl with embarrassed amazement.

A friendly message to Dave: Chill the ad hominems, mkay? Crying “FUD!” at the mere sight of someone who you perceive as poo-poo’ing an aspect of your interest doen’t typically translate into a well thought-out rebuttle. You took the low road and tried to convey Jeff as being some instrument of some nefarious, Mr. Burns-like person at Sun. Is vilifying instead of cool-headed technical discourse really your desired style? Has anyone come biting at you saying “oh, he’s a Linux kernel developer, so he has an agenda”?

Technorati Tags: linux, solaris, opensolaris, storage, fud, zfs, fuzzy bunnies

From their announcement:

• Sun StorageTek 5800 storage system (Honeycomb) client interfaces along with the Honeycomb software developer kit (SDK) and Honeycomb emulator/server. Honeycomb is a third-generation digital repository solution for data capture and management.
• SAM-FS (Storage Archive Manager) provides data classification, policy based data placement, protection, migration, long-term retention, and recovery capabilities for organizations to effectively manage and utilize data according to their business requirements. SAM-FS is used exensively in security/surveillance, digital video archiving, and medical imaging data environments.
• QFS Sun’s shared file system software delivers significant scalability, data management, and throughput for the most data-intensive applications. Well known today in the traditional high performance computing (HPC) arena, QFS is increasingly being used in commercial environments that require multiple host, high speed access to large data repositories.

Also, and pretty awesome yet:

In addition, today QLogic is contributing their Fibre Channel HBA driver code to the OpenSolaris storage community. For the first time, developers have access to an I/O stack from the application through to the operating system.

Now how cool is that? That last sentence says it all… openness from from the app to the HBA. Most excellent. Thank you to Sun and a really surprised thank you to Qlogic!

Technorati Tags: solaris, opensolaris, sun, qlogic, storage