Mercury gets a HBA upgrade

Dale Ghent | August 3, 2008

mercury.elemental.org is the server which hosts my $HOME and this website. It’s my Solaris 10 play-box, and I guess you can say that maintaining it is something of a hobby.

Its hardware is a quad core Xeon-equipped Dell PowerEdge 860, a small 1u server. Its pair of internal drives are Seagate SATA2, and were connected to the on-board Intel ICH7-based SATA controller. But there was something fishy about this in that the Solaris ahci SATA driver never attached to it and instead the drives ran in IDE mode. Despite my best efforts, I couldn’t change this. I eventually found out the reason – Dell crippled the SATA controller in the system BIOS to allow only IDE mode!

So this server was sold with “SATA drives”, which would imply a fully functioning SATA controller to drive them… but not quite. IDE mode means there were no benefits of SATA NCA and other niceties.

To fix this, I got a LSI SAS3041E-R controller – a 4x PCIe card that uses the LSISAS1064E chipset and offers 4 SATA ports. In Solaris land, this card would be driven by the mpt driver, a proven driver as the LSI SAS 1064 and 1068 chipsets are used to drive the on-board hard drives in pretty much every current Sun x86 and Niagara-based SPARC systems.

I installed this card in the single 8x PCIe slot in the PE860, and ran a 24″ SATA cable from it to HDD1, and used the existing Dell cable that connected the on-board controller to HDD1 to connect HDD0 to the card. After some fiddling in /boot/solaris/bootenv.rc to tell the kernel the new device path to its boot drive, the mpt driver attached and I was good to go.

I kicked off a SVM mirror resync as a basic test of sequential IO, and I hit 75MB/s reading from one drive and writing to the other. Not bad. A zpool scrub of my mirrored ZFS pool of 66.5GB of data (pool is 444GB in size) took just over an hour.

So if you’re thinking about a 4 or 8 port SAS/SATA card, consider the LSI SAS3041 or SAS3080/3081 cards, respectively. Both come in PCI-X and PCIe flavors and are supported by Solaris (and OpenSolaris) just fine.

/usr/X11/bin/scanpci output:
pci bus 0x0001 cardnum 0x00 function 0x00: vendor 0x1000 device 0x0056
LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS

Kernel boot messages:
scsi: /pci@0,0/pci8086,2779@1/pci1000,3090@0 (mpt0): Rev. 8 LSI, Inc. 1064E found.
scsi: /pci@0,0/pci8086,2779@1/pci1000,3090@0 (mpt0): mpt0 supports power management.
pcplusmp: pciex1000,56 (mpt) instance 0 vector 0x38 ioapic 0xff intin 0xff is bound to cpu2
scsi: /pci@0,0/pci8086,2779@1/pci1000,3090@0 (mpt0): mpt0 Firmware version v1.17.2.0 (IR)
scsi: /pci@0,0/pci8086,2779@1/pci1000,3090@0 (mpt0): mpt0: IOC Operational.
scsi: /pci@0,0/pci8086,2779@1/pci1000,3090@0 (mpt0): mpt0: Initiator WWNs: 0x500605b0000fa840-0x500605b0000fa843
pcie_pci: PCIE-device: pci1000,3090@0, mpt0
genunix: mpt0 is /pci@0,0/pci8086,2779@1/pci1000,3090@0
scsi: sd4 at mpt0: target 4 lun 0
genunix: sd4 is /pci@0,0/pci8086,2779@1/pci1000,3090@0/sd@4,0
genunix: /pci@0,0/pci8086,2779@1/pci1000,3090@0/sd@4,0 (sd4) online
scsi: sd3 at mpt0: target 5 lun 0
genunix: sd3 is /pci@0,0/pci8086,2779@1/pci1000,3090@0/sd@5,0
genunix: /pci@0,0/pci8086,2779@1/pci1000,3090@0/sd@5,0 (sd3) online

OpenSolaris 2008.11 – A Preview For The Storage Admin

Dale Ghent | July 16, 2008

Many reviews have been written about OpenSolaris since its release, but all of them barely tread beyond the desktop aspect, with the obligatory screenshots of the GNOME environment and a high-level description of only the major features most are already familiar with, or at least have heard of.

I’d like to take a different approach with this review, one that descends below the GUI to highlight aspects that server administrators in particular would be more interested in.
Read the rest of this entry »

Making Solaris HFS-aware

Dale Ghent | March 3, 2008

I’ve started a project of my own to port the HFS/HFS+ filesystem driver from Apple’s XNU kernel to OpenSolaris/Nevada.

Hopefully this will work well enough to allow Solaris users to read and write to HFS or HFS+ formatted disks and disk images. This includes iPods that were initialized on a Mac. Please check out the page I made for it and lend a hand if you’re interested!

Technorati Tags: , , , , ,

Comments
Comments Off
Categories
Geekery, Solaris
Tags
, , , , ,
Comments rss Comments rss
Trackback Trackback

Server upgrade time – elemental.org gets modern

Dale Ghent | November 17, 2007

After almost 8 years of running elemental.org mail, mailing lists, shell accounts, many websites (such as this one), database servers and essentially being a one-server ISP, the Sun Ultra 2 which ran all those things as lithium.elemental.org was retired and replaced this past weekend with a new server. Say hello to mercury.elemental.org.

Mercury is a Dell PowerEdge 860 with a Intel Xeon X3220 (quad core, 2.4Ghz) and 4GB 8GB of 667Mhz DDR2 RAM. Unlike lithium, mercury’s storage is entirely internal in the form of two mirrored 500GB SATA drives. This is to keep the entire package in 1 rack unit of space to keep colocation costs down.

What really excites me about this new server is that it is running Solaris 10 8/07 (lithium was running a very patched Solaris 8 FCS!). Solaris installed without a hitch and the 860’s onboard BCM5721 NICs are recognized by the bge driver, as are its IPMI baseboard controller by the bmc driver. The chipset on this system is the Intel ICH7 and unfortunately the Solaris ahci driver supports only the ICH6 at the moment, so the drives are running just fine in IDE compatibility mode.

This upgrade wasn’t just a mere update of hardware and OS. I also completely changed how the mail storage works and also make use of ZFS file systems for each user home directory and virtual web site:

  1. Out with uw-imap, in with Cyrus. All mail is delivered to Cyrus, so there are no more maildir-style spools sitting in each person’s home directory.
  2. To take advantage of Cyrus’s features, elemental.org is now operating its own Kerberos realm, ELEMENTAL.ORG. This is my first time running my own Keberos KDC, and I love it. Cyrus and Sendmail, via SASL, now offer GSSAPI authentication. Using Solaris’s pam_krb5_migrate.so.1 PAM module, as people log in with their UNIX passwords, a Kerberos principle is made for them and they are granted tickets. Pine is configured to connect to Cyrus and authenticate with GSSAPI, so shell users don’t have to type in or save their password when accessing their email!
  3. As I mentioned, all user data is now stored on a mirrored ZFS pool. Each user and virtual website gets their own ZFS file system and this will allow me to keep tabs on disk usage (and easily delete a user or site if the need should arise.) The zpool’s net size is 442GB.
  4. All incoming email is goes through greylist, ClamAV, and finally SpamAssassin milters.
  5. I’m more at ease and familiar with Solaris’s SMF facility now, having made a point to write SMF manifests for the services I’m running rather than plain old init scripts.

In addition, I’m now monitoring several aspects and services on the new system using Cacti.

Here’s to another 8 years of hopefully trouble-free operation!

Technorati Tags: , , ,

Comments
Comments Off
Categories
Geekery, Solaris
Tags
, ,
Comments rss Comments rss
Trackback Trackback
This page took 0.234 seconds to generate.