Technology

A colleague found an interesting piece of information on page 23 in the "Release Notes of AIX 6.1":

AIX Version 6.1 with Technology Level 6100-04 supports a maximum of 64 processor cores and a maximum of 128 logical processors. The logical processors correspond to the total number of Simultaneous Multithreading (SMT) threads. This means that the maximum supported configuration is 64-core POWER5 or POWER6 in SMT2 mode (2 threads per core).

Perhaps this is the reason why there are just a few published benchmark for a larger machine. Of course you could use LPARS, but then the SAP-SD benchmark would be a 3-tier result and not a 2-tier one fore example. And as far as others and i've read through all the service pack docs, this limit is still in AIX 6.1, albeit the newest AIX 6.1 TL4 SP3 is said to be able to capable to use up to 256 threads, but didn't found this info in the docs so far, the info is just in an IBM preso. The newest available release note at the IBM site is still stating the limit, but it's from November. To use more than that, the support doesn't go into 5.3 or 6.1, you will need AIX 7.1 for that ... will be interesting to see the ISV reaction on that. However, this looks like they have to do more in-depth work to scale up to 1024 procs to me ...

PS: The processor of a plain-standard Solaris 10? Just look here.

In some discussions in mail and in some community forums linking to the articles about deduplication and hashing there was a slight misunderstanding. I should explain some things.

Frequent readers of my blog know that i'm using an Notebook via iSCSI to emulate a SSD. At the moment my SSD is an old Acer Aspire notebook. It works reasonably well, it's much faster than a hard disk and i can test the behavior of applications with SSD without buying one.

On 16th December i will talk about Hadoop on Sun at the Apache Hadoop Get Together in Berlin. The talk will be divided in three parts:

• Advantages of using Hadoop on CMT systems like the T5120
• Using Hadoop in conjunction with Solaris Features like Container and Resource Management in non-dedicated architectures
• Hadoop and Grid Engine

There will be short live demonstrations in the second and third part of the presentation.

My presentation will not be the only one. Richard Hutton from nugg.ad talks about data processing with hadoop in his presentation "Moving from five days to one hour." Nikolaus Pohle from Nurago talks about data analysis with Hadoop for online market research.

Yesterday the usual news outlets speculated about the rumours circulating about the reasons for the data loss at Danger. For the uninitiated: Danger is the the company behind the Sidekick.

TGDdaily started by reporting the information provided by a reader with the article: "Oracle, Linux, Sun to blame for Sidekick Danger data loss":

TG Daily reader Tommy T informed us earlier today that: “Microsoft was running on Sun Solaris/Linux/Oracle-based Danger servers at the Verizon Business Center in San Jose as part of a contractual obligation to T-Mobile.”

That isn't really a surprise. Despite all the press fuzz, many many sites still deploy and use Solaris for their mission critical stuff, while using Linux for stuff that scales well on clusters.

Of course Microsoft was asked for the reasons and just reported

Sidekick runs on Danger’s proprietary service that Microsoft inherited when it acquired Danger in 2008. The Danger service is built on a mix of Danger created technologies and 3rd party technologies.

I think that's a good answer, we don't blame anyone but we tell the world, the application wasn't developed at Microsoft and the infrastructure wasn't Microsoft as well. Especially as Microsoft doesn't need additional doubts about the feasibility of their operating and infrastructure environment after Microsoft loosed the .net implementation at the London Stock Exchange in favor of of a Solaris/Linux implementation. I don't expect Microsoft to shed more light into the situation.

And it isn't that way, that they are the only shop on the world using RAC on Sun. The Inquirer expresses this situation in a nice way - "Oracle and Sun fingered for Sidekick fiasco"

That would all be well and good but for the fact that the Sun-Oracle combo is not exactly rocket science and is popular among those who manage very large, mission critical corporate databases.

I think, the truth about this outage is much more complex, as the various media outlets state. IT systems are called systems for a reason. They are large complex interrelationships of components. There is a database, there is server, there is storage, there is a SAN, there is a network. And there are many devils of the detail in any system. Whoever says something different, simply lies.

But at the end it's pretty irrelevant. It's like with aircraft crashes. There isn't a single reason why an aircraft crash happens. When there is a major CNN moment at a service, there are several resons why it happened. It may start with a small hick-up and other effects leads to a disaster.

At the end it's a large heap of rumors surrounding this situation. Daniel Eran Dilger writes in "Microsoft’s Pink/Danger backup problem blamed on Roz Ho":

According to the source, the real problem was that a Microsoft manager directed the technicians performing scheduled maintenance to work without a safety net in order to save time and money. The insider reported:
In preparation for this [SAN] upgrade, they were performing a backup, but it was 2 days into a 6 day backup procedure (it’s a lot of data). Someone from Microsoft (Roz Ho) told them to stop the backup procedure and proceed with the upgrade after assurances from Hitachi that a backup wasn’t necessary. This was done against the objections of Danger engineers.

As far as I'm understanding the rumor, they made a backup, deleted they only other backup in the course of this procedure, stopped the backup and did a firmware update. Sorry ... i don't even do an update of of notebook without starting a backup before doing so and my blog replicates it data every six hours to a system in my office at home.

A backup is a safety net. You don't do any task that could put your data at risk without having an backup. And you should better test the restore. Of course this takes time, at my blog a few minutes ... at Danger a few days. But you are sure that you don't appear at CNN. (okay, i would appear at CNN, but i would get some question in our coffee kitchen at work). Sh.t happens, customers understand this, but they don't accept that you don't value their data. Perhaps this is the real story of the this mishap.

“In preparation for this [SAN] upgrade, they were performing a backup, but it was 2 days into a 6 day backup procedure (it’s a lot of data). Someone from Microsoft (Roz Ho) told them to stop the backup procedure and proceed with the upgrade after assurances from Hitachi that a backup wasn’t necessary. This was done against the objections of Danger engineers.

I really can't imagine a single reason why a HDS/HP/EMC/Sun storage support engineer says "Hey. You don't need a backup. We will do the firmware upgrade without it". I just can imagine that the the engineer said: "Well ... i did several upgrades so far and there wasn't a problem that sent me to the tapes"

By the way: There is one thing, that keeps me really puzzled. It's a piece of information at TheRegister:

It involves 20 or so CentOS Linux servers [...]

Sorry, CentOS is a nice variant of Linux, but i don't really think that's a good idea to use a distribution of Linux with at least questionable commercial support (The "Commercial Support" menu item at centos.org delivers a page stating that there is no content) in a mission critical environment. I know that an enterprise Red Hat or Suse distribution/support agreement costs you some bucks, but in such an environment i would opt for the professionally supported variant. But again: This is a example of an attitude. And this attitude is not one of valuing the data of the users.

A hickup in a service has most often their reason in the realm of technology, but real disaster has more reasons, it has deeper reasons. Just 10% are technological reasons. Or to say it more precisely: A disaster can be root caused by a technical reason, but the step from a hickup to a disaster needs always reasons outside the realm of technology. And given the rumors surrounding the Danger outage are true, this outage is one of the best examples to show it.

Sun has now announced the F5100 officially. The specs are really impressive: The largest version with 80 FMods yields 1.6 Million read IOPS and 1.2 million write IOPS with 4k blocks. 12.8 GB/sec read and 9.7 GB/sec write. 1,920 GB capacity. It uses 386 watts while active. More specs are available at the Sun specs page for the Sun Storage F5100 Flash Array. Furthermore the documentation of this devices should reappear soon at docs.sun.com. Until that moment you will find some additional information about the device at Sun's website. This device isn't that expensive as you may think. The device starts at $45,995 (with 20 Flash Modules giving you 480 GB, 397 K IOPS read and 304 K IOPS write).

When you look in such a device you will see 80 densly packed FMods.

In the back of the device you will find the SAS connectors for the device. There are four domains with 4 SAS connectors each. For a maximum performance configuration you configure 5 FMods and a SAS connector into a zone.

Most important: The FMods doesn't have a battery on each module to protect the caches. This is done centrally by super capacitors in so called Energy Storage Modules. You plug them into the front of these devices.

Just a warning: Those ESM are quite potent super caps. Read the manual with care before handling those modules. You need a lot of power to protect all the caches and those ESM are capable to deliver this power.

Robin Harris points to an interesting study about DRAM failures in his blog storagemojo (BTW: Robins blog is really a great read). He points to the paper "DRAM Errors in the Wild: A Large-Scale Field Study"( written by Bianca Schroeder (University of Toronto), Eduardo Pinheiro and Wolf-Dietrich Weber (both from Google)) in his article.

Some of the numbers are really terrifying: 4.15% unrecoverable errors for of of the platforms are much more then i had thought and i'm somewhat conservative in my thinking how far i trust hardware. Furthermore hard errors (as in "bit permanently flipped and put it to the trashbin") are vastly more common reasons for errors as most people think.

As a sidenote: After the discussion about DRAM prices in the M3000 i've got some flak because of memory prices at different quality and got many comments "there never failed a dimm at my home pc". But given the point that Google is said to use cheaper hardware and the amount of errors (especially the unrecoverable ones) there may be a point behind the fixation of Sun in regard of memory quality

A reader think it makes no sense to use the STEC SSD, and we should switch to the Intel X25E drives. That sounds reasonable at first as the X25E is much cheaper as the STECs. But as usual the devil is in the detail. So why do many people still use some of the more expensive STECs? Do they have too much money? Are they morons? I will tell you a dirty little secret. No, not at all. When you take it really seriously the X25E aren't enterprise ready. At least in their default setting.

Nice to see, that i'm not the only one who thinks, that IBM will run in the same challenges like Sun in regard of massive multicore processors. Great to see, when this position is somewhat confirmed by someone who isn't known as friendly to Sun and is said to be firmly on the blue side.

BlueToTheBone writes in his column "The Four Hundred" about Moore's Law and the Performance Wall:

Well, with the Power7 chips coming next year, IBM has to get the multithreading fixed in DB2 for i or get a whole lot of excuses ready for why customers buy more cores and threads, running at lower clock speeds, and don't see performance go up.

But he points to an even more interesting point, that isn't really a known territory to an open systems guy like me. Besides of this "bytecode-compiled is slower than iron-compiled" stuff (which isn't true since the invention of Just-in-time-compilers) he has a very valid point. Much of the software is really old, and it wasn't written for a environments with hundreds of cores. We learned the hard way, that there were a vast amount of pitfalls in the Open System world which thinks multithreaded/multiprocess for quite a time now. Now software developed in RPG and COBOL (and many code lines originate from a time when many of us weren't even a glint in the eyes of our parents) hits with Power7 on an environment that doesn't fulfill on the promise of ever-increasing single-thread performance. BlueToTheBone comes to a similar conclusion than the one i've made a while ago. Perhaps many applications stay on Power6 while newer developments can move to Power7:

. It might even mean putting off a move to Power7 iron and sticking with Power6 or Power6+ boxes as you dig through your code and see how parallelization can and cannot be used to make your applications run faster as well as offer more capacity to support more work.

Looks like IBM is somewhat concerned about the Exabyte V2 announcement and that announcement that is like to happen on October 14th. At least there is a loud "We have it, too - kind of" from Armonk. They tout their product IBM DB2 Pure Scale. Power machines with AIX, some infiniband gearand special edition of DB2.

Some educated guesses

Mr. BlueToTheBone is right. Details are sparse at the moment. But just to write down some points, you remember for having a deeper look at after an official announcement. Those points are purely speculative given that the information in the ElReg article are really sparse:

• It will be interesting how this solution with a centralized lock management will compare with the distributed lock management in RAC. Perhaps this will be the scaleability stopper for the IBM solution. I think we will see a solution with fewer, but bigger nodes. Given that the job of lock management is consolidated on one node, i would say that the front end node will be a really big machine.
• But have the same opinion like Chris Mellor (hell freezes over, i have the same opinion): IBM has a storage problem. There was the 1 Million IOPS Quicksilver marketing stunt. But that system were just 40 Fusion-IO cards in a lot of servers. So that doesn't look like something playing in the same ballpark like the Sun F5100

  The QuickSilver flash was composed of one or two 250GB Fusion-io ioDrives with PCIe interfaces in 29 host servers connected by Fibre Channel to the 13 SVCs. There were 41 ioDrives in total.

  Just for the uninitated: The SVC (Storage SAN Volume Controller) is a server as well. A full rack of hardware for 1 Million IOPS. But perhaps they just plug the Fusion-IO in the database servers and don't use SVC at all... who knows
• The p6 pSeries systems doesn't have PCI Express 2.0. With the PCIe 1.0 variant you have 240 MBytes/s. Thus you have somewhat around 2 GByte/s on the x8 PCIe of a p520 for example. That isn't enough for QDR. Some systems like the 520 provide GX busses for better connectivity, but nevertheless there doesn't seem to be QDR cards supported in the systems. But those 520 are 4 RU systems, don't know if they use them for such a cluster. Thus i would expect to see DDR infiniband in those systems. I assume they will have half the bandwidth in the fabric
• There isn't a system in the pSeries lineup that allows to put 12 disks on 2 rack units. That's the other side of the storage problem, albeit Chis Mellor at El Reg doesn't mention it. That's the point why the X4275 is such an interesting system. You can have 12 Disks and 2 sockets on 2 rack units. I assume the database servers have seperate storage arrays.

There are some other points, but those are much deeper in the realm of speculation, so i just wait for the announcement for further comments.

Frankenbase

Some people say, that Exabyte V2 in its current incarnation is just rushed together in the light of the upcoming merger. But the stuff that IBM told exclusively in the ears of Bluetothebone sounds like kludged together just to counter an competitors announcement. I'm sure that IBM will show of a strange configuration, calling it DB2 Pure Scale. But Frankenstein creation will look as a beauty beside this solution.

The cryptographic algorithm AES is everywhere. You find it in your router, you find in your OS. Your WiFi network use it. And probably the three-letter agency in your country,too . But how does it works? Jeff Moser wrote (well, it's more a comic) the "Stick Figure Guide to the Advanced Encryption Standard (AES)". A really great explanation of AES.

Asterisk is a cool piece of software. You can use it to do almost everything in the realm of telephony except of doodling on your notepad while listening to a boring telco. But i hadn't the idea to implement something like that: The Annoyatron Telemarketer Torture. I want this on my cellphone when someone calls me to sell me a private health insurance. Well ... as i switch into "maximum unfriendliness", they don't call me often ... looks like i have some negative mark in their data warehouse

As my colleague Ingo said a while ago: "It doesn't matter how fast you stir your meal while cooking when you wait for your sweetheart to search the wine. You can't eat a minute earlier". And i found a another example for this: The newly published TPC-H 3 TB benchmark for the IBM p595, the biggest system in the Power series.

But i will go into the past before: In 2007 (in IT thats halfway to infinity) Sun benchmarked the E25K with 72 sockets at 1.8 GHz with two cores per socket and 256 GBytes of memory. This benchmark yielded 114,713.7 QphH.

2 years, 224 GBytes main memory and 3.2 GHz later the biggest machine available from IBM is just 34 percent faster (to be exact: 154,115.8 QphH) than the biggest machine available from Sun from two years ago ... based on a design from the beginning of this century.

By the way: Just in case you wonder about the memory configuration - 512 GB is the maximum configuration that can be used with 667 MHz. More downclocks the memory. Using 4 TB of memory would downclock it to 400 MHz.

Perhaps the challenge gets a little more clear, when we stay in the same shop: Almost 4 years ago IBM published a result of the p5-595. This system yielded with 1.9 Ghz Power5 CPU 100,512.30 QphH. 4 years of development, 256 GByte more memory and 2.6 times the frequency gives you just roundabout 50% more performance. Nice, but not that impressive, especially given the effort put into this CPU in regard of cycles.

An thats perhaps the most important takeaway from this point: Ingo was right. Just increasing the frequency isn't sufficient. It gets clearer with every published benchmark. You have to do more or different. Perhaps the future of large SMP systems is in a more efficient communication between the components, more real bandwidth (not all bandwidths added into a number for marketing purposes) and less bandwidth between the CPUs and between CPU and bandwith.

But i have a request to our benchmarking team: I would like to see an TPC-H result for the M9000 at 3TB. Giving the existing results for the M9000, putting some knowledge into the equation and doing some extrapolations i don't think that that we have to fear this competition. But thats just an educated guess.

There is a nice example for my point, that BlueToTheBone tries to downplay everything coming Sun, even when doing so makes a complete fool out of him.

There is an interesting piece in the hardware description as mentioned on the FAQ page of the Exdata system:

2 quad-core Intel Xeon E5540 (2.53GHz) processors 24GB memory
4X96 GB Sun Aura Smart Flash Accelerator F20 PCU cards
Disk Controller HBA with 512MB Battery Backed Write Cache
InfiniBand Connectivity Dual-Port QDR (40Gb) InfiniBand Host Channel Adapter
Power Supplies - Dual-redundant, hot-swappable
Remote Management Sun Embedded Integrated Lights Out Manager (ILOM)
Disk Drives 12 x 600 GB 15,000 RPM SAS.

The flash drive isn't in the disk slots, it's on this cards. Thus each of the cells comes with 384 GB of flash. You don't have to sacrify space for speed here. Neat.