The IT Business

$2.07 billion for 3PAR - a company just working in the storage area - ... now i'm sure that $7.4 billion for Sun was a steal.

I sat a on my fingers for all the weekend but now i can't take all this comments in the blogs and the mailing lists any longer. Ten years ago, i would have gone ballistic long ago... but the fuse got longer of the past few years. However i'm still able to detonate, however despite the undirected explosion 10 years ago, it's more directed today.

And this article is such a directed explosion. I worked a while on this article and this was one of the reasons why it was relatively silent on this blog the last few days. At first i didn't wanted to publish it, but some events of today led me to think otherwise. I don't know if this is a wise move, because the torch of this rant will burn down some beards. On the other side, i think the article is worth the publication, but you have to judge about that.

However i try to prevent me from exploding thus i used this weekend to play with my new toy (i purchased an iPad last week and it's really fscking cool, i just thought "Like Enterprise ... i just wouldn't gave them away like they do in TNG" ) and worked a lot on the planing on the modifications on my newly purchased building.

However the mail by Garrett d'Amore asking about the removal of SVM was a drop. It was one of those drops, that led to a severe spill-over. So this blog article will be a long rant. If you don't like rants, just skip this.

And keep in mind, it's a rant ... it's not meant to be fair or objective ... think of it as a way to write down my frustration about a lot of events in the last few weeks and especially in the last few days.

Last week i've reported about the point that the X-Force numbers regarding unpatched disclosures could be sorted in a different way to yield a completely different view on the data. More interesting is a recent development: After reassessing the data, many of the vulnerabilities had to be sorted into different categories. So the numbers were fundamentally incorrect as well.

The list changed a lot due to this changes: Sun went from 9% high+critical to 0%. IBM leads the pack with 29% unpatched high+critical vulnerabilities without patches. However 22% for Oracle doesn't look that good as well. You will find the updated list in the blog entry " Mid-Year 2010 X-Force Trend and Risk Report - Update to Unpatched Vulnerabilities Chart".

At the moment you read a lot about this X-Force report and Sun is said to keep more vulnerabilities unpatched. But More interesting than the number of unpatched number of patches is the number of "Percentage of Critical and High 2010 H1 Disclosures with no patch" on page 20 in this report.

1. Google: 33%
2. IBM: 29% (the owner of X-Force)
3. Oracle: 22%
4. Linux: 20%
5. Microsoft: 11%
6. Novell: 10%
7. Sun: 8%

Some code is so old that it predates the "invention" ( ) of free software by the FSF. Such an example is the RPC code provided by Sun to the world under a permissive license in 1984 and it was use in many implementations like the one used in Linux to provide NFS services. This was possible due to the licensing Sun choose at that time:

Sun RPC is a product of Sun Microsystems, Inc. and is provided for
unrestricted use provided that this legend is included on all tape
media and as a part of the software program in whole or part. Users
may copy or modify Sun RPC without charge, but are not authorized
to license or distribute it to anyone else except as part of a product or
program developed by the user.

However it isn't technically free software, as it was freeded before free software was formally defined ... and out of some strange reasons, people found the license now free enough. In some Linux distributions this situation was considered as a serious bug (I would file a bug against something different in regard of this, but that's a different story ) However due to whatever reasons, this particular issue wasn't resolved for years ...

As Tom Callaway wrote in a recent blog entry, this situation has been resolved now:

So, we restarted the effort with Oracle, and on August 18, 2010, Wim Coekaerts, on behalf of Oracle America, gave permission for the remaining files that we knew about under the Sun RPC license (netkit-rusers, krb5, and glibc) to be relicensed under the 3 clause BSD license.

Sometimes charts look to perfect to be measured. I had this feeling when i saw the rperf numbers of the 795 and put them into a chart. I'm a very visual person, so i put everything in a chart just to get a feeling for numbers.

At first i thought i was paranoid, but then my colleague Jan Brosowski mailed to me that he had thought the same, albeit he approached the problem from the mathematical point of view. Okay ... that left me with a lot of questions and so i did some quick bullshit-testing math on the datapoints.

Some math

After reading his mail i wanted to do some tests on my own. So i did a short test with the numbers. At first i've put the data of the 3,7 Ghz P7 into my favorite statistical programm R.

> procs <- c(24,48,72,96,120,144,168,192)
> rperf <- c(273.51,547.02,820.53,1094.04,1367.55,1641.06,1914.57,2188.08)
> fm <- lm (rperf ~ procs)
> fitted.values(fm)
      1       2       3       4       5       6       7       8 
 273.51  547.02  820.53 1094.04 1367.55 1641.06 1914.57 2188.08 
> residuals(fm)
            1             2             3             4             5 
-7.886829e-14 -1.045378e-14  6.540129e-14  6.414338e-14  3.446377e-14 
            6             7             8 
-2.363755e-14 -2.489546e-14 -2.615336e-14 
> coefficients(fm)
 (Intercept)        procs 
1.607775e-13 1.139625e+01
> summary(fm)

Call:
lm(formula = rperf ~ procs)

Residuals:
       Min         1Q     Median         3Q        Max 
-7.887e-14 -2.521e-14 -1.705e-14  4.188e-14  6.540e-14 

Coefficients:
             Estimate Std. Error   t value Pr(>|t|)    
(Intercept) 1.608e-13  4.241e-14 3.791e+00  0.00906 ** 
procs       1.140e+01  3.499e-16 3.257e+16  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 5.442e-14 on 6 degrees of freedom
Multiple R-squared:     1,	Adjusted R-squared:     1 
F-statistic: 1.061e+33 on 1 and 6 DF,  p-value: < 2.2e-16

Strange ... the linear model leads to coeffcients able to predict the rperf value per core with minimal residuals. And i learned not to trust data with a R-squared of 1. Okay ... let's check for the 4.0 GHz P7 perf numbers:

> procs <- c(32,64,96,128,160,192,224,256)
> rperf <- c(372.27,744.54,1116.81,1489.08,1861.35,2233.62,2605.89,2978.16)
> fm <- lm (rperf ~ procs)
> residuals(fm)
            1             2             3             4             5 
 1.627514e-13 -1.454981e-13  4.364426e-16 -4.677910e-15 -3.821397e-14 
            6             7             8 
-1.490662e-14 -1.052861e-13  1.453949e-13 
> fitted.values(fm)
      1       2       3       4       5       6       7       8 
 372.27  744.54 1116.81 1489.08 1861.35 2233.62 2605.89 2978.16 
> coefficients(fm)
  (Intercept)         procs 
-3.215549e-13  1.163344e+01 

Again ... minimal residuals. Okay ... last check ... for the 4.25 GHz procs.

> procs <- c(24,32,48,64,80,96,112,128)
> rperf <- c(347.36,463.14,694.71,926.28,1157.85,1389.42,1620.99,1852.56)
> fm <- lm (rperf ~ procs)
> residuals(fm)
            1             2             3             4             5 
 3.163842e-03 -1.638418e-03 -1.242938e-03 -8.474576e-04 -4.519774e-04 
            6             7             8 
-5.649718e-05  3.389831e-04  7.344633e-04 
> fitted.values(fm)
        1         2         3         4         5         6         7 
 347.3568  463.1416  694.7112  926.2808 1157.8505 1389.4201 1620.9897 
        8 
1852.5593
> coefficients(fm)
 (Intercept)        procs 
 0.002429379 14.473100282

Sorry ... that's is looking to perfect to me.

When you put the 64-cores LPARs data into the same system, you will see for 4.0 GHz:

> procs <- c(128,256)
> rperf <- c(1406.36,2812.72)
> fm <- lm (rperf ~ procs)
> coefficients(fm)
  (Intercept)         procs 
-3.215549e-13  1.098719e+01

And now for the 4.25 GHz P7:

> procs <- c(64,128)
> rperf <- c(777.09,1554.18)
> fm <- lm (rperf ~ procs)
> coefficients(fm)
  (Intercept)         procs 
-1.607775e-13  1.214203e+01

Both times the intercept is 0 (i assume the small intercept is owed to rounding in data point by IBM or by the challenges of floating point arithmetic on computers.

That's totally unreasonable for measured data. When you just assume 99% of the performance for the 256 cores datapoint (thus an practically impossible scaling factor) you would have an intercept in the range of 28.13.

> procs <- c(128,256)
> rperf <- c(1406.36,2812.72*0.99)
> fm <- lm (rperf ~ procs)
> coefficients(fm)
(Intercept)       procs 
   28.12720    10.76744

Conclusion

At the moment i don't believe that IBM has really measured all the data it provides in the rperf list. The data fits to perfect in a linear model. The interesting question is: "Which data points were really measured?" All the data provided for the configurations look computed/guessed or something like that and not measured. Even when you want to assume that IBM found a way to the holy grail of linear scaleability, an R-squared of 1 and residuals at 0 are just ridiclious. I would really like to know what data points were really measured.

The new "IBM Power Systems Performance Report POWER7, POWER6 and POWER5 results" holds an interesting piece of information. A reoccuring question be colleagues and befriended admins is the impact of LPARs to the performance. It looks like IBM needs the LPARS to get some speed out of their larger systems.

Just a few examples: When using a 795 with 4.25 GHz and 64 cores a configuration with 4 LPARS a 16 cores yield a relative performance of 926.28. The same system with just 1 LPAR with 64 processors yield a relative performance of 777.09. So leaving the scaling to the OS instead of dividing it into 4 small systems gives you just 83.89% of the performance. When using a 795 with 4.25 GHz and 64 cores a configuration with 8 LPARs with 16 cores each yields a rperf value of 1852,56. With 2 LPAR with 64 cores each you get 1554,18. Interestingly is 83,89% again.

At first i thought "16 cores are easily fitting on a processor book (with 4 procs each). A 64 cores LPAR has to use two processor books. So when you use a configuration larger than a processor book you will leave 16,11% on the way". But doing the same calculation with some other data showed otherwise. But the move from 32 to 64 core lpars just reduce the performance by 5,7 percent respectively 5,4 %. 32 cores fit on a processor book, too. Thus the difference should be similar to the 64 to 16 cores situation.
So my interpretation is a little bit different: The scalability of AIX seems to have sweet spot between 16 and 32 cores. I thought a moment about an intra-book bottleneck, but the CPUs on the book are fully meshed (1 hop from each CPU to every other), so i don't think it's a problem.



When you look into this chart (please click into the image for a larger version), you may find some interesting points. The light blue line is a hypothetical perfectly scaling 4.0 GHz Power7 in a 795. The data is based on rPerf number of 103.41 for an 8 core system(source: Page 20 of the document). Please look at the right side of the chart at 256 cores. You end up at 81,9% of the hypothetical performance when you use 64 core LPARs and at 86% of the hypothetical performance when using 32 core LPARS (the difference is interestingly pretty much the same as computed before for 64 cores instead). At 64 cores your load is distributed at 8 processors, thus just 2 processor board. Still there is an serious impact of almost 20%. Will be interesting to further dig into this topic.

However it's important to know, that the operating system is limited to 64 core SMP no matter how many cores are in the system by the LPARs configuration. So this numbers doesn't factor in scalability challenges of AIX above 64 cores as the os has not to scale above this point while generating this rperf numbers. The numbers for the large core number configurations are not single OS image numbers. Further penalties for the OS scaling comes on top. Furthermore this benchmark is a pure CPU/memory benchmark. As IBM explained in their own description of the benchmark, there is no I/O and no networking involved.

That said, a number of really interesting data points are missing in the pdf:

• The rperf number of a fully blown unpartitioned system
• The rperf number on a fully blown system with just one LPAR in the size of the complete system
• Somehow i have the impression, IBM is hiding something. When you look into the mentioned pdf there is SPEC number for an unpartitioned system, but no rperf for it. The existence of the SPEC numbers hints to the point, that they had indeed an OS that was able to scale to 256 cores. On the other side, there is no SPEC number for the partitioned systems, but the rperf numbers. Just call it a presentiment ...

Just found this article about a new SSD from Micron. This article reminded me of my thought that there should be a third category in SSD. In the harddisk market there are consumer drives, enterprise drives and near-line drives. A consumer drive is that stuff you include in your desktop that is designed to run for a limited number of hours a day and mostly sits idle in the computer casing. An enterprise drive is designed to to work 24/7 with being busy most of the time (80% for example). Nearline drives are the one that can run for 24/7 but they are just designed for 20% busy time. Of course you you could use any disk for each task, but it's a simple equation: You swap busy time for the lifetime of the drive. You can use your SATA home user drive for your database but expect to see your storage array very often.

That said, i think we need a similar distinction in the SSD area. At the moment there are just enthusiasts drives and enterprise drives. Out of a strange reason there are enthusiasts drives that got the enterprise moniker, but thats a different story. But there is something similar important to the busy time for SSD, even when they aren't susceptible to mechanic wear. It's something i would like to call "write time". It's specified as the amount of time the drive is writing data to the media. That's unimportant for rotating rust, but for Flash based solid state drive it's completely different.

Let's do some calculation: Microns new drive is said to deliver 275 Megabyte/s. 275 Megabyte 60 seconds 60 minutes 24 hours 365 days * 5 years equates to 40,3840095 Petabyte. However the endurance of the drive is specified to be 4.5 Petabyte in 5 years. Thus you could only write in roughly 1/10 of the time data onto the disk. It has an allowable write time of roughly 10%.

So that may not a problem for usage if normal filesystems are used, where read and write load occurs on the disk, but now think of the ZFS separated log ... this is a write-only device for almost all practical purposes. You never read from it in normal operation,so there are no reads to relief the drive from the write load.

So you should not size the amount of SSD disks not only for the expected write bandwidth, you should also take into consideration how this influences the lifetime of your drive and how long you expect the server to stay in operation. At 275 Megabytes per second you would brick a drive designed for 4.5 PB within five months. As well as you choose the right disk type by thinking about the busy time, you have to factor in the write time for your SSD.

Where should someone set the differences between the types? Enthusiast drives are easy to spot ... MLC drives and all drives without capacitors in case they have caches. Near-line and enterprise SSD? I would draw the line somewhere around 50%. When you can't get through the warranty time at max speed with 50% writes, it's a nearline drive. Above that it's an enterprise drive. But that's only my personal opinion and a first idea.

After reading the article "Amerikas Superreiche starten Spendenrevolution" at spiegel.de, i found Mr. Ellisons name on the givingpledge.org-website.

The Giving Pledge is an effort to invite the wealthiest individuals and families in America to commit to giving the majority of their wealth to the philanthropic causes and charitable organizations of their choice either during their lifetime or after their death.

Bryan wrote an interesting article about his move to Joyent:

Add it all up — the history in the cloud space, the disposition to solving tough cloud problems that I want to solve like instrumentation and observability, and the exciting development of node.js — and you have a company in Joyent that I believe could be the next great systems company and I’m deeply honored (and incredibly excited) to be a part of it!

The world celebrates the 11th SysAdminDay today. To all the users out there: This is a nice opportunity to thank your admin staff for doing migrations at night to keep your business running at day, for making seemingly impossible requirements possible, for keeping the systems up and running, for answering your questions, that would be able to answer yourself with google in the time you need to pick up the phone

And keep in mind, your admin would do everything for you to restore the operations of your datacenter:

Is this a new trend in IT sales? Companies offering AMEX gift cards for attending a sales presentation. Got the second mail of this kind. Greenbytes offered $50 for listening to them, a few minutes ago i've got a similar mail from a company called Exagrid offering $100 for an half hour web presentation.

Somehow i'm not sure what i should think about that: Should i translate this "We are so sure that you will buy the product that we pay you to take a sales call!" or "We are so desperate that we need to pay people just to take a sales call...".

Perhaps you could make a living out of professional sales attending ... given that such examples are representing a trend ... or auction an hour a day on eBay in half-hour increments for such paid sales calls

With this article i want to learn a thing or two and i want to start a discussion.

Whenever you express, that a mainframe hasn't no advantages over a high-end Unix system, the first comment or reply to a comment is "Oh, but IBM has Parallel Sysplex and delivers availability xyz can only dream of". Often the discussion ends there, because few people know, what this "Parallel Sysplex" is. I had to read into a lot of documentation to learn more about the Parallel Sysplex.

I've learned a few things in that: A sysplex doesn't seem to be much different from something like we simply call a cluster in Unix. A parallel sysplex is an extension of this and enables the system to act as one large system. However you can't consider it as a tool that is able to distribute arbitrary tasks on sub-process level onto several systems.

As far as i understand the concept of the Parallel Sysplex, it's not the way that any and every software is enabled to be a part of a parallel sysplex as it has to use certain facilities that make a parallel sysplex out of a sysplex. Your application have to use a thing called the coupling facility. And parallel sysplex exactly contributes this facility to the applications running on the mainframe.

Parallel Sysplex doesn't help you to make your zLinux more available. You have to work with the Linux cluster-frameworks to do so, albeit there seems to be some workarounds

I understand that the parallel sysplex makes DB2 on z/OS more scalable and available, but where is the difference to Oracle RAC on Solaris on a 2 big node cluster? I understand that a parallel sysplex makes CICS more available and scaleable, but where is that different for example from using Oracle Tuxedo in a highly available configuration with clustering support? I understand that that parallel sysplex can make storage more available, but how is that different from the Availability Suite with it's Remote mirror facility, from SamFS/QFS, from the global filesystem in Sun Cluster.

I know, all these things working differently, but that's not the point. I want to know, why some people think "Oracle can just dream of". What's so special about that ... the parallel sysplex is an ... well ... interesting architecture, however i fail to see any reason why it's special that everyone talking about mainframes is just pointing to it.

Perhaps it's just the way that mainframe people doesn't understand the open systems world to the same extend as open systems people understand the mainframe world: Barely to an extend at all. Additionally i have the impression that many services on a mainframe are so ubiquious in that world that they consider properties of this software components as properties of the mainframes. Another hypothesis is, that most mainframe guys doesn't know what's possible on Unix (real Unix, not the pengiun variant) and thus consider their technologies as superior. I talked to a mainframe guy on a conference some time ago and i had to explain that SunCluster has gone great lengths since Sun Cluster 2.

We talk about synchronous storage replication in Solaris for quite a time now (2000), service restarting isn't a problem since SMF. Instruction Retry on hardware basis is available since the introduction of the M-Series, the same for memory mirroring. Hot swap of I/O, memory and CPUS is possible since last century.

May be i just don't get get, but may you are able to help me: I want to postulate "There is nothing in a Parallel Sysplex that a equivalent configuation on the basis of an M9000 isn't capable to do". But i'm a open systems guy from the core, so i ask the mainframe aware people to explaon me, why Parallel Sysplex is considered as superior to Unix-HA clustering when used with applications developed for high availability. Please don't answer with "You have no idea, ParSys is so superior" ... i want some real reasons.

There is one point in the media reporting that is really upsetting me: Some news outlets were touting the hybrid mainframe as the development that will save this technology for decades. IBM gets an overly positive press for placing a rack besides the mainframe.

Sun was talking about the concept of workload computing for years now: Use x86, UltraSPARC T* and SPARC64 where it fits best and use Solaris as the overarching umbrella. The same OS on all components.

However nobody was touting this as a major step forward ... Just as a reason to bash Sun, because they interpreted is as uncertainity about their technology. There were even people seeing this as an excuse for the single thread performance of T1/T2.

They didn't looked at it as the basic point behind it: That one size doesn't fit all and one proc doesn't fit all needs, and a proc can't fulfill all needs.

Just got the news, that Oracle Solaris Premier (24x7 Support) Support is available on HP servers via HP again. You have to order it via HP Software License Management Services. For details ask your friendly HP sales rep. However:there is (and was) a way to buy Support for Solaris on HP via Oracle, about that way ask you friendly Oracle sales rep.