Planet Sysadmin               

In my last entry, "CL" posted a comment wanting more to see more examples of Windows media redirection.  Here's another video showing a variety of clips.  Talking heads, animation,  a couple of VC-1 live streams.  Enjoy.  Still have the dog barking at the tripod problem, so I apologize for some of the shaky parts.  Take a Dramamine if you get queasy.  If you want to watch this in a bit better quality, follow this link and click on the "watch in high quality" link once you get into YouTube (right under the "Views" number).

It's been over a month (and three hurricanes in America) since I've posted a blog. More than a few of you've noticed - thanks for the prodding...

It's been a busy summer, on nearly every front. Customer activity hasn't slowed down, and the good news surrounding the (otherwise unfortunate) economic crisis embroiling many customers (especially those in the financial services industry, a heavy concentration for Sun) is that it's whipping up the winds of change. Customers facing spending pressure, or tiring of vendor price increases have new options, and there's a new appetite to explore those options (nothing like mandates from the CEO to reduce spending by 50%).

One of my more interesting recent meetings wasn't with a customer, though, it was with an equity analyst from a global financial institution. Equity analysts publish research that feeds the investment community - their (free) research and financial analysis accompanies buy/hold/sell recommendations to investors (who hopefully generate trading fees for the analyst's employers).

This one analyst hadn't historically followed Sun, and was in the process of developing his first rating. He wanted to focus on our storage plans - more and more of the customers whom he interviewed were focused on storage, and many were talking up a specific open source software technology: ZFS. (Before meeting with me, he'd talked to colleagues in his own IT shop, and was impressed to find some who admitted to running ZFS at home - nothing like touching your customers where they live... if you'd like to have ZFS sent to you, click here or on the LiveCD shown at right.)

Granted, you can see an increasing focus on storage at Sun - the acquisition of MySQL is as much a storage acquisition, as an enhancement to Sun's developer offerings. Discussions of flash memory, the economics of archiving, the Lustre parallel file system, all point to an increasing focus on what Sun sees as an exceptional opportunity for customers (and thus, investors). Storage and computing are converging - and we're about to bring the trends that transformed the server industry a few years ago (mass engagement in open development communities, and scale achieved via clusters of commodity parts vs. proprietary technologies) to the historically closed and proprietary storage industry.

Now, the notion of "engaging customers in open development communities" doesn't sit well among some traditional storage analysts (or our competition) who believe "Storage is too mission critical to tolerate open source software." Although I appreciate that wisdom and experience, I think the market's more nuanced than that - mission critical environments don't tolerate unsupported software, true, which is why we offer 24x7 commercial support for ZFS (on Sun hardware, and Dell, even). But broad global adoption of key open source projects will continue to drive change deep into the world's datacenters. Gartner's prediction that 90% of world's companies will run open source software didn't specify where they'd be running it - "everywhere" is the safest bet.

But back to the equity analyst - he patiently asked, "Great theory, but when will you see revenue results?"

"Last year," I responded. "You're seeing it accelerate."

As many folks know, we shipped our first ZFS based storage systems in 2007 - known as Thumpers. Thumpers finished up this last year generating around $100m in billings, up 80% year over year. From a capacity perspective, we delivered roughly 90 petabytes of Thumper storage in FY2008, to some of the most demanding storage installations on earth (up ~200% y/y). What's fueling the growth? Adoption of ZFS is a clear driver (this chart gives you a sense of where we're seeing adoption - thus revenue opportunity). But ultimately, customers are recognizing they can save money, space and power. Thumpers are roughly twice the capacity in half the space at half the cost of the competition - $1.20/Gigabyte. (They also run Windows and Linux with the same hardware economics).

Now, our view is "OpenStorage" (systems built from commodity parts and open source software) will grow far faster than the proprietary storage market. We plan on driving that growth, and over the next few months, you'll see a tremendous amount of storage innovation targeting the growing breadth of customers wanting better/faster/cheaper/smaller options. Expect to see flash, zfs, dtrace, and good old fashioned systems engineering play a very prominent role in an aggressive push into the storage market.

And in case you missed our announcement last week, our progress was validated by industry analysis - IDC said customers are growing their disk storage business with Sun far faster than with any of our proprietary competition. And at three times the rate of the overall market's growth. A great place to start.

If you'd like to know more, and might be interested in taking a Thumper system for a free trial run, just click here and pick the country in which you're located. We supply most systems at Sun for free trials across the globe (yes, we even cover shipping to you). If you like the system, please buy it. If not, we'll take care of getting it returned to Sun, you owe us nothing. (That's the closest we can get to free hardware downloads...)

As I said to the analyst, you need only look to the results we're already delivering to see the linkage between open innovation and revenue growth. ZFS won't transform demand for our legacy products, but it'll certainly transform the opportunity and industry unfolding before us. But don't just get our opinion, the best folks to validate our approach aren't at Sun, they're among the storage buyers finally feeling the winds of change - at their backs.

To obtain certain lighting effects, you sometimes need to attach things to your flash head, such as gel filters or bounce cards. Normally, this involves fussing around trying to strap things down with a rubber band or opting to gum up your equipment with sticky-backed velcro or tape. Instead you can create a cheap speed strap that's both easily removable and simple to attach things to.

There are commercial solutions for this that can be a bit pricey, but Jake O'Connell found a way to make one on the cheap for about $3. Velcro came out with something called a "Design Strap", which is basically a zip-tie with Velcro on one side. Put a rubber band around your flash and then wrap a couple Design Straps over that and you have an attachment system that stays put, comes off easy, and is simple to attach things to. Just add velcro to your gels, bounce cards or whatever and you can quickly affix them to the flash however you like.

DIY 3 Dollar 10 Second Speed Strap

[Prologue] SysAdmin received this email, looks like the request has been escalated to my boss. L1/L2, Could you please assist Matt ASAP! This is pertaining to UNIX acct so we need to iron out. The SC is 14270227. Make this high priority please. Thanks Issue root filesystem (/) is currently 100% full. Quick Response Team require free disk space to perform some activities [...]

Google Chrome is an open source web browser developed by Google. The name is derived from the graphical user interface frame, or “chrome”, of web browsers.Chromium is the name of the open source project behind Google Chrome,released under the BSD license.

(...)
Read the rest of Install google chrome with wine in Ubuntu (329 words)

---

© admin for Ubuntu Geek, 2008. | Permalink | No comment | Add to del.icio.us digg
Who's linking ? Technorati BlogPulse Google
Want more on these topics ? Browse the archive of posts filed under General.

Related Articles

• Webmin Installation and Configuration in Ubuntu Linux (2)
• Webalizer - Apache web server log file analysis Tool (3)
• Upgrade Ubuntu Server 6.10 (Edgy Eft) to 7.04 (Feisty Fawn) (3)
• Update IP addresses at dynamic DNS services Using ddclient (12)
• UFW (Uncomplicated firewall) For Ubuntu Hardy (21)
• Ubuntu Linux + Apache2 + Virtual Hosts + Syslog Server (6)
• Ubuntu 8.04 (Hardy Heron) LAMP Server Setup (34)
• Ubuntu 7.10 (Gutsy Gibbon) LAMP Server Setup (29)
• Ubuntu 7.04 (Feisty Fawn) LAMP Server Setup (15)

Last week's launch of Google Chrome generated some discussion over the legal language in our new browser's terms of service (TOS). As we noted in a subsequent post on Google Chrome's terms of service:

"... Under copyright law, Google needs what's called a "license" to display or transmit content. So to show a blog, we ask the user to give us a license to the blog's content. (The same goes for any other service where users can create content.) But in all these cases, the license is limited to providing the service."

We've also some seen discussion on a few blogs about how our universal terms of service apply to other products, with some users worried that Google is trying to claim ownership of the content they generate. To be clear: our terms do not claim ownership of your content -- what you create is yours and remains yours. But in lawyer-speak, we need to ask for a 'license' (which basically means your permission) to display this content to the wider world when that's what you intend. This issue is not unique to Google; it applies to lots of other Internet companies that display and transmit user content. You can see some other terms of service here from Amazon, eBay, and Facebook.

In some of our products, such as Gmail and Google Docs, we have included additional terms to make it clear that we do not claim ownership of the content. But even without those additional clarifications, we still wouldn't be claiming ownership of your content -- just a license that gives us your permission to use the content to provide the service. The additional terms are there to reassure our users that they still own their own content, even after giving us the permission we need to help them share and collaborate with others, whether via Gmail, Blogger, YouTube, Google Docs, or other services.

Because, in the end, that's what's most important: making sure you're comfortable using our services to share, publish, and store your stuff. We'll continue to look at our terms of service to make them as clear and user-friendly as possible, because at the end of the day if you're not comfortable, our products won't succeed -- and we know it.

Posted by Mike Yang, Senior Product Counsel

Bob Payne interviewed me at Agile 2008. We spoke about my initial plans for Agile 2009, and my (in-writing) project portfolio book. The link is here: Agile 2009 - Johanna Rothman - Agile Portfolio Management and Agile 2009. I had a blast with Bob.

If you’re wondering why it sounds like I’m chewing my cud (!), it’s because I was shivering. I was wearing a nice shirt and a nice jacket, but it was so cold in the area Bob had set up for the interviews, my teeth were chattering and my mouth was dry because I was so cold. Next year, I will bring my fleece, and possibly a scarf and gloves.

I got an Asus eee pc (model 900) the other day (I ordered it through buy.com). I'm going to a conference next week (zendcon), and I didn't want to lug around my laptop (it gets pretty heavy after a few hours). I didn't need much--something with a Web browser, an ssh client, something to pull images off of my camera (a Kodak Easyshare v1003), and a text editor for taking notes. The eee pc fits that nicely, and it only weighs about 2.5 pounds.

It's got an 8" screen which I'm finding to be perfectly readable. The keyboard is pretty small, but I'm getting used to it (I'm posting this from the eee pc).

It has an SD/MMC card reader, 3 USB 2.0 ports, and built-in wireless (802.11 g/b). It has 1GB of RAM and a 20GB solid-state hard drive (the thing boots to the login prompt in about 30 seconds). It even has an integrated 1.3Mpixel Web cam.

I'm finding the eeeuser wiki to be very helpful (especially a post about getting gwenview to import photos from my camera).

One little wrinkle I experienced was that the wireless adapter was inadvertently disabled, and it took me a little while to figure out what had happened. The eee pc typically recovers well from going to standby mode, but once when I tried to resume, the screen looked like static and I couldn't get it to do anything. I had to hold down the power button and reboot. When it came back, it refused to join my wireless network. I finally figured out (with the help of the diagnostic tools) that the wireless adapter had been disabled (dunno if I accidentally did that, or if it was a fluke). Anyway, holding down the special function key ("Fn" in the lower left) and hitting F2 toggles the adapter, and I was back in business.

I noticed a couple of blog posts written in response to my (somewhat) recent php|arch article:

• An alternative way of EAV modeling
• EAV Modeling - Square Peg in a Round Hole?

Reading these was initially discouraging, as both authors are critical of the concept of my article. Although I don't necessarily agree with them, they both make good points in their posts, and I encourage you to read them both if you are interested in the topic. I particularly appreciate the point about table-level locking in MyISAM tables, and how that might affect performance in the frequently-updated value tables.

Although this is not a very useful rebuttal, about all I can say is that the EAV method has worked well for me. As with anything, your mileage may vary. I'm not running reddit or facebook, and the systems I've built on EAV don't have thousands of simultaneous users. So I really can't say how well it would perform in a large-scale deployment.

After I sort of got over being defensive about the whole thing, I'm just glad that people found the article interesting enough to talk about it.

For reasons unknown to me, the Trib published the vital statistics column today, a Sunday, instead of on its regular Monday schedule. That means you don’t get to start the work week with Bad Baby Names. Instead you get to leisure over them during a lazy Sunday!

Marysa and Shaylyn are the traditional Y-attack names. While bad, these are getting to be a little old and tired.

Luckily Xayton was born to come wake us up. Xayton sounds like a name from the future, possibly the name of an evil robot overlord. In that sense it’s a cool name, but it’s no name I’d want to saddle a little boy with.

The emotional favourites for today don’t really have a bad name, they’re just appropriate for the day. Two sets of parents named their kids Angel. One boy, one girl. Isn’t that nauseating?

But the winner is Ja’miesha. Complete with apostrophe. Jamiesha isn’t even a good name on its own, but throwing in that apostrophe brings things to a whole new level. In Hawaii the
http://en.wikipedia.org/wiki/Okina">
http://en.wikipedia.org/wiki/Okina">
http://en.wikipedia.org/wiki/Okina">ʻokina is used to represent a glottal stop. Does this mean that we’re supposed to pronounce Ja’miesha with a glottal stop in there? Or were the parents just idiots and thought that their little girl’s name would look pretty with an apostrophe (it’s not even an ʻokina because Jamiesha isn’t anywhere near Hawaiian)? I vote the latter.

Even before the operating system is booted, GRUB enables access to file systems. Users without root permissions can access files in your Linux system to which they have no access once the system is booted. To block this kind of access or prevent users from booting certain operating systems, set a boot password. As the user root, proceed as follows to set a boot password:

Now that I have my weblog looking reasonably consistent between Gecko and WebKit based browsers, I’ve taken another look at Opera.  Opera doesn’t have support for border-radius, but does have support for background images in SVG, which can be used to provide the same effect.  My Nav Bar on my test site now employs this technique, and it requires two separate images: 039 on CCD and CCD on FFF.

More complex effects are also possible, with only slightly more work.  I’ve applied some of those techniques here.

Frankly, my first reaction to this was mixed.  No two ways about it: it requires more work and more page fetches to produce the same result as could be done in CSS.  This might be slightly better if one could somehow embed the SVG in the CSS file itself, as many of the SVG files are engineered for a single purpose.  This lead me to initially think that having CSS continue to capture the common cases, leaving comparatively advanced techniques like SVG for special purpose or complex effects was the right way to go.

But then I saw this (screenshot) and even (&deity; forbid) this (screenshot), and it occurs to me that SVG opens the door to unubtrusive special purpose and (when judiciously employed) not-so-complex effects.  It would be fairly easy to give my Nav Bar a more clearly defined frame, a more tapered shape, and use filter effects to vary the color.

The pluses for SVG in CSS is that it doesn’t require either adjusting your markup or JavaScript to achieve these effects, a desirable characteristic that generally the other techniques don’t share.  Nor does it require that your page be XHTML.  And there are lots of good techniques for effectively caching static files.  The only remaining issue is that this technique works best for backgrounds instead of borders, as backgrounds images (if present) are displayed on top of the background color, making the background color an effective fallback for browsers that don’t support SVG in CSS.  Of course, the background can draw a border within the padding, but any CSS provided borders would still show up outside of the background.

Meanwhile, Robert O’Callahan has been exploring other ways to integrate these technologies.

Hadoop is a Java-based distributed application and storage framework that's designed to run on thousands of commodity machines. You can think of it as an open source approximation of Google's search infrastructure. Yahoo!, in fact, runs many components of its search and ad products on Hadoop, and it's not too surprising that they are a major contributor to the project.

MapReduce is a method for writing software that can be parallelized across thousands of machines to process enormous amounts of data. For instance, let's say you want to count the number of referrals, by domain, in all the world's Apache server logs. Here's the gist of how you'd do it:

1. Get all the world to upload their server logs to your gigantor distributed file system. You might automate and approximate this by having every web administrator add some javascript code to their site that causes their visitor's browsers to ping your own server, resulting in one giant log file of all the world's server logs. Your filesystem of choice is HDFS, the Hadoop Distributed Filesystem, which handles partitioning and replicating this enormous file between all of your cluster nodes.
2. Split the world's largest log file into tiny pieces, and have your thousands of cluster machines parse the pieces, looking for referrers. This is the "Map" phase. Each chunk is processed and the referrers found in that chunk are output back to the system, which stores the output keyed by the referrer hostname. The chunk assignments are optimized so that the cluster nodes will process chunks of data that happen to be stored on their local fragment of the distributed file system.
3. Finally, all the outputs from the Map phase are collated. This is called the "Reduce" phase. The cluster nodes are assigned a hostname key that was created during the Map phase. All of the outputs for that key are read in by the node and counted. The node then outputs a single result which is the domain name of the referrer, and the total number of referrals that were produced from that referrer. This is done hundreds of thousands of times, once for each referrer domain, and distributed across the thousands of cluster nodes.

At the end of this hypothetical MapReduce job, you're left with a concise list of each domain that's referred traffic, and a count of how many referrals it's given. What's cool about Hadoop and MapReduce is that it makes writing distributed applications like this surprisingly simple. The two functions to perform the example referrer parsing might only be about 20 lines of code. Hadoop takes care of the immense challenges of distributed storage and processing, letting you focus on your specific task.

Since Hadoop is written in Java, the natural way for you to create distributed jobs is to encapsulate your Map and Reduce functions into a java class. If you're not a Java junkie, though, don't worry, there's a job wrapper called HadoopStreaming which can communicate with any program you write with the usual STDIN and STDOUT. This lets you write your distributed job in Perl, Python or even a shell script! You create two programs, one for the mapper and one for the reducer, and HadoopStreaming handles uploading them to all of the cluster nodes and passing data to and from your programs.

If you want to play around with this, I really recommend a couple of howtos written by German hacker Michael G. Noll. He put together a walkthrough for getting Hadoop up and running on Ubuntu, and also a nice introduction to writing a MapReduce program using HadoopStreaming (with Python as an example).

Are any Hackszine readers using Hadoop? Let us know what you're doing and point us to more information in the comments.

Hadoop
Running Hadoop On Ubuntu Linux
Writing An Hadoop MapReduce Program In Python

Scobleizer — Tech geek blogger » Blog Archive The Superbowl of Startups «:

"Blogging is NOT reporting. It’s the single voice of a person. When you read me here you are reading me the way I’d talk to you at a cocktail party. You’re hearing my opinions. If I’m doing ‘reporting’ then you’ll know, because of how I source it."

I really like this explanation for blogging. Maybe we need "Blogging is NOT reporting" T-shirts.

via Youtube

via Youtube

While I've always been a bit of a perl guy I don't want this post to be "perl has x and python doesn't" in tone. Which is lucky really as Python has exceptions and threading as first class features where as perl has... ahem.

So after spending a chunk of today reading a python book and spending some time writing code here's my initial short list of gripes -

• except IOError
• print adding newlines
• Significance of whitespace in blocks. But not like that.
• The lack of ++

Considering how picky I can be that's a very short list so Python must sit well with me so far. Now, in order, I can't help but read except IOError as 'catch everything apart from IOError'. This one bugs me more than it should but considering how happy native exceptions in the language made me this just felt mean.

Secondly, print adding newlines. While this might seem trivial every other language I use on a daily basis has a print function that doesn't print a newline so this feels weird. At least it's not called say ;)

Now to the one that I'll get no sympathy on - whitespace in blocks. First up let me say I don't mind about the enforced indentation. I indent anyway so it's not a big deal. I guess I'll hit the odd case when it annoys me (probably involving heredocs) but I've got nothing against it. What does irk me is the lack of block delimiters - whitespace just doesn't cut it for me.

I like my { and } delimited blocks, a nasty voice in my head is telling me to add them but just comment them out ( if x == y: # { ) but that seems very wrong. I've always looked at those examples in C programming books that say...

# incorrect
if ( something )
  print("All's well");
  wellness++;

# this is wrong because wellness is a separate statement
# and not part of the if

... and thought - "just add the damn braces, you'll be back to add more code later anyway." Now I'm learning a language that seems to want me to slip up like this. I'll either get used to this or move to ruby.

Lastly we have the lack of ++ and --. I know the arguments, I've read them before. I disagree. I've never done anything insane with ++ and where I have used it it's saved me typing. Can we have ++ and remove nested ternary ( ? : ) instead please?

I like Python and I think I'll be investing more time in to learning it.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Summary: JavaScript Benchmarks aren't adapting well to the rapid increase in JavaScript engine performance. I provide some simple tests for verifying this and propose a modified setup which could be used by all JavaScript Benchmarks to achieve high-quality results.

There now exists three, what I would consider to be, major JavaScript performance benchmarks. Each are released by a major browser vendor. WebKit released SunSpider, Mozilla released Dromaeo, Google released the V8 Benchmark.

Each suite has a variety of tests and a test runner. I'm currently interested in one thing: The quality of the test runner that each of these suites provides.

There are three points that any test runner tries to achieve:

1. Retrieving accurate numbers. Another way to phrase it: "Retrieving stable numbers." If you run the test suite multiple times will you get identical, or near-identical, numbers?
2. Reducing the possible error. Does the suite attempt to quantify how much possible error there could be in their results? How large is the error?
3. Reducing run time. How long does it take to run each test?

The ideal suite would be one that's capable of running an individual test as quickly and accurately as possible with virtually no error. However, in order to get those numbers you need to carefully chose what style of tests you wish to run.

I quantify the current field of tests into three categories:

Slow-running tests: These tests generally take a couple hundred milliseconds on an average consumer machine. This is the style of tests that you generally see in SunSpider and occasionally in Dromaeo. These tests have a distinct advantage: They are generally quite accurate. You don't need to run them very many times (say about 5) in order to get a consistent picture of how the test will run.

Moderate-running tests: These tests take less than one hundred milliseconds. You'll see these tests in Dromaeo. These tests need to be run quite a few times (20-30) in order to get a better picture of their speed.

Fast-running tests: These tests take less than ten milliseconds. You'll see these tests in the V8 Benchmark and in Dromaeo. These tests must be run many, many, times (usually close to a thousand) in order to weed out any possible error. If you were to run one 10ms test 5 times - and get a single result of 11ms that would introduce a significant level of error into your results. Consider tests that take 0-1ms. A deviation within that range can instantly cause error levels around 50% to occur, if enough test iterations aren't completed.

Looking at the above categories the solution seems obvious: Use slow-running tests! You get to run them fewer times and you get accurate results - everything's peachy! But here's the problem: The speed of JavaScript engines are increasing at a rate faster than test suites can adapt. For example, SunSpider was initially developed with all tests running at equal speeds on a modern computer. Now that speed improvements have come along, though, (WebKit improvements, then Mozilla, then SquirrelFish, then TraceMonkey, then V8) those results don't even remotely resemble the tests of old. Most of the results have moved down into the moderate-running range of tests, some even into the fast-running range - but here's the problem: They're still only running the originally-designed number of loops. An example of the difference:

This means that a browser is running a test for 5-10 loops (in both SunSpider or Dromaeo) but the speed of the test no longer matches that assigned number of iterations. At first glance you could say one of two things: 1) Increase the difficulty of the tests or 2) Have them run for more iterations. There are problems with this, though.

While you certainly could increase the complexity of existing tests the result would be a never-ending battle. Improvements would have to land every couple months in order to keep up with the pace of improvement. This would work ok in Dromaeo (since it has versioned tests) but not all suites can handle this behavior. Additionally this now makes the tests less-useful for measuring slower browsers (the tests now take an absurd amount of time to complete as opposed to the very-plausible numbers from before).

Additionally, you could increase the number of test iterations that would occur but not without assigning specific iteration counts to each individual tests. And this is the full problem: How do you know what numbers to choose? Raising the number to 20 iterations may help one browser - but what about another browser which will need 100 iterations to get a proper count?

This leaves us in a bind. Browsers keep getting faster at tests, test suites do the wrong number of iterations, causing the error level to continually increase:

We should take a step back and look at what the test suites are doing to counter-act the above trend from happening - if anything at all.

SunSpider was originally dominated by long-running tests running 5 times each. The tests use to be long-running but are now only in to the medium to fast-running range (depending on the browser). This has caused the accuracy to decrease and error level to increase. Increasing the number of iterations would help (but hinder older browser performance).

Dromaeo has a range of tests (fast, moderate, and long-running) each running 5-10 times each. Dromaeo attempts to correct the number of iterations run, right now, but kind of fails when doing so. It looks at the results of past iterations (especially the error level generated by the results) and decides to run more tests until a stable error level is achieved. The problem with this is the samples are no longer being independently determined. Whereas test runs 1-5 were independent, test runs 6-10 were not (they're only being run due to the fact that previous test runs provided poor results). So while the results from Dromaeo are hyper-stable (they're the most stable performance test that we run at Mozilla) they're not determined in a proper statistical manner. Thus Dromaeo needs to be changed in order for people to be able to gather accurate results without sacrificing its statistical integrity.

The V8 Benchmark takes a completely different strategy for its fast-running tests: Instead of running for a pre-determined number of iterations each test is run continuously until a second of time has passed. This means that individual tests frequently run anywhere from 50-200 times (depending on the machine and browser they run on). Currently the V8 Benchmark does suffer from one shortcoming: There is no error calculation done. Both SunSpider and Dromaeo fit the results to a t-distribution and compute the possible error of the results whereas the V8 Benchmark just leaves them as is.

However, the V8 Benchmark does bring up a very interesting strategy. By picking tests that are simpler (and, arguably, most current complex tests will become "simple" as engines rapidly improve) and running them more times (relative to the complexity of the test) the results become much more stable.

Consider the result: Fast-running tests end up having a smaller error range because they are able to run more within a given allotment of time. This means that the test runner is now self-correcting (adjusting the number of iterations seamlessly). Since all JavaScript engines are getting faster and faster, and complex tests are running in shorter and shorter amounts of time, the only logical conclusion is to treat all tests as if they were fast tests and to run them a variable number of times.

We can test this hypothesis. I've pulled together a simple demo that tests the computational accuracy of the three styles of test (Simple - or Fast-running, Moderate, and Complex - or Slow-running) against the two different types of suites: "Max" style (the style implemented by the V8 Benchmark) and "Fixed" style (the style implemented by SunSpider and Dromaeo). The results are quite interesting:

Fast and moderate-speed tests are incredibly accurate with the max-style of test running. Often their error is no more than 2% (which is really quite acceptable). The quality degrades for the complex tests, but that's ok. Complex tests could be tweaked to consume less resources (thus allowing them to iterate more times and become more accurate).

Right now the accuracy of slow-to-moderate tests in fixed run test suites are suffering from increased rates of error as the engines get faster (as can be seen in the above chart).

The major change that would need to be made to the flexible style of execution (at least the one implemented in the V8 Benchmark) would be some form of error checking and t-distribution fitting. In V8 a series of tests are run over the course of one second. I propose that the resulting number (the total number of tests executed) be saved and the tests are repeatedly run again (perhaps 5 times, or so). The resulting set of 5 numbers (representing, potentially, thousands of test runs) would then be analyzed for quality and error level. I have this measure completely mocked up in my sample test measurement.

The relevant portion of the code is here:

  setTimeout(function(){
    var start = (new Date).getTime(), diff = 0;

    for ( var n = 0; diff < 1000; n++ ) {
      test();
      diff = (new Date).getTime() - start;
    }

    runs.push( n );

    if ( r++ < 4 )
      setTimeout( arguments.callee, 0 );
    else {
      done(name, runs);
      if ( next )
        setTimeout( next, 0 );
    }
  }, 0);
}

Switching to a flexible style of text execution has many benefits:

• It'll help to keep the rate of error small for most tests (especially if the tests are tailored to be smaller in nature).
• It'll allow the suite to gracefully adapt over time, as engines speed up, without sacrificing the ability to work on old engines.
• The maximum number of runs (and, thus, the maximum amount of accuracy) will occur for the engines that are able to complete the tests faster. Since the greatest amount of competition is occurring in these high numbers (as opposed to in older user agents, where there is no progress being made) granting them the most accurate numbers possible makes this ideal.

I plan on prototyping this setup into Dromaeo and releasing it shorting. It'll take much longer to run the full test suite but the result will be quite worth it.

vs

That's DJB on the left and TXS on the right. I've never seen either of them in the same room at the same time, so there is still a chance they are the same entity. Maybe TXS is out teachng classes, writing software, or doing other academic things when he should be sleeping or maybe DJB is out having TXS' life while he should be sleeping. This could all be one big real life Fight Club scenario (without the fighting). The world may never solve the mystery that is TXS vs DJB.

If you're in the insanely small population that has not seen Fight Club or read the book, then I'm sorry I ruined it for you. Maybe you should get with the modern era if you're going to continue to read my drivel.

I dug that picture of DJB out from archive.org.

Apple's Time Machine is a great feature in their OS, and Linux has almost all of the required technology already built in to recreate it. This is a simple GUI to make it easy to use. Installation (Ubuntu/Fedora/Opensuse) To use, make sure you have the following packages installed: Ubuntu: $ sudo apt-get install python python-glade2 python-gnome2 python-sqlite3 python-gconf rsync Redhat/Fedora

The following example shows the structure of a GRUB menu file. The example installation has a Linux boot partition under /dev/sda5, a root partition under /dev/sda7, and a Windows installation under /dev/sda1.  gfxmenu (hd0,4)/boot/message color white/blue black/light-gray default 0 timeout 8 title linux root (hd0,4) kernel /boot/vmlinuz root=/dev/sda7 vga=791 resume=/dev/

Malware (for "malicious software") is any program or file that is harmful to a computer user. Thus, malware includes computer viruses, worms, Trojan horses, and also spyware, programming that gathers information about a computer user without permission. Ignoring the threat of malware is one of the most reckless things you can do in today's increasingly hostile computing environment. Malware is

I read Gunter Ollmann's post in the IBM ISS blog with interest today. Gunter is "Director Security Strategy, IBM Internet Security Systems," so he is undoubtedly pro-outsourcing. Here is his argument:

[S]ecurity doesn’t come cheap. While individual security technologies get cheaper as they commoditize, the constant influx of new threats drives the need for new classes of protection and new locations to deploy them...

If you were to examine a typical organizations IT security budget, you’d probably see that the majority of spend isn’t in new appliances or software license renewals, instead it’ll lie in the departments staffing costs...

This is at odds with the way most organizations normally deal with specialized and professional skill requirements... Just about every organization I deal with (including some of the biggest international companies) relies upon external agencies to provide these specialist services and consultancy – as and when required – it’s more cost effective that way.

With that in mind, why are organizations building up their own highly-trained (and expensive) specialist internal security teams? Granted, some of the security technologies being deployed by organizations are relatively complex, but do they really require a Masters degree and CISSP certified experts to babysit them full-time...

Nowadays you can tap in an incredibly broad range of expertise – ranging from hard-core security researchers capable of helping you evaluate the security of new products you’re thinking of buying and deploying throughout your enterprise, through to 24x7 security sentinels; so knowledgeable about the security product you’ve deployed that they’re capable of guaranteeing protection with money-back SLA’s...

Organizations should take a closer look at their security budgets and evaluate whether they’re getting the right value out of their internal teams and whether their skills investment meets the daily need of the business. (emphasis added)

By highlighting the focus on "security products," you can probably predict my response to Gunter's post. Sure, you can get hire experts that may (or may not) be cheaper than internal staff, and they may be smarter in individual products or even defensive tactics, but they are poor with respect to the most critical aspect of modern security: business knowledge. It does not matter if you are the world's greatest packet monkey if you 1) don't know what matters to a business; 2) don't know business systems; 3) don't know what is normal for a business... do I need to continue?

This is the biggest challenge I see for consultants, having been one and having hired them. It's easier to hire a consultant to help configure a security product than it is to figure out if that product is even needed, which to buy, how to get approval and business buy-in, how to support it operationally, and a dozen other decisions.

I agree that certain specialized tasks merit outside support. That list changes from organization to organization. However, beware arguments like Gunter's.

A sestina is a poetic form. It consists of six-line stanzas, with each stanza's lines ending in the same six words, in a different order for each stanza. Then there is a final stanza, called the envoi, in which each line contains two of the six words.

You can see examples of sestinas here, or provide your own six words to see what form comes out.

It is incredibly hard to write a sestina that doesn't sound forced, and hardly anybody ever manages it. A really good sestina, when read aloud, is not immediately identifiable as a sestina. It just sounds like there's a rhythm in there, but you can't quite place it until you read it that third or fourth time, and see it on a page.

Most sestinas, however, work for the first stanza, and possibly the second, but after that you feel that the author is just saying any old nonsense just to stay in the form.

Sestinas work best when they are about a repetitive topic. Examples might be a child's game, or an addiction, or a daily event. So I thought that the latest topic on Inspire Me Thursday - Breath - would be ideal for it. Unfortunately, so far, it just sounds like, after the first stanza, I'm merely babbling to fit the form.

I've had a really hard time writing lately. Everything feels forced, both fiction, poetry, and nonfiction. I keep hoping that if I force it long enough, it'll start to flow. But the pump refuses to be primed.

I read a name I hadn't seen in years today when I read Kim Zetter's story Israeli Hacker Known as "The Analyzer" Suspected of Hacking Again:

Canadian authorities have announced the arrest of a 29-year-old Israeli named Ehud Tenenbaum whom they believe is the notorious hacker known as "The Analyzer" who, as a teenager in 1998, hacked into unclassified computer systems belonging to NASA, the Pentagon, the Israeli parliament and others.

Tenenbaum and three Canadians were arrested for allegedly hacking the computer system of a Calgary-based financial services company and inflating the value on several pre-paid debit card accounts before withdrawing about CDN $1.8 million (about U.S. $1.7 million) from ATMs in Canada and other countries. The arrests followed a months-long investigation by Canadian police and the U.S. Secret Service.

The Analyzer was the "mastermind" behind Solar Sunrise, one of the original "so easy a Caveman could do it" intrusions -- back in 1998. Solar Sunrise was huge and it was one of several very rude awakenings I remember while serving in the Air Force that decade.

Seeing The Analyzer back in law enforcement custody reminds me of the post I made about Max Ray Butler and somewhat of my post Intruders Selling Security Software. It's all about trust.

Grand Opening is occurring this weekend.  Took a tour yesterday, looks modern, clean, and should attract more businesses into downtown.

The facilities looks to be about a quarter the size of the combined Moscone Center (i.e., North, South, and West combined).  I’d wager that the weak link in attracting major (international or even national) conferences is the airport, as Raleigh is neither a major destination nor a national hub.

One unique feature is the shimmer wall which is fun to look both at night and day.

Brian Trammell and Bill Yurcik were kind enough to ask me to deliver the keynote at the 1st ACM Workshop on Network Data Anonymization (NDA 2008). The one day event takes place 31 October 2008 at George Mason University in northern VA. My talk will discuss the trials and tribulations of OpenPacket.org, and changes planned for the project.

In Part 1 we discussed core analysis in general and some basic mdb commands for high level investigation. When you dig deeper things can get confusing and complex because everything is referenced by address. This is where the Solaris Crash Analysis Tool comes in.

Solaris CAT has been around for a long time, but only as of version 5.0 released on June 18th of this year has it been available for Solaris X86/X64. You can find the Solaris CAT 5.0 Release Notes here.

To get started, download CAT 5.0, uncompress and install the package:

# bunzip2 SUNWscat5.0-GA-i386.pkg.bz2
# pkgadd -G -d ./SUNWscat5.0-GA-i386.pkg 

The following packages are available:
  1  SUNWscat     Solaris Crash Analysis Tool (5.0 GA SV4622M)
                  (i386) 5.0

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: 1

Processing package instance  from 

Solaris Crash Analysis Tool (5.0 GA SV4622M)(i386) 5.0
...

The package will, by default, install into /opt/SUNWscat. There are two binaries we're really interested in, found in the bin/ directory: scat and blast. The scat tool is the CLI interface to Solaris CAT and provides a shell which is a human friendly re-implementation of mdb (no "::" prefixing commands, etc.) The blast tool is a really nice Java GUI interface to the CLI which adds a lot of "just click here" functionality and is excellent for testing and playing around. I highly recommend you point your browser at /opt/SUNWscat/docs/index.html, which includes some minimal but extremely useful HTML documentation.

Authors note: I'm resisting a "scat" joke with amazing strength. Seriously... resisting.... so.... hard....

We'll focus on the CLI here. Invocation is a little unusual; add /opt/SUNWscat/bin to your path and then change to the directory containing your dumps (usual /var/crash/hostname/), for the .0 dumps use "scat 0", for the .1 dumps use "scat 1", and so on. You'll fine the "online help" within the CLI exceptional, lets look:

# export PATH=$PATH:/opt/SUNWscat/bin
# cd /var/crash/ev2-r01-s10/
# ls -l
total 14205330
-rw-r--r--   1 root     root           2 Aug 25 07:49 bounds
-rw-r--r--   1 root     root     1444762 Aug 25 07:43 unix.0
-rw-r--r--   1 root     root     7268106240 Aug 25 07:49 vmcore.0
# scat 0

  Solaris[TM] CAT 5.0 for Solaris 11 64-bit x86
    SV4622M, Jul  3 2008

  Copyright © 2008 Sun Microsystems, Inc. All rights reserved.
  Use is subject to license terms.

  Feedback regarding the tool should be sent to SolarisCAT_Feedback@Sun.COM
  Visit the Solaris CAT blog at http://blogs.sun.com/SolarisCAT

opening unix.0 vmcore.0 ...dumphdr...symtab...core...done
loading core data: modules...symbols...ctftype: unknown type struct panic_trap_info
CTF...done

core file:      /var/crash/xxxxxxxx/vmcore.0
user:           Super-User (root:0)
release:        5.11 (64-bit)
version:        snv_67
machine:        i86pc
node name:      xxxxxxxxxxxxxxxxxx
system type:    i86pc
hostid:         xxxxxxxx
dump_conflags:  0x10000 (DUMP_KERNEL) on /dev/dsk/c0t0d0s1(24.0G)
time of crash:  Mon Aug 25 07:41:00 GMT 2008 (core is 13 days old)
age of system:  91 days 22 hours 49 minutes 50.97 seconds
panic CPU:      1 (8 CPUs, 31.9G memory)
panic string:   page_free pp=ffffff0007243bd8, pfn=11228e, lckcnt=0, cowcnt=0 slckcnt = 0

sanity checks: settings...vmem...
WARNING: FSS thread 0xffffff097d1e3400 on CPU2 using 99%CPU
WARNING: FSS thread 0xffffff09fddbab40 on CPU3 using 99%CPU
sysent...clock...misc...
NOTE: system has 54 non-global zones
done
SolarisCAT(vmcore.0/11X)> 

When CAT is unleashed on a dump several "sanity checks" are run which can point out glaring known issues. There is an HTML document in the docs/ directory which outlines all the various sanity checks. These checks alone make CAT a must-have tool! Sanity check output will come in two varieties, "WARNING" which indicates something out of whack that may have been the cause or contributor to the crash, and "NOTE" which is unlikely the cause but of interest. We can see in the example above two warnings telling me that 2 threads were consuming 99% of a CPU... thats handy! It also notes that I'm running 54 zones.

The available commands a broken down into categories which you can see using the "help" command. The first group are for "Initial Investigation:" and include: analyze, coreinfo, msgbuf, panic, stack, stat, and toolinfo. Lets look at the "analyze" commands output:

SolarisCAT(vmcore.0/11X)> analyze

core file:      /var/crash/xxxxxx/vmcore.0
user:           Super-User (root:0)
release:        5.11 (64-bit)
version:        snv_67
machine:        i86pc
node name:      xxxxxx
system type:    i86pc
hostid:         xxxxx
dump_conflags:  0x10000 (DUMP_KERNEL) on /dev/dsk/c0t0d0s1(24.0G)
time of crash:  Mon Aug 25 07:41:00 GMT 2008 (core is 13 days old)
age of system:  91 days 22 hours 49 minutes 50.97 seconds
panic CPU:      1 (8 CPUs, 31.9G memory)
panic string:   page_free pp=ffffff0007243bd8, pfn=11228e, lckcnt=0, cowcnt=0 slckcnt = 0


==== panic thread: 0xfffffffef4ce5dc0 ==== CPU: 1 ====
==== panic user (LWP_SYS) thread: 0xfffffffef4ce5dc0  PID: 10156  on CPU: 1 ====
cmd: /opt/local/sbin/httpd -k start
t_procp: 0xffffffff06595e50
  p_as: 0xffffffff093490e0  size: 47374336  RSS: 3125248
  hat: 0xffffffff092a9480  cpuset: 1
  zone: address translation failed for zone_name addr: 8 bytes @ 0x3

t_stk: 0xffffff00486bcf10  sp: 0xffffff00486bc880  t_stkbase: 0xffffff00486b8000
t_pri: 3(FSS)  pctcpu: 0.380035
t_lwp: 0xfffffffefe61ab60  lwp_regs: 0xffffff00486bcf10
  mstate: LMS_SYSTEM  ms_prev: LMS_SYSTEM
  ms_state_start: 2 minutes 31.229022230 seconds earlier
  ms_start: 2 minutes 31.343582414 seconds earlier
psrset: 0  last CPU: 1  
idle: 0 ticks (0 seconds)
start: Mon Aug 25 07:41:00 2008
age: 0 seconds (0 seconds)
syscall: #131 memcntl(, 0x0) ()
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_PANIC - thread initiated a system panic
        T_DFLTSTK - stack is default size
tpflg:  TP_MSACCT - collect micro-state accounting information
tsched: TS_LOAD - thread is in memory
        TS_DONT_SWAP - thread/LWP should not be swapped
        TS_RUNQMATCH
pflag:  SMSACCT - process is keeping micro-state accounting
        SMSFORK - child inherits micro-state accounting

pc:      unix:vpanic_common+0x13b:  addq   $0xf0,%rsp

unix:vpanic_common+0x13b()
unix:panic+0x9c()
unix:page_free+0x22e()
unix:page_destroy+0x100()
genunix:fs_dispose+0x2e()
genunix:fop_dispose+0xdc()
genunix:pvn_getdirty+0x1f0()
zfs:zfs_putpage+0x129()
genunix:fop_putpage+0x65()
genunix:segvn_sync+0x39f()
genunix:as_ctl+0x1f2()
genunix:memcntl+0x709()
unix:_syscall32_save+0xbf()
-- switch to user thread's user stack --

This output provides a vast array of useful details, including:

• System summary, including OS release and version, architecture, hostname, and hostid; as well as number of CPU's and memory
• Time of crash and previous uptime ("age of system")
• The panic string and CPU that it occurred on
• The thread that caused the panic and its details, including the command (argc &argv), its memory footprint (size & rss), and zone
• The threads state information, run time, start time, current syscall
• The call stack

As noted in Part 1, what most people are really looking for when doing core analysis is to determine which application was responsable, and this output provides that data in great clarity. Lets dig into it a bit more explicitly... based on the above "analyze" output we can see that....

• The system is an 8CPU X86 box running snv_67 (Solaris Nevada Build 67) in 64bit mode with 32GB of RAM.
• System crashed on Aug 25th at 7:41AM GMT, it was previously up for 91 days
• System paniced on "page_free" call, on CPU 1
• The running thread was "httpd -k start"... an Apache worker process.
• The process had the PID 10156, consumed 3.1MB of Physical Memory (RSS) and had a virtual size of 47MB
• The process was using less than 1% (pctcpu) of CPU 1, was using the Fair Share Scheduler (FSS), on Processor Set (psrset) 0.
• The process started on Aug 25th at 7:41AM GMT, it was 0 seconds old when it crashed... possibly a forked worker gone bad.

For many administrators this might be as much as you wanted to know, right there. But lets look at a couple more commands.

You'll recall that during the sanity checks at startup it noted 2 threads consuming full CPU's. We can feed the thread address to the "thread" command to get details on them:

SolarisCAT(vmcore.0/11X)> thread 0xffffff097d1e3400
==== user (LWP_SYS) thread: 0xffffff097d1e3400  PID: 27446  on CPU: 2 ====
cmd: nano svn-commit.tmp
t_procp: 0xffffffff2e908ab0
  p_as: 0xffffffff10402ee0  size: 2772992  RSS: 1642496
  hat: 0xffffffff102f6b48  cpuset: 2
  zone: address translation failed for zone_name addr: 8 bytes @ 0x2

t_stk: 0xffffff004e47ef10  sp: 0xffffff003d3fcf08  t_stkbase: 0xffffff004e47a000
t_pri: 26(FSS)  pctcpu: 99.306175
t_lwp: 0xffffffff202a78b0  lwp_regs: 0xffffff004e47ef10
  mstate: LMS_SYSTEM  ms_prev: LMS_USER
  ms_state_start: 2 minutes 31.228983791 seconds earlier
  ms_start: 39 days 19 hours 11 minutes 8.989252296 seconds earlier
psrset: 0  last CPU: 2  
idle: 9 ticks (0.09 seconds)
start: Wed Jul 16 12:30:07 2008
age: 3438653 seconds (39 days 19 hours 10 minutes 53 seconds)
syscall: #98 sigaction(, 0x0) ()
tstate: TS_ONPROC - thread is being run on a processor
tflg:   T_DFLTSTK - stack is default size
tpflg:  TP_TWAIT - wait to be freed by lwp_wait
        TP_MSACCT - collect micro-state accounting information
tsched: TS_LOAD - thread is in memory
        TS_DONT_SWAP - thread/LWP should not be swapped
        TS_RUNQMATCH
pflag:  SMSACCT - process is keeping micro-state accounting
        SMSFORK - child inherits micro-state accounting

pc:      unix:panic_idle+0x23:  jmp    -0x2     (unix:panic_idle+0x23)

unix:panic_idle+0x23()
0xffffff003d3fcf60()
-- error reading next frame @ 0x0 --

So using the "thread" command we can get full granularity on a given thread. In fact, using the "tlist" command you can dump this information for every thread on the system at the time of crash.

Another nifty command is "tunables". This will display the "current value" (at time of the dump) and the default value. If someone's been experimenting on the production systems this will clue you in.

SolarisCAT(vmcore.0/11X)> tunables   
    Tunable Name     Current   Default Value  Units      Description
                     Value                               
    physmem          8386375   *              pages      Physical memory 
                                                         installed in system.
    freemem          376628    *              pages      Available memory.
    avefree          338943    *              pages      Average free memory 
                                                         in the last 30 seconds
.........

Using the "dispq" command we can look at the dispatch queues (run queue). This answers "what other processes were running on CPU at the time of the crash", again, using the thread address we can dig into them with "thread":

SolarisCAT(vmcore.0/11X)> dispq
      CPU                  thread               pri        PID cmd
  0 @ 0xfffffffffbc26bb0   0xffffff003d005c80    -1            (idle)
               pri  60 -=> 0xffffff004337dc80    60          0 sched
  1 @ 0xfffffffec6634000 P 0xfffffffef4ce5dc0 P   3      10156 /opt/local/sbin/httpd -k start
  2 @ 0xfffffffec662f000   0xffffff097d1e3400    26      27446 nano svn-commit.tmp
  3 @ 0xfffffffec66f4800   0xffffff09fddbab40    25      21329 java -jar xxxxx.jar --ui=console
  4 @ 0xfffffffec66ea800   0xffffff003d414c80    -1            (idle)
               pri  60 -=> 0xffffff0048b12c80    60          0 sched
  5 @ 0xfffffffec6770800   0xffffff003d4b0c80    -1            (idle)
  6 @ 0xfffffffec6770000   0xffffff003d53bc80    -1            (idle)
  7 @ 0xfffffffec6762000   0xffffff003d58fc80    -1            (idle)

      part                 thread               pri        PID cmd
  0 @ 0xfffffffffbc4eef0

There are far too many to go through in a blog entry... but lets look at my personal favorite, "zfs". The "zfs" command can show us the pool(s), their configuration, read/write/checksum/error stats, and even ARC stats!

SolarisCAT(vmcore.0/11X)> zfs -e
ZFS spa @ 0xfffffffec6c21540
    Pool name: zones
    State: ACTIVE
       VDEV Address      State    Aux   Description
    0xfffffffec0a9e040  FAULTED    -       root

            READ   WRITE   FREE   CLAIM   IOCTL  
    OPS        0      0     0      0      0 
    BYTES      0      0     0      0      0 

    EREAD       0
    EWRITE      0
    ECKSUM      0

            VDEV Address      State    Aux     Description
         0xfffffffec0a9eac0  FAULTED    -    /dev/dsk/c0t1d0s0

                  READ      WRITE     FREE   CLAIM   IOCTL  
         OPS     74356305  578263155     0      0      0 
         BYTES       757G      10.4T     0      0      0 

         EREAD       0
         EWRITE      0
         ECKSUM      0
SolarisCAT(vmcore.0/11X)> zfs arc

ARC (Adaptive Replacement Cache) Stats:

    hits                       77708247444
    misses                         1930348
    demand_data_hits           74303514929
    demand_data_misses             1325511
    demand_metadata_hits         620388795
    demand_metadata_misses          160708
    prefetch_data_hits          1361651307
....

I hope this helps you get an idea of how easy it is to really dig deeply into your core dumps using Solaris CAT to hide the oddities of mdb from you. Its a powerful and robust tool, and I'm glad that we have it.

Happy dump divin'! You'll be amazed how much you'll learn about your system.

A friend of mine is working on digital defense strategies at work. He is interested in your commentary and any relevant experiences you can share. He is moving from a "deny bad, allow everything else" policy to an "allow good, deny everything else" policy.

By policy I mean a general approach to most if not all defensive strategies. On the network, define which machines should communicate, and deny everything else. On the host, define what applications should run, and deny everything else. In the browser, define what sites can be visited, and deny everything else. That's the central concept, although expansions are welcome.

My friend would like to know if anyone in industry is already following this strategy, and to what degree. If you can name your organization all the better (even if privately to me, or to him once the appropriate introductions are made). Thank you.

Rob Lee was kind enough to ask me to deliver the keynote on the second day of the SANS WhatWorks in Incident Response and Forensic Solutions Summit. The two-day event takes place 13-14 October 2008 at Caesars Palace in Las Vegas, NV. The conference agenda looks great, with training classes available before and after the summit. The tuition fee is $1,595 if paid by 10 Sep or $1,845 thereafter. I am very much looking forward to attending this event.

Rob also pointed out the new SANS Computer Forensics and E-discovery Community and SANS Forensics Blog.

Solaris is one of the most stable operating systems available... but lets face it, stuff happens. Solaris does panic, but I want everyone to be clear, a "panic", despite the seemingly contradictory name, is by its nature a controlled event. When the kernel encounters behavior that is uncorrectable and will cause irreparable harm to the running system or, even worse, corrupt data, the system will voluntarily tap out using the panic system call to get the system down quickly, hopefully leaving a core dump in its wake for post-mortem analysis.

In this blog entry we'll discuss core dumps and panic's in general. In part 2 we'll discuss a tool to make life just a little easier, the Solaris Crash Analysis Tool, or "Solaris CAT".

I want to point out that post-mortem core analysis is really the task of a kernel engineer. The fact is, way less than 1% of us who ever engage in core analysis are actually going to have any real idea of what the hell we're doing. And thats ok! I guarantee that you'll post something from an analysis to a mailing list and you'll get some asshole who forgets that he's been paid to work on the Solaris kernel for the last 20 years while you work a job which is now on hold because of said core dump, with replies like "We can clearly see that due to the memory address in this register that you are a moron...." The point here is, if you don't know what your doing, don't be discouraged. What we, mere mortals, are trying to do is not necessarily solve the problem but provide clues which will help us guide our search, either by posting a stack trace to a mailing list, or send the dump to Sun Support, or to take a panic string and search the bug database or Google for. The cuddletech rule of crashes is:

An unexpected crash is unacceptable; An unexplained crash is inexcusable.

If you're reading this you've probably lived through a panic before, but lets recap. The best explanation of a "crash" event and resulting dump can be found in the dumpadm(1M) man page:

     A crash dump is a disk copy of the physical memory
     of  the computer at the time of a fatal system error. When a
     fatal operating system error occurs,  a  message  describing
     the  error  is  printed to the console. The operating system
     then generates a crash dump by writing the contents of  phy-
     sical  memory to a predetermined dump device, which is typi-
     cally a local disk partition. The dump device can be config-
     ured  by way of dumpadm.  Once the crash dump has been writ-
     ten to the dump device, the system will reboot.

     Fatal operating system errors can be caused by bugs  in  the
     operating system, its associated device drivers and loadable
     modules, or by faulty  hardware.  Whatever  the  cause,  the
     crash  dump  itself  provides invaluable information to your
     support engineer to aid in diagnosing the problem. As  such,
     it  is  vital  that the crash dump be retrieved and given to
     your support provider. Following an operating system  crash,
     the  savecore(1M)  utility  is executed automatically during
     boot to retrieve the crash dump from the  dump  device,  and
     write it to a pair of files in your file system named unix.X
     and vmcore.X, where X is an integer  identifying  the  dump.
     Together,  these  data  files form the saved crash dump. The
     directory in which the crash dump is  saved  on  reboot  can
     also be configured using dumpadm.

I encourage you to read both the savecore(1M) and dumpadm(1M) man pages. You'll find that with savecore -L you can create a dump of a live system, so if you don't have a crashed system around to play with, use that. Alternatively, you can use reboot -d to dump a core and reboot.

At this point we'll assume you have a dump available. By default you'll find them in /var/crash/hostname/, you'll have dumps in pairs: vmcore.0 and unix.0. We feed these two files to mdb, the (-k, kernel) Modular DeBugger, to preform our analysis like so:

# mdb -k unix.0 vmcore.0 
Loading modules: [ unix krtld genunix specfs dtrace cpu.AuthenticAMD.15 uppc pcplusmp ufs ip sctp usba lofs zfs random ipc md fcip fctl fcp crypto logindmux ptm nfs ]
>

You are now free to move about the dump. mdb commands are strange and unusual at first, it takes a lot of time to get comfortable with it, but there are a couple of debugger commands that can give us the essence of what we need. Lets walk through them.

The ::status command will display high level information regarding this debugging session. Of usefulness here is the dumps "panic message" and OS release.

> ::status
debugging crash dump vmcore.0 (64-bit) from hostname
operating system: 5.11 snv_43 (i86pc)
panic message: BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference
dump content: kernel pages only

The ::stack command will prove you with a stack trace, this is the same thing trace you would have seen in syslog or the console.

> ::stack
atomic_add_32()
nfs_async_inactive+0x55(fffffe820d128b80, 0, ffffffffeff0ebcb)
nfs3_inactive+0x38b(fffffe820d128b80, 0)
fop_inactive+0x93(fffffe820d128b80, 0)
vn_rele+0x66(fffffe820d128b80)
snf_smap_desbfree+0x78(fffffe8185e2ff60)
dblk_lastfree_desb+0x25(fffffe817a30f8c0, ffffffffac1d7cc0)
dblk_decref+0x6b(fffffe817a30f8c0, ffffffffac1d7cc0)
freeb+0x89(fffffe817a30f8c0)
tcp_rput_data+0x215f(ffffffffb4af7140, fffffe812085d780, ffffffff993c3c00)
squeue_enter_chain+0x129(ffffffff993c3c00, fffffe812085d780, fffffe812085d780, 1, 1)
ip_input+0x810(ffffffffa23eec68, ffffffffaeab8040, fffffe812085d780, e)
i_dls_link_ether_rx_promisc+0x266(ffffffff9a4c35f8, ffffffffaeab8040, fffffe812085d780)
mac_rx+0x7a(ffffffffa2345c40, ffffffffaeab8040, fffffe812085d780)
e1000g_intr+0xf6(ffffffff9a4b2000)
av_dispatch_autovect+0x83(1a)
intr_thread+0x50()

The ::msgbuf command will output the message buffer at the time of crash; the message buffer is most commonly used by sysadmins through the "dmesg" command.

> ::msgbuf
MESSAGE                                                               
....
WARNING: IP: Hardware address '00:14:4f:xxxxxxx' trying to be our address xxxx
WARNING: IP: Hardware address '00:14:4f:xxxx' trying to be our address xxxx

panic[cpu0]/thread=fffffe80000adc80: 
BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference

sched: 
#pf Page fault
Bad kernel fault at addr=0x0
.... blah blah, snipped for brevity.

The ::panicinfo command will give you lots of fun cryptic counter information, of most interest is the first 3 lines, which contain the CPU on which the panic occured, the running thread, and the panic message. You'll notice these are commonly repeated and the most useful pieces of information.

> ::panicinfo
             cpu                0
          thread fffffe80000adc80
         message BAD TRAP: type=e (#pf Page fault) rp=fffffe80000ad3d0 addr=0 occurred in module "unix" due to a NULL pointer dereference
             rdi                0
             rsi                1
             rdx fffffe80000adc80
             rcx                0
              r8                0
              r9 fffffe80dba125c0
             rax                0
             rbx fffffe8153a36040
             rbp fffffe80000ad4e0
             r10              3e0
             r10              3e0
             r11 ffffffffaeab8040
             r12 ffffffffb7b4cac0
             r13                0
             r14 fffffe820d128b80
             r15 ffffffffeff0ebcb
          fsbase ffffffff80000000
          gsbase fffffffffbc27850
              ds               43
              es               43
              fs                0
              gs              1c3
          trapno                e
             err                2
             rip fffffffffb838680
              cs               28
          rflags            10246
             rsp fffffe80000ad4c8
              ss                0
          gdt_hi                0
          gdt_lo         defacedd
          idt_hi                0
          idt_lo         80300fff
             ldt                0
            task               60
             cr0         80050033
             cr2                0
             cr3        10821b000

In my opinion, the koolest command is ::cpuinfo -v. Truth be told, if you run multiple applications on a server the most common question people (especially managers) want answered is "which application did it?", being translated into geek-esse "who do I blame?" This command will help you determine that by displaying, complete with beautiful ASCII art, the threads and process names running on each CPU (NRUN). In the following example, we know the event occured on CPU 0, thus thats the one we want to look at. Note that the "sched" process should be interpreted as "kernel".

>  ::cpuinfo -v
 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  0 fffffffffbc2f370  1b    1    0 165   no    no t-1    fffffe80000adc80 sched
                       |    |    |
            RUNNING --+    |    +--> PIL THREAD
              READY         |           6 fffffe80000adc80
             EXISTS         |           - fffffe80daab6a20 ruby
             ENABLE         |
                            +-->  PRI THREAD           PROC
                                   99 fffffe8000b88c80 sched

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  1 ffffffff983b3800  1f    1    0  59  yes    no t-0    fffffe80daac2f20 smtpd
                       |    |
            RUNNING --+    +-->  PRI THREAD           PROC
              READY                99 fffffe8000bacc80 sched
           QUIESCED
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  2 ffffffff9967a800  1f    2    0  -1   no    no t-0    fffffe8000443c80
 (idle)
                       |    |
            RUNNING --+    +-->  PRI THREAD           PROC
              READY                99 fffffe8000b82c80 sched
           QUIESCED                60 fffffe80018f8c80 sched
             EXISTS
             ENABLE

 ID ADDR             FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD           PROC
  3 ffffffff9967a000  1f    1    0  -1   no    no t-0    fffffe8000535c80
 (idle)
                       |    |
            RUNNING --+    +-->  PRI THREAD           PROC
              READY                60 fffffe8000335c80 zsched
           QUIESCED
             EXISTS
             ENABLE

The ::ps command allows us to see all running processes. Several flags are supported, including -z to display Zone ID's.

> ::ps -z
S    PID   PPID   PGID    SID  ZONE    UID      FLAGS             ADDR NAME
R      0      0      0      0     0      0 0x00000001 fffffffffbc25900 sched
R      3      0      0      0     0      0 0x00020001 ffffffff9970d928 fsflush
R      2      0      0      0     0      0 0x00020001 ffffffff9970e558 pageout
R      1      0      0      0     0      0 0x42004000 ffffffff9970f188 init
R  20534      1  20533  20533    24   1006 0x42010400 ffffffffb246f9b8 ruby
R  20532      1  20531  20531    24   1006 0x42010400 fffffe8109674308 ruby
R  20529      1  20528  20528    24   1006 0x42010400 fffffe80dc5602f0 ruby
...

We can use ::pgrep to search for processes and use the appropriate address for further digging. In the following example I'll find a Java process and then determine which zone that process was running in:

> ::pgrep java
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R   3628      1   3620   3574      0 0x42004400 fffffe80deeb3240 java
> fffffe80deeb3240::print proc_t p_zone->zone_name
p_zone->zone_name = 0xffffffffae0cef00 "testzone03"

There are many more tools and way to dig into your dumps using mdb. It can be confusing because you need to reference things by address, but you get more comfortable with it as you play around. If you are interested in learning more I highly recommend reading Eric Lowe's "Examining the Anatomy of a Process", which digs into the topic of process examination via mdb.

One thing you'll notice in all this is that the messages at the time of crash on the console or in syslog contain almost everything you need to know without digging too deeply. Therefore, assuming you have those messages, the most useful thing most people will extract from the core files is the output of the ::cpuinfo command to see what process was on the offending CPU at the time of the crash. Knowing what processes, zones, etc, were running at the time of crash are interesting but rarely mean much if they weren't directly involved in the panic.

As I said, once you start getting into referencing memory addresses to deepen your analysis things get sticky and tricky very very quickly... thats where Solaris CAT comes in, which we'll talk about in part 2.

Here.

via techcrunchit

Relational databases that speak SQL are the data-storage backbone for most developers. Unfortunately, but most of the data that's created outside the control of the technology caste at a typical workplace is in Excel format. Because of this, being able to procedurally read and write Excel documents with a familiar language can open up a whole world of possibilities for automation and data migration.

Assuming you're attempting to read and write standard text (Ie. not binary/graphic) data from Excel worksheets, this is actually fairly doable in PHP and Perl.

A recent article by Mike Diehl at Linux Journal peaked my interest in this. He shows off some of the features of the Spreadsheet::ParseExcel Perl module, which can be used to pull data and even formatting information from cells in an Excel worksheet. Once you have your hands on the data, you can do what you want with it: output it to XML, toss it in a database for subsequent querying, or even convert it into other Excel documents (oh, the shame).

Perl Excel Libraries and Information
Spreadsheet:ParseExcel - Read from Excel 95/97/2000 documents
Spreadsheet:WriteExcel - Write to Excel 97/2000/2002/2003 documents
Linux Journal - Reading Native Excel Files in Perl

There are libraries for dealing with native Excel files in PHP as well. The following two seem to be the only options for binary Excel documents.

PHP Excel Libraries
PHP Excel_Reader - Read Excel 95 and 97 documents
Spreadsheet_Excel_Writer - Write Excel 5.0 documents
Reading and Writing Spreadsheets with PHP

With the most recent version of Excel, there is an XML file format option that will allow you to read and write data in a worksheet by directly interacting with the saved file's DOM. IBM has a document that details doing this with PHP, and it would be straightforward to apply this technique to Perl as well.

Read/Write XML Excel Data in PHP

Finally, if all you need to do is output a document that can be read in Excel, a standard CSV-format file will usually do the trick. Escaping can be a bit tricky, however, and my preferred format has become a plain-old HTML table. Just create a file that contains a TABLE element (no BODY or HTML tags necessary), with any number of TR rows and html-escaped data in the TDs, and save it out. If you use the XLS file extension, it will open directly in Excel with a double-click and Excel never seems to mind reading in the data.

Do you have any other Excel programming hacks? Give us a shout in the comments.

The following addons and utilities will add a whole bunch of extra features to iWeb, allowing you to create even more robust web sites.

1. iTweak
2. Easy iWeb Publisher
3. iWebSites

iTweak

iTweak adds boasts that it is the “ultimate companion for iWeb”. I’m not sure about that, but it sure does add a lot of features. The highlights -

iTweak can back up both iWeb 06 and iWeb 08 sites in a snap

It can add a Favicon via a simple drag and drop method

You can integrate a Google Search bar into and for your site

You can add Google Analytics, Statcounter and any other hitcounter in the world

Add a powerful PHP contact form

Search & Replace the HTML that iWeb publishes

iTweak can password-protect your site

iTweak is donationware - so if you find it super-useful, donate some money to the author.

click to enlarge

Easy iWeb Publisher

Easy iWeb Publisher is very similar to the previously covered w2w utility. It allows you to upload your iWeb created site to a non-.Mac account. After configuring it to use your Web Hosting account details, you can just drag and drop folders onto the Dock icon and Easy iWeb Publisher takes care of the rest.

click to enlarge

iWebSites

iWebSites only has one feature, but it’s a really useful one. It allows you to create multiple websites that are completely independent of each other (it does this by manipulating the website files that iWeb creates).

---
Related Articles at Simple Help:

• How to upload a website built with iWeb to a non .mac/MobileMe account
• Update: Gmail increases storage to 4.2GB