Planet Sysadmin               

          blogs for sysadmins, chosen by sysadmins...
(Click here for multi-language)

October 24, 2014

Chris Siebenmann

In Go I've given up and I'm now using standard packages

In my Go programming, I've come around to an attitude that I'll summarize as 'there's no point in fighting city hall'. What this means is that I'm now consciously using standard packages that I don't particularly like just because they are the standard packages.

I'm on record as disliking the standard flag package, for example, and while I still believe in my reasons for this I've decided that it's simply not worth going out of my way over it. The flag package works and it's there. Similarly, I don't think that the log package is necessarily a great solution for emitting messages from Unix style command line utilities but in my latest Go program I used it anyways. It was there and it wasn't worth the effort to code warn() and die() functions and so on.

Besides, using flag and log is standard Go practice so it's going to be both familiar to and expected by anyone who might look at my code someday. There's a definite social benefit to doing things the standard way for anything that I put out in public, much like most everyone uses gofmt on their code.

In theory I could find and use some alternate getopt package (these days the go to place to find one would be godoc.org). In practice I find using external packages too much of a hassle unless I really need them. This is an odd thing to say about Go, considering that it makes them so easy and accessible, but depending on external packages comes with a whole set of hassles and concerns right now. I've seen a bit too much breakage to want that headache without a good reason.

(This may not be a rational view for Go programming, given that Go deliberately makes using people's packages so easy. Perhaps I should throw myself into using lots of packages just to get acclimatized to it. And in practice I suspect most packages don't break or vanish.)

PS: note that this is different from the people who say you should eg use the testing package for your testing because you don't really need anything more than what it provides and stick with the standard library's HTTP stuff rather than getting a framework. As mentioned, I still think that flag is not the right answer; it's just not wrong enough to be worth fighting city hall over.

Sidebar: Doing standard Unix error and warning messages with log

Here's what I do:

log.SetPrefix("<progname>: ")
log.SetFlags(0)

If I was doing this better I would derive the program name from os.Args[0] instead of hard-coding it, but if I did that I'd have to worry about various special cases and no, I'm being lazy here.

by cks at October 24, 2014 05:16 AM

October 23, 2014

Ubuntu Geek

Eric – A Full featured Python and Ruby editor and IDE

Sponsored Link
Eric is a full featured Python and Ruby editor and IDE, written in python. It is based on the cross platform Qt gui toolkit, integrating the highly flexible Scintilla editor control. It is designed to be usable as everdays' quick and dirty editor as well as being usable as a professional project management tool integrating many advanced features Python offers the professional coder. eric4 includes a plugin system, which allows easy extension of the IDE functionality with plugins downloadable from the net.
(...)
Read the rest of Eric – A Full featured Python and Ruby editor and IDE (405 words)


© ruchi for Ubuntu Geek, 2014. | Permalink | No comment | Add to del.icio.us
Post tags: ,

Related posts

by ruchi at October 23, 2014 11:14 PM

Aaron Johnson

Day 8: The Longest Drive: Iceland

We thought going into the trip that there’d be a couple days of long driving but for the most part we were able to make a bunch of stops every day and see a bunch of things… except this day. Think the Google Maps estimate for this day was north of 4 hours so we tried to get an early start so that we could get somewhere and maybe do something in the latter half of the day. The guesthouse we stayed at didn’t have a formal breakfast but he provided tokens for us to use at a local “bakarí”, which was fantastic. We had donuts and ham and cheese croissants at Aðalbakarí and then hit the road.

Our first stop was for an easy to find geocache that was at a statue in the middle of nowhere, everyone got to stretch their legs for a bit and then we packed it in again and drove on roads like this:

at which point I must now pause and say that driving around Iceland was like watching a really long and slow but extremely beautiful nature movie. I thought a number of times that Iceland is like the island that resulted from Hawaii (volcanoes, oceans, etc.) and Alaska (glaciers, snow, fishing, etc..) and some state in the middle of the US (cows, sheep, horses, etc..) all getting together and saying “let’s make an island that has the best parts of all of what we have.”, which is to say that driving wasn’t a chore at all, except for the gravel roads in some places… OH and the precarious cliffs that we drove right on top of to get out of Siglufjörður, other than that though, amazing.

Second stop, which I can’t remember how we found (think it was the navigator) turned out to be really cool. I think she was looking for geocaches as a place for us to stop and she found this geocache at an abandoned house (had been abandoned for 70 years) that you had to drive off road to even get close to. Apparently it’s name is Svarðbæli, but you can read more about it here. We drove out the 4×4 road to about 1/2 mile away and then hoofed it on the gravel road the rest of the way. The geocache itself was a bear to find (had to enter the really old house, climb up to the second floor and then it was hidden away in the rafters, I couldn’t find it, Karen found it later) but the views and the walk were brilliant:

Everyone got to horse around a bit and get their wiggles out which was nice. Didn’t see an option to put an offer down on the house but if I was a hermit, I think I’d want to live here:

We made our picnic lunch in the back of the car, if I remember correctly this was the day that someone decided that they didn’t want a PB&J for lunch which meant that they had to wait until dinner for food. Doesn’t pay to be fussy in our family.
And then we drove:

until our next stop, a hill called Helgafell, which is called “holy mountain” and is a 227 meter high volcanic cone that, drum roll, had a geocache on top, which we found and then got to enjoy the views from the top:

A short drive later and we were at our “hotel” for the evening in another little town, this one called Stykkishólmur, in probably the smallest of all the rooms we had on the trip but we made do. We got there a bit early hoping to find something to do but there wasn’t much open at all (more stuff in the summer) so we hiked up to the top of the lighthouse:

which is called Súgandisey and tried to make sure no one fell off a cliff, then hoofed it back a mile or so into the little town center where the only place to eat that was open at 5pm was a little pizza shop called Stykkið, which ended up having GREAT pizza.

Kids went to bed, I missed a geocache on top of the lighthouse and had to go back at night with my headlamp to find it:

which wasn’t too hard to do, short of the wind and cold.
Stats:

  • Light houses: 1
  • Abandoned houses: 1
  • Geocaches: 4

by ajohnson at October 23, 2014 09:59 PM

Day 7: Húsavík, Goðafoss, Akureyri and Siglufjörður: Iceland

Already back from the trip, didn’t have time at night to do days 7, 8, 9 and 10 but I’m trying now. It’s never as good days after when you can’t remember all the little details though. Either way, Day 7 started early just like every morning on this trip since everyone was in the same room and the littlest person that lives with us just cannot keep his wiggles and sounds to himself and MUST share them with EVERYONE else in the room as soon as it turns 6am, sometimes earlier. I tried to keep him in bed and quiet for as long as possible, didn’t work all that well.

We had breakfast at our “hotel”, which was really a house converted into a “hotel”, which was like most of the places we stayed. I remember that pickled herring was on the table set out for folks if they were in the mood, I had a pickled herring plate for dinner on the first night we were in town but could never pull the trigger for breakfast.

After breakfast I took the boys out for a short walk around the harbor in Húsavík both because it a short walk and because there were a couple geocaches but mostly because it’s impossible to pack up the room while three crazy horses are trampling everything. We found two on our short walk, one an old church built out of Norwegian wood supposedly in a Swiss style although that’s been debated in the comments on the geocache log (you can see pictures there) and the other right outside the whale musuem, which we would have loved to have visited but it opened at 10am and we had plans for the day (lots of driving!). Here’s a shot of one of the boats in the harbor that we walked / skipped / ran by:

The dudes and I walked around a bit more (saw some REALLY interesting looking shops right on the waterfront where guys were working on fixing boats) and then headed back to the guesthouse to help Mommy finish the packing.

Our first stop for the day was at another amazing waterfall (Goðafoss), which also happened to be REALLY cold, so cold in fact that the mist from the waterfall was frozen on top of the sand and rocks leading up to the waterfall which made walking a bit of an adventure.

The biggest one and I went and tracked down the geocache that was at this waterfall and we took a bunch of pictures:

got some wiggles out and then packed it in for the long drive up to the northern most point that we’d hit on the trip. We needed a break on the way and so we ended up spending a bunch of the afternoon walking around the second biggest “city” in Iceland (Akureyri) where we had a great lunch at a hostel / restaurant, took pictures with a couple trolls (Karen hasn’t uploaded her pictures yet), found a couple geocaches (had to walk the dogs, err kids) and then went to this “christmas shop” (in Icelandic: Jólahúsið) which was supposed to be really great for kids , which ended up being ok for kids but not something that was going to keep their attention for longer than 15 minutes but that was long enough for us to get our Christmas ornament for the year (hi Grammie!).

Finally, we did the drive up to Siglufjörður (66° north!), which is a teensy little town right on the water that I read later is sometimes not accessible at all in the winter and it was here that we had our first bit of rain (we were very lucky the entire trip and didn’t get much rain at all even though October is supposedly the rainiest of all seasons in Iceland). We got in pretty late (6:30pm), dropped off our bags and walked down to the harbor to find some dinner:

The guy that rented us our house for the night said there were only two restaurants open in the winter, one that had homemade food and another that had fried food and pizza. Sadly the homemade food place was closed but we ended up having a great experience at Veitingastaðurinn Torgið where the hostess and waitress brought us out crayons and TOYS and generally made our dinner really fun. Food was good and we ended up having to run back up the hill to the house because it started hailing on us.

We stayed the night at the The Herringhouse, which if you ever get a chance to stay at, is very nice and has a shower that’s to die for.

If I ever get to go back, it looks like there are some really really neat hiking trails that are easily accessible from the town, would be amazing to hike up into the hills at sunset in the summer:

Stats:

  • Trolls: 2
  • Waterfalls: 1
  • Amazing showers: 1
  • Geocaches: 5

by ajohnson at October 23, 2014 09:09 PM

Chris Siebenmann

The clarity drawback of allowing comparison functions for sorting

I've written before about my unhappiness that Python 3 dropped support for using a comparison function. Well, let me take that back a bit, because I've come around to the idea that there are some real drawbacks to supporting a comparison function here. Not drawbacks in performance (which are comparatively unimportant here) but drawbacks in code clarity.

DWiki's code is sufficiently old that it uses only .sort() cmp functions simply because, well, that's what I had (or at least that's what I was used to). As a result, in two widely scattered spots in different functions its code base contains the following lines:

def func1(...):
    ....
    dl.sort(lambda x,y: cmp(y.timestamp, x.timestamp))
    ....

def func2(...):
    ....
    coms.sort(lambda x,y: cmp(x.time, y.time))
    ....

Apart from the field name, did you see the difference there? I didn't today while I was doing some modernization in DWiki's codebase and converted both of these to the '.sort(key=lambda x: x.FIELD)' form. The difference is that the first is a reverse sort, not a forward sort, because it flips x and y in the cmp().

(This code predates .sort() having a reverse= argument or at least my general awareness and use of it.)

And that's the drawback of allowing or using a sort comparison function: it's not as clear as directly saying what you mean. Small things in the comparison function can have big impacts and they're easy to overlook. By contrast, my intentions and what's going on are clearly spelled out when these things are rewritten into the modern form:

   dl.sort(key=lambda x: x.timestamp, reverse=True)
   coms.sort(key=lambda x: x.time)

Anyone, a future me included, is much less likely to miss the difference in sort order when reading (or skimming) this code.

I now feel that in practice you want to avoid using a comparison function as much as possible even if one exists for exactly this reason. Try very hard to directly say what you mean instead of hiding it inside your cmp function unless there's no way out. A direct corollary of this is that sorting interfaces should try to let you directly express as much as possible instead of forcing you to resort to tricks.

(Note that there are some cases where you must use a comparison function in some form (see especially the second comment).)

PS: I still disagree with Python 3 about removing the cmp argument entirely. It hasn't removed the ability to have custom sort functions; it's just forced you to write a lot more code to enable them and the result is probably even less efficient than before.

by cks at October 23, 2014 04:15 AM

October 22, 2014

Adrian C.

SysV init on Arch Linux, and Debian

Arch Linux distributes systemd as its init daemon, and has deprecated SysV init in June 2013. Debian is doing the same now and we see panic and terror sweep through that community, especially since this time thousands of my sysadmin colleagues are affected. But like with Arch Linux we are witnessing irrational behavior, loud protests all the way to the BSD camp and public threats of Debian forking. Yet all that is needed, and let's face it much simpler to achieve, is organizing a specialized user group interested in keeping SysV (or your alternative) usable in your favorite GNU/Linux distribution with members that support one another, exactly as I wrote back then about Arch Linux.

Unfortunately I'm not aware of any such group forming in the Arch Linux community around sysvinit, and I've been running SysV init alone as my PID 1 since then. It was not a big deal, but I don't always have time or the willpower to break my personal systems after a 60 hour work week, and the real problems are yet to come anyway - if (when) for example udev stops working without systemd PID 1. If you had a support group, and especially one with a few coding gurus among you most of the time chances are they would solve a difficult problem first, and everyone benefits. On some other occasions an enthusiastic user would solve it first, saving gurus from a lousy weekend.

For anyone else left standing at the cheapest part of the stadium, like me, maybe uselessd as a drop-in replacement is the way to go after major subsystems stop working in our favorite GNU/Linux distributions. I personally like what they reduced systemd to (inspired by suckless.org philosophy?), but chances are without support the project ends inside 2 years, and we would be back here duct taping in isolation.

by anrxc at October 22, 2014 09:28 PM

SysAdmin1138

Getting stuck in Siberia

I went on a bit of a twitter rant recently.

Good question, since that's a very different problem than the one I was ranting about. How do you deal with that?


I hate to break it to you, but if you're in the position where your manager is actively avoiding you it's all on you to fix it. There are cases where it isn't up to you, such as if there are a lot of people being avoided and it's affecting the manager's work-performance, but that's a systemic problem. No, for this case I'm talking about you are being avoided, and not your fellow direct-reports. It's personal, not systemic.

No, it's not fair. But you still have to deal with it.

You have a question to ask yourself:

Do I want to change myself to keep the job, or do I want to change my manager by getting a new job?

Because this shunning activity is done by managers who would really rather fire your ass, but can't or won't for some reason. Perhaps they don't have firing authority. Perhaps the paperwork is too much to bother with firing someone. Perhaps they're the conflict-avoidant type and pretending you don't exist is preferable to making you Very Angry by firing them.

You've been non-verbally invited to Go Away. You get to decide if that's what you want to do.

Going Away

Start job-hunting, and good riddance. They may even overlook job-hunt activities on the job, but don't push it.

Staying and Escalating

They can't/won't get rid of you, but you're still there. It's quite tempting to stick around and intimidate your way into their presence and force them to react. They're avoiding you for a reason, so hit those buttons harder. This is not the adult way to respond to the situation, but they started it.

I shouldn't have to say that, but this makes for a toxic work environment for everyone else so... don't do that.

Staying and Reforming

Perhaps the job itself is otherwise awesome-sauce, or maybe getting another job will involve moving and you're not ready for that. Time to change yourself.

Step 1: Figure out why the manager is hiding from you.
Step 2: Stop doing that.
Step 3: See if your peace-offering is accepted.

Figure out why they're hiding

This is key to the whole thing. Maybe they see you as too aggressive. Maybe you keep saying no and they hate that. Maybe you never give an unqualified answer and they want definites. Maybe you always say, 'that will never work,' to anything put before you. Maybe you talk politics in the office and they don't agree with you. Maybe you don't go paintballing on weekends. Whatever it is...

Stop doing that.

It's not always easy to know why someone is avoiding you. That whole avoidant thing makes it hard. Sometimes you can get intelligence from coworkers about what the manager has been saying when you're not around or what happens when your name comes up. Ask around, at least it'll show you're aware of the problem.

And then... stop doing whatever it is. Calm down. Say yes more often. Start qualifying answers only in your head instead of out loud. Say, "I'll see what I can do" instead of "that'll never work." Stop talking politics in the office. Go paintballing on weekends. Whatever it is, start establishing a new set of behaviors.

And wait.

Maybe they'll notice and warm up. It'll be hard, but you probably need the practice to change your habits.

See if your peace-offering is accepted

After your new leaf is turned over, it might pay off to draw their attention to it. This step definitely depends on the manager and the source of the problem, but demonstrating a new way of behaving before saying you've been behaving better can be the key to get back into the communications stream. It also hangs a hat on the fact that you noticed you were in bad graces and took effort to change.

What if it's not accepted?

Then learn to live in Siberia and work through proxies, or lump it and get another job.

by SysAdmin1138 at October 22, 2014 08:00 PM

Ubuntu Geek

Everything Sysadmin

Katherine Daniels (@beerops) interviews Tom Limoncelli

Katherine Daniels (known as @beerops on Twitter) interviewed me about the presentations I'll be doing at the upcoming Usenix LISA '14 conference. Check it out:

https://www.usenix.org/blog/interview-tom-limoncelli

Register soon! Seating in my tutorials is limited!

October 22, 2014 02:28 PM

Google Blog

An inbox that works for you

Today, we’re introducing something new. It’s called Inbox. Years in the making, Inbox is by the same people who brought you Gmail, but it’s not Gmail: it’s a completely different type of inbox, designed to focus on what really matters.

Email started simply as a way to send digital notes around the office. But fast-forward 30 years and with just the phone in your pocket, you can use email to contact virtually anyone in the world…from your best friend to the owner of that bagel shop you discovered last week.

With this evolution comes new challenges: we get more email now than ever, important information is buried inside messages, and our most important tasks can slip through the cracks—especially when we’re working on our phones. For many of us, dealing with email has become a daily chore that distracts from what we really need to do—rather than helping us get those things done.

If this all sounds familiar, then Inbox is for you. Or more accurately, Inbox works for you. Here are some of the ways Inbox is at your service:



Bundles: stay organized automatically
Inbox expands upon the categories we introduced in Gmail last year, making it easy to deal with similar types of mail all at once. For example, all your purchase receipts or bank statements are neatly grouped together so that you can quickly review and then swipe them out of the way. You can even teach Inbox to adapt to the way you work by choosing which emails you’d like to see grouped together.

Highlights: the important info at a glance
Inbox highlights the key information from important messages, such as flight itineraries, event information, and photos and documents emailed to you by friends and family. Inbox will even display useful information from the web that wasn’t in the original email, such as the real-time status of your flights and package deliveries. Highlights and Bundles work together to give you just the information you need at a glance.
Reminders, Assists, and Snooze: your to-do’s on your own terms
Inbox makes it easy to focus on your priorities by letting you add your own Reminders, from picking up the dry cleaning to giving your parents a call. No matter what you need to remember, your inbox becomes a centralized place to keep track of the things you need to get back to.
A sampling of Assists
And speaking of to-do’s, Inbox helps you cross those off your list by providing Assists—handy pieces of information you may need to get the job done. For example, if you write a Reminder to call the hardware store, Inbox will supply the store’s phone number and tell you if it's open. Assists work for your email, too. If you make a restaurant reservation online, Inbox adds a map to your confirmation email. Book a flight online, and Inbox gives a link to check-in.

Of course, not everything needs to be done right now. Whether you’re in an inconvenient place or simply need to focus on something else first, Inbox lets you Snooze away emails and Reminders. You can set them to come back at another time or when you get to a specific location, like your home or your office.

Get started with Inbox
Starting today, we’re sending out the first round of invitations to give Inbox a try, and each new user will be able to invite their friends. If Inbox can’t arrive soon enough for you, you can email us at inbox@google.com to get an invitation as soon as more become available.

When you start using Inbox, you’ll quickly see that it doesn’t feel the same as Gmail—and that’s the point. Gmail’s still there for you, but Inbox is something new. It’s a better way to get back to what matters, and we can’t wait to share it with you.



Cross-posted from the Official Gmail Blog

by Emily Wood (noreply@blogger.com) at October 22, 2014 11:03 AM

Chris Siebenmann

Exim's (log) identifiers are basically unique on a given machine

Exim gives each incoming email message an identifier; these look like '1XgWdJ-00020d-7g'. Among other things, this identifier is used for all log messages about the particular email message. Since Exim normally splits information about each message across multiple lines, you routinely need to reassemble or at least match multiple lines for a single message. As a result of this need to aggregate multiple lines, I've quietly wondered for a long time just how unique these log identifiers were. Clearly they weren't going to repeat over the short term, but if I gathered tens or hundreds of days of logs for a particular system, would I find repeats?

The answer turns out to be no. Under normal circumstances Exim's message IDs here will be permanently unique on a single machine, although you can't count on global uniqueness across multiple machines (although the odds are pretty good). The details of how these message IDs are formed are in the Exim documentation's chapter 3.4. On most Unixes and with most Exim configurations they are a per-second timestamp, the process PID, and a final subsecond timestamp, and Exim takes care to guarantee that the timestamps will be different for the next possible message with the same PID.

(Thus a cross-machine collision would require the same message time down to the subsecond component plus the same PID on both machines. This is fairly unlikely but not impossible. Exim has a setting that can force more cross-machine uniqueness.)

This means that aggregation of multi-line logs can be done with simple brute force approaches that rely on ID uniqueness. Heck, to group all the log lines for a given message together you can just sort on the ID field, assuming you do a stable sort so that things stay in timestamp order when the IDs match.

(As they say, this is relevant to my interests and I finally wound up looking it up today. Writing it down here insures I don't have to try to remember where I found it in the Exim documentation the next time I need it.)

PS: like many other uses of Unix timestamps, all of this uniqueness potentially goes out the window if you allow time on your machine to actually go backwards. On a moderate volume machine you'd still have to be pretty unlucky to have a collision, though.

by cks at October 22, 2014 04:21 AM

October 21, 2014

Ubuntu Geek

Yellow Bricks

What is coming for vSphere and VSAN? VMworld reveals…


I’ve been prepping a presentation for upcoming VMUGs, but wanted to also share this with my readers. The session is all about vSphere futures, what is coming soon? Before anyone says I am breaking NDA, I’ve harvested all of this info from public VMworld sessions. Except for the VSAN details, those were announced to the press at VMworld EMEA. Lets start with Virtual SAN…

The Virtual SAN details were posted in this Computer Weekly article, and by the looks of it they interviewed VMware’s CEO Pat Gelsinger and Alberto Farronato from the VSAN product team. So what is coming soon?

  • All Flash Virtual SAN support
    Considering the price of MLC has lowered to roughly the same price as SAS HDDs per GB I think this is a great new feature to have. Being able to build all-flash configurations at the price point of a regular configuration, and with probably many supported configurations is a huge advantage of VSAN. I would expect VSAN to support various types of flash as the “capacity” layer, so this is an architects dream… designing your own all-flash storage system!
  • Virsto integration
    I played with Virsto when it was just released and was impressed by the performance and the scalability. Functions that were part of Virst such as snapshots and clones these have been built into VSAN and it will bring VSAN to the next level!
  • JBOD support
    Something many have requested, and primarily to be able to use VSAN in Blade environments… Well with the JBOD support announced this will be a lot easier. I don’t know the exact details, but just the “JBOD” part got me excited.
  • 64 host VSAN cluster support
    VSAN doesn’t scale? Here you go,

That is a nice list by itself, and I am sure there is plenty more for VSAN. At VMworld for instance Wade Holmes also spoke about support for disk controller based encryption for instance. Cool right?! So what about vSphere? Considering even the version number was dropped during the keynote and it hints at a major release you would expect some big functionality to be introduced. Once again, all the stuff below is harvested from various public VMworld sessions:

  • VMFork aka Project Fargo – discussed here…
  • Increased scale!
    • 64 host HA/DRS cluster, I know a handful of customers who asked for 64 host clusters, so here it is guys… or better said: soon you will have it!
  • SMP vCPU FT – up to 4 vCPU support
    • I like FT from an innovation point of view, but it isn’t a feature I would personally use too much as I feel “fault tolerance” from an app perspective needs to be solved by the app. Now, I do realize that there are MANY legacy applications out there, and if you have a scale-up application which needs to be highly available then SMP FT is very useful. Do note that with this release the architecture of FT has changed. For instance you used to share the same “VMDK” for both primary and secondary, but that is no longer the case.
  • vMotion across anything
    • vMotion across vCenter instances
    • vMotion across Distributed Switch
    • vMotion across very large distance, support up to 100ms latency
    • vMotion to vCloud Air datacenter
  • Introduction of Virtual Datacenter concept in vCenter
    • Enhance “policy driven” experience within vCenter. Virtual Datacenter aggregates compute clusters, storage clusters, networks, and policies!
  • Content Library
    • Content Library provides storage and versioning of files including VM templates, ISOs, and OVFs.
      Includes powerful publish and subscribe features to replicate content
      Backed by vSphere Datastores or NFS
  • Web Client performance / enhancement
    • Recent tasks pane drops to the bottom instead of on the right
    • Performance vastly improved
    • Menus flattened
  • DRS placement “network aware”
    • Hosts with high network contention can show low CPU and memory usage, DRS will look for more VM placements
    • Provide network bandwidth reservation for VMs and migrate VMs in response to reservation violations!
  • vSphere HA component protection
    • Helps when hitting “all paths down” situations by allowing HA to take action on impacted virtual machines
  • Virtual Volumes, bringing the VSAN “policy goodness” to traditional storage systems

Of course there is more, but these are the ones that were discussed at VMworld… for the remainder you will have to wait until the next version of vSphere is released, or you can also sign up for the beta still I believe!

"What is coming for vSphere and VSAN? VMworld reveals…" originally appeared on Yellow-Bricks.com. Follow me on twitter - @DuncanYB.


Pre-order my upcoming book Essential Virtual SAN via Pearson today!

by Duncan Epping at October 21, 2014 12:55 PM

Chris Siebenmann

Some numbers on our inbound and outbound TLS usage in SMTP

As a result of POODLE, it's suddenly rather interesting to find out the volume of SSLv3 usage that you're seeing. Fortunately for us, Exim directly logs the SSL/TLS protocol version in a relatively easy to search for format; it's recorded as the 'X=...' parameter for both inbound and outbound email. So here's some statistics, first from our external MX gateway for inbound messages and then from our other servers for external deliveries.

Over the past 90 days, we've received roughly 1.17 million external email messages. 389,000 of them were received with some version of SSL/TLS. Unfortunately our external mail gateway currently only supports up to TLS 1.0, so the only split I can report is that only 130 of these messages were received using SSLv3 instead of TLS 1.0. 130 messages is low enough for me to examine the sources by hand; the only particularly interesting and eyebrow-raising ones were a couple of servers at a US university and a .nl ISP.

(I'm a little bit surprised that our Exim doesn't support higher TLS versions, to be honest. We're using Exim on Ubuntu 12.04, which I would have thought would support something more than just TLS 1.0.)

On our user mail submission machine, we've delivered to 167,000 remote addresses over the past 90 days. Almost all of them, 158,000, were done with SSL/TLS. Only three of them used SSLv3 and they were all to the same destination; everything else was TLS 1.0.

(It turns out that very few of our user submitted messages were received with TLS, only 0.9%. This rather surprises me but maybe many IMAP programs default to not using TLS even if the submission server offers it. All of these small number of submissions used TLS 1.0, as I'd hope.)

Given that our Exim version only supports TLS 1.0, these numbers are more boring than I was hoping they'd be when I started writing this entry. That's how it goes sometimes; the research process can be disappointing as well as educating.

(I did verify that our SMTP servers really only do support up to TLS 1.0 and it's not just that no one asked for a higher version than that.)

One set of numbers I'd like to get for our inbound email is how TLS usage correlates with spam score. Unfortunately our inbound mail setup makes it basically impossible to correlate the bits together, as spam scoring is done well after TLS information is readily available.

Sidebar: these numbers don't quite mean what you might think

I've talked about inbound message deliveries and outbound destination addresses here because that's what Exim logs information about, but of course what is really encrypted is connections. One (encrypted) connection may deliver multiple inbound messages and certainly may be handed multiple RCPT TO addresses in the same conversation. I've also made no attempt to aggregate this by source or destination, so very popular sources or destinations (like, say, Gmail) will influence these numbers quite a lot.

All of this means that this sort of numbers can't be taken as an indication of how many sources or destinations do TLS with us. All I can talk about is message flows.

(I can't even talk about how many outgoing messages are completely protected by TLS, because to do that I'd have to work out how many messages had no non-TLS deliveries. This is probably possible with Exim logs, but it's more work than I'm interested in doing right now. Clearly what I need is some sort of easy to use Exim log aggregator that will group all log messages for a given email message together and then let me do relatively sophisticated queries on the result.)

by cks at October 21, 2014 03:28 AM

October 20, 2014

Everything Sysadmin

See you tomorrow evening at the Denver DevOps Meetup!

Hey Denver folks! Don't forget that tomorrow evening (Tue, Oct 21) I'll be speaking at the Denver DevOps Meetup. It starts at 6:30pm! Hope to see you there!

http://www.meetup.com/DenverDevOps/events/213369602/

October 20, 2014 04:28 PM

Mark Shuttleworth

V is for Vivid

Release week! Already! I wouldn’t call Trusty ‘vintage’ just yet, but Utopic is poised to leap into the torrent stream. We’ve all managed to land our final touches to *buntu and are excited to bring the next wave of newness to users around the world. Glad to see the unicorn theme went down well, judging from the various desktops I see on G+.

And so it’s time to open the vatic floodgates and invite your thoughts and contributions to our soon-to-be-opened iteration next. Our ventrous quest to put GNU as you love it on phones is bearing fruit, with final touches to the first image in a new era of convergence in computing. From tiny devices to personal computers of all shapes and sizes to the ventose vistas of cloud computing, our goal is to make a platform that is useful, versal and widely used.

Who would have thought – a phone! Each year in Ubuntu brings something new. It is a privilege to celebrate our tenth anniversary milestone with such vernal efforts. New ecosystems are born all the time, and it’s vital that we refresh and renew our thinking and our product in vibrant ways. That we have the chance to do so is testament to the role Linux at large is playing in modern computing, and the breadth of vision in our virtual team.

To our fledgling phone developer community, for all your votive contributions and vocal participation, thank you! Let’s not be vaunty: we have a lot to do yet, but my oh my what we’ve made together feels fantastic. You are the vigorous vanguard, the verecund visionaries and our venerable mates in this adventure. Thank you again.

This verbose tract is a venial vanity, a chance to vector verbal vibes, a map of verdant hills to be climbed in months ahead. Amongst those peaks I expect we’ll find new ways to bring secure, free and fabulous opportunities for both developers and users. This is a time when every electronic thing can be an Internet thing, and that’s a chance for us to bring our platform, with its security and its long term support, to a vast and important field. In a world where almost any device can be smart, and also subverted, our shared efforts to make trusted and trustworthy systems might find fertile ground. So our goal this next cycle is to show the way past a simple Internet of things, to a world of Internet things-you-can-trust.

In my favourite places, the smartest thing around is a particular kind of monkey. Vexatious at times, volant and vogie at others, a vervet gets in anywhere and delights in teasing cats and dogs alike. As the upstart monkey in this business I can think of no better mascot. And so let’s launch our vicenary cycle, our verist varlet, the Vivid Vervet!

by mark at October 20, 2014 01:22 PM

Google Blog

DISTRICT VOICES: Inside Panem with our finest citizens

Meet District Voices, the latest campaign in our Art, Copy & Code project—where we explore new ways for brands to connect with consumers through experiences that people love, remember and share. District Voices was created in partnership with Lionsgate to promote the upcoming release of The Hunger Games: Mockingjay Part 1. -Ed.

Greetings, Citizens of Panem!

The Capitol has joined forces with Google and YouTube to celebrate the proud achievements of our strong, lively districts. Premiering today on YouTube, a new miniseries called DISTRICT VOICES will take you behind the scenes to meet some of Panem’s most creative—and loyal—citizens.

At 4 p.m. EDT/ 1 p.m. PDT every day this week, one of your favorite Citizen creators from YouTube will give you a never-before-seen tour of their districts. First, the Threadbanger textile experts of District 8 will show how utility meets beauty in this season’s fashion—plus, you’ll get a look at a new way to wear your Capitol pride. Tomorrow, District 2's Shane Fazen will provide a riveting demonstration of how we keep our noble peacekeepers in tip-top shape. On Wednesday, Derek Muller from District 5—Panem’s center of power generation—will give you a peek at a revolutionary new way to generate electricity. Thursday The Grain District’s own Feast of Fiction will show you how to bake one of beloved victor Peeta Mellark’s most special treats. And finally, iJustine, District 6’s liaison to the Capitol, will give you an exclusive glimpse at the majestic and powerful peacekeeper vehicles in action.

Tune in at CAPITOL TV. And remember—Love your labor. Take pride in your task. Our future is in your hands.

by Emily Wood (noreply@blogger.com) at October 20, 2014 10:05 AM

Tech Teapot

New Aviosys IP Power 9858 Box Opening

A series of box opening photos of the new Aviosys IP Power 9858 4 port network power switch. This model will in due course replace the Aviosys IP Power 9258 series of power switches. The 9258 series is still available in the mean time though, so don’t worry.

The new model supports WiFi (802.11n-b/g and WPS for easy WiFi setup), auto reboot on ping failure, time of day scheduler and internal temperature sensor. Aviosys have also built apps for iOS and Android, so you can now manage your power switch on the move. Together with the 8 port Aviosys IP Power 9820 they provide very handy tools for remote power management of devices. Say goodbye to travelling to a remote site just to reboot a broadband router.

Aviosys IP Power 9858DX Closed Box Aviosys IP Power 9858DX Open Box Aviosys IP Power 9858DX Front with Wifi Aerial Aviosys IP Power 9858DX Front Panel Aviosys IP Power 9858DX Rear Panel Aviosys IP Power 9858DX Read Close Up #2

 

The post New Aviosys IP Power 9858 Box Opening appeared first on Openxtra Tech Teapot.

by Jack Hughes at October 20, 2014 07:00 AM

Chris Siebenmann

Revisiting Python's string concatenation optimization

Back in Python 2.4, CPython introduced an optimization for string concatenation that was designed to reduce memory churn in this operation and I got curious enough about this to examine it in some detail. Python 2.4 is a long time ago and I recently was prompted to wonder what had changed since then, if anything, in both Python 2 and Python 3.

To quickly summarize my earlier entry, CPython only optimizes string concatenations by attempting to grow the left side in place instead of making a new string and copying everything. It can only do this if the left side string only has (or clearly will have) a reference count of one, because otherwise it's breaking the promise that strings are immutable. Generally this requires code of the form 'avar = avar + ...' or 'avar += ...'.

As of Python 2.7.8, things have changed only slightly. In particular concatenation of Unicode strings is still not optimized; this remains a byte string only optimization. For byte strings there are two cases. Strings under somewhat less than 512 bytes can sometimes be grown in place by a few bytes, depending on their exact sizes. Strings over that can be grown if the system realloc() can find empty space after them.

(As a trivial root, CPython also optimizes concatenating an empty string to something by just returning the other string with its reference count increased.)

In Python 3, things are more complicated but the good news is that this optimization does work on Unicode strings. Python 3.3+ has a complex implementation of (Unicode) strings, but it does attempt to do in-place resizing on them under appropriate circumstances. The first complication is that internally Python 3 has a hierarchy of Unicode string storage and you can't do an in-place concatenation of a more complex sort of Unicode string into a less complex one. Once you have compatible strings in this sense, in terms of byte sizes the relevant sizes are the same as for Python 2.7.8; Unicode string objects that are less than 512 bytes can sometimes be grown by a few bytes while ones larger than that are at the mercy of the system realloc(). However, how many bytes a Unicode string takes up depends on what sort of string storage it is using, which I think mostly depends on how big your Unicode characters are (see this section of the Python 3.3 release notes and PEP 393 for the gory details).

So my overall conclusion remains as before; this optimization is chancy and should not be counted on. If you are doing repeated concatenation you're almost certainly better off using .join() on a list; if you think you have a situation that's otherwise, you should benchmark it.

(In Python 3, the place to start is PyUnicode_Append() in Objects/unicodeobject.c. You'll probably also want to read Include/unicodeobject.h and PEP 393 to understand this, and then see Objects/obmalloc.c for the small object allocator.)

Sidebar: What the funny 512 byte breakpoint is about

Current versions of CPython 2 and 3 allocate 'small' objects using an internal allocator that I think is basically a slab allocator. This allocator is used for all overall objects that are 512 bytes or less and it rounds object size up to the next 8-byte boundary. This means that if you ask for, say, a 41-byte object you actually get one that can hold up to 48 bytes and thus can be 'grown' in place up to this size.

by cks at October 20, 2014 04:37 AM

October 19, 2014

Ubuntu Geek

Configuring layer-two peer-to-peer VPN using n2n

n2n is a layer-two peer-to-peer virtual private network (VPN) which allows users to exploit features typical of P2P applications at network instead of application level. This means that users can gain native IP visibility (e.g. two PCs belonging to the same n2n network can ping each other) and be reachable with the same network IP address regardless of the network where they currently belong. In a nutshell, as OpenVPN moved SSL from application (e.g. used to implement the https protocol) to network protocol, n2n moves P2P from application to network level.
(...)
Read the rest of Configuring layer-two peer-to-peer VPN using n2n (416 words)


© ruchi for Ubuntu Geek, 2014. | Permalink | No comment | Add to del.icio.us
Post tags: , , ,

Related posts

by ruchi at October 19, 2014 11:20 PM

Evaggelos Balaskas

SatNOGS - Satellite Networked Open Ground Station

What started as a Nasa Space App Challenge now becomes an extraordinary opensource achievement on the top five finalist of hackaday.io.

What is SatNOGS in non technical words: imagine a cheap mobile openhardware ground station that can collaborate through the internet with other ground stations and gather satellite signals all together, participating in a holistic opensource/opendata and public accessible database/site !

If you are thinking, that cant be right, the answer is that it is!!!

The amazing team behind the SatNOGS is working around the clock - non stop ONLY with openhardware and free software to do exactly that !

A fully modular system (you can choose your own antennas! or base setup) you can review the entire code on github, you can see in high quality videos and guides for every step, every process, you can participate via comments, emails or even satellite signals !

satnogs_02.jpg

3D Printing is one of the major component in their journey till now. The have already published every design they are using for the satnogs project on github! You just need to print them. Every non-3d printing hardware are available to every hardware store near by you. The members of this project have published the Arduino code and schematics for the electronics too !!

Everything is fully documented in details, everything is open source !

AMAZING!

satnogs.jpg

It’s seems that i may be bias, so dont believe anything i am writing.
See for your self and be mind-blowing impressed with the quality of their hardware documentation

Visit their facebook account for news and contact them if you have a brilliant idea about satellites or you just want to get a status of their work.

How about the team ?

I’ve met the entire team at Athens Hackerspace and the first thing that came into my mind (and it is most impressive) is the diversity of the members itself.

Not only in age (most of them are university students, but older hobbyists are participating too) but also in the technical area of expertise. This team can easily solve every practical problem they can find in the process.

SatNOGS, as I’ve already mentioned, is fully active and that all started (with the bing bang of-course) with an idea: To reach and communicate with the Space (the final frontier). Satellites are sending signals 24/7 and the ground stations cant reach every satellite (i am not talking to geo-static satellites) and there is no one to acknowledge that. The problem that the satnogs is solving is real.

And i hope with this blog post, more people can understand how important is that this project scale to more hackerspaces around the globe.

To see more, just click here and you can monitor the entire process till now.

Tag(s): SatNOGS

October 19, 2014 09:28 PM

Ferry Boender

Bexec v0.8: Execute a vim buffer and capture output in split window

I released v0.8 of my Bexec vim plugin. The Bexec plugin allows the user to execute the current buffer if it contains a script with a shebang (#!/path/to/interpreter) on the first line or if the default interpreter for the script's type is known by Bexec. The output of the script will be grabbed and displayed in a separate buffer. 

New in this release:

  • Honor splitbelow and splitright vim setting (patch by Christopher Pease).

bexec

Installation instructions:

  1. Download the Vimball
  2. Start vim with: vim bexec-v0.8.vmb
  3. In Vim, type: :source %
  4. Bexec is now installed. Type :Bexec to run it, or use <MapLeader>bx

 

 

by admin at October 19, 2014 01:22 PM

Server Density

Chris Siebenmann

Vegeta, a tool for web server stress testing

Standard stress testing tools like siege (or the venerable ab, which you shouldn't use) are all systems that do N concurrent requests at once and see how your website stands up to this. This model is a fine one for putting a consistent load on your website for a stress test, but it's not actually representative of how the real world acts. In the real world you generally don't have, say, 50 clients all trying to repeatedly make and re-make one request to you as fast as they can; instead you'll have 50 new clients (and requests) show up every second.

(I wrote about this difference at length back in this old entry.)

Vegeta is a HTTP load and stress testing tool that I stumbled over at some point. What really attracted my attention is that it uses a 'N requests a second' model, instead of the concurrent request model. As a bonus it will also report not just average performance but also on outliers in the form of 90th and 99th percentile outliers. It's written in Go, which some of my readers may find annoying but which I rather like.

I gave it a try recently and, well, it works. It does what it says it does, which means that it's now become my default load and stress testing tool; 'N new requests a second' is a more realistic and thus interesting test than 'N concurrent requests' for my software (especially here, for obvious reasons).

(I may still do N concurrent requests tests as well, but it'll probably mostly be to see if there are issues that come up under some degree of consistent load and if I have any obvious concurrency race problems.)

Note that as with any HTTP stress tester, testing with high load levels may require a fast system (or systems) with plenty of CPUs, memory, and good networking if applicable. And as always you should validate that vegeta is actually delivering the degree of load that it should be, although this is actually reasonably easy to verify for a 'N new request per second' tester.

(Barring errors, N new requests a second over an M second test run should result in N*M requests made and thus appearing in your server logs. I suppose the next time I run a test with vegeta I should verify this myself in my test environment. In my usage so far I just took it on trust that vegeta was working right, which in light of my ab experience may be a little bit optimistic.)

by cks at October 19, 2014 06:04 AM

October 18, 2014

SysAdmin1138

For other Movable Type blogs out there

If you're wondering why comments aren't working, as I was, and are on shared hosting, as I am, and get to looking at your error_log file and see something like this in it:

[Sun Oct 12 12:34:56 2014] [error] [client 192.0.2.5] 
ModSecurity: Access denied with code 406 (phase 2).
Match of "beginsWith http://%{SERVER_NAME}/" against "MATCHED_VAR" required.
[file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "1425"] [id "340503"] [rev "1"]
[msg "Remote File Injection attempt in ARGS (/cgi-bin/mt4/mt-comments.cgi)"]
[severity "CRITICAL"]
[hostname "example.com"]
[uri "/cgi-bin/mt/mt-comments.cgi"]
[unique_id "PIMENTOCAKE"]

It's not just you.

It seems that some webhosts have a mod_security rule in place that bans submitting anything through "mt-comments.cgi". As this is the main way MT submits comments, this kind of breaks things. Happily, working around a rule like this is dead easy.

  1. Rename your mt-comments.cgi file to something else
  2. Add "CommentScript ${renamed file}" to your mt-config.cgi file

And suddenly comments start working again!

Except for Google, since they're deprecating OpenID support.

by SysAdmin1138 at October 18, 2014 09:46 PM

Rands in Repose

Chris Siebenmann

During your crisis, remember to look for anomalies

This is a war story.

Today I had one of those valuable learning experiences for a system administrator. What happened is that one of our old fileservers locked up mysteriously, so we power cycled it. Then it locked up again. And again (and an attempt to get a crash dump failed). We thought it might be hardware related, so we transplanted the system disks into an entirely new chassis (with more memory, because there was some indications that it might be running out of memory somehow). It still locked up. Each lockup took maybe ten or fifteen minutes from the reboot, and things were all the more alarming and mysterious because this particular old fileserver only had a handful of production filesystems still on it; almost all of them had been migrated to one of our new fileservers. After one more lockup we gave up and went with our panic plan: we disabled NFS and set up to do an emergency migration of the remaining filesystems to the appropriate new fileserver.

Only as we started the first filesystem migration did we notice that one of the ZFS pools was completely full (so full it could not make a ZFS snapshot). As we were freeing up some space in the pool, a little light came on in the back of my mind; I remembered reading something about how full ZFS pools on our ancient version of Solaris could be very bad news, and I was pretty sure that earlier I'd seen a bunch of NFS write IO at least being attempted against the pool. Rather than migrate the filesystem after the pool had some free space, we selectively re-enabled NFS fileservice. The fileserver stayed up. We enabled more NFS fileservice. And things stayed happy. At this point we're pretty sure that we found the actual cause of all of our fileserver problems today.

(Afterwards I discovered that we had run into something like this before.)

What this has taught me is during an inexplicable crisis, I should try to take a bit of time to look for anomalies. Not specific anomalies, but general ones; things about the state of the system that aren't right or don't seem right.

(There is a certain amount of hindsight bias in this advice, but I want to mull that over a bit before I wrote more about it. The more I think about it the more complicated real crisis response becomes.)

by cks at October 18, 2014 04:55 AM

Giri Mandalika

Blast from the Past : The Weekend Playlist #7

Previous playlists:

    #1 (50s, 60s and 70s) | #2 (80s) | #3 (80s) | #4 (80s) | #5 (80s) | #6 (90s)

Audio-Visual material courtesy: YouTube. Other information: Wikipedia.

1. Fatboy Slim / Norman Cook - Brimful of Asha (1998)

A remix. Original by UK band Cornershop.

2. Vanilla Ice - Ice Ice Baby (1990)

3. Beck - Loser (1993)

4. Primus - Mr. Krinkle (1993)

5. Tool - Stinkfist (1996)

if you don't mind watching dark videos, look for Stinkfist official video on youtube.

6. P.M. Dawn - Set Adrift On Memory Bliss (1991)

7. Primitive Radio Gods - Standing Outside A Broken Phone Booth (1996)

no traces of official video anywhere on web, for some reason.

8. Blues Traveler - Run-Around (1995)

Grammy winner.

9. KoRn - A.D.I.D.A.S. (1997)

Under Pressure mix. Another dark song that has nothing to do with sportswear brand, Adidas.

10. Chumbawamba - Tubthumping (1997)

one hit wonder.

by Giri Mandalika (noreply@blogger.com) at October 18, 2014 01:00 AM

October 17, 2014

Everything Sysadmin

Usenix LISA early registration discount expires soon!

Register by Mon, October 20 and take advantage of the early bird pricing.

I'll be teaching tutorials on managing oncall, team-driven sysadmin tools, upgrading live services and more. Please register soon and save!

https://www.usenix.org/conference/lisa14

October 17, 2014 05:28 PM

Standalone Sysadmin

VM Creation Day - PowerShell and VMware Automation

I should have ordered balloons and streamers, because Monday was VM creation day on my VMware cluster.

In addition to a 3-node production-licensed vSphere cluster, I run a 10-node cluster specifically for academic purposes. One of those purposes is building and maintaining classroom environments. A lot of professors maintain a server or two for their courses, but our Information Assurance program here goes above and beyond in terms of VM utilization. Every semester, I've got to deal with the added load, so I figured if I'm going to document it, I might as well get a blog entry while I'm at it.vmware_ia_spinup

Conceptually, the purpose of this process is to allow an instructor to create a set of virtual machines (typically between 1 and 4 of them), collectively referred to as a 'pod', which will serve as a lab for students. Once this set of VMs is configured exactly as the professor wants, and they have signed off on them, those VMs become the 'Gold Images', and then each student gets their own instance of these VMs. A class can have between 10 and 70 students, so this quickly becomes a real headache to deal with, hence the automation.

Additionally, because these classes are Information Assurance courses, it's not uncommon for the VMs to be configured in an insecure manner (on purpose) and to be attacked by other VMs, and to generally behave in a manner unbecoming a good network denizen, so each class is cordoned off onto its own VLAN, with its own PFsense box guarding the entryway and doing NAT for the several hundred VMs behind the wall. The script needs to automate the creation of the relevant PFsense configs, too, so that comes at the end.

I've written a relatively involved PowerShell script to do my dirty work for me, but it's still a long series of things to go from zero to working classroom environment. I figured I would spend a little time to talk about what I do to make this happen. I'm not saying it's the best solution, but it's the one I use, and it works for me. I'm interested in hearing if you've got a similar solution going on. Make sure to comment and let everyone know what you're using for these kinds of things.

The process is mostly automated hard parts separated by manual staging, because I want to verify sanity at each step. This kind of thing happens infrequently enough that I'm not completely trusting of the process yet, mostly due to my own ignorance of all of the edge cases that can cause failures. To the right, you'll see a diagram of the process.

In the script, the first thing I do is include functions that I stole from an awesome post on Subnet Math with PowerShell from Indented!, a software blog by Chris Dent. Because I'm going to be dealing with the DHCP config, it'll be very helpful to be able to have functions that understand what subnet boundaries are, and how to properly increment IP addresses.

I need to make sure that, if this powershell script is running, that we are actually loading the VMware PowerCLI commandlets. We can do that like this:


if ( ( Get-PSSnapin -name VMware.VimAutomation.Core -ErrorAction SilentlyContinue ) -eq $null ) {
Add-PSSnapin VMware.VimAutomation.Core
}

For the class itself, this whole process consists of functions to do what needs to be done (or "do the needful" if you use that particular phrase), and it's fairly linear, and each step requires the prior to be completed. What I've done is to create an object that represents the course as a whole, and then add the appropriate properties and methods. I don't actually need a lot of the power of OOP, but it provides a convenient way to keep everything together. Here's an example of the initial class setup:


$IA = New-Object psobject

# Lets add some initial values
Add-Member -InputObject $IA -MemberType NoteProperty -Name ClassCode -Value ""
Add-Member -InputObject $IA -MemberType NoteProperty -Name Semester -Value ""
Add-Member -InputObject $IA -MemberType NoteProperty -Name Datastore -Value "FASTDATASTORENAME"
Add-Member -InputObject $IA -MemberType NoteProperty -Name Cluster -Value "IA Program"
Add-Member -InputObject $IA -MemberType NoteProperty -Name VIServer -Value "VSPHERE-SERVER"
Add-Member -InputObject $IA -MemberType NoteProperty -Name IPBlock -Value "10.0.1.0"
Add-Member -InputObject $IA -MemberType NoteProperty -Name SubnetMask -Value "255.255.0.0"
Add-Member -InputObject $IA -MemberType NoteProperty -Name Connected -Value $false
Add-Member -InputObject $IA -MemberType NoteProperty -Name ResourcePool -Value ""
Add-Member -InputObject $IA -MemberType NoteProperty -Name PodCount -Value ""
Add-Member -InputObject $IA -MemberType NoteProperty -Name GoldMasters -Value ""
Add-Member -InputObject $IA -MemberType NoteProperty -Name Folder -Value ""
Add-Member -InputObject $IA -MemberType NoteProperty -Name MACPrefix -Value ""
Add-Member -InputObject $IA -MemberType NoteProperty -Name ConfigDir -Value ""
Add-Member -InputObject $IA -MemberType NoteProperty -Name VMarray -Value @()

These are just the values that almost never change. Since we're using NAT, and we're not routing to that network, and every class has its own dedicated VLAN, we can use the same IP block every time without running into a problem. The blank values are there just as placeholder, and those values will be filled in as the class methods are invoked.

At the bottom of the script, which is where I spend most of my time, I set per-class settings:


$IA.ClassCode = "ia1234"
$IA.Semester = "Fall-2014"
$IA.PodCount = 35
$IA.GoldMasters = @(
@{
vmname = "ia1234-win7-gold-20141014"
osname = "win7"
tcp = 3389
udp = ""
},
@{
vmname = "ia1234-centos-gold-20141014"
osname = "centos"
tcp = ""
udp = ""
},
@{
vmname = "ia1234-kali-gold-20141014"
osname = "kali"
tcp = "22"
udp = ""
}
)

We set the class code, semester, and pod count simply. These will be used to create the VM names, the folders, and resource groups that the VMs live in. The GoldMaster array is a data structure that has an entry for each of the gold images that the professor has created. It contains the name of the gold image, plus a short code that will be used to name the VM instances coming from it, and has a placeholder for the tcp and udp ports which need forwarded from the outside to allow internal access. I don't currently have the code in place that allows me to specify multiple port forwards, but that's going to be added, because I had a professor request 7(!) forwarded ports per VM in one of their classes this semester.

As you can see in the diagram, I'm using Linked Clones to spin up the students' pods. This has the advantage of saving diskspace and of completing quickly. Linked clones operate on a snapshot of the original disk image. Rather than actually have the VMs operate on the gold images, I do a full clone of the VM over to a faster datastore than the Ol' Reliable NetApp.

We add a method to the $IA object like this:


Add-Member -InputObject $IA -MemberType ScriptMethod -Name createLCMASTERs -Value {
# This is the code that converts the gold images into LCMASTERs
# Because you need to put a template somewhere, it makes sense to put it
# into the folder that the VMs will eventually live in themselves (thus saving
# yourself the effort of locating the right folder twice).
Param()
Process {
... stuff goes here
}
}

The core of this method is the following block, which actually performs the clone:


if ( ! (Get-VM -Name $LCMASTERName) ) {
try {
$presnap = New-snapshot -Name ("Autosnap: " + $(Get-Date).toString("yyyMMdd")) -VM $GoldVM -confirm:$false

$cloneSpec = new-object VMware.Vim.VirtualMachineCloneSpec
$cloneSpec.Location = New-Object VMware.Vim.VirtualMachineRelocateSpec
$cloneSpec.Location.Pool = ($IA.ResourcePool | Get-View).MoRef
$cloneSpec.Location.host = ($vm | Get-VMHost).MoRef
$cloneSpec.Location.Datastore = ($IA.Datastore | Get-View).MoRef
$cloneSpec.Location.DiskMoveType = [VMware.Vim.VirtualMachineRelocateDiskMoveOptions]::createNewChildDiskBacking
$cloneSpec.Snapshot = ($GoldVM | Get-View).Snapshot.CurrentSnapshot
$cloneSpec.PowerOn = $false

($GoldVM | Get-View).cloneVM( $LCMasterFolder.MoRef, $LCMASTERName, $cloneSpec)

Remove-snapshot -Snapshot $presnap -confirm:$false
}
catch [Exception] {
Write-Host "Error: " $_.Exception.Message
exit
}
} else {
Write-Host "Template found with name $LCMasterName - not recreating"
}


(apologies for the lack of indentation)

If you're interested in doing this kind of thing, make sure you check out the docs for the createNewChildDiskBacking setting.

After the Linked Clone Masters have been created, then it's a simple matter of creating the VMs from each of them (using the $IA.PodCount value to figure out how many we need). They end up getting named something like $IA.ClassCode-$IA.Semester-$IA.GoldMasters[#].osname-pod$podcount which makes it easy to figure out what goes where when I have several classes running at once.

After the VMs have been created, we can start dealing with the network portion. I used to spin up all of the VMs, then loop through them and pull the MAC addresses to use with the DHCP config, but there were problems with that method. I found that a lot of the time, I'll need to rerun this script a few times per class, either because I've screwed something up or the instructor needs to make changes to the pod. When that happens, EACH TIME I had to re-generate the DHCP config (which is easy) and then manually insert it into PFsense (which is super-annoying).

Rather than do that every time, I eventually realized that it's much easier just to dictate what the MAC address for each machine is, and then it doesn't matter how often I rerun the script, the DHCP config doesn't change. (And yes, I'm using DHCP, but with static leases, which is necessary because of the port forwarding).

Here's what I do:

Add-Member -InputObject $IA -MemberType ScriptMethod -Name assignMACs -Value {
Param()
Process {
$StaticPrefix = "00:50:56"
if ( $IA.MACPrefix -eq "" ) {
# Since there isn't already a prefix set, it's cool to make one randomly
$IA.MACPrefix = $StaticPrefix + ":" + ("{0:X2}" -f (Get-Random -Minimum 0 -Maximum 63) )
}
$machineCount = 0
$IA.VMarray | ForEach-Object {
$machineAddr = $IA.MACPrefix + ":" + ("{0:X4}" -f $machineCount).Insert(2,":")

$vm = Get-VM -name $_.name
$networkAdapter = Get-NetworkAdapter -VM $vm
Write-Host "Setting $vm to $machineAddr"
Set-NetworkAdapter -NetworkAdapter $networkAdapter -MacAddress $machineAddr -Confirm:$false
$IA.VMarray[$machineCount].MAC = $machineAddr
$IA.VMarray[$machineCount].index = $machineCount
$machineCount++

}
}
}

As you can see, this randomly assigns a MAC address in the vSphere range. Sort of. The fourth octet is randomly selected between 00 and 3F, and then the last two octets are incremented starting from 00. Optionally, the fourth octet can be specified, which is useful in a re-run of the script so that the DHCP config doesn't need to be re-generated.

After the MAC addresses are assigned, the IPs can be determined using the network math:


Add-Member -InputObject $IA -MemberType ScriptMethod -Name assignIPs -Value {
# This method really only assigns the IP to the object.
Param()
Process {
# It was tempting to assign a sane IP block to this network, but given the
# tendancy to shove God-only-knows how many people into a class at a time,
# lets not be bounded by reasonable or sane. /16 it is.
# First 50 IPs are reserved for gateway plus potential gold images.
$currentIP = Get-NextIP $IA.IPBlock 2
$IA.VMarray | ForEach-Object {
$_.IPAddr = $currentIP
$currentIP = Get-NextIP $currentIP 2
}

}
}

This is done by naively giving every other IP to a machine, leaving the odd IP addresses between them open. I've had to massage this before, where a large pod of 5-6 VMs all need to be incremental then skip IPs between them, but I've done those mostly as a one-off. I don't think I need to build in a lot of flexibility because those are relatively rare cases, but it wouldn't be that hard to develop a scheme for it if you needed.

After the IPs are assigned, you can create the DHCP config. Right now, I'm using an ugly hack, where I basically just print out the top of the DHCP config, then loop through the VMs outputting XML the whole way. It's ugly, and I'm not going to paste it here, but if you download a DHCPD XML file from PFsense, then you can basically see what I'm doing. I then do the same thing with the NAT config.

Because I'm still running these functions manually, I have these XML-creation methods printing output, but it's easy to see how you could have them redirect output to a text file (and if you were super-cool, you could use something like this example from MSDN where you spin up an instance of IE:


$ie = new-object -com "InternetExplorer.Application"
$ie.navigate("http://localhost/MiniCalc/Default.aspx")
... and so on

Anyway, I've spun up probably thousands of VMs using this script (or previous instances of it). It's saved me a lot of time, and if you have to manage bulk-VMs using vSphere, and you're not automating it (using PowerCLI, or vCloud Director, or something else), you really should be. And if you DO, what do you do? Comment below and let me know!

Thanks for reading all the way through!

by Matt Simmons at October 17, 2014 03:16 PM

Google Blog

Through the Google lens: search trends October 10-16

Diet secrets from Zach Galifianakis, and cord cutting from a cable company?! Here's a look at another topsy-turvy week in search.

A cast of characters
Search will always have its fair share of characters and this week was no different. First up, moviegoers learned who’s next in line for Hollywood’s superhero treatment when Ezra Miller, star of The Perks of Being a Wallflower, landed the title role in the 2018 film The Flash. And whispers are swirling in Tinseltown that Gal Gadot's already impressive resume—she’s set to play the world’s most famous Amazonian, Wonder Woman—will soon get another stellar addition, the lead female role in a remake of Ben-Hur.

But they weren’t the only celebrities to get the Internet buzzing. Comedian and fan favorite Zach Galifianakis caused a stir on the trends charts after he revealed a much thinner version of himself on the red carpet of the New York Film Festival. When a reporter asked Galifianakis if he had made any lifestyle changes to lose the weight, he responded with a straight face, “No, I'm just... I'm dying.” Clearly Galifianakis isn’t sharing his weight loss secrets.

Out with the old, in with the new
HBO has seen the light! This week the premium television network announced that they will launch a new stand-alone service for fans of its TV shows. Soon, homes without a cable subscription can sign up for HBO Go and get their fill of Game of Thrones and other HBO shows with just an Internet connection—leading people to wonder if this is the beginning of the end for cable providers.

Consumers also had a lot of new mobile devices to choose from this week, starting with our own line of Nexus gadgets like the Nexus 6 running the latest version of Android, 5.0 Lollipop. Meanwhile, Apple announced an updated version of the iPad.
The show’s just getting started
Is it awards show season already? It’s not—but that’s not stopping searchers from looking ahead. The Internet rejoiced when How I Met Your Mother and Gone Girl star Neil Patrick Harris said “Hosting the 2015 Academy Awards? Challenge accepted!” But with the Oscars red carpet still months away, searchers had their sights set on another celebrity bash: Paul Rudd's keg party… at his mom’s house… in the suburbs of Kansas City. What else are you supposed to do when mom’s out of town and the KC Royals just punched a ticket to the World Series after a nearly 30-year hiatus?

Tip of the week
‘Tis the season for pumpkin spice beers? Next time you’re in a new town and looking to grab a cold one just say “Ok Google, show me pubs near my hotel” and find your new favorite haunt.


by Emily Wood (noreply@blogger.com) at October 17, 2014 02:36 PM


Administered by Joe. Content copyright by their respective authors.