Blogging Notes

It's really sad to learn that spammers and other bottom-feeders are so prevalent. Even in the blogging world, you can't really go purely by your analytics to determine readership. For instance, the biggest two regular readers of this blog are unwanted. Basically spammers. adsense and vampirestats. Please don't look them up.

If you should blog, or just are generally interested, you might want to read further about them here:

excerpt:

If you have a blog and check to see where the traffic to your blog originates, you may notice traffic from websites called Vampirestat.com, Adsensewatchdog.com, Villainstat.com, Uglystat.com and/or Zombiestat.com. First off, don't click on them to find out why they are sending you traffic.

Neither Adsensewatchdog, nor any of these others have anything whatsoever to do with Google or Google AdSense and are essentially spam sites that use automated traffic to blogs to attract clicks to their own sites from blog owners such as you. Once you're at their site, at a minimum, you'll be fed ads. 

I Remain,

TheHackerCIO

Throwing A Monkey Into the Works

Not a MonkeyWrench.

But throwing either into the works isn't the normal way to treat a production system.

But here is yet another example of the need to check your assumptions in the face of technological change.

Netflix did just that when they created the Chaos Monkey. It was released to the world at large a little over a year ago, in 2012.  It's now available, along with several other variants on github, as the SimianArmy. The Chaos monkey is a service that randomly seeks out AWS ASGs (Autoscaling groups), then finds virtual instances in them and kills them, thus applying the ultimate acid test of your failover capability.

The idea is to fail early, fail often.

And to see how your failover works in this context.

The Chaos Monkey can be configured to run during normal working hours, so that the problems which may result from its chaos can be addressed by the staff during regular hours, rather than at 3am.

And, in fact, the reason for throwing this monkey into the works is to avoid those 3am calls. And to get used to planning for failure.

This is where the re-thinking comes into play. The new environment for technology is one which derives from enormous data management, not to mention millions or even billions of users, where systems need to be:

• available 24x7x365 without a maintenance window 
• flexibly scalable, ideally linearly
• based on commodity hardware, subject to failure and outage
• resilient within this context; failover should be automatic and transparent to the user

Instead of designing with the assumption of avoiding component failure at all costs, the Netflix approach says we deal with failure like a RAID system. We build using cheap commodity hardware. That's the "I" in RAID, by the way. (Inexpensive) We build in a lot of redundancy. (That's the "R"). Then, we automate failover and make it transparent to the user.  But, in this new approach, we need to know how dependable our system is.

Unless something is measured, it is an unknown quantity. The more we deal with failure on a regular basis, the more prepared we are for the unexpected. It's like regular fire drills. Or terrorism simulations. 

So here is the paradoxical need for the Chaos Monkey. The real threat is that commodity hardware is too dependable. It's so dependable that it can lull us into not properly planning for failure. But our whole approach with cheap commodity hardware  and full, 100% uptime is predicated on and assumes regular failure. Solution: produce failure regularly, but at random. 

Perfect example of putting new wine into new bottles.

You can read more about all the members of the Simian Army here. 

And just last month, the released the Cassandra Chaos Monkey, so that NoSql database instances can experience the same Chaos as your other tiers. The announcement is here. 

I Remain,

TheHackerCIO

An Evening's Evangelism

Last night was spent playing hookey from the Geeky Book club. But only because a particularly special speaker was in town. Patrick McFadin, chief Evangelist for Apache Cassandra was speaking at DreamWorks in Glendale.

So, TheHackerCIO slogged through an hour and a half of LA traffic to get out to Glendale in time to see the talk. Not to mention hearing it.

Patrick is a good presenter, so the talk was well organized and interesting. His purpose was to convince us that C* [the semi-official abbreviation for Cassandra] was the best persistence tier for your application.

He predicated this on the tunable consistency available in C*; pointing out that if you were willing to specify ALL, and take the performance hit, you could construct the most consistent distributed database system possible. One where every node had to acknowledge before an operation completed.

The talk was too long to go too in-depth, but I was particularly interested by the architecture of writing all files out immutably. Even compaction is accomplished by reading in the fragmented files and writing a new compressed one. So, in theory, you could always recover -- even from programmatic database corruption. Ideally, you use a snapshot to do point-in-time recovery, followed by writing a script to extract "post-point-in-time" updates from the files and apply it where required.  

He mentioned that the joke among C* cognoscenti is that CQL has a UPSERT statement, because update and insert are so very similar. If a row doesn't exist, update will insert it and if it exists insert will replace the data in it! UPSERT is a fun way to remember this similarity of statements.

Patrick also pointed out that Netflix -- the poster boy for C* -- has just released the Chaos Monkey for C*! He challenged the Mainframe person attending to introduce the Chaos Monkey to the Mainframe systems, and see how they compare in terms of failover and availability.  If you don't know about the Chaos monkey, tomorrow I'll fill you in on it. Because it's important.

To summarize his talk, I liked his zinger the best: Use Oracle to count your money; Use Cassandra to make it.

I Remain,

TheHackerCIO

Reactive Programming

It's been recommended that I take a look at a Coursera course on Functional Reactive Programming. I wish I had time to add something more into the mix, because the reactive programming seems to mesh well with Functional programming.

To handle this class, I'd need an extra thirty hours a week in my schedule. I'd need to fulfill the prerequisite by learning Scala, as a functional programming language, rather than the Clojure I've been working on. So, it's not likely to happen.

But the idea of reactive programming is fairly simple to get an intuition of. It's the model used by spreadsheets. When a cell changes, all the calculations are updated.

This fits in well with the Functional Programming paradigm, where functions are composed to effect solutions rather than step-by-step imperative algorithm development. It's definitely worth a closer look.

To learn more about reactive programming, read the Reactive Manifesto.

I Remain,

TheHackerCIO

Is Management Structure Even Necessary?

How about a different "model" for management? Perhaps the Polaroid corporation management model? To challenge the conventional WhizDumb & perhaps even your assumptions about setting up a standard management structure and lots of process, consider the success of Polaroid, in creating the 60-second instant camera. To quote from an article that frequently compares Edwin Land with Steve Jobs:

There was no managerial structure supervising the diverse groups involved. There were no written specifications that had to be accomplished. There was never any scheduled plan for when any task had to be completed. Yet one person, knowledgeable in every field involved, orchestrated this endeavor by challenging the available technology and the ingenuity of the many persons involved and expanding the boundaries of both. That person was Dr. Edwin H. Land.                                        [ref: here; viz. 1994 Optics & Photonics News]

Let's break this down point by point:

• no managerial structure
• no supervision structure
• no required written specifications
• no schedule
• no plan

Can you imagine such a working environment? I doubt it. It requires a Land. It also represents the opposite project management approach to that of the Big 4, where "Methodology" and "Process" are co-Regents, and guarantee automatic success, except of course, when the project ends up in cost and time over-runs, cancellation, failure, and litigation. Somehow I've never seen those last 5 process "pathways" explored, or even described in the Method/1 documentation, although almost every project I've seen them work on has ended up traversing them. 

TheHackerCIO will leave the implications of this as an exercise for the reader. A sort of meditation exercise. But the point TheHackerCIO takes away from this is that all the apparatus of the Big-8-6-5-4/MBAhole establishment: process, managerial structure, specification, scheduling, planning, managers who know nothing about anything but management, and, of course, the Litigation Department as a Profit Center -- all of that is not a necessary condition for success. 

And it's interesting to ponder: is a lone innovator orchestrating an entire project, pushing others to achieve what they thought was impossible, is that the ideal model for innovation? Given the extraordinary achievements of both Land and Jobs under this model, is certainly enough to make one consider it.