May 26, 2010

Don’t SPOF me, Bro’!

Posted in Financial Markets, Foolishness, Social Commentary tagged , , , at 4:49 pm by Doug Brockway

A SPOF, or “single point of failure,” is a part of a system which, if it fails, will stop the entire system from working. They are undesirable in any system whose goal is high availability, be it a network, software application or other industrial system.

Sometimes high availability is desirable as every minute a system is down money is lost.  If an airline’s reservation system is down they still own the planes but they can’t sell the tickets.  Sometimes high availability is desirable because having the system fail also means significant additional costs.  The owners of Three Mile Island and BP both understand this all too well.

Some in management understand the dangers and the costs of SPOFs quite well.  This brings us to two stories about Edward Crosby “Ned” Johnson 3rd, the CEO of Fidelity Investments.  Johnson’s reputation (I’ve never met him) is that of a particularly hands-on, operationally knowledgeable person.  In the early 1980’s Fidelity used to have its main data centers in Boston.  Ned was told by the manager of a data center there that it was secure, that no one, not even Ned, could get unauthorized access.  The story goes that Ned climbed over the computer window desk and into the data center.  Having made his point security was upgraded and over the next few years the data centers were moved out of the city center.

Not too many years later Fidelity established a facility in the Dallas area, including a data center.  Mainframe computers of the time, and for the most part now, were cooled by the circulation of water through pipes inside the machines.  On a tour of his new data center Johnson is reputed to have asked to see the back-up to the water supply.  Having seen that he asked to see the back-up to the back-up.  There was none.  Ned asked how much water was involved.  Given a number he concluded it was about the volume one has in a reasonable swimming pool so a pool was constructed on-site, for employee and family use, attached by pipes to the back-up to the water supply.

I don’t have the data on the volumes of trades or dollars that Fidelity was managing to in those times.  Suffice it to say that as the dominant, the largest mutual fund company, trade and dollar volumes were, and remain, quite high.  Johnson knew the near term value of lost transactions and the long-term reputational risk and revenue risk if his customers came to believe Fidelity was not a reliable partner.

He likely used different words but his message to management was, “Don’t [SPOF] me, Bro’!”

Wouldst that the management of BP took the same view in the Gulf of Mexico.  As I write they are attempting the “top kill” method to stop the gushing oil leak that has plagued us for over a month.  They had a controller on the blow-out preventer fail and no plan to deal with it.  They were drilling five miles down “where no man has gone before” and no at-hand, tested solutions to outages that are reasonably expectable.  The now famous Minerals Management Service blithely gave BP authority to go ahead without the plans and reports that are designed to prevent or control for disasters.  One can only wonder what any of these people were thinking.

One thing is clear, they weren’t thinking “Don’t [SPOF] me, Bro’!”

Causal factor of unavailability
Lack of best practice change control
Lack of best practice monitoring of the relevant components
Lack of best practice requirements and procurement
Lack of best practice operations
Lack of best practice avoidance of network/system failures
Lack of best practice avoidance of internal application failures
Lack of best practice avoidance of external services that fail
Lack of best practice physical environment
Lack of best practice network/system redundancy
Lack of best practice technical solution of backup
Lack of best practice process solution of backup
Lack of best practice physical location
Lack of best practice infrastructure redundancy
Lack of best practice storage architecture redundancy

May 25, 2010

The Quality of Loans is not Strained

Posted in Financial Markets, Social Commentary tagged , , at 3:26 pm by Doug Brockway

I did something fairly “wonkish” today. I read a newly published document from Fannie Mae titled “Lender Letter LL-2010-03, An Introduction to Fannie Mae’s Loan Quality Initiative” (LQI). As stated in the document’s introduction:

“Historically, many issues related to compliance with Fannie Mae selling policies are not detected until after loans are delinquent or through the foreclosure process. Loan repurchase requests to lenders have increased in the past three years, highlighting the need for an improved approach for working with lenders to deliver loans that meet Fannie Mae’s underwriting and eligibility guidelines. Fannie Mae conducted an extensive analysis to determine the primary drivers of repurchase requests and is launching the Loan Quality Initiative (LQI) to identify and implement policy, process, and technology enhancements to improve the compliance with underwriting and eligibility guidelines and mitigate repurchase risk.”

For loan investors (Fannie in this case) and their counterparties (mostly mortgage bankers, conduits and securitizers) the big issue is so-called “repurchase risk.” If the information submitted to Fannie Mae for their approval of a loan is incorrect or inaccurate than in many cases the banker must buy back that loan from Fannie Mae.

Some important things to know:

  1. Most mortgage bankers take out loans from large institutions and use those borrowed dollars to fund loans at your closing. They then turn around and sell the loan, often to Fannie Mae, and generate the cash to do it again,
  2. The loans they must repurchase are by definition, by “reputation” tainted whether there is actual fraud or error. The inaccuracy in the loan application may be immaterial to actual risk but the repurchase must happen in any case
  3. Since the loans are tainted the mortgage bank won’t be able to sell them to a third party.
  4. Mortgage banks don’t have the cash lying around to buy whole loans so the repurchase cash comes out of profits.

Many mortgage bankers were caught in this squeeze when investors stopped buying sub-prime mortgages at all. According to Implode-o-meter, “since late 2006 383 major U.S. lending operations have “imploded.””

Now, many face the prospect of going out of business because the loans they wrote in recent years have problems. Sometimes the issue is outright fraud, sometimes a mis-calculation of a formula, sometimes a missing document. The reason for the LQI is to reduce this activity in the future and create a more stable system, and that’s a good thing. That will work itself out for investors and bankers, for Fannie Mae, Freddie Mac and the rest.

What I find “curious” is the nature of the holes that the Loan Quality Initiative is aiming to fix. Here’s some of the issues they’re tackling:

  • Confirmation of Borrower Identity and Occupancy
  • Validation of Qualified Parties, and Borrower Credit Profile
  • Confirmation of Borrower Occupancy
  • Identification of Property Unit Number
  • Loan Delivery Enhancements
  • Validation of Loan Eligibility at Delivery

Reading down the list, with these, and other updates Fannie will only fund loans to people who actually exist, they’ll only do it with brokers and banks who aren’t on lists of crooks and incompetents, if you say you’re going to live in a house they’ll really be sure,… really, if it’s a condo they’ll make the loan on the right condo in the building or complex, and the loan that they fund will actually be a qualifying loan.

All systems and procedures have error rates. After all of the LQI initiatives are in place we’ll probably still be concerned about borrower identity and the rest. Still and all, it’s a sobering list. In my interactions with the mortgage industry as a customer the rules and the controls were paramount. At your closing you sign and sign and sign, document after document.

We were told that the mortgage process was solid. It wasn’t. We were told that BP understood deep sea drilling. They don’t. Many people are evoking the name of George Bailey from “It’s a Beautiful Life” lately. I often feel a bit more like another Jimmy Stewart character, Elwood P. Dowd….

May 13, 2010

Rock Icons and IT Thought Leadership

Posted in Foolishness tagged , , at 1:16 pm by Doug Brockway

Mechanical calculators or computers date at least as far back as the 150-100 BC with the creation of the Antikythera mechanism.  In more modern times Charles Babbage created his mechanical difference engine in the 1800’s. John Von Neumann outlined the architecture of modern computers in the first half of the 20th Century based in part on ENIAC, a military computer developed by J. Presper Eckert and John Mauchly.  They then made it generally practical with the first commercial computer, the UNIVAC-1Admiral Grace Hopper made that invention accessible in business by inventing COBOL, COmmon Business Oriented Language, the first business move away from ones and zeros, computing for the business masses.

These relatively technical accomplishments were soon complimented by people who studied the effect of computing on business and how to manage computing in business.  In the 50’s John Diebold coined the use of the word “automation” to reflect the use of computers in this way.  Based in part on the Diffusion of Innovations concepts developed by Everett Rogers in the 60’s, in the 70’s Richard Nolan and Chuck Gibson wrote “Managing the Four Stages of EDP Growth” signaling an understanding that a business’ ability to manage computing, to manage IT, had predictable evolutionary steps.  In the 80’s Michael Hammer led the revolution in Business Process Reengineering.

These were and are the great thought leaders in IT and IT Management.  We all stand on their shoulders.  But of significant concern is where are the subsequent generations of thought leaders for how IT and business interact?  It’s been quite some time since “reengineering.”  As is often the case Rock and Roll provides the answer.

One of the first generally recognized rock icons who wrote on the role of IT in business was John Lennon.  Expressing a visionary viewpoint he wrote:

Imagine there’s no hunger,

It’s easy if you try

Computers are transformative

No need to wonder why.

As it happens these original lyrics were edited in studio.

Mick Jagger was contemporaneously writing practical observations on the yin and yang between business goals and technical visions and what can actually be done within a given time period, with a given set of IT skills and a given budget.  Influenced heavily by Fred Brooks and The Mythical Man Month Jagger famously (and repetitively) wrote:

You can’t always git what you want

You can’t always git what you want

You can’t always git what you want

But if you try sometime

You might just find

You get what you need

Jerry Garcia was a keen observer of this scene.  He was especially interested in CIOs, their careers and what it took for them to succeed.  He would immerse himself in the life of a CIO-at-a-time, chronicling their struggles, how they overcame obstacles, and the successes they settled for.  Garcia’s most telling commentary on this was from Truckin’ with the line “What a long, strange trip it’s been.”  As a side note Jerry Garcia and Tom Davenport have never been seen in the same place at the same time.

As we entered the age of outsourcing Warren Zevon described the optimal approaches to defining which services to outsource, which vendors to consider, how to choose a vendor and how to negotiate with the vendors.  This last was summed up with:

I’m the innocent bystander
Somehow I got stuck
Between the rock and the hard place
And I’m down on my luck

Send lawyers guns and money!

More recently Lady Gaga has been writing about the difficulties of outsourcing contracts gone bad with:  “Want your bad romance, caught in a bad romance. Rah-rah rah-ah-ah.  Roma roma-ma,” though the meaning of the last phrase remains unclear.

The Foo Fighters have been writing on the conflicts between reinvesting in current systems or “going greenfields” and replacing existing applications and infrastructures:  “Well we all want something ‘better than,’ we wish for something new.”

IT professionals are constantly being sold by vendors.  They’re constantly being besieged by users.  They’re constantly being reviewed by auditors and CFO’s.  Whatever certifications they have are always going out of date as the technology relentlessly marches on.  Still, IT generally succeeds.  Much is written about how IT may or may not be strategic.  The proof is in the widespread consistently improving use of technology in business.  On these points, in the song “Handle Me with Care” The Traveling Wilbury’s wrote:

I’ve been fobbed off and I’ve been fooled

I’ve been robbed and ridiculed

In data centers and night schools

Handle me with care….

I’ve been uptight and made a mess

But I’ll clean it up myself, I guess

Oh, the sweet smell of success

Handle me with care!

(originally posted at

May 12, 2010

But its legal!

Posted in Financial Markets, Social Commentary tagged , , at 10:02 am by Doug Brockway

This came across the e-mail today in a newsletter I read.  Its about the dealings of Goldman Sachs and their ilk in the mortgage market:  “It didn’t help, of course, that the rating agencies gave AAA ratings to many securities that were not AAA – even Enron carried an investment-grade rating days before it collapsed.”….

To me this is where one crux, at least, lies.  The sellers of CDO’s and buyers of related swaps overtly engineered high risk bonds to fool rating agencies’ standards and models and then used market pressure to get rating agencies to put their once good name on what may be legal but on the street corner is fraud.  The sellers invested a lot of time, days and weeks, in each security to do this and buyers, stupidly, made their decisions in minutes, not enough time to uncover the fraud.

My nephew the trader says they’re all big boys and should have known.  In “The Big Short” Michael Lewis shows time and again that when a bond trader calls you up to sell you something your first question is “how are they trying to f#%k me?”  But, as the hot market of the early 2000’s turned to the bubble of ’05-’07 we lost all honor among thieves.

Even if you say that’s OK, that it was “understood” (and I can’t get myself there, but leave that alone for a while), having that game played, and those risks taken based on those lies with post-Glass Stegal “main street” deposits, your and my bank deposits, is beyond disreputable.  When people suggest that its “legal” I think that so were the actions of King John,… technically….

May 10, 2010

Do Not Mail

Posted in Useful day-to-day tagged , , at 2:09 pm by Doug Brockway

Borrowers know that applying for a loan such as a mortgage involves the lender pulling a credit report.  What they may not know, however, is that this allows credit bureaus to sell a borrower’s information to third party vendors – which include other mortgage companies.

To stop this you can register yourself on the “Do Not Call” list for telephone calls:

BUT, in addition,  you should sign up for “Opt Out Prescreen” which will stop the four credit bureaus (Equifax, Experian, Innovas, and Trans Union) from selling their name as a trigger lead for five years. Go to

And while you’re at it, to cut down on junk mail go to

Thanks to Rob Chrisman:

May 8, 2010

Might Facebook be on the Fundamentally Wrong Path?

Posted in Facebook, Marketing 2.0, Social Media tagged , , at 2:17 pm by Doug Brockway

In early May Dan Yoder posted this article on Business Insider:  10 Reasons To Delete Your Facebook Account.  One of the key points made in the analysis is that “Facebook’s Terms Of Service state that not only do they own your data (section 2.1), but if you don’t keep it up to date and accurate (section 4.6), they can terminate your account (section 14).”

In defense of their privacy changes last January Facebook said ‘People have really gotten comfortable not only sharing more information and different kinds, but more openly and with more people. That social norm is just something that has evolved over time.’ In introducing the Open Graph API Facebook says ‘… the default is now social,’ meaning Facebook not only wants to know everything about you, and own that data, but to make it available to everybody.

“At the same time that they’re telling developers how to access your data with new APIs, they are relatively quiet about explaining the implications of that to members.”  Claims Yoder, “Since they are in the business of monetizing information about you for advertising purposes, this amounts to tricking their users into giving advertisers information about themselves.”

And lastly to my point, “At this point, all your data is shared with applications that you install. Which means now you’re not only trusting Facebook, but the application developers, too, many of whom are too small to worry much about keeping your data secure.”

There are many data privacy concerns that come to mind reading this.  I’ll add that the approach is simply wrong-headed from the point of view of marketing.

Yoder’s article made me think of a fantastic post by David Meerman Scott.  In Social media marketing explained in 61 words.  Scott writes, “

  • You can buy attention (advertising)
  • You can beg for attention from the media (PR)
  • You can bug people one at a time to get attention (sales)

Or you can earn attention by creating something interesting and valuable and then publishing it online for free: a YouTube video, a blog, a research report, photos, a Twitter stream, an ebook, a Facebook page.”

When Facebook claims ownership of our data and makes that data available to under managed and under controlled developers and  advertisers, they are using a social media medium to buy, beg and bug me for attention.  They are violating the basic tenets of social media marketing and being a world-class pain in the ass to boot.

Yoder makes other points.  He’s not technically impressed with Facebook, says it’s really hard to delete your account, that Facebook doesn’t support the Open Web, and their user interface [is sub-standard].  I certainly appreciate the last.

My guess is that Facebook will experience solid success for a while but that can, likely will, come to a screeching halt if their user-base ties the link between the “land grab” for personal data and annoying, unrelenting, old-style push marketing and advertising.

In its own way this is similar to Goldman Sachs et al claiming that they’re creating liquidity that makes life possible while at the same time taking every dime from every source they can in any manner that crosses their minds.  The concepts clash, or, as they say in Harvard Square, “that there’s one heckuva’ lotta’ cognitive dissonance.”