by

Essential Skill for Enterprise Software Development – Turn Uncertainty into Certainty

In most software development work at the office, I exercise a skill, which I can only think of as ‘turn uncertainty into certainty’.

Uncertainty

Some months ago, I was one of the ‘architects‘ on a project. The project lead told us that this was an ‘assessment’ project. We were not sure what that required us to do. We asked the powers that be. Their answers did not remove the confusion.

After a bit, this much became clear; we had to discover information that would help the client approve or reject the project. In addition, the project lead also wanted these; a Business Requirements Document (BRD), an Architecture Document, a Test Strategy Document, a Cut-over Strategy Document. What information these documents must contain, always seems vague to me.

Our business domain is vast. The BRD can get pretty detailed. How much detail do you need for a yay or nay decision? The analysts did not know. The project lead’s answers did not help much.

Regardless, we had to deliver something. Our jobs were on the line. My project lead needed something that the client could accept. Her job was on the line. The client needed something that was more than they already knew. Otherwise why engage us at all?

Certainty

What do you do? I sort of knew what to do. I had been to this rodeo before.

  • Give your lead something quick. It must be clear enough to let her immediately know if she can use it. Take the feedback into account, and make another small, quick experiment. Continue that till you arrive at something adequate.
  • Accept the burden of research and discovery yourself. Present the information such that the audience only has to say yes, or no. Don’t keep asking, what do you need. Instead, show what you have produced, and ask, does this work for you. Give folks something concrete to react to.
  • Keep the frequency of this interaction high. She has to see you digging, and adjusting. That is good for morale, all around. And if you go off on a tangent, both of you have to discover this fast.
  • Finally, you have to be ready to make a mistake. Yea, the work takes some courage. If you are lucky, your organization will make it safe to learn by trial and error. If not, well, who told you this was easy?

I know the types of information that characterize software solutions to a business problem.

  • Essential business requirement.
  • Specification of a solution to that essential business requirement. Broadly, you can often describe the solution in two different ways
    • A technology agnostic description
    • A technology-centric description

I realized that this conversation related to the technology-centric specification of the solution. So I studied the matter in question till I was able to put together a couple of technology-centric views of the solution.

  • A system-centric specification of the business process, in BPMN 2.0
  • A UML Component Diagram, which laid out software components that folks readily recognized; this also laid out how the components depended on each other.

As luck would have it, both the project lead and the client found this useful. This became the Architecture document.

Progress

I was still not certain that I had enough information for the ‘assessment’. I would wager that at that moment, neither were my project lead nor the client. It did not matter though. They saw that I knew the material that I presented. We went from uncertainty to some certainty. And they decided that if I did it once, I could do it again. That was progress. Onward.

by

Architect in a COTS heavy project

Does an architect have a place in a COTS software implementation? I was part of one in the recent past. It was confusing, difficult, and eventually, instructive.  

No Choices To Make

Architecture, as I understood it, seemed to be already in place. 

My operative view of architecture is Grady Booch’s notion.

To paraphrase, a solution to a business problem is a collection of choices. Some of these choices are expensive and risky to undo. These choices are architecture.

When the project started, the most consequential decision was already made. The bulk of our business runs on a massive COTS application. This COTS software has gaps, which we have filled with home grown software. Now, we were going to replace one of these homegrown solutions with a new COTS module.

Further, the vendor already documented hardware and software requirements for the COTS module. There are no choices to make there either.

If there are no choices to make, where is the need for an architect?

Kept Away From Business Domain

At the very beginning, they told me that business domain fell outside my lane. They excluded me from conversations related to business requirements. This confused me. See, my view of software engineering includes knowledge of the business.

I am an engineer. I devise, construct, and deliver solutions for business problems. At its simplest, this work has two prerequisites. I must know the business problem, and I must know the tools with which I will solve the business problem.

By the way, these tools include human beings. To put it another way, if you don’t know what makes people tick, you cannot be an effective engineer.

Architecture is a category of choices that make up the solution to the business problem. How can I make those choices, any choices, if I don’t know the business?

From that perspective, I could not fathom turning a blind eye to the business domain.

They wanted a machine honcho

By and by, I began to see what they wanted from me, the ‘architect’.

See, we have a business to run. People offer to sell us machines that will help run the business.

The vendor says that machine can perform business activities, X, Y, and Z.

  • You can send out Batch Enrollment Queries to CMS.
  • You can experiment with different rating rules, before committing to one.
  • You can communicate with customers via Email, Twitter, and Instagram
  • And so on …

Business folks verify the business promises that the vendor pitches to us. I, as an ‘architect’, am not expected to contribute to that effort.

The vendor also says the machine has some technical characteristics, P, Q, R.

  • This lawn mower runs only on premium unleaded gas.
  • Any certified J2EE application server can host this application.
  • Needs an IBM MQ Client installed on the same computer
  • The web pages will only work on the Safari browser
  • And so on ….

These attributes are agnostic to the business purpose that the software serves. Business folks could not care less about this stuff. Business folks lack the expertise to evaluate these properties of the machine. Someone has to do it. Enter, the ‘architect’.

The ‘architect’ discovers all relevant, technical, business-agnostic, characteristics of the new machine. The ‘architect’ evaluates the implications, cost, risk posed by this machine. Is the machine an easy fit into the technical environment in place currently? Does the machine conform to some grand enterprise strategy? The ‘architect’ must supply answers of that kind.

  • We are a .NET shop. The new machine is a Java web app. We do not have in-house expertise in managing a Java app.
  • This software needs 24 GB of RAM in Production.
  • We use Chrome. Chrome and Safari use the same rendering engine. Web apps that work on Safari will work on Chrome. So the browser will not be a roadblock.
  • And so on …

Not an exercise in solution design

They wanted someone that knew the machines well. They wanted someone that was familiar with the ecosystem of machines; current, and future.

They did not want someone that was well-versed in the art and practice of designing solutions. Sadly for me, there lay the sweet end of architecture.

by

Interview Question – Sort the common sorting algorithms based on Big O analysis

This interview question will make me laugh. The Big O notation for sorting algorithms indeed!

Short answer; I have been a generalist application programmer for going on 20 years. In all those years, I have never had to look up a sorting algorithm’s Big O score.

If I were a systems programmer, working on a compiler, or a DBMS, I presume I would use that knowledge. Am I producing consumer software that millions of users access at the same time? Are you processing tons of data, as in social media? We may need knowledge of sorting algorithms. Build or maintain in-house business software that 300 CSRs use, on a good day? At an insurance company that is only marginally more agile than the DMV in Zootopia? Big O notation is a faint echo.

My long answer; I can tell you how I approach performance, and load handling issues, if you wish. There is much I can do before I am forced to dredge my college memories for Big O balderdash.

At Business Analysis, and Solution Design

Business says, build this. Or business says, we have a problem, do something. Parse business’ lament, and you’ll find two categories of information. Essential business requirement, and a solution that satisfies the requirement. Further, we can view the solution from two perspectives; technology agnostic, and technology-centric.

Nailing down the work in the above manner will reveal performance problems. By the same token, lack of this analysis, can make you miss performance problems.

There was once a web app. The user would push a button, which initiated an action on a bucket of 100K documents. More often than not, the action took too long to complete, and the web page would time out.

Much unhappiness ensued. Smart people tweaked the application code, and the database. It was not enough.

Finally, someone asked the right question. Shouldn’t long lived actions be asyncrhonous? The user should not have to wait at that web page till we completed the work. The system should tell the user, thanks for the work, you can go now, we’ll let you know when we finish.

Big O analysis did not raise that question.

Do you know what did? Knowing what business purpose we were trying to serve, so we could find an alternative way to get there. Courage to point out the error in system design to senior folk who allowed the error in the first place. You could also consider it, caring enough about the work to speak up. Earning the trust of the business folks and your peers in tech, so they are willing to lend you their ear. Finally some technical skills in the User Experience, and Asynchronous Processing areas.

Loose Coupling or Bust

Next, in so far as there is room for it, ensure that the parts that comprise your solution are loosely coupled. In plan English, I should be able to tinker with one part without affecting any other part. Each part must know as little as possible about the other parts. Make this happen at both large-grained and fine-grained levels.

Dave Thomas, Pragmatic Programmer, on loose coupling.

At the fine grained level, this typically puts me in Bob Martin’s Clean Code territory.

At the large grained level (aka ‘architecture’), I want to make architecture disappear. I am referring to Grady Booch’s definition of architecture.

Grady Booch's definition of Software Architecture

To paraphrase, architecture is those choices that are expensive to change. All our choices must be inexpensive to change.

Construct, Measure, Improve

Next construct the solution.

Now measure performance. Throw load at the system. Locate the bottlenecks. Remove the bottlenecks

If we got the design, and construction right, it should be easy to locate the bottlenecks. Furthermore, we can remove the inefficiency with minimal impact to the rest of the system. Remember, the parts of our solution are loosely coupled.

Improving Performance, Without the Big O

Look at I/O

About performance itself, I know one broad truth. The delays caused by disk based I/O, and network latency overwhelms anything you might do in memory.

Here are three examples from my recent experience.


Several activities in our QA environment were crawling. We noticed that log files were huge. We were at a very fine level in QA. We raised the logging level, which wrote fewer logs to disk. QA sped up to acceptable levels.


One activity was slower than acceptable in production. We discovered that we were making close to 1500 trips to the DB. We consolidated the query into 3 trips to the DB. We were good to go.


Last, we have a batch framework. It breaks a large task into small chunks, and puts the chunks on a queue. Listeners pull the chunks off the queue, and work on them in their own time. Some tasks took forever to complete, seized up even. We found that the queue choked when it received a lot of messages very fast. 750K records broken into 7500 chunks of a 100 records each, ran for 15 hours once. Increase the chunk to 20000 records, which put about 38 chunks on the queue. This worked like a charm.

In each of the above cases, I did not need Big O analysis. Someone built the Logging Framework that we used. And the DB. And the Messaging Platform. That person needed the Big O.

Learn the machine

At some point, we will need to fine tune a machine; a DB, a JVM, a CLR, a Messaging Server. For this you need in-depth specialized knowledge of the particular machine. I don’t have that. Remember, I am a generalist. I know what questions to ask. I can dig up answers. And then forget, because I don’t use that knowledge everyday.

More to this than we can see

Last, at this level, there are often a lot of variables, which interact in ways that is not always easy to predict. For instance, there are tons of JVM parameters, and DB switches. You do what you need to do to get past the current problem. Then you watch how things go, till you have to intervene again.

This is what I know about performance.

See, I don’t remember enough Big O analysis to save my life. Yet, life goes on.


by

Business Agnostic Software – Buy rather than Build

A solution that you can apply to any business at all, is what I might call business agnostic software. If you feel the urge to build such software, think again.  Buy it instead.

Examples of business agnostic software

An insurance company maintains correspondence with the insured.  The company buy’s or subscribes to an email service.  It does not build the email service from scratch.

A bank generates statements, and various notices that the government mandates.  For this, a bank might need a template engine that generates PDF.  Buy that template engine.  The bank itself will have to build the templates for these business documents.

All companies have business processes that they wish to start, track, execute, and end.  This is what BPM (Business Process Management) engines are for.  Buy that BPM engine.  The business can implement its business processes using the BPM engine it buys.

Health insurance companies that are in the Medicare business, receive beneficiary information from the government.  Many rules determine how the company responds to this information.  A company could use a rule engine to manage these rules.  Buy that rule engine.   The company should stick to only implementing the rules.

Wait, here is the simplest example.   Why do you buy a database management system?   You don’t build an alternative to SQL Server, or Oracle, do you?

Why avoid building business agnostic software

The primary purpose of an Enterprise IT shop is to support the business. Learning the business, analyzing it, modeling it, and implementing it, is a significant challenge in itself.  Focus on that.  Master that before taking on other problems.

Business agnostic software focus on more general problems.

Take rule engines for instance.

  • What is a rule?
  • How can rules be combined to produce new truth?
  • Devise an algorithm that can execute 10,000 rules in a few seconds.

An insurance company could use a machine that answers those questions.  Does a health insurance company want to spend time and money on building that machine?  For most companies there will be little value in it.

People spend their lifetimes studying these fundamental questions.  There are PHds galore in rule systems.  Let them build the rule engine. You confine yourself to identifying an effective rule engine, and using it well.

We do build business agnostic software though

I am surprised by how often Enterprise IT shops end up building business agnostic tools.

In my work experience, I have seen home grown solutions to these problems, which are best left to the experts.

  • A web framework
  • An Service Locator/IOC container
  • A rule engine
  • A BPM engine
  • A scheduler

We can never avoid this completely.  Start with management that lacks engineering savvy, throw in gung-ho developers, and mix with institutional inertia.  Less than optimal decisions will happen. So it goes.

Exception

Some companies do focus on fundamental problems.

They have deep pockets.  They have the means to invest in fundamental research.  Their business model, their operating ethos, includes such effort.

Their business problems are of such scale and complexity that currently available solutions are inadequate.  They must invent new solutions themselves.

Unix and C came out of  AT&T.  Ericcson created Erlang.   UML came out of research at GE, and IBM, among other places.  Xerox pioneered Graphical User Interfaces, and Object-Oriented Programming.

Ask yourself.  Are you one of these companies? I am confident most Enterprise IT shops would say, no.

by

Nobody told me I had to be a teacher

A Tech Lead has to be a teacher? Why didn’t someone tell me?

We have this C# method. It used to be good. Small. 25 lines or less. All at one abstraction level. See Bob Martin.

One of the kids, from offshore as it happens, had a bright idea. Next morning I wake up to find that the method is now a 100 lines.

I want to tell the kids. The method is too large. No method should be more than 25 lines. And they will chop it down pronto. They are good kids. But. There is no telling how they will chop it down. If they know how to tell a story in simple, clear steps, I have not seen evidence of it yet. They can’t seem to find the shortest, most direct route, between A and B.

Ask them how to go from D.C. to Baltimore.

They will come up with this. I-495 from National Airport to I-270 and Frederick. Then take I-70 to Breezewood. Break East on the PA turnpike to Harrisburg. Finally, come down I-83 to Baltimore.

That was fun. We got to write a lot of code. Oh what a lovely route it was.

They won’t stop, look around, and find 295, the Baltimore Washington Parkway, a straight shot between the hearts of D.C and Baltimore.

So I have to teach them how to chop the method down. But there is a problem. I know how to do the work. I don’t know how to teach it to another person.

I can say, make sure all of the code is at a single level of abstraction. They have no idea what I am saying. I don’t blame them. What does the word ‘abstraction’ even mean? I don’t know how to describe it man, I just know it when I see it. You know the button that starts your car. That’s an abstraction. You know what I mean? You don’t? Well, bloody hell.

So now on to Plan C. I redo the method myself. You know, I give them advice, and an example. Guess what happens. Nothing. Two days later the silly drama repeats itself.

It turns out, they could not care less about Clean Code. They know the programming language. They know some libraries. They are decent at problem solving. They get a kick out of flipping switches and seeing results. They are having fun. They feel no inclination to examine what they are doing. Stop, step back, dig deep, see under the surface, unlearn old habits, cultivate new habits. Further, after a couple of Sprints, they also learn that I will clean up the code myself.

It took me a long time to understand how to write Clean Code. It took me much practice to do it instinctively. Do you know how much I rewrite? How am I going to get the kids to adopt a regimen of study and practice? How am I going to get them to show interest and sustain it? How am I going to get them to care?

I am only a working engineer. You are asking me to not only teach, but also to motivate.

All of this, I have to do while a project is going on; under the gun, to deliver something.

It is not going to happen.

You are not going to get Clean Code.

Unless. You hire the right people.

by

Focus on skills rather than role

The project manager said, that is not your role, stay in your lane. I held my tongue. But it struck me that I no longer understand the fixation on roles.

While developing software, we must make certain choices. Some of these choices are expensive to change if you get them wrong. These choices constitute architecture. See Grady Booch. Architectural decisions must happen. How does it matter who makes these decisions, as long as they know what they are doing? We need Architecture. We don’t need Architects.

The same argument applies to other work. We need business analysis, we don’t need business analysts. We need testing, we don’t need testers. You catch the drift.

If one person can do business analysis, and solution design, why not let them do it? If one person can write code and test, why not let them do it? With each fewer person in the team, you avoid an information hop. With each fewer information hop, you avoid some information loss.

Here is another argument. A designer must know the business that she creates solutions for. Why not let her learn the business first hand? Let her do the business analysis. A developer, makes better choices in the code, when he knows the essential business. Why not let the developer talk to the business? If the developer has the necessary skills, drop the middle man. Knowing what Agile has taught us, do you still want a developer that cannot test? Let the developers test; eliminate a whole moving part in pipeline.

If one person can do a job, why would you want three? If three people can do a job, why would you want six?








by

Don’t make users learn codes in place of business terms

Here is a good example of bad user experience I came across recently.

In training, I was told the various status codes that health claims would be in.

  • 125 for a pending claim.
  • 483 for a paid claim.
  • And so on.

I was also told codes that indicate the results of business jobs

  • 0 means a claim has been priced completely.
  • 58 means claim has an unknown provider.
  • 26, 27,28 are violations of Wyoming state laws regarding Medicaid claims for black lung disease.
  • And so on.

Then they tested us on these codes.  I did poorly on the test. Who can remember all these damn codes?

So, I have a question.

Why can’t a pending claim simply be marked ‘pending claim’? Why must you come you convert that plain English status to a number?

If there are technical reasons for representing that status as a number, fine, do so.  However, why must I, the business user, be exposed to that number? Why can’t you keep the number to yourself, and tell me the claim’s status in plain English?

Why are you making me remember arbitrary numbers, when natural and easy to understand business terminology exist?

And last but not least, if the business user did not see these numbers, you can avoid a whole chunk of training.   Is that anything but a good thing?

This is an example of “Design is an act of communication“.

This is user experience.

by

Architecture I believe, architects not so much

So I am part of the Enterprise Architecture group now.  That makes me laugh.  In my admittedly narrow experience, architects have never seemed relevant to my responsibilities.   That is part of why I took the job.  See if I could figure out what this is all about.

Engineer

I have come to think of myself as an engineer.  I build things that people use.   Point me to a business problem.   I can find my way from the problem to a solution, soup to nuts.  Part of this work is architecture, the decisions that are expensive to change.   The work also includes business analysis, and solution design (tech-agnostic, and tech-centric).  Then there is construction of the solution – coding, and testing.  Last but not least, deploy the solution, and support it.  Drive this whole ship forward with agility.

On a day to day basis, my job is to do whatever needs done to get work out the door.  Every day finds me doing this, that, or all the work mentioned above.

Architect

I have been a working software engineer in several different environments.   The architects that I have come across in that journey have been of little help.

  • Have architects had knowledge that I did not? Not that I ever saw.  Anything they told me, I could pick up off the internet in a couple of hours.  Often, a little due diligence on my part would reveal the weakness in their choices.
  • Had the architects created solutions, which I could use, as is, to do my work?  In my personal experience, no. Many of them considered themselves above what they called, ‘development‘.
  • Did the architects have skills that I could borrow?  Could I say, look, I have a problem, can you solve it for me, or can you teach me to solve it?  This never happened either.

So, why take the job

There has to be more to being an architect than I have seen.  If that is true, I would like to find out what I have missed.

Architects seem to have more access to both business and executive IT management.  It would be useful to see the world from their point of view.

An engineer solves business problems.   The architect role gets me closer to the essential business.  Engineering starts there.  Now, fewer people will  be surprised when I ask, wait, what business purpose does this serve; why must we do this.

In any event, the die is cast.  Let’s see how it rolls.

 

by

Build vs Buy – asking for an engineer

When a build vs buy discussion comes up, a few questions start to rattle around my head.  I suspect this is the engineering side of my brain acting up.

Skills you don’t want

Supporting a business with a computing solution is an intricate dance between several players.

It is analysis of the business problem, large and small-grained design of a solution, construction, testing and delivery of the solution, all enabled by effective communication between people, agile planning and execution.

Which of the above work are you expecting to avoid by buying rather than building?

What skills will no longer be necessary, when you buy instead of build?

A business machine

A person buys insurance from us.  Over the years, as that person and we do business, a lot of information accumulates.   This information is in various media.

  • Paper that we send back and forth
  • Electronic text that we send back and forth
  • Still pictures
  • Moving pictures – video
  • Audio recording

To run our business, we must be able to do the following.

  • This information must reside somewhere.
  • We want to be able to search for stuff in this information.
  • We want to view this information when necessary.

Now, consider a machine that promises this.

  • Here is a box.  The box has a hole.  Drop your information in this hole.
    • Anyone can do this.   My 3 year old niece can do this.
      • This is your information repository.
  • The box has a screen, and a keyboard.  When you want to search for something, ask the question on the screen.  We’ll find and show you what we think might answer your question.
    • Anyone that knows English and your business can do it.
      • This is your search
  • When you decide what you want to look at, the box will bring up that material to you, as you put it in the box.  It will come out of a second hole.
    • This is your view.

If this machine existed, I would buy it.

The interesting question is this. Look at the skills in play.  You only need people that know the business.  And you need the cash to buy the machine.  A guy with a trolley to wheel the machine in would also be useful.  You need none of the drama that we think of as IT.

What skills do you want to acquire, maintain, and manage.  There is your build vs buy decision.

Mistakes you want to avoid

What mistakes, errors, failures will buying avoid?

What mistakes are you looking to avoid?

The analyst gave the developer a table of numbers, which affects a customer’s bill.  The developer discovered that the industry specifies a formula, which generates those numbers.   He used the formula and generated the table himself.   He found that his table and the analyst’s table were different.

The developer asked the analyst to review the tables.   The analyst said she did, and the table was fine.  She was talking about her table.  The developer thought she meant his table.   The developer implemented his table.

The developer had made a subtle error in floating point calculation.    His numbers were wrong.   This led to a small increase in the bill that customers got.  We got wind of it only when the customer complained.

The developer and the analyst built something wrong.  What can you buy that will make their work, and associated error, unnecessary?

Communication problem

When two people collaborate, they will misunderstand each other sometimes.   You can avoid that ubiquitous human snafu only if they never have to communicate with each other.   What can you buy that will accomplish that?

What makes a man a man

We speak to people within and without the organization.   We use phones.  We buy phones.  We don’t build them from scratch.

We communicate via written word.   We buy paper, and pens.  We don’t make paper.   We buy a word processor.  We don’t build a word processor.

We want to maintain a correspondence with people.   We buy or subscribe to an email service.  We don’t build an email server and client from scratch.

We built a web application that accepts a beneficiary’s enrollment application.  We did not buy this application.

What makes the word processor and that enrollment web application different, which caused us to buy one, but build the other?

by

Developers must know the business domain – I

Knowledge of the business domain is essential to developers.  They must know the business as well as the business folks know it.

We have all worked in environments where the business analyst’s word is gospel.  The analyst specifies a solution, and the developer translates it verbatim into code.

This approach to software development is less then optimal.

Blind football

Imagine sending a running back onto the field blindfolded.  We tell him, never mind that you cannot see a thing.  The analyst will relay  instructions to your ear.  Follow them to the letter and Bob’s your uncle.

  • Run 6 yards on a 37 degree angle to the left.  Your left, not mine.
  • Now tack right, 82 degrees, and run 15 yards at a speed of 26 miles per hour.
  • Wait, 300 pound linebacker at 7 ‘o’ clock.  Wheel, dummy, wheel.
  • Uh oh, too late.  No, wait.  The linebacker got tripped by his own cornerback!

See, the defense has blindfolds on too.  In fact, all 22 players on the field are blind as bats.  They are unseeing puppets, lumbering about, whose strings the business analyst is pulling.  Don’t you think there will be stumbles?  You bet there will be.  The business analysts have to be superhuman to get everything right all the time.  Your team must loose the blindfolds.

Pre-empt the bug

Business folks and analysts are as fallible as the next person.  There will be holes in the information they provide.  These gaps can cause  errors.  Developers must recognize and fill those gaps so that they can avoid those errors.  The alternative is to let the error happen, hope that someone catches it, then fix the error.   How can this ever be better than avoiding the error in the first place?