Posted on 11/29/2007 3:14:47 AM by Justin Etheredge
One mistake that many companies fall into is trying to completely automate certain business processes. Sure there are a lot of business processes that are just screaming to be automated, and some of them seem to have been invented just to be automated, but sometimes a bit of thought should be put into whether or not we should automate them.
A lot of programmers want to automate anything that is able to be automated, no matter how convoluted the process may be. They will jump through hoop after hoop trying to shove a business process that involves a small amount of human intuition into a complex algorithm that tries to simulate what a human would do. In some cases, especially for large companies, dealing in large volumes, this can make sense to try and write hugely complex algorithms to solve relatively simple problems since the savings are multiplied huge numbers of times.
For example, look at credit card applications. The process of reviewing a credit application can be very relative. Someone could have blemishes on their credit that are of little concern to a credit card company. I used to work a lot with credit scores years ago (when writing insurance software) and one thing I found interesting was that people will often get negative items on their credit histories because of healthcare insurance companies. What will happen is that someone will go to a doctor to have a procedure performed and the doctor will then send a bill to the insurance company. The insurance company will drag their heels on the bill while they are deciding whether or not they are going to pay or how they can get out of paying it. The doctor doesn't get paid for months and decides to send a bill to the patient. They don't care how they get their money, only that they get it. Well, the patient gets a bill for, lets say, 40,000 dollars. The patient obviously does not have this kind of money and calls their insurance carrier who just talks them in circles. The doctors office eventually gets fed up and sends the bill to collection which puts a negative mark on the person's credit. Eventually (hopefully) the insurance company relents and pays the fees, but not before damage has been done to the person's credit. This doesn't happen as much anymore as it used to, because most of the time all of this is worked out before the procedure happens, and there are better processes in place to make sure that these things don't happen.
The whole point of this though, is that for a lot of purposes these medical related negative items on your credit history are irrelevant. So, lets say a credit card company decides that they are going to devise a computer program that is going to try and approve their applications for them. They really have two options, they can devise an algorithm where they keep piling on rule upon rule upon rule in order to attempt to automate the process more and more. In this way they are cutting out the need for a human to ever look over the application, or they will devise an algorithm that covers the 90% case, and then passes on the last 10% that it cannot process to a human in order to have it reviewed. In the case of the medical negative marks you would have to either work into the system an algorithm that would selectively factor in different marks on their credit history (based on the customer pulling the score and for what purpose they are pulling it) in order to determine a credit score, or they would have to have someone at the office pulling the credit score review each credit response and try to determine if there is anything that they can safely ignore. Obviously for these credit agencies this choice is easy, they calculate thousands of credit scores per day. They would never be able to keep up if they were processing these manually, so they have invested huge amount of resources in order to completely automate every last bit of the credit scoring process.
So, what is the best option for your process?
Well, like all things in life, there is no clear answer. There are numerous factors that go into whether or not you would want to automate a process like this. In this situation, here are just a few of them:
1) Is this process able to be automated at all? Is there some part of the process that cannot be calculated and can only be decided by a human being? If so, how often will this case come up? Some processes simply cannot be automated because it requires a judgement call that does not fit into an algorithm. Other processes don't have these cases, or if they do, they can be replaced by an approximation of this judgement call that is good enough. If you replace the judgement call with a approximation, how much will it save versus the potential loss in accuracy of your decision?
2) Should we automate this process? This is a hard one for developers, because when it comes to software, everything should be automated. Leaving anything to a human should be avoided at all cost because humans make mistakes and when dealing with software there is so much minutiae. This doesn't translate to business processes though. Think about how much you hate those automated call systems that most companies have setup. They try to trap you in a web of questions in order to try and get you to not talk to a person. This person is the most likely avenue to have your issue resolved, but every second you are on the phone with them you are costing them money. So they looked at the business and they did their cost calculations and they said, if we automate the call system and we can solve 50% of problems without our customers talking to a person then we can cut our call center staff by 50%. And that is true as long as they can actually solve 50% of calls without someone talking to a person. The loss here is immeasurable though. Every time I have to call up Verizon on the phone I lose just a bit more of my patience for them. Eventually I will just start off every call screaming. The gain here is measurable, but the loss is not easily quantifiable, and so the business is likely to take the one with numbers. This leads me to my next point...
3) What is the potential loss/gain if we do not automate it? This not a strictly monetary decision as we looked at in the last number. This question can involve many factors that are very hard to quantify. The only thing that I can really say is to try and decide what values are most important to your company and go with those. If your first and foremost value is customer service, then if you are in a situation where one of your variables is a negative effect on customer service, then even though this may be hard to quantify, you need to put a large emphasis on that variable. Many companies claim to be customer oriented, and it is these kinds of decisions that really show their true colors. Hopefully the processes that you are involved in don't have a lot of politics, non-quantifiable variables, and other complications in calculating benefits and losses.
4) Will it cost more money to automate it than it does to do it manually? Programmers may cringe at that questions, because we innately want to automate everything. There has been times where I would have been willing to program replacements for all of my co-workers! :-) The real question remains though. If there is a task that only occurs once a month and it takes someone 20 minutes to do it, then how much does it cost to manually perform the task versus how much time and resources would it take to automate it? If you are a small company then other questions might come into play, such as whether you even have the resources available to automate it if you wanted to. Some tasks could be automated, and even potentially save time or money, but how long would those savings take to achieve versus how much can you afford to spend your resources now?
5) Will we lose accuracy in our decision making if we do it? Sometimes you will be in a situation where automating a process will cause you to have to make less accurate or informed choices. If automating the process will cause a huge savings in resources while only losing a small amount of accuracy then the choice is simple, but if automating a process would cause numerous poor decisions to be made then the choice might not be so simple. Especially if the decision that is being made is core to the business. So how do you make this decision? Well, you have to decide how important the decision is, how much accuracy you will lose, and how much savings you will get from automating. These numbers are of course all relative, but if one of these variables is significantly higher than the others then the decision will be fairly simple for you.
6) Will we lose our "human touch?" This decision is actually far more in depth than most people make it out to be. Some people see a process and say, yep, this can be automated and it will save us X dollars per year in labor, so lets do it. But consider this simple example, which is not really a business process per se, but it illustrates the point. An online store has a customer checking out who enters in failing credit card information. For most companies the answer is simple, display an error to the customer telling him that his credit card information failed and that he needs to try again. But for other (more thoughtful) companies, this is not a clear cut decision. Some companies would rather let through the failing card so that the merchant themselves can contact the customer to get proper payment. This way the customer cannot just "give up" on the order and abandon it. By letting the customer place the order, the merchant is securing a sale that could have potentially been abandoned. What if the failure was a temporary problem at the card issuers bank? The customer would have had little opportunity to fix the problem and would have moved on. To a small merchant, this sale might represent a good chunk of that day's earnings and therefore they would rather accept the order and spend the time to follow up, rather than potentially lose the sale. On the flip side though, a large store who does a huge volume may not see this as a good way to have their employees spend time. The way they see it, they might have an occasional lost sale, but if we have to hire an additional customer service rep to follow up on all the failed cards (most of which would probably have been fixed and resubmitted by the user) then we would lose money. So, is your business personal with your customers? If so, then this is very important, otherwise (if you were something like a military contractor) it might not be much of a factor for you.
7) Will we lose agility? Once a business process has been written into software it is much harder to change than an informal process. Especially in larger companies, changes in software are usually done by months and quarters, not days or weeks. Once you codify a business process, it may take you a long time to change it, or you may resist changing it, because you don't want to go through the hassle of changing the software. If you try to account for this, and devise a system that non-technical people can modify, how much will this cost you? Creating a system that will allow non-technical users to modify processes could add quite a bit of complication to a project. It is good though that recently with the rise of DSL's and customer modifiable workflows like the Windows Workflow Foundation, this is becoming a more surmountable task, but is still far from being an insignificant development cost.
Often decisions about whether to automate a process are not always as easy as we want them to be. Generally speaking a lot of companies will look at a process and see how much time it takes, how much effort will take to automate it, and then make a decision based on those two variables. Hopefully this article will get you thinking a bit more about the processes that you have to automate in your business, and the most important part of any business equation is to think!
Posted on 11/28/2007 10:37:07 AM by Justin Etheredge
Earlier this week I was talking about the difference between 80% and 20% developers, since the topic was brought up again by Jeff Atwood. I was saying that the difference between the 80 and the 20 is simply their want and desire to learn and perfect their craft. So, with that topic fresh on my mind, I was perusing dotnetkicks yesterday and came across this post by Buddy Lindsey. When I saw the title, "Real World Interfaces in C#", and saw that it already had 6 kicks in less than two hours I went ahead and clicked on it to see what it was all about.
I started reading it and Buddy started off talking about how he thought that he finally had a good grasp on how you would want to use interfaces in real life. At first I was a bit confused, I was trying to figure out where the cool new tidbit of information was, or the cool new technology I had never heard of, or the new development technique that I could use. But I kept reading further and I have to admit that I was a little embarrassed with myself because I had some of the "what is hard about this?" thoughts. I click on his "About Me" page to find out that he was a beginnermediate (sic) developer, and it all made sense.
This was one of the 20% developers that I had been talking about in my last post! He was a new guy that was blogging. Most 20% developers that are blogging have years and years of experience and they flood the blogosphere with tons of information about IoC, TDD, functional programming, C# 3.0, Extension Methods, Object-relational mapping, etc... but here is a guy blogging about a topic as basic as interfaces and there is a great response to it! Bravo!
So, my question to you guys is, do we really need to start focusing on blogging more for the beginners? I know personally I would be happy to throw in a post about some topic that I had problems with in my early early days as a developer. I wish that blogging was as popular as it is now when I was first learning, the huge amount of knowledge that comes out of so many people makes learning about new things so easy. But would there have been enough blogs for me to read that would have helped me through the basics? Let me know what you think, and after my functional programming series wraps up I may look at a few issues that I had as a young aspiring developer.
Posted on 11/28/2007 3:14:00 AM by Justin Etheredge
So, I finally got the release version of Visual Studio 2008 installed on my home box. And what do I get whenever I pretty much try to do anything within the IDE, including close it? This error:
"Visual Studio has encountered an unexpected error." I tried a repair install, but that didn't affect anything. I was getting ready to give up and go to bed, when I decided that even though it was a completely useless error maybe, just maybe, if I search for it in the context of Visual Studio 2008 I might get some kind of useable result. Well, as luck would have it I got this as my first result:
Seems innocuous enough, but in the post he said that he found it to be VisualSVN (don't let it fool you, VisualSVN rocks hard) that was causing the error. I just happened to have this installed, and now, it works perfectly. Thank you google and this post.
Posted on 11/27/2007 9:24:54 PM by Justin Etheredge
In the last two parts of this series I went over the Map and Fold (reduce) methods and so in this part I figured that I would touch on the Filter higher order function. Don't worry, this will be the last higher order function that we will go over before we start getting into some other functional concepts.
The filter function takes a list and a function that returns a boolean. It loops through each item in the list and applies the function to determine inclusion in the result. The result is simply a filtered version of the input list. We are going to create this function as an extension method of IEnumerable.
public static IEnumerable<T> Filter<T>(
this IEnumerable<T> list,
Func<T, Boolean> func)
{
List<T> result = new List<T>();
foreach (T item in list)
{
if (func(item))
{
result.Add(item);
}
}
return result;
}
As you can see, this method only has one Generic parameter because the result list is always the same type as the items in the input list. This method returns "IEnumerable<T>" which is the filtered list. The function that is being passed to it "Func<T, Boolean> func" takes an item of the types in the list and returns a boolean which determines whether or not that particular item is included in the result. We simply loop through each item, using an if statement that runs the passed function on each item.
As an example of using this, I am going to use the same example list as I have in previous parts of this series.
//declare our integer list
List<int> numbers = new List<int> { 1, 2, 3, 4, 5, 6, 7, 8 };
var result = numbers.Filter(x => x % 2 == 0);
Here we have our list of integers and then we call Filter on it passing in a method that checks each integer to see if it is even. We get a list back with the values 2, 4, 6, and 8. This can be used for far more than just finding even numbers in a list though.
First we will declare a simple Person class.
public class Person
{
public string FirstName { get; set; }
public string LastName { get; set; }
}
Then we will use the new Collection initializers to declare a short list of Person objects.
var people = new List<Person>
{
new Person { FirstName = "Bob", LastName="Parker" },
new Person { FirstName = "Fred", LastName="Thompson"},
new Person { FirstName = "Bob", LastName="Smith"}
};
Then we decide that we want to find all of the Bobs. (Oh that made me think of Office Space). We could find all of the Bob's like this:
var Bobs = people.Filter(x => x.FirstName.ToUpper() == "BOB");
Now that we have gone through all of these higher order functions I will now have to break it to you that Linq provides a built in method that will do pretty much exactly what all of these function do. The method for Filter is called "Where" and it takes a Predicate as a parameter, which is just a delegate declared like this:
public delegate bool Predicate<T>(T obj);
Essentially it is the same as the Func<T, Boolean> that we used earlier, and could have been used instead but I didn't want to throw it in without explaining it. You would use this method like this:
var Bobs = people
.Where(x => x.FirstName.ToUpper() == "BOB").ToList();
This works almost exactly the same as our "Filter" method with the exception that we have to add "ToList" to the call in order to have the query executed immediately, otherwise it will be turned into an expression tree that can later be executed.
Well, that wraps up our discussion of the "Filter" higher order function. As usual, I hope that you learned something from this post. In the next post we are going to discuss a fairly simple concept that a lot of people try to make super complicated, Memoization. So, until next time, happy programming.
Posted on 11/26/2007 5:07:04 PM by Justin Etheredge
Welcome back after a long Thanksgiving break. If you were celebrating it, I hope it was a good one!
I was going through my usual feeds in Google reader and I came across a new post title "The Two Types of Programmers" by Jeff Atwood. In this post Jeff seems to be completely retracting his post that he made earlier on how books such as "Teach Yourself Asp.net in 24 Hours" cheapens our craft. Honestly I didn't agree with Jeff's first post, and now I also don't agree with the conclusions that Jeff makes in this post either.
In Jeff's first post he starts talking about Scott Mitchell's new book "Teach Yourself Asp.net 2.0 in 24 Hours." and how titles like that cheapen our craft because it takes so much longer than that in order to really learn any technology. To quote the post:
Yes, the book title is just marketing hype to drive sales. It isn't meant to be a rational statement of expectactions. The implication that you can learn a giant swath of technology like ASP.NET 2.0 in 24 hours, much less become competent in it, is funny to those of us who know better. That part is obvious. But on a deeper level, it's also offensive. It implies that the field of software development is so shallow that a complete beginner can become competent in 24 hours. Yes, we know better, but not everyone does. And the type of people buying this book most certainly won't know what they're getting themselves into.
The last two sentences are what really sticks me. Saying "Yes, we know better" and "the type of people buying this book most certainly won't know what they're getting themselves into" is making a pretty broad generalization that people buying these books actually think that they can learn Asp.net in 24 hours. Somehow I doubt that the majority of the people buying these books actually believe that. While I have only bought one of these books ever (It was called Teach Yourself XML in 24 hours, and actually that almost sounds doable :-) ) I can imagine that some people are drawn to the fact that the books give a very brief overview of a technology while breaking it down into discrete steps that can be accomplished in small amounts of time (most of them are broken down into 24 one hour lessons).
The reason that I purchased the "Teach Yourself XML in 24 hours" book so many years ago was because I had not dealt much with XML at the time, and I needed something to give me a quick high level overview of the technology without a lot of background and fluff (or what I thought to be fluff at the time). I looked at a bunch of books and at the time I honestly didn't care about SGML and the history of this stuff (I was a newbie back then) I just wanted someone to show me how to form proper XML documents. And for its purpose, the book served it well. But was I an 80% developer at the time? No, of course I wasn't, I was green, but I sure wasn't an 80% developer. I was reading and absorbing anything I could get my hands on. I was less concerned with the background because at the time I was more concerned with learning the technology than learning what was behind it.
So, how does having books like this cheapen our craft? Is an Amish furniture maker worried about the guy working in the La-z-boy factory? Of course not. They serve entirely different customers. One person is looking for a decent piece of inexpensive furniture, and one person is looking for a quality, hand-built piece of furniture. Saying that having inexpensive furniture cheapens the craft of furniture making seems backwards to me, it seems like it would give it more value. Right now I am sitting on a piece of crap couch that still cost me 1000 dollars, and if at the time I could have afforded to buy something better I certainly would have. I appreciate nice furniture more because I have seen bad furniture.
Now, if you are operating on the assumption that having books like this is going to create cheap programmers, that do cheap work, then you are probably incorrect. But there is a *huge* market for cheap work and cheap software. Ignoring the fact that this market exists will only send more and more jobs overseas. (I am *not* saying that overseas programmers are bad, just that overseas labor is less expensive. There are many many skilled developers all over the world.) Some people simply will not pay to have quality software built, and for them the price is much more important than the quality of the software. I am not saying that this is a good thing, it is just the reality of business sometimes.
Jeff is working over at Vertigo and I would assume that they work with large clients with very deep pockets, and it is sometimes easy to forget that almost one fourth of all U.S. workers are employed by businesses with 20 or less employees. I personally work at one of these companies and we deal with these companies all day long. We do consulting for a lot of them and it is hard because we try to only employ alpha developers and so therefore we have to charge high hourly rates in order to turn a profit, but this is in direct contrast to what most of our customers want. A lot of them want "good enough" software at rock bottom prices, something that they have to go elsewhere for.
My point is that we need these two levels of developers because they serve two different purposes. One is (often) a higher paid employee who is given more challenging tasks, and the other usually does jobs that are more planned out and formulaic. While almost every field is made up of a majority of these types of people, in few fields beyond programming is there so much contempt for them. The one thing that we do have to realize though is that these are different types of people and you can't really interchange them.
Some people refer to these people as Morts or 9 to 5ers, but I find "Morts" offensive and I find no reason to call them this just because they do not share our passion. I prefer the term 9 to 5ers because it carries less of a negative connotation.
This leads me to the statement that I really didn't agree with in Jeff's latest post:
I often think we're wasting our time writing blogs which are largely read by the same 20%. In my experience, there's precious little trickle-down effect from the alpha programmers to everyone else.
This statement really doesn't make much sense to me. Why would we be wasting our time when we are aiming our blogs at the 20% of our field that are going to actually read what we write? I think that any developer who is actively reading development books and blogs is not an 80% developer. And of course there is very little trickle down to the 80% because they don't share our passion! They might have a passion for music, books, surfing, ...whatever. They program to keep the lights on and the food in the fridge, but they don't have a passion for software. Why would they want to spend their time learning about Linq when they can pick it up over time by looking at chunks of code that the alpha developers wrote?
So, in the end I don't think that you can really reach the 80% developer because by definition if they starting reading and programming in their free time then they are becoming a 20% developer. This is not something that you can teach though, at their very start they have to want to learn and better themselves and they have to have the internal drive to do this. No book or blog post is going to change that, because the real different between an 80 and a 20 developer is not whether or not they can write a Lisp compiler, but whether they have the passion and determination to learn and perfect their craft. So keep on programming, reading, and learning because you don't care as much as you do because someone talked you into it one day, you care as much as you do because you love the challenge and you can't imagine it being any other way.