1.) How do you determine individuals are “qualified and prepared”?

2.) Assuming you can consistently solve point 1, if the individual does not care, is not motivated, then all preparedness and qualifications, and even professionalism wont help.

3.) Assuming all the above are covered, there still is the human condition. Meaning we are creatures of interpretations, and this adds to the complexity.

I think design is important, but seldom does the designers have the full picture of what’s being designed. We want to approach software design, as an architect approaches the design of a building. The problem is that the best architects, do an extreme amount of research before starting the design process. Things like:

• Location.
• Elevation
• Soil
• Dispersion,
• Region
• Are earthquakes common place in this region?
• Are hurricanes or tornadoes common place here?

Among many other things they consider before putting pencil or pen to paper, or going into CAD. When the above is decided they work out the best materials for the location, and design they envision. They then analyze costs and what not, make compromises and arrangements to mitigate costs.

After all this they start what we (developers/software engineers) call the design process, and that’s making drawings or mock ups, showing it to the client, changing to the tastes/needs of the client, when everything is agreed upon, then the plans are made and drawn, there may be a need to adjust the budget here. Then we have peer reviews in the form of all the revisions, etc.. needed to get approval for construction.

So this is a very time consuming process, and one that most Software designers don’t even bother with. Most don’t know or care where the application is going to run, or an idea of the resources need for said application, or whether the application really benefits from a heavily threaded system, or if going that route may be even more detrimental performance wise. They just don’t know or care. I’m sure there are numerous exceptions to this, and there are quite capable designers that take all the above in consideration, but I believe they are the minority.

Some of you may ask: So, If I want to design an application, I should take into consideration and know about, the Operating System, the Web server, whether threads may help or make it worst, etc?

My answer is a conclusive yes! You should research and know about all of this to make inform decision on what is needed and why. For example: IIS and Apache are completely different servers, and have different requirements maintenance overheads, performance tweaks, etc.. Same goes for the DB, and how you deal with replication, etc..

If you do all the above, it’s easier for the DB engineers to prepare and optimize querys, etc.. for the application, because there is a clear picture, of what is expected of the DB. Same goes for the other parts of the equation.

So yes, properly designing software is hard, but really what that’s worth having or building isn’t hard? The problem in my view is not the silver bullet (I personally don’t think there is one) but I do think the approach described in worst is better is better than the approach that’s being nurtured in the enterprise, working but needs occasional reboots it’s great to good. Everything below that is good to ok.

This is a very interesting topic, and complex topic, hence all the papers and research around it.

This particular project was (is) of the utmost importance for that company, and in his development team he’s the only one that was working on this specific application (a server). To the suffering of all the involved in his department this particular piece of code had to be written in C, and there was just no way around it for several technical and design reasons that are not important at this moment.

I was in charge of testing his application (not unit testing), I had to performance test his application, to do this, one needs to know the architecture, and all that the developer needs to know to develop it, and more. I don’t have to tell you that this was a painful experience that mostly drove me crazy.

Just seeing who the email was from and subject surprised me to no end. As it was, they were having some issues, more to the point he was having some issues –as we were friendly, which basically means he was the only person I didn’t tell to f..k off. He asked for my number which I gave to him, and a few minutes later, his boss (whom I would have gladly frag’d) was on the line, telling me, how my NDA still applies and what not… at this point I just calmly stated you called me. What do you want?

The server has been having issues, and they have to reboot it every 72 hours or so… at this moment they have put the old code base (which was the one I worked with) back in production, but need the new functionality ASAP, and basically wanted me to check it out. Now, I was curious, I have that problem, I’m too curious so I said yes, sure.

They sent me the vmstat of the last 72 hours of the server, along with several log files, and after a cursory glance, it was obvious there were memory leaks. This they already knew (they are not dumb) but couldn’t find the spots where they were happening, and it’s here where the story gets interesting and where we might learn a few things, so this doesn’t happen to you.

First a disclaimer, this particular developer is quite good, and diligent. Also I’m not as good or diligent, and I don’t know much more than anybody else, but I have had the grand advantage of working with people that do know more than me, and that are willing to show me, so I learn and get better, this is my way of giving back, and saying thanks to all of them.

Malloc

The first issue when I saw the code, was the scheme he was using for memory allocation. I believe that with regards to memory allocation, if there is a failure allocating memory, aborting right then and there is the right thing to do, of course there are exceptions, but for most situations this is ideal, a well done policy, is also something to look into, for this type of scenarios.

His scheme was like this:

• Allocate a good chunk of memory, that will be shared among threads.
• Each thread allocates the memory needed to do it’s operation, write the results to the shared chunk, and clean after itself.

I wouldn’t do it this way, but this scheme works as long as everyone plays nice. As it happens all threads are playing nice, the problem lies with the allocation done internally by the threads.

He is over writing the pointer that malloc returns, and while he is freeing it, the portion returned by malloc is never free(), this goes on for all the threads.

Some of you might be thinking What? Over writing? How, why? Well the thing is that he didn’t noticed or know he was doing it, which is not good, but it’s a common mistake with C programmers (newbies, and not so newbies).

Another nice little side effect, is that every time, a thread wrote to the shared memory chunk, this was sent out to the appropriate interfacing systems, upon a successful completion, the memory chunk was de-allocated and the re-allocated, meaning he free() and then malloc() again and again. Which also brought the nifty problem, that if a thread was waiting in place to write to the chunk, the reference it had to the chunk had gone bye bye, so to avoid a segmentation fault, he worked around this problem in a very creatively unnecessary way (that I would rather soon forget).

Don’ts:

• Do not over write (change where it points) the pointer returned by malloc(), don’t do it ever.
• Wrap your malloc() call in a way that it checks (once is enough) no out of memory errors, if there is exit immediately.
• Don’t blindly accept that the arguments to your methods are what you are expecting, or valid data.
• When returning an error state from a function, for God’s sake, don’t abuse NULL!

Do’s:

• Learn how to use lint.
• Use lint often.
• Run lint, after every compilation, add it to the build.
• Check and double check all of lint’s warning and errors, fix them.
• Run lint again after fixing the errors.
• Read your code, go thru it in your head, or which ever way you prefer.

I’m in agreement with Thomas Ptacek. If my friend would’ve done that, this problem would have surfaced in the pre-production stages, because the system would have crashed earlier, enabling them to find the culprit much sooner.

There are other things that contributed to the problems, but the good news is after an hour of going thru the code, the system is back up, and it’s been over 72 hours and it hasn’t crashed, and it’s memory consumption is minimal as it should’ve been from the start.

Update:

Forgot to mention I provided them with a nice little Ring buffer implementation that captures the steps/states of the threads, from beginning to end, in case of an error, it dumps it’s content to disk.

That way they will be able to pin point more or less what triggered the crash, where in the code this happen, and the time this happened.

For more info on ring buffers, check out this links:

• http://en.wikipedia.org/wiki/Circular_buffer
• http://www.ibm.com/developerworks/aix/library/au-buffer/index.html
• http://visibleworkings.com/trace/Documentation/ring-buffer.pdf

One of the beauty of said hardware spec, is that there is a set of instructions that needs to be run in sequential order, checked, and continue to the next instruction. Now each instruction is a separate function, because they are huge and are reused in other parts of the code. The spec had a list of all the functions names (of course the list was in a spread sheet) so I just copied and pasted the functions name, to the code (I don’t do copy and paste, but after only having slept for an hour, It looked might attractive).

So in my laziness I came up with this:

    1 void *(*p_foo[])() = {
    2         function1,
    3         function2,
    4         function3,
    5 };
    6
    7 for (i = 0; i < ILIMIT; i++)
    8         trace_error(p_foo[i](param1,param2));

Now this works, because all the functions share the same prototype, and return a pointer to void. The value of ILIMIT is huge, so I saved myself a lot of typing. I don’t remember seeing that idiom anywhere, I’m not claiming I made it up, or is my invention or anything of the sort. It’s not that creative to begin with, and can be made a lot better.

But if you ever find yourself with a similar problem, and don’t want to write code to generate the function calls for you (or are to tired to do so :)). This is a clean alternative.

Going to sleep now.

In the process, I have been trying out several distributed revision control systems. So far I have tried:

• Bazaar
• Mercurial
• Darcs
• Arch
• monotone
• svk

I think they are all good at what they do, and picking one above the others, is really a matter of personal preference.

I liked bazaar, git, and mercurial the most. I decided on Git, because of the ability to interact with svn out of the box, it’s svn integration is really good. I actually liked bazaar the most, but it’s svn support is not as robust as git’s. Now let me be clear I hate git’s command interface, but it’s growing on me. The fact that there is a emacs vcs mode makes it a lot simpler for me.

With this I’m not suggesting ditching svn for a distributed system. At work we use svn and it’s not going to change any time soon. If it were not for the fact that I have the bad habit of using svn branches for development, I wouldn’t care about merging all that much, while merging in svn it’s not really as bad, as some people make it sound, you do have to be careful, and svn it’s not very helpful with conflict resolutions, besides that, the fact that I don’t have to run a server for my local repository is a wonderful thing. I also like the simplicity and multiple methods for publishing my Git branch.

If you haven’t tried a distributed revision control system, and you are a svn user, I suggest you start with bazaar, as it’s user interface is very much like svn so the learning curve is really small, Mercurial is also a nice system to start with, though a bit different than svn.

If you do try them out, I would love to hear about your experience.

* This has nothing to do with warez keygens, it’s part of the validation process of our software.

My answer: Who said I was a good programmer?

To be honest, I’m not sure if I am a good or bad programmer, and I never thought about it until he asked.

Coincidentally I found a few blogs talking about this very subject. They go to explain how, good programmers are more efficient and productive than bad programmers, and how good programmers take ownership of their projects, and don’t need to be micro-managed etcetera.

In my opinion it is true that good programmers tend to be more efficient, productive, and effective. I also agree that most good programmers are capable of taking ownership of a project and running with it.

But, I think in the real world things are not black and white, as the above statements seem to imply.

There are circumstances that can make a good programmer, behave like a bad programmer, the implications that good programmers are more disciplined, is also not necessarily true.

Humor me a bit more, and let me outline the variables, that I think are important, and may break the above perceptions:

Work Environment: The work environment is a very important variable. Productivity, efficiency and reliability depends highly of your work environment, and what you as a programmer need.

Some programmers need peace and quiet, others need to be surrounded by people, others need open spaces, other needs music, or a combination of some or all of the above, we are complex beings.

If you put someone that needs to work in a open space, in an office, he’s productivity will go down, we humans can get used to almost any environment, but there is always a trade off, and it is the programmer that has the short end of the deal. I believe this is why Google works the way it does, there are offices, cubes, open spaces, recreation rooms, conference rooms, and much more. They don’t care where you do your work (at least that is what googlites tells us). That right there is a productivity booster.

Projects: The type of projects you work on, can motivate you and inspire you to be better, and study more, or they can stifle you and bore you. This is a big problem, in several fronts, the truth is that no matter what the company needs the work has to be done, boring or not, and usually they need it by yesterday.

The problem for the programmer, is how does he motivates himself to do the work, and do it well?

I know this seems like a childish or amateurish proposition, and most professionals will roll their eyes at it. Don’t get me wrong it has to be done, it is your job, whether you like it or not, but it depends on the character of the programmer, and his own self pride, whether the level of the result is to par, with the results, of work he finds interesting and challenging.

Most managers overlook this issue, and in the end, either the programmer leaves the company or department, or he stops caring.

It is not an easy problem to solve, some companies have worked out this issue by given programmers, time slots to work on whatever they want, others will sit all the programmers down, explain the projects at hand, and let them volunteer for the different projects (this has a lot of potential problems).

But the truth is that most companies can’t afford to that, and they have to resort to other types of incentives, sometimes they work sometimes they don’t.

In this life, people have to do a lot things they don’t like, and they have to do it well, I’m not making excuses here, I do believe you need to be realistic, and know, that no matter what you do, or where you work, you will have to do things you do not like, and when that time comes, you have to make tough decisions, you can do it or leave.

But my personal opinion is that life is to short, to waste your time on things you don’t care about, or don’t want to do, sometimes we don’t have a choice, we have families to provide for, but if you do have a choice, use it wisely.

Management: A good manager, is a blessing. He is the first and in some cases the last line of defense or defiance.

A good manager, knows his team, and knows what they are capable of doing, he knows how to push them to be better, faster, and efficient. In short he knows: the good, the bad and the ugly, of his team, and is aware of the role that each member currently plays, and what they could play, he fights and stand up for his team

When upper-management puts out a ridiculous deadline to a highly critical project, it is usually the manager, and the Senior staff, that comes with a plan of action to deliver in time.

Sadly, this is hardly ever the case.

Most managers are ill prepare to handle the different personalities, and work ethics of his employees, most managers don’t understand, and don’t even try to understand the realities of each team member, sometimes there isn’t time for him to do that, after all he/she does have a life.

A lot of people think they can be a manager, and that being a manager is easy, well, being a bad manager is a piece of cake, all you need is to be good at watching your own back, and obvious denial ability, because nobody likes excuses.

But being a good manager, is a very tough job, you are dealing with people, different cultures, and personalities, your boss has certain expectations, usually this expectations are unrealistic, and brought by lack of knowledge or by politics.

Managers, –good one at least, have the task of bringing clueless upper-management requests, to reality. It’s difficult to explain; that the project they want deployed in three months, actually needs a year. More so when s/he has to wrestle with the managers and directors of the other departments, who say that’s the design is ready, corrected, and certified, that he’s just giving excuses, because his team, are a bunch of whining hacks!

Of course, when the problems start flying, you would hear the requirement people, shouting that, that feature is not correct, or a feature was overlooked, a procedure is not following all the steps outlined in the documents, designer will then come out, defend themselves in which ever way possible and point the finger else where.

And so the cycle continues.

This is the reality that some, most programmers live, and deal with.

In an ideal environment, the:

Good Programmer = Productivity + Efficiency + Self dependence.

Is true, and even in not such and ideal environment, it can hold, but then there is the human aspect, that’s never taken into account.

Now, if a bad programmer is one that is:

• Unproductive.
• Inefficient.
• Slow, or dependent on others

Then there is no problem, because, if they want, they can learn to be the opposite of that. You the good programmer can teach them.

The truth is that, if you think about it, you can’t rely on the above conditions to differentiate the good from the bad.

I read a blogger, who said that:

You can know a good programmer from a bad or not so good programmer, because when the project is assigned a bad programmer, will start looking for sample code, and waste time looking for how others have tackled the problem, while the good programmer just starts working on it.

I think, that is not a fair or realistic statement. If I’m given a problem, that I know it was solved, efficiently and intelligently by someone else, I would go and look and talk to the person, who did it and if I can use the code unchanged, I would use it, and give due credit.

I don’t know if all this makes sense, I’m sure it doesn’t.

I have to say that all this has to be taken with a grain of salt. I have had been told, by some that I’m a good programmer, and by others that I suck. Truth be told, neither of the opinions affect me in any way, I love what I do, and will keep on doing it, with or without pay. I think that alone makes me a decent programmer.

I’m very interested on your thoughts in the matter, it does not matter if you are not a programmer, I think this extends to other careers.

Right now if you Google programmer vs. developer, you will get articles on how developers:

• Design the application from the ground up.
• Write specs, and documentation.
• Write test cases.
• Write unit tests.
• Write code.

Instead a programmer:

• Write codes.
• Fix bugs; if you are lucky.

Well I don’t know about you but that does not seems like a real picture to me. People can call themselves whatever they want, you can call yourself a programmer, developer, engineer etcetera. Hell there was guy at my former employer that referred to me as “Engineer”, let me tell you I’m not an Engineer don’t have the degree, don’t have the know how. The point is that the word developer to my knowledge is a relative new term, while programmer is a much older term. Maybe the reasons of such distinction is to differentiate the real programmers, from the posers? I don’t know.

The truth is that when I hear the world programmer I think of people like:

• Dennis Ritchie
• Ken Thompson
• Bjarne Stroustrup
• Bill Joy
• James Gosling
• Linus Torvalds
• Keith Bostic
• Alfred Aho
• Brian Kernighan
• Rob Pike
• Jamie Zawinski (jwz)
• Theo de Raadt
• Paul Graham
• John McCarthy
• Guy L. Steele Jr.
• Larry Wall
• Eric S. Raymond

For a more complete list please go here.

If you take the time to visit each of the links above, you will see some are referred to as:

• Computer Scientist
• Software Engineer
• Computer Programmers

Those who know me, might be perplexed I left out from the list, who to me is a great programmer and someone that I admire, that is Donald Knuth. Dr. Knuth is to me is a master, any other title in my opinion falls short of his achievements.

Now I’m sure you are asking yourself, what is my position, if I failed to make my position clear from the statements above let me just say, I don’t believe in labeling, that said I don’t consider myself to be one or the other, but if you push me I have to go with Programmer.

I know a lot of people that ware the title “developer or programmer” and are neither. Programming is a lot more than writing code in your favorite language, and to me there are only: good programmers or bad programmers –of course the posers, but those are easier to spot.

I leave you with this little snippet from Donald Knuth:

Languages come and go much faster than I can write books. That’s why I chose to explain algorithms in English, not in the language of the moment. Readers learn a lot by converting from English to their favorite language; The Art of Computer Programming emphasizes things that are independent of languages. No matter what programming language is hot, you need good ideas to express in those languages. If you want your algorithms to be prepackaged, fine, but then my books aren’t written for you.
Actually I’m extremely glad to see the continuing development of languages, not only because programming languages are getting better and better in important ways, but also because such work soaks up a lot of people’s energy-therefore computer scientists don’t write papers that I would otherwise have to read, and I can get my books finished a lot sooner.

“Oh, but I can do that!, it just paint splashed in a canvas! How hard can it be?”

The same happens when they hear a cool Rock song:

“Oh! I’m going to be a guitarist, and land me some chicks! How hard could it be to play guitar?”

Well as it turns out, it’s easier to play three chords on a guitar, than to paint like Jackson Pollock or Picasso. But the premise of art being simple, does not make it easy, the same way, that it being complex, does not make it hard. Learning a programming language, is not an easy task, it takes weeks or days to “learn”, but years to use it correctly; just like some one can learn the notes on a pentagram, and the scales, in weeks, but it will take them years for them to write a Symphony.

There is a lot more to just sitting down and banging words out on a keyboard, you need to:

• plan or design.
• structure the information your code is going to handle (data structures)
• the efficient use of said information (algorithms).

There are those that believe, that all the above is only needed for big projects –They believe so because they read articles on how Linus wrote GIT on a few days, or how “X” wrote “Y” in three days. They are wrong.

The truth is that you won’t see me with a UML program, doing diagrams, use case scenarios, and class diagrams, not because I don’t like them, I like them, and use them, when communicating ideas to others, but for the most part I the design in my head, and can visualize it. Before I even consider writing code, I work out the different scenarios in my head, this is easy to do for me, but it might not be easy for you, and that’s fine, sometimes I need to do an outline, or a brain map, before writing code, this is all part of the process of writing.

It’s the same way for a composer when he’s got a commission for a new piece, composers can’t wait for the muse to come when they have a due date for the work –oh, yes composers, writers and painters have due dates.

Composers devise ways of making themselves efficient, and be able to complete their work with enough time, to proof read, re-write, revise, etcetera.. It’s here where some programmers deviate, once the code is written, compiles and runs as expected, only a bug , will make the programmer look at the code again.

Programmers don’t proof read, and they sure as hell don’t like to re-write, and revising is not liked much either. Some do unit testing, which in some cases can cause the programmer to go back, and refactor/revise the code, but in most cases, the programmer just fixes the bug, and does not even care to investigate why the bug was there in the first place, also programmers don’t really look for the most efficient way to code, they avoid making their life easier.

This brings me to the fact that most programmers don’t think about their craft; there are hundreds of books about the craft of music composition, writing a novel, and very few that deal with the craft of programming.

Most programmers don’t even realize that programming, goes beyond the language being use, the language is just a tool; you don’t see painters having flame wars on whether the smaller brush, is better than the bigger brush, for doing a sky line? I’ve never seeing construction workers arguing about which brand of hammer is the best for the job, but you will see that with programmers.

After a long time working on different teams, with different tools (languages) I’ve concluded that most programmers don’t take their craft seriously, and in most cases they sabotage themselves, by not taking the time to look for ways of being more efficient with their time, and their code.

Programming is an art, there is no question about it in my mind, but for it to reach the level of recognition, of the other arts, there is still a long way to go. The knowledge of algorithms, logic, design, and the abilities to do this things well, are worth nothing, if they are not accompanied by thorough, efficient and intelligent coding, do not fool yourself into thinking, that you will be able to make a finished product in a few days. Care for your code, at the end it’s your creation, it exists because you exist, and you made it.

Frederick Brooks had the following to say in his book, “The Mythical Man Month: Essays on Software Engineering”:

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures.”

• Git: According to Linus and some kernel developers, it’s IT. There is nothing better, oh and SVN and CVS suck and you are either a sucker or a moron if you are using CVS or SVN! (That’s the gospel of Linus, not mine.)  I agree with him that merging is the key.
• Darcs: Is another distributed SCM, this one uses, smart; which according to the site, is a unique algebra of patches.
• Mercurial: Yet another distributed SCM, there are quite a few projects using it.
• Bzr: Which is yet another, though I find it quite nice in Linux, because it lets you work with existing CVS and SVN repositories, of course you need a plugin that only works in Linux.

And there are many, many, more, commercial, shareware, open source, etc.. The premise of all this, is that CVS sucks, and SVN is not good enough, to tell you the truth I feel these guys are making CVS a more desirable alternative, because while CVS has a lot of issues, they are well known and documented issues, with well known and documented work arounds. I feel more attached to CVS every passing day, It’s true that I’m used to CVS, but I have and still am giving the SCM mention above a shot, and I’m currently using SVN too.

I used to love SVN, until I had to do some real work, that needed to use the full power of the SVN, and it worked ok, but things that I was told were easy, were in fact hackish, and not all that simple. Setting SVN up is a simple affair, but the way to manage keywords and properties is not trivial, as it should be, and there are a few others things that should be simple and easy to use or administer, but in turn are quite difficult.

CVS is alive and well on many projects, both Open and closed, and I believe, it will keep a good share of the market. I’m not saying the above software is not good, it is, but they are bombarding us, and some projects are using different SCM for different modules for reasons I don’t know and will probably wont understand.

I do agree that the distributed model is very appealing and in theory is ideal, that git and the others like it, make merging trivial, and it’s quite smart too.

But that does not mean CVS or SVN is less than the above, in fact some of the largest code bases are served and managed in CVS and SVN (Yes code bases bigger than the linux kernel, a lot bigger), without problems, yes they have policies and standards in place to minimize issues, but that’s not because of CVS, it’s the nature of a big project, you need consistency, and safe guards. So in a organized work model, the distributed system may be more chaotic, and inefficient than CVS or SVN.

The sad part, is the new crop of SCM looks more of a recursive re-envention of the wheel, with very little new or unique features, that would merit one over the other. It’s like a collective exercise on how to make a distributed SCM.

Let’s see what will come out of all this SCM crazyness, hopefully something will make our collective life easier.

Here is the video of Linus on Git.

[youtube=http://www.youtube.com/watch?v=4XpnKHJAok8]