Innovation, Creativity, Software and Success

It is amazing just how many software companies are "creative" and "innovative". In addition, by reading CV's and online profiles, it is amazing just how many people are also "creative" and "innovative".

At this point, let's loosely define our three terms "Innovate", "Creativity" and "Success":

• Innovate - Provide a new solution to problem that is distinctly different from existing approaches
• Creativity - The desire, ability and resourcefulness to create 'something'.
• Success - The measure by which an individual or organization can be said to have achieved its objectives.

So it might seem reasonable to expect that the world is full of innovative, creative companies that are full to the brim of innovative and creative people. And that all of these innovative and creative companies are creating innovative new stuff....

Thus it stands to reason that the majority of these companies are NOT successful, because most new 'stuff' is created by the same relatively small number of companies.

In most cases, not being a creative and innovative company is perfectly fine. For example, a restaurant does not have to be particularly creative or innovative to make a profit (although I am not saying it may not help).

However, if you are a software company and you are not genuinely creative or innovative I would bet that you are probably not successful either. Even the big mega-corporations like Apple, IBM, Oracle, Microsoft, Facebook and SAP were at some point in their history creative and innovative and to varying degrees continue to be so.

So if these mega-corporations still continue to be innovative and creative, then your small software company DEFINITELY has to be.

But most software companies are neither innovative nor creative. And I wonder if this is because most people are not actually innovative or creative, despite what they say.

Companies do need different types of people, but software companies do need a genuine element of creativity and innovation which, if not always, mostly takes precedence over the best practices, rules of thumbs, processes and procedures that 'most' people would govern by.

So find genuinely innovative and creative people and let them do their thing...

IMified - Please dont shutdown!

This morning I received the following email from IMified informing me that the service intends to shut down due to lack of profit and what seems like continual problems with the IM companies. The real shame is that IMified is an awesome service with absolutely no alternative - please do correct me if I am wrong, I'd love to know.

For those of you that are not aware of IMified (http://imified.com), it is a service which allows application developers to create 'bots' which are essentially IM users on networks like AOL, Google Chat, MSN, XMPP etc and provides a single consitent API to allow asynchronous communication with a client. This service is absolutely brilliant for notification services like Femtoo.com (http://femtoo.com - online web page tracker and notification service). Maybe IMified could have a future as an open source, cloud-based service. Eitherway, I really hope that we can save IMified.

Here is the email for your information:

===================

Dear IMified Customer,

The purpose of this email is to alert you to a change in the IMified service.

Over the last several months, it has become apparent that running an IM bot hosting service is something that cannot be done profitably. When we created IMified several years ago, the popular IM networks had programs for running bots - bypassing limits in contact lists and message volume, for example. This made it feasible to run a large scale IM bot. However, over the last year, these programs have started to disappear. Our contacts at IM providers have left the companies or been reassigned. It's become clear that this is a business that public IM providers no longer want to be in.
We've tried to keep IM support operational despite the fact that not only have we had no support from IM networks, but we have also had to fight the networks actively stopping and shutting down IM bots. Continuing to do this requires a significant amount of resources, and does not bring in enough revenue to justify the battle. So we're shutting down the service.
The logical question that many of you will have next is "what can I do instead?" Unfortunately, we're not aware of any viable IM bot hosting services. Those that we knew of have all gone away. One alternative is to host one yourself. Building a basic XMPP (Jabber) bot isn't hard. You just need to sign up with an XMPP host, someone like Gmail or Jabber.org will work for low-traffic bots. There are numerous tutorials and frameworks for building XMPP bots in your language of choice. A discussion thread on Quora has some other ideas. Once you have a XMPP bot up, you can connect it to other IM networks using XMPP Gateways (also known as Transports). These sign on to an IM network for you, and relay messages to and from an XMPP account. There's not much in the way of public gateways available, so to go this route, you may need to host your XMPP server and your gateways yourself. Spectrum and Kraken are two popular gateways, and Kraken is available as an easy-to-install plugin for Openfire, an open source XMPP server.

If you have concerns or questions about the shutdown, please email shutdown@imified.com

- The IMified Team

===================

Software development and the dangers of the word "Done"

People like using the word "done" and with good reason - if something is "done" it means we no longer have to worry about it and we can get on with the next task - its a decisive and positive word. But this word is usually misleading and in the real world, software development tasks are very rarely "done".

To say that a software development task is "done" tends to send a message to the project manager and client that the specific user story or feature will never need any revision and is completely free from errors. This is almost certainly not the case. It's a good assumption that new code will contain errors, even if you haven't found them yet and you can be pretty sure that when the new feature is demonstrated to the client or as soon as real people start to use the system, changes will be required.

And at the engineering level, well organized code will generally adhere to the DRY (dont repeat yourself) principal. This means that engineers will try to 'abstract' code, functions and structures to be as general as possible so that they are used as often as possible. This implies that although a feature may appear unchanged from one version to the next, the engineers really know that actually a lot of the underlying code may well have changed and although unit testing can 'positively' test and confirm consistent behaviour, a good dose of manual testing will do no harm.

But this article is not about code organization or testing techniques, it is about why we should try not to use the word "done". When we say this word, we probably really mean: "done to the point where another task is of higher priority". If this definition of "done" is generally accepted then project managers and clients can continue as usual, but should not be alarmed when something marked as "done" contains bugs, needs changing or the code requires rework.

The Internet - and what it could be

The Internet is still basically a mass of insular websites that do not interact with each other. Websites are still primerily designed to be read/browsed by a human, regardless of whether it is done from a PC, mobile or tablet device. Most websites are still just like books in a library which do not communicate and share information with one another.

Websites have got prettier and include more media, like music, videos and animations but essentially, websites have not changed and the Internet as a whole as not significantly changed for many years.

But the Internet has an amazing potential, behind all of these insular websites are computers capable of complex data processing and hi-speed communications, but for the most part, this potential power is just used to serve up pre-defined static content.

I think it is now time to build the REAL "web 2.0". For me, this implies websites that communicate and share information automatically with other websites and services in (pseudo) real-time. The end goal of which is to turn the Internet into something really powerful - a single massively parallel computer. The hard part has been done for us with technologies like HTTP, DNS, TCP/IP, JSON etc already defined and well supported by tried and tested technologies - all the building blocks of a new Internet are in place.

But to create this new Internet, we must create a mechanism such that all data currently formatted and published for humans (the majority) is also available to other web services. Consider my own personal website:

http://tomcarnell.com

If you visit this website from a standard web browser, you would be presented with a mixture of graphics, text and hyperlinks organised to be human readable and formatted to be attractive (like any other website). Unfortunately, this type of content is completely unsuitable for a third party web service to make use of, let alone use in a meaningful way. So, lets consider the following two proposed URL's:

http://tomcarnell.com/services.json
http://tomcarnell.com/data.json

Now, a third party web service can read the contents of 'services.json' and can learn that the 'tomcarnell.com' site has a service called 'contact_me', a programatic version of the 'contact form' (http://tomcarnell.com/contact_me) which allows the service to submit a contact request. The services.json file will also detail how the 'contact_me' service should be called, what type of data it returns and maybe some human readable text to explain what the service does and how it should be used. We could also implemenent a 'subscribeToService_xxx' service which means that before allowing just anything to use the 'contact_me' service, one must first subscribe. The rules and details of how these services work are completely up to the website owner.

Our third party web service could also decide to read the contents of 'data.json' which contains semantically structured data. This data might contain the details of my portfolio, my curriculum, my contact details, links to social profile pages and any of the text found on the human readable website. It could also detail daily contractor rates, availability, location and phone number, which a third party recruitment service might find useful when searching for an available web developer in the Madrid area with experience with jQuery for example. Again, the contents of 'data.json' are completely up to the website owner.

Online shops could employ the same standard to publish special offers and could provide services to allow 3rd application to check an order status or product stock levels.

A news website could publish their latest headlines in 'data.json' and maybe provide a search service in 'services.json' which could return a list of news articles that fulfill the search criteria.

But the fun really starts when we start to combine these services. We could have language translation services, hotel reservation services, transport reservation systems, services that interact with social networks, services to access and query government and official data and all this opens us up to the possibility of creating sophisticated 'problem specific' services, for example a service that knows how to plan an entire wedding and knows which other services it needs to locate available venues, check guests calendars, send and recieve invitations, book a chauffeur driven car, obtain estimates for food, gift lists etc...

It would be really great to move in this direction, but it will take the support of lots of people. However, as a stepping stone, website owners can start by using http://livedirectory.org/ - a service which provides the functionality to publish and manage an arbitrary set of data and allow 3rd party applications to subscribe to this data, which means that LiveDirectory will send change notifications to all subscribers when this data is updated. For example, I have implemented the equivilent of the 'data.json' file as a LiveDirectory profile:

http://livedirectory.org/profile/tom-carnell

The above LiveDirectory profile details information about me and allows anybody to 'subscribe' to any part of that data structure. Another example is the jQuery profile:

http://livedirectory.org/profile/jquery

Where services can subscribe to be notified whenever a new version of jQuery is released, which is specifically located here:

http://livedirectory.org/profile/jquery/latest_version

The LiveDirectory.org aims to provide a simply but very powerful API based data manangement, retrieval and subscription system to help the Internet take one step toward becoming a more useful tool for everyone.

Why database column names should be globally unique

After years of working with relational databases (RDBMS's), I have just started to realise the importance of globally unique names for database columns. Generally I use the term 'globally unique' to mean unique within the database schema, but I it could potentially extend to being unique across all schemas.

The problem that arises from using repeating column names is when two or more tables are used in a single query. For example, if we have two tables: 'customers' (which has a column 'id') and 'customer_orders' (which also has the column 'id') when we perform the following query:

"SELECT * FROM customers c, customer_orders o WHERE c.name = 'tom';"

The "*" clause means we will return every column of both tables. So considering a single row returned from the query, what would be the value of 'id'? Would it be the 'customers.id' field or the 'customer_orders.id' field? What if our application needed to use both values? Of course, we could change our query and give aliases to the columns, which would also mean we would need to explicitly list the columns we wish to be returned:

"SELECT c.id customer_id, o.id order_id, c.name, o.value FROM customers c, customer_orders o WHERE c.name = 'tom';"

So we can now reference either the customers.id value and the order.id value. But this solution makes our query fragile and susceptible to schema changes - what if we later rename the column 'c.name' to 'c.first_name'? The query would then break. In addition, and for the purposes of consistancy of any 'data interface' we might define, we would be forced to use these explict column aliases in all our queries, which would reduce the readability of the SQL and introduce the same problems associated with any form of code duplication.

In reality, the first SQL query would be perfectly good had we used unique column names. For example, 'customers.cst_id' instead of 'customer.id' and 'customer_orders.cso_id' instead of 'customer_orders.id'. Here I have chosen a three character 'prefix' for column names that gives some clue to the name of the containing table. Therefore 'customers.name' would also become 'customers.cst_name'.

Implementing a simple column naming convention across the entire database schema will greatly improve the effectiveness of SQL queries and will reduce the likelihood of application errors that may occur by accidentally referencing the wrong data within an application.

Delivering software projects: Drupal, Joomla or just go bespoke?

Imagine you are building a house and the time has come to install the electrics. Which of the following do you do:

1. Buy an 'off-the-shelf', 'one-size-fits-all' complete home eletrical system and then hire an electrician to customize and tailor the system to fit your needs, or
2. Hire an electrician straight-off to build the electrical system for you?

The first option may seem like a good way to save time and possibly money by buying a pre-built solution and hiring an expert engineer to tailor the system. However, such pre-built electrical systems are not available in shops because customizing a system that does not do what it is supposed to can often be more expensive that simply building the exact system to requirements.

Importantly, the second option works. An expert is hired to use their skills, experience and tools knowledge to deliver exactly what a client needs - nothing more and nothing less. This is the general approach taken not just in small-scale or private engineering, but also in large scale commercial engineering projects. And this is also the reason why 'off-the-shelf', 'one-size-fits-all' systems are simply not available for purchase - nobody builds them because nobody would buy them.


So why is the world of software engineering projects so obsessed with using off-the-shelf one-size-fits-all software like Drupal and Joomla which almost certainly do not do what the client wants out of the box and will almost certainly need extensive customization and tailoring? Why not follow the rest of the engineering world and deliver client software by using expert engineers to build tailored applications that do exactly what is required using the most appropriate tools and techniques available?

I believe there are three reasons to explain the software engineering obsession with using 'off-the-shelf' systems:

1. Building systems like Drupal and Joomla is actually pretty easy, which explains why there are hundreds, if not thousands of these types of 'one-size-fits-all' systems on the market. And the sheer number of these systesm gives an overall impression that we should be using them.
2. Self preservation - if a software project fails, it is difficult to blame the guy that chose to use Drupal, whereas it is probably much easier to blame the guy that chose to build a bespoke system.
3. In a few cases, for simple systems like very basic websites an 'off-the-shelf' system can help speed things along if very minimal customization is required - which of course is simply not the majority of high-value business systems.

A fourth reason is the often stated benefit that building a system based on an off-the-shelf product will avoid technology lock-in and avoid the problems of supporting proprietory software. However, I believe this goal is rarely, if ever achieved. It is highly likely that the parts of an off-the-shelf system that have been customized and tailored for a particular client are also likely to be the parts of the system that add most value to the client (by the very nature of those parts having been written to satisfy a specific client need).  So in many cases, the most valuable parts of system are also likely to be 100% proprietory with no support available from either the off-the-shelf product team or the open source community.

It is unlikely that designing and building an RDBMS from scratch would be a necessity for a given software project, therefore it is likely to be a good idea to use an off-the-shelf solution for this type of specific system component. The same goes for other system components like computer languages, operating systems, communication protocols etc. However, depending on client needs, the design and implementation of a new type of RDBMS maybe exactly what is required. This is to say that even with a bespoke project a vast amount of functionality can be provided by pre-built technologies. However, if a key requirement can not be serviced by pre-built technology then it will have to be built from scratch - drupal or no drupal.

There are many great and often free tools that software engineers can use to build high-quality applications quickly and effectively. An experienced and expert software engineer should be able to choose which of these tools and technologies best fits a clients needs and deliver a bespoke system faster than using a 'one-size-fits-all' product.

I wonder if as the software engineering industry matures that we will see fewer 'off-the-shelf' 'one-size-fits-all' packages and move toward boutique software consultancies that delier high-quality solutions quickly and cost effectively. And more importantly that the software engineering industry follows in the foot steps of far more mature engineering industries.

Web usability and HTML: Uploading files and photos

More and more people use online document and photo sharing websites (dropbox, picasa, facebook etc). However, I am still amazed at the number of websites that require files to be uploaded and that do not offer a way to link to an online resource.

A classic example is adding a photo to an online profile or attaching a document to an application form. It would be really helpful to be able to either upload a file, or simply enter a URL to an online file. This is something applications developers could easily implement.

However, the ideal solution would be to change the HTML5 upload file selector to allow the selection of either a disk-based resource/file or a publically available web-based resource (via a url). This change would immediately make ALL web applications that perform file uploading more flexible with zero effort required from developers.

An obvious progression from this would be to allow browsers to access secured resources (private picasa galleries or files on ftp sites etc) - of course, only with the owners permission and with appropriate authentication.