Metadata driven development, the Holy Grail of software development.

In life, there are thinkers, and there are doers.

I recently had the privilege of having a discussion about “the holy grail of software development” with a colleague of mine, who I consider to be one of the greatest thinkers I have ever met.    Unfortunately, after the conversation, I was left with a weird taste, which I later explained by the fact that even the greatest among the thinkers, can only think I terms of what they already know.  Of course, the more bits and pieces of information they know, the more combinations they can make, and the more puzzles they solve.  But still, it is extremely hard, if not nearly impossible, for anyone to think beyond the limits of one’s current knowledge.

Similarly, when people see LightSwitch for the first time, they tend to label it with labels that they already own.  They compare it to what they already used before, or read about.  They try to fit it in known boxes, and unfortunately, for most software developers, that’s either the box labeled “software made by expert software engineers”, or the one leaning more towards “rapid application development”.  LightSwitch however, does not fit in the first box, but surprisingly, does not fit in the RAD box as well.  LightSwitch fits in a third box, called “metadata driven development”, which unfortunately is a box unknown to most…

In this article, I’d like to take the time to explain my vision on metadata driven development, and even if you don’t consider it to be the “Holy Grail” of software development (which I personally do), simply learning about a this ideology will offer anyone the opportunity to make more combinations, and solve more puzzles…

Traditional software development vs metadata driven development (MDDD).

Think about your current  projects.  Suppose you have to expand the software product you are currently working on to support your customer’s need to be able to manage “people”.    From your customer’s needs, each person has a last name, first name, phone number and an email address to be managed.  When you are asked to develop this, some of you will take the WET approach (“we enjoy typing”) and actually type out this classes from scratch.  Others, who use RAD tools, will design the entities in a more graphical environment, which in turn will generate these classes for you.  Maybe you will have combined the best of both worlds, and start by partially coding some classes and have some tools (like the powershell commands in the mvc scaffolding nuget packages) generate the rest.  Whatever road you took, the end result of most of us will almost always be a combination of classes in the trend of “PeopleProcessService”, “PeopleRepository”, “PeopleManagementView”, … Almost always, you will rely on some frameworks (sometimes developed in-house), offering some common abstractions, such as a “ManagementScreen”, or even simply an “IScreen”, for your “PeopleManagementView”.

Excluding testing and deployment, there’s three (well: two and a half) important components to consider in MDDD, steps on the road to take, phases of development, before the software can run and “people can be managed”.  Each of these components contains one single version of the truth, following the DRY principle perfectly.

  • The heart of an MDD application, contains the single version of the truth which is your business logic.  To help you visualize this, imagine an XML file which contains metadata, describing everything from entities, their corresponding business logic and validation rules, users and how different user levels offer different possibilities or actions to perform, to a rough layout of your screens and a description of the user processes, wizards, steps to take, …  It doesn’t stop there, a true MDD application in his metadata, will include limited documentation, input for different test cases, or even partially describe the desired architecture of the application, stating whether CQRS should be used for some entities or not, whether some entities should be eagerly cached, lazily cached, or loaded on demand, …  It goes without saying that this metadata needs very good tooling to be managed, preferably with different levels.  Project managers and support staff should be able to tweak or configure the business logic (such as: the minimum and maximum limits on one account transaction, …) in tools they are comfortable using, software engineers (doers) will design the entities, screens, … and architects (or the thinkers among the software engineers in a smaller company) will tweak the architectural parts.
  • Next component, is the compile time process.  During this process, a very limited amount of C# specific classes (or of course, the programming language of your choice), will be generated to serve the single purpose of supporting the human flaws in the former and the latter phase.  Without proper tooling, it’s hard to read and understand metadata, and even harder to effectively debug it.  To enable the framework team (more about this later) to do their job properly, the metadata needs to be translated into something they can understand, in our example, a “person” class.  Another flaw is that no metadata language will ever be able to express every business rule, and sometimes some additional functional programming will still have to be done, which again creates the need for POCO (plain old C# objects) for the functional developers to work against.
  • The final phase to consider, is the runtime phase.  Here, a carefully designed framework loads up both the metadata and the generated classes, and brings them together in an application.  This phase contains the single version of the truth which are your technological specific choices, combined into the user experience you want to offer the end users.
Before reading on, I’d like to explicitly draw attention about the separation of technology specific choices.
  • The first phase is technology independent.   It’s written in its own metadata language.
  • The second phase is only dependent on the development language of choice.  (Very likely to be C# if you’re a reader of my blog.)
  • The last phase is completely technology dependent.
Another point noteworthy here is the difference between MDDD and what most people consider Rapid Application Development.  In RAD, there’s only two phases, the metadata phase and the compilation phase.  Everything expressed in metadata, results in generated classes, and there’s no difference at runtime between an application where everything is carefully hand coded or that’s generated by RAD tools.  Compared to RAD applications, MDDD has a much smaller amount of classes generated (there will be a “Person” class, but not a “PeopleManagementView”, no “CreateNewPersonCommandValidator”, or “PeopleRepository”), and is defined by the fact that the metadata lives on into the runtime phase.
We are already used to the abstraction that a PeopleManagementView “is a” ManagementView, which in turn implements “IScreen”.  MDDD just takes the abstraction one level further, saying there’s only some particular instance of “Screen”, loaded with that metadata of how you would like to represent the management of people, hooked up to the web service / DAL that can perform query actions or persist changes on these specific entities.

MDDD has a lot of similarities with whatever road you took as a traditional software engineer, to develop the “people” functionality, but the end result will be different.

The benefits of metadata driven development.

  • More fun for software engineers.  There’s a good chance, you as a software engineer are not working on a anti missile satellite shield operating system, or working out a new physics engine for Battlefield 4.  More likely, you are working on a payroll application, maintaining a huge CRM,  or developing a real estate management suite. Writing functional software is boring, because most software engineers are thinkers, and not doers.  They have little interest in the actual business they write software for, but prefer thinking and brainstorming about user experience, tweaking performance, discussing architectural practices and studying design patterns.  MDDD allows for a greater part of the team to work on these things they find exciting, and not on functional programming.  It probably requires quite some organisational changes in your company, but i suspect this will be worth it in the end.  (I now find it interesting to see how one of Michael Washington’s firsts posts about LightSwitch also envisions that LightSwitch could bring a serious organisational change…)
  • Faster time to market.  Granted, MDDD requires more of an initial investment in the teams that create and maintain the metadata language and the teams that create the technology specific frameworks to handle the metadata at runtime, but the effective amount of functional programming done, is ridiculously small, where as this is exactly what our customers pay for.  Same functionality, developed in less time equals happier customers and more profit.
  • Faster time to market, again.  In traditional software development, “if you can express it, you can develop it”.  This means, if the salesperson has an agreement with the customer, a functional analyst can write it in a document, an architect can design it, and a software engineer can code it.  Another reason why MDDD speeds up development (and thus lowers costs), is that both functional developers, salesmen, project managers, … work on the same metadata, just on different levels of detail.   “If you can express it, you developed it”.
  • Lower cost of change.  For both functional as technological choices, there is only one version of  every piece of truth.  Compare these requests to change something, to how long it would take you in your current project:
    • The maximum length of a person’s last name should be 150, instead of 120 characters long.  This should be changed in the database, should be checked on the server-side, and visible to the end-user by limiting the amount of characters in each textbox where the last name is shown.  In MDDD, this means changing one element in your metadata.
    • Your UX responsible decided that each button ( / command) should have a tooltip, which you didn’t implement.  Your current software suite has 5000+ buttons.  In MDDD,this means changing the one place where the decision is made to represent commands as a button, and creating a button that has a tooltip instead.
    • Because of the sheer amount of users, the servers of some of your applications are lagging and your team decides all server calls should be done asynchronously from now on.  In MDDD, this means making one change in the one place where is decided on how a command is routed to a specific server call.
    • Your QA team wants to create automated (coded) UI tests, and asks you to add a dependency property to each of your links, buttons and rows.  In MDDD, there are three places in the framework that will require a small change: where links, buttons and rows respectively are mapped to the metadata.
    • Management would like fully working WP7 aswell as a Windows 8 version, and an HTML5 website written in ASP.NET MVC3, out of the 30+ WPF applications you have previously created, to remain competitive in an rapidly moving forward market.  In MDDD, this is still a huge change, because your company will need to set up 3 new frameworks (or framework teams).  However, after this investment, not only can users choose to run the WPF desktop, WP7, Win7 version or browse the web app, but they can do so for all 30+ previously created applications as well.

How LightSwitch fits in.

Although I did out a lot of critique on LightSwitch implementations on my blog in the past, let it be clear that deep down inside I am a LightSwitch fan of the first hour.  LightSwitch itself is targeting small business application development, and imho. accomplished that goal with glance.  Although at first sight, LightSwitch seems to be a RAD tool, it’s actually a MDDD tool, environment and framework.  Spend some time with it, give it a chance if you haven’t done so already, don’t just look at the tool but look at the API of their framework, and soon you will see the beauty of it.  Even if you don’t use it for your own personal or professional development, let it aid you in showing you what MDDD is about, let it teach you that not everything fits into those boxes of knowledge you already posessed, let it help you combine what you already know into more pieces of the puzzle that forms your development career.
I unfortunately have to admit that LightSwitch is the closest to MDDD I have ever seen accomplished, it is not MDDD “pur sang” (or more correct: what I envision MDDD “pur sang” to be).  LightSwitch Markup Language (LSML, the metadata language used to write the ApplicationDefinition.lsml file, the heart of any LightSwitch application), is a closed format.  No extensibility in the metadata language, implies that the pure separation of the three phases described above will be violated.  This is effect surfaces in a technology dependent extensibility model, losing a lot of the MDDD flexibility.  If they made the LSML an open format, and people could express their custom shells or controls in metadata, the community could write an HTML, WinRT or WP7 framework that would operate at runtime on the exact same metadata and generated classes, as the current Silverlight 4 applications that are currently created with LightSwitch…  I’m sure that software component vendors like ComponentOne, Telerik and DevExpress would all be thrilled to bring not only their Silverlight components, but the corresponding HTML or WinForms ones to the LightSwitch developers aswell.  Ofcourse, I must admit that this counterargument is based on a healthy guess, and thus only valid until someone proves me wrong.
LightSwitch is a good sign however, that there are people out there that think alike, and one day we’ll fully separate our business logic from our technological and implementation choices, making our applications stand the test of time and technological improvements…

Q&A.

Three questions were asked to me when explaining my ideas around MDDD the first time, I’d like to include them and properly answer them below.  If you would have any others, leave me a comment, I’d be happy to discuss them with you.  I’m a man of 0’s and 1’s, and often struggle express my thoughts into human language, let alone English (which is not my native tongue).
  • Is MDDD your idea, or are you just describing LightSwitch?  MDDD is not my idea, I just combined answers I already had with answers LightSwitch offers to some common development issues, with some unanswered questions.  The colleague mentioned in the intro, told me later that there was a name for this way of software development: MDDD, which has apparently been around for quite some years aswell, but to my knowledge, never been as closely approached as the LightSwitch team has accomplished.
  • What’s wrong with the “PeopleManagementView”, ie. describing how the screen should look in a language that was intended to do just that, Silverlight XAML for example?  Describing how a screen should look and feel, should be done in the metadata language, so it can survive the technological choice of that moment.  If it’s not in the metadata, your metadata is not reusable and a substantially large amount of work will have to be redone with every change across your application, be it some kind of UX thing or the entire technology used to render your UI.  If it is in your metadata, but that metadata is used to generate a “PeopleManagementView” (the RAD way), then I consider this to be a duplication.  Two versions of the truth to maintain, once again reducing the flexibility of MDDD.  Using the metadata at runtime is exactly what defines MDDD, and makes it so powerful.
  • How does MDDD compare to the SOLID design principles, architectural patterns such as SQRS, … ?  I fail to see how MDDD opposes these good practises and well thought patterns. MDDD is more of an ideology, a way of making software.  Both the limited amount of generated classes that serve as a translation of the metadata, as the framework that works with the metadata at runtime, as the tooling used to maintain the metadata, should embrace these design principles, patterns and practices.
About these ads

3 thoughts on “Metadata driven development, the Holy Grail of software development.

  1. Hi Jan, are you aware of short courses that teach the principles of MDDA/MDDD ? I am looking for alternatives to present this concept to a small group of IT professonals in Panama and have had no luck so far. Thanks, Henri

    • Hey Henri,
      I’m not really aware of any short courses. Myself I became a ‘believer’ after reading the ‘DRY – Do not Repeat Yourself’ chapter of the book “The pragmatic programmer: from journeyman to master’. They have a fun and persuasive way of bringing their content, so perhaps it could help you find the right words for your presentation.

      Good luck!

      Jan

      PS: Panama? Wow lucky!!

      • Jan,
        Thank you so much. I have also applied it way back but learned it the hard way. I will check out the book.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s