Programming for Designers

Over the recent years, the field of programming has interestingly opened up from being an exclusive specialist profession (’the programmer’) into becoming more an more a normal toolset for non-programming specialists. Be it biologists, aeronautic or insurance specialists, or as well – designers and artists.

There are three major reasons for this development: a) the amount of raw data (databases, spreadsheets, texts, files, …) available to any profession is growing rapidly, especially with the web, so the number of ‘please, i don’t want to do this by hand all weekend!’-cases increased exponentially, b) completely new design possibilities brought by tools and different thinking, and c), so called High Level Languages have been developed in a growing manner. But what is that?

Low Level vs. High Level

In the early days of programming, everyone who wanted to program a computer just had to know very detailed facts and pitfalls about the hardware of the computer itself. You had to know about pointers and registers, and about claiming memory and freeing it up again if not needed anymore. I still remember about this vaguely from my old C64 Assembler book, which i never finished or even fully understood. I just remember that as being somewhat scary and incredibely tedious. Programming close to the hardware is called Low Level. The most classic low level language still widely used today is C/C++. So you might ask, why should anyone ever want to go that tedious and error prone path? There are three reasons: speed, speed and speed. Programming close to hardware (the ‘metal’ as some call it) and set everything up manually gives those low level languages a speed level other languages simply cannot reach today even remotely. So if your job is to write or extend a 3d render engine or an operating system like MacOS X, Linux or Windows that is surely very important for you. Might change a bit though with multicore processors and network computing, but that is a different topic of its own.

So that was low level. Now high level: if your job is to administrate a dataset of a few thousand employees for example, check if phone numbers are correctly formatted and such, you rather want to concentrate on that job, and not deal with memory or pointers. And performance is not so much of an issue, as it does not really matter if your script runs 20 seconds or 120 seconds. You just run it once a day 4 o’clock in the morning anyway. Those languages usually have a very clear and easy to understand syntax and grammar, and deal with translations to the hardware automatically. That way i.e. the list of the employees is removed from computer memory after the script is finished, which is called garbage collection. This might seem a normal thing for normal users, but it is not really. One had to add, that any of those high level languages run on a low level layer themselves, mostly C – but the user does not have to know that unless he really wants to. Examples for high level languages are Ruby, Python, Actionscript and many many more. Those code is parsed and translated word by word for the machine when the script is run, that why it takes some time. This is called interpretation.

As we see the issue with high and low languages is always a trade off between performance and ease of use. I guess one should mention something like a third hybrid category between high and low, the most common of which is Java and the Java VM (Virtual Machine). The concept behind this technology is, to have a more or less high level language which can be learned and used easily like Java, but still have the best performance to come with that. After you write Java code and execute it, a compiler translates that code in the background into so called bytecode, which can run on the Java VM. The VM then deals with the hardware, that means Java code runs on any computer platform without having to adjust the code at all, which is a big deal. The code in the VM can run much faster than if Java code had to be interpreted, and on top of that with the help of sophisticated analyzers in the VM, the bytecode is even compiled further down to machine code if necessary. That is called the Just-In-Time compiler, or JIT. So it might be a complex Java program with many loops and calculations might run faster after a few minutes than it did in the beginning. Yes, i agree – this seems very clever. As it is such a clever concept, many other classic or new high level languages find their ways into the Java VM, as for instance JRuby, Jython, Groovy or Scala. On the other end the Java VM opens up more and more for those languages, whereas the performance is still far below that of the pure Java, because the JIT optimization has a hard time figuring out what’s going on in those loops of high level dynamic languages running there in its box.

On a sidenote: one should not mix up Java, the Java VM, and JavaScript. Java is the syntax of the language to originally program for the Java VM. The Java VM is more a platform, think of it as a layer which always looks the same from the top side, but fits on each different computer or operation system from the bottom side. And JavaScript is just a high level language mainly used for webdesign. It does not have to do anything with Java at all, but was called JavaScript because Java was a business buzzword 10 years ago. Today it might have been named ‘Social2dot0Script’ or so.

Okay, but enough for introduction – now let’s get to the evaluation. Please note that the technical introduction above is important to understand the details later on, so you might jump back to that if needed. Further everything here is completely subjective through my own pair of glasses and use cases. You might have very different needs, so please take these paragraphs just as a colleague thinking out loud, and not as a universal ranking system by any means. Take this as a starting point to perform your own evaluations. It’s fun!

Codebased Realtime 2D: Director, Actionscript, AIR, Flash/FX, JavaFX and Silverlight

The well known and widespread complex usually refered to as ‘Flash‘. It is a commercial solution offered by Adobe, consisting of a combo made of a free webbrowser player (Flash Plugin), a free browser and desktop player (AIR), and a commercial authoring environment (Flash). Flash offers graphic authoring, import from Photoshop and Illustrator and so on, and got its own high level (see above) embedded script languages (Actionscript/FX). Flash established itself as something like the animated interactive form of PDF for the web. Everything is very reliable, vector antialiasing and performance work well, it’s got a great video codec – but it is mainly a 2D environment. Add-ons like Papervision offer 3D capabilities, but it is not what Flash was made for in the first place. Flash is geared towards interfaces and webpages, which more and more grow into ‘rich internet applications’ (RIA). Interactive web content, movie websites, webshops, and in the future mobile interfaces are the main target market. I never looked deeper into Flash, as i am not inside this target market. If i would, this surely would be a player i could not get around. Some designers use Flash outside the web, i.e. for art installations or VJing, but those are rather rare cases from users already firm in the language as opposed to looking to pick up a new one. In the 90ies, the first commercial media programming environment Macromedia Director and its language Lingo had the Share which today Flash got, in the beginning more CD-authoring-based, later on for the rising internet. Adobe Director still exists, but is not found on the web very frequently anymore.

So Flash/FX basically owns the market it is in, and as this is a market with a lot of money involved (think webshops and the interface of future TV sets), competitors are trying hard to catch up at least a little. JavaFX is a new scripting language introduced 2008 by Sun Microsystems, who basically are the Java folks themselves. It is not really open source, but comes free as Java and its IDEs do as well. At first look JavaFX offers some interesting features, i.e. the language is easy to read and write, you have nice commands for timelines, or you got a bind command which let’s you create a autoupdating network of values. Today JavaFX is 2D based, as they claim to introduce hardware accelerated 3D in future releases. On the downside: the language seems so very eager to win market from Flash, that is feels more like a medley of other languages designers are supposed to like, as opposed to come up with something really original. If you look at the samples and you know Processing you know what i mean. FlashFX tries to catch up rather than to jump to the wide open. The biggest downside for a designer might be, is that the Java community, as nice and polite and loveable as they really are (JavaPosse), is not a design community. We have to adress a very common misunderstand in the non-design world here: this article is on programming for designers, which is NOT exclusively web designers at all! There might be artists or product designers out there who are very concerned with prototyping interfaces, as they know that will make up most of future products surface, but do not care at all about CSS-Stylesheets and AJAX. So do not expect to find a peer group intuitively understanding where you are heading with your project. The focus in all communities is geared towards using FlashFX to put interfaces, GUI buttons and videos on a browser or cellular canvas, which run on existing Java code and maybe a database like Amazon or YouTube does. Nothing less, but nothing more. Oh, and Silverlight is basically Microsofts way of trying to get market share from Adobe Flash. The language itself is based on is C# and you have to use authoring software by Microsoft, other than that most of the above applies as well.

Codebased Realtime 2/3D: Processing and Java

Processing is a script language based on Java with a simplified Java syntax, and was created by Ben Fry and Casey Reas based on John Maeda’s ‘Design by Numbers’ at the MIT. It is widely spread amongst media artists, university students and graphic designers. It is save to say that this language is the one who was the first to adress programming for designers, especially when related to data visualisation and infographics. The language, being open source, spun off some interesting derivates, like i.e. a mobile edition or one for interfacing with Do-It-Yourself electronic devices. The concept and the attitude of the software is very well developed, and one got to point out that the online documentation, learning examples as well as the community are outstanding. If a designer is looking for learning material and a peer group to understand his issues – here it is par excellence. There are modules to extend the functionality of Processing, i.e. for exporting PDF or DXF, for video and tracking, and for many more. Once used to the language, the introduced term ’sketch’ for writing a small programm fits very well.

So far, so good – but there are downsides: the introduction to Processing states it is mainly a prototyping tool, and, running on the Java VM, it is reasonably fast, but is not geared towards a full featured production environment. That is understandable, though in real life people efficient with the language might probably sooner or later get to a point when they are asked to create something very complex in a production situation, i.e. for a museum or a trade fair. What then? In my (not overly complex) études Processing performed very nicely, though when going from web to beamer resolution and introducing lots of things to draw, one feels there certainly could be issues with framerates once. Especially tweaking and debugging is hard in code, and certainly something you might not want to do if a client sits next to you. And if you are urged to do things like a more complex application, classes, databases, multiscreen-display and debugging OpenGL display, you quickly get to a point where you feel lost as a designer without a solid programming and Java background. As joyful it is to create smaller personal experiments and sketches in Processing, be aware that if you are planning to grow with the language over the years (and that’s what we want, right?), you will sooner or later be forced to deal and become efficient with Java programming itself. That is, let’s say, less enjoyable for a designer, and might mean hiring specialists for production projects. On top of that, looking around in the Java community, it seems that even professional Java developers are looking into ways to get away from Java itself for technical reasons, and are flirting heavily with currently hyped languages like JRuby or Scala. So digging your way as a newbie into pure Java in 2009 might just not be the best of all ideas it seems.

Nodebased Realtime 2/3D: MAX/MSP, Quartz Composer and vvvv

If a designer wants to utilize a programming environment, but does not want to get into typing code and therefore avoid pitfalls like typos and spending an evening on looking where exactly the bracket and the semicolon is missing, these toolboxes come to mind. All of them have in common, that they are so called graphical programming environments. That means instead of typing commands and variables as text, you arrange blocks with specialized functions (so called ‘nodes’) in a flowchart-manner and draw connections between them. As there is no code to be interpreted involved, the nodes are usually written in a low level language (which the user does not have to bother with), and therefore really fast and responsive, even with large networks. Setting up, testing and understanding programs by working with a diagram is certainly very helpful, while having a few downsides as well in exchange. All of the three above applications offer graphical programming environments. I will not go into deep on MAX/MSP, but it has to be mentioned because it is the ancestor of graphical programming environments. Its origins are in art installations, and especially in sound – so the package for doing graphical output called Jitter has been put on top later on. The one thing which makes working with MAX/MSP unconfortable is that you have an edit and a run mode, meaning you always have to switch back and forth for testing and changing. Quartz Composer and vvvv on the other hand offer valuable realtime feedback on things you do to your network of nodes (called the ‘patch’).

Quartz Composer (QC) is part of Apple’s Developer Tools, and is free to use. The interface is what we know of Apple products, very clean and efficient. QC comes with a big set of nodes, though most of them are ready-made GPU effects and filters like bloom or color correction. The performance of the graphics system running on the MacOS X Quartz system is remarkably, and the text rendering engine is just very clean. One can extend 3rd party nodes and plug them into QC. The product lives in the apple world though: it does only run on a Mac, and there it runs in Quicktime, QT-related products like Motion or Final Cut, as Standalone or i.e. as a Screensaver. There is no webplayer for it and it does not run in a browser’s Quicktime, as it is so tightly integrated with the Mac’s graphic system. It performs really well! But there are downsides: in production use, you might have to deploy your QC patch. A patch could i.e. make a great Kiosk system or drive a huge media wall on a fair, and if it is very graphics hungry and does not run well on a Mac Mini’s graphics card – what hardware is it you quote your client to lock into the dark plywood-boxes of a dozend terminals…? But besides financial issues, there is one big bugger you might look into before you spend too much time on QC. In a QC patch, you have to go a rather complicated way of nesting patches into patches into patches. There is no easy way i found to keep good overview over just a medium complex patch and navigate efficiently, and it can feel a little like being blindfolded. That way it can be pretty tedious to dig through those nestings after a few weeks to find out what you once did.

vvvv is a very powerful ‘pay-when-you-earn-money’-ware created and maintained by a german studio called Meso. Like Processing it originated from a thesis project, and started as a inhouse-tool for their own media projects. In terms of platforms it is somewhat the opposite of QC, it only runs on windows and nowhere else. While this might seem politically incorrect for some people, the reasons are rather pragmatic than ideologic: a) Windows hardware can be bought really cheap without a case and built into an installation, and be equipped with b) one or more mass-market and longtime 3D-shooter-hardened screaming graphics cards. Building on DirectX is morally disputable, but the technology and drivers have just seen any game-triangle-madness out there. With that in your back, expect to see very cutting edge performance and stabilty in vvvv, combined with a node based graphical programming environment. There are as well ways to render 2D graphics like vectors and text, though these seem to be more a sideline, and offer not as well suited features as Processing or QC do. The second remarkable feature of vvvv besides 3D is I/O – vvvv can out of the box basically talk to almost anything you might think of. Networks, lighting installations, musical instruments, file sytems, Wii controllers, whatever. Similar to Processing, the community is a very friendly and open bunch, and documentation is similarly well done.

Nodebased Non-Realtime 3D: Maya, Houdini and Softimage ICE

Softimage offers since 2008 a new and innovative (at least if you don’t talk to Houdini users) graphical development framework called ICE. On the other hand Houdini has been there for a long time, and was always somewhat of a niche product – but a very highly regarded one. Both products are full blown 3D/CG design packages, so they have steep learning curves and a certain price tag, which make them suitable for high-end professional use and not really for occasional sketching. Nevertheless the concepts are interesting to look at, as they might trickle down over the years into other applications. In appearance and usage these node-based systems are very similar to QC or vvvv. The difference ist though, 3D applications always have been programms with a very rich feature set (many buttons), each of which was geared to once special purpose. These graphical programming frameworks change this approach: the common tools like ‘make helix’ or ‘push surface normals’ are still there, but internally they already have been prebuilt with the same nodes the artist can access. That way one can open the hood and either extent or change existing tools, or build very complex and custom ones for oneself, say ‘make 99 animated helixes’ and ‘push surface normals in a sine pattern’. Any way, leveraging these custom tools one with another can create incredibely elaborated and complex designs or animations. This complexity though can be sometimes overwhelming and wants to be mastered – but that is true for all programming tasks.

Codebased Non-Realtime 2/3D: General languages and design applications

To close with this article i will go from all those interesting specialised tools back to the beginning of this article – high level languages. They are easy, they are common, and almost any serious design application got one for sure, just look for it in the documentaion. There are a few standard ones, but unfortunately not one single common standard for embedded high level languages. But maybe that is how it’s got to be, survival of the fittest. So it is best to start to look with the application you use the most of the time, and just jump into that language. Going from one language to another language is easier than getting into the first one anyway. If you are mainly using Illustrator, you might look into JavaScript, if you are a SketchUp guy, your starting point would be Ruby, if Maya is your thing you would go for Python. In the 3D world it seems Python generally is a good choice, as there is hardly any application not offering support for Python or another open high level language these days. And of course once you learned that, you might find yourself to use the language as well for different and less artsy tasks, as i.e. making file copies on a backup drive. Or distilling yourself an abstract from this article, which got longer than i originally intended to. Thanks for reading!

Disclaimer on Mathematics

Even if up to here this text is a software overview, one obvious but very fundamental thing applies to all of the above  - you do not have to be a master of mathematics to create your own design programs, but you should have interest. Basically no matter which tool or language you are using – the logic of the code has to be thought of by a human (you) first. Most of the time if not always that is the hardest part, and mathematics and logic are always sitting on your lap. At the beginning this can be rather tricky for us designers as this was not a central part of our training, but – there is no way around it.