The Cloud ... and why aren't we using it properly?

I had a random thought while walking the dog this evening, it went a little like this:

• Cloud computing is fairly ubiquitous 
• Everyone is developing applications that are "Cloud" based (e.g: App Engine, EC2, Azure ...)
• Where are they developing this software?

So the final point is the bit I'm interested in.

Being the hack that I am, I use a mixture of Eclipse, Xcode, TextMate and various Database tools to write the stuff that I do. All of this is based on the fact that the machine in front of me is a Mac. I like Mac's, they compliment the way I work. I use Windows/Linux at work, it compliments what I do there.

But the point of the matter is that I'm developing on a platform in a "platform independent" language thats eventually going to be living in the Cloud. Does it matter what platform I'm using to develop on?

This lead me to another question:

Should I even be developing software on my desktop?

The answer to this (for the Cloud anyway) is: No.

If I'm writing something that lives in the cloud, why am I not developing it in the cloud itself? Why is there not an IDE that lives in the cloud? Why isn't there an equivalent of Eclipse that is web based? Hell, it means I wouldn't have to find where I left the jar for x.y.z version 1.2.3 .... because its already out there ... 

This is what I'd like:

1. Something usable - much like Eclipse or XCode - a useful IDE
2. Make that puppy web based 
3. When I do something like the following:
   AdminClient theClient = new AdminClient();
   It gets the correct jar file from ... the internet!
4. It means I can use a browser anywhere to do what I need to do ...
5. It means that we are one step closer to not requiring anything more than a browser

I know that its probably quite impractical, the amount of JavaScript or other enabling language would be immense to accomplish this and the load times would be immense. But they wouldn't have to be.

Google just released Google Web Toolkit 2.0, it has a snazzy feature called code splitting. It only ships you the bits of javascript that you need at the time. If I dont hit ctrl-space to autocomplete a snippet, then dont ship the autocomplete to me. If I'm not playing with build configurations - dont send me the js to do that.

Make this lean and mean and all it means is that I'll just need a netbook to do everything, then I can ditch the power hungry behemoth on my lap (although it is shiny and looks very nice). Why not give me a tablet-esque thing with a touchscreen so i can use it like a pad and paper, the local hardware can do all of the pen -> character conversion, and the cloud can do the heavy lifting.

Just a thought ...

-- Update: 22nd September 2010

So then Rails 3.0.0 comes along and adds bundles. You have a gemfile that automagically install all of your required gems from the interweb. Use it on Heroku and your reuiqrements are satisfied immediately .... Nice.

Hudson, iPhone, Xcode and Unit Testing

After much messing about I have finally managed to get Hudson to work with xcodebuild, this is for an Objective C iPhone project, so the things below were modified to fit our requirements

I have the following things running

• Automated Builds (SCM polled every 15 minutes for changes)
• Unit Testing (courtesy of google toolbox for mac)
• Profiling (a modified version of the gcovr python script + some nice compiler flags)
• Code Coverage (courtesy of Cobertura, gcovr, profiling and scan-build)
• Static analysis reports (CLang)
  

This is what I have:

• mac mini (continuous integration server)
• Snow Leopard Server (10A433)
• Apache Tomcat (Application Server)
  
• Hudson (Continuous integration)
• Hudson Cobertura Plugin
  
• Subversion (SCM)
• gcovr (slightly modified to fit)
• A whole bunch of Apple documentation (and some dead trees)

onward ... to the setup!!

Having used Hudson quite a bit in the past as a CI environment, I was careful to ensure that my HUDSON_HOME was located in a separate location from the tomcat installation, This makes installing updates to hudson relatively easy (just dump the new war file into the WEBAPPS directory and restart Tomcat).

I segregated the user which runs the tomcat environment (not the standard _appserver user), its just a habit I have and it lets me be able to pick out my processes a lot easier. On an OSX Server you'll want to add a new group and user (_tomcat for the group and _tomcat for the user), try and stay below 500 for the userid, that way you its not a real user (to the operating system).

Getting your Tomcat environment to Auto start on boot and restart if it dies require the use of a plist file (we'll look at that later) and launchctl to load it. The plist also needs to be copied to /System/Library/LaunchDaemons so that it will restart on boot.

Here is what you will need to acquire before beginning:

1. The hudson war file
2. A copy of gcovr (you will need to do some modfication to make this work if you have no unit tests)
3. An iPhone project
4. Some actual Unit tests defined as a build setting (I used UnitTests -please tell me you wrote tests for your code?)
5. xcode installed
6. iPhone SDK installed
   
7. Your Mac
8. Patience
9. Time

Here we go ...
Here is my handy dandy script to go add the user/group, fix up some permissions, create some directories and fire up Tomcat with Hudson installed, just make sure you have you hudson.war file in the directory you run this script from, you'll also either need to be root (sudo su -) or sudo to run it (sudo ./configure_tomcat.sh).

#configure_tomcat.sh
#!/usr/bin/env bash

if [ ! -f ./hudson.war ];
then
echo "You are missing the hudson war file in this directory"
exit 1
elif [ ! -f ./server.xml ];
then
echo "You are missing the server.xml file in this directory"
exit 1

elif [ ! -f ./org.apache.tomcat.plist ];
then
echo "You are missing the org.apache.tomcat.plist file in this directory"
exit 1

else
echo "+ Ready to install .."
fi


# Is tomcat already running? If so - unload it, reconfigure then reload

TCRUN=`launchctl list org.apache.tomcat 2> /dev/null | wc -l`

if [ $TCRUN != 0 ];
then
# Tomcat is running already - unload it
echo "+ Tomcat appears to be running, unloading ready for configuration"
launchctl unload org.apache.tomcat.plist
else
# Tomcat isn't running
echo "+ Tomcat not running, ready to configure"
fi

# Determine if our users and groups already exist, if so - don't recreate them, else, go and make them

ISTOMCATGRP=`dscl . -list /Groups | grep _tomcat`
ISTOMCATUSR=`dscl . -list /Users | grep _tomcat`

if [ ! -d /usr/local/ ];
then
mkdir /usr/local
fi

if [ "$ISTOMCATGRP" == "" ];
then
# create the tomcat user for hudson to run as ...
echo ""
echo "+ Creating group _tomcat as PrimaryGroupID 300"
echo ""
dscl . -create /Groups/_tomcat PrimaryGroupID 300
dscl . -create /Groups/_tomcat RealName "Tomcat Users"
dscl . -create /Groups/_tomcat Password \*
sleep 5
fi

if [ "$ISTOMCATUSR" == "" ];
then
# Create the _tomcat user in the _tomcat group
echo ""
echo "+ Creating the _tomcat user as UniqueID 300, PrimaryGroupID 300, NFSHomeDirectory /usr/local/tomcat"
echo ""
mkdir /usr/local/tomcat
chgrp _tomcat /usr/local/tomcat
chmod g+w /usr/local/tomcat
dscl . -create /Users/_tomcat UniqueID 300
dscl . -create /Users/_tomcat PrimaryGroupID 300
dscl . -create /Users/_tomcat NFSHomeDirectory /usr/local/tomcat
dscl . -create /Users/_tomcat UserShell /bin/sh
dscl . -create /Users/_tomcat RealName "Tomcat Administrator"
dscl . -create /Users/_tomcat Password \*
sleep 5
fi



# chgrp the entire /Library/Tomcat directory to _tomcat

chgrp -R _tomcat /Library/Tomcat

echo ""
echo "+ Copying server.xml and hudson.war to required locations"
echo ""

cp server.xml /Library/Tomcat/conf
cp hudson.war /Library/Tomcat/webapps

sleep 5

echo ""
echo "+ Installing the org.apache.tomcat service via launchctl"
echo ""

chown root ./org.apache.tomcat.plist
launchctl load ./org.apache.tomcat.plist
cp ./org.apache.tomcat.plist /System/Library/LaunchDaemons

echo "Install complete - Hudson is available here http://`hostname`:8080/hudson"
echo ""

Get a copy of the server.xml file from the /Library/Tomcat/conf directory, find the server.xml file and copy it to the same directory as the above script.

Modify the server.xml to have the following in the connector section

URIEncoding="UTF-8"

Finally, get a copy of the org.apache.tomcat.plist file from /System/Library/LaunchDaemons and modify it as follows:

• Add HUDSON_HOME as a key in the Environment Variables section
• Set the string for HUDSON_HOME to be /usr/local/hudson
• Change the UserName to be _tomcat
  
• Modify the Disabled key to be Enabled

Now that we have the above (saved to your machine) we should be ready to proceed

run the script (the first listing) - this will add groups, users etc, and setup your application environment and launch hudson.

That was easy :)

Next step - configuring your hudson environment to compile your code (ala xcodebuild)

Jump into Hudson (http://localhost:8080/hudson) and do the following

• Click on "Manage Hudson" -> "Manage Plugins" -> "Available"
  
• Add the following plugins

1. Hudson Cobertura plugin
2. Hudson Disk Usage plugin
3. Hudson DocLinks plugin

Once the plugins are downloaded hit the restart button in hudson

Now that we have these plugins available, we need to get a copy of scan-build (static analysis reporting) and gcovr (python based code coverage output in cobertura style XML format)

I extracted the scan-build checker package and installed it into /usr/bin (scan-build and scan-view) all of the libs into /usr/lib, the libexecs to /usr/libexec and the man pages into /usr/share

If you have no unit tests, you may need to modify gcovr to work, just edit line 178 and change the extension from gcda to gcno.

Now that we have all of the bits and pieces installed, its time to configure a job:

Go back to the main hudson page, click on "Manage Job"

Add a new job with the following options:

• Build a free-style software project
• Give your project a name and click on OK
  
• Select your SCM (I use Subversion)
• Enter the details for your repository location
• Set your build triggers (I poll the scm every 15 minutes for changes */15 * * * *)
• Make the build paramaterized (I set it as a "Choice" with the name TARGET and the options UnitTest and the Project Name as the second choice)
  
• Add a build step (Execute shell) with the following
• 
  

  cd Whatever your project is called in the repository && scan-build xcodebuild clean -target "${TARGET}" -configuration Debug -sdk iphonesimulator3.0 GCC_GENERATE_TEST_COVERAGE_FILES=YES PREBINDING=NO GCC_OPTIMIZATION_LEVEL=0 && rm -rf html

• Add another Build step (execute shell) with the following
• 
  

  cd Whatever your project is called in the repository && scan-build -o html xcodebuild -target "${TARGET}" -configuration Debug -sdk iphonesimulator3.0 GCC_GENERATE_TEST_COVERAGE_FILES=YES PREBINDING=YES GCC_OPTIMIZATION_LEVEL=0 DEAD_CODE_STRIPPING=YES GCC_DEBUGGING_SYMBOLS=full GENERATE_PROFILING_CODE=YES GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES OTHER_LDFLAGS=" -L//Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator3.0.sdk/usr/lib/gcc/i686-apple-darwin10/4.2.1/ -fprofile-arcs -lgcov -prebind "

• Add one last build step (execute shell) with the following
• 
  

  cd Whatever your project is called in the repository && /usr/bin/gcovr -r . -x -b -e /Developer 1> html/coverage.xml 2>/dev/null

A brief explanation about what happens above

1. Cleans out your build directory
2. scan-build the code (CLang static analysis) and add profiling and test coverage
3. Profiling will occur once the unit tests begin and generate gcda files analyzing the code you actually invoke (cobertura style too)
4. Finally we generate the cobertura information using gcovr

Now - we need to add the static analysis output

• Click on publish documents
• Give it a title and description (static analysis is a good choice)
• Set the directory to be Whatever your project is called in the repository/html

Now add the cobertura coverage

• Click on publish cobertura coverage report
• Set the xml report pattern to **/html/coverage.xml
• Modify the coverage settings to suit
• Click on consider only stable builds

That should be it?!?

Now save the project and start it up - you should now get some funky code coverage, and static analysis. If it all goes belly up, click on the build and look at the console ouptut, this should point you in the right direction.

You should really be using google toolbox for mac (with the iPhone Unit testing enabled) this allows for some really good profiling of your application.