MacPorts headaches... follow-up

Some of these issues apparently have to do with the order in which things are done and may also depend on the fact that some of these packages require native C-language extensions.

I wanted to work with nokogiri to do some XML parsing. Given my previous post, I figured I'd better install this using MacPorts, which I did:

     sudo port install nokogiri

That was pretty straight-forward and worked quickly. However, when I went to irb to check things out, I got the following message:

irb(main):001:0> require 'nokogiri'

HI.  You're using libxml2 version 2.6.16 which is over 4 years old and has

plenty of bugs.  We suggest that for maximum HTML/XML parsing pleasure, you

upgrade your version of libxml2 and re-install nokogiri.  If you like using

libxml2 version 2.6.16, but don't like this warning, please define the constant

I_KNOW_I_AM_USING_AN_OLD_AND_BUGGY_VERSION_OF_LIBXML2 before requring nokogiri.

 

So I quit irb and installed libxml2

     sudo port install libxml2

which also went just fine.  Unfortunately, nokogiri still gave me that long warning message.

What I had to do was re-install nokogiri so that it would recognize the new version of libxml2 (which was now in the /opt/local tree) and then everything worked fine.

Seems like I can get things to work now, but having to remember previous dependencies and re-install packages to pick up those new items isn't going to be fun.

MacPorts headaches...

Recently I started doing some of my development work on a little MacBook. Pretty nice as I drop down into terminal, have all the power of *nix (FreeBSD-based in this case), and still have Mac niceties available to me, such as MacVim for editing.

Everything is rocking along just fine: I've got Cucumber and RSpec running and working my way thru "The RSpec Book" (pretty useful so far). However, I needed to install FXruby so that I could do some graphical work and figured this would be no problem. Turned out to be a little tricky and I had to remember an issue I ran into a while back on a Linux system.

First off, this won't work:

     gem install fxruby

It'll install that gem alright, but that's not enough to get things running: you have to get Fox installed as well and that doesn't come along with the gem.

Of course, to do that, you have to install MacPorts which is actually relatively painless (see the MacPorts Guide for how to do this).

Once you've done this, then you can install the complete FXRuby collection of stuff (see the FXRuby Book at PragProg.com)...

     sudo port install rb-fxruby

and it's a very large collection indeed: it took over 1 hour on my machine.

The only hitch I ran into was having too old a version of the Mac XTools package, so that was another 800Megs of download and an install. If you do this make sure you get the right version: XTools 3.1 is for OS X 10.5 and XTools 3.2 is for 10.6. Check this first or you'll spend most of an hour installing fxruby only to have it complain about the wrong XTools version and you'll have to start over.

MacPorts updates your ~/.profile or ~/.bash_profile file (note that the MacPorts install docs only mention ~/.profile and I had to dig a bit to find that they had properly modified my ~/.bash_profile file) so that your system can find the added programs...

But that's not quite everything you have to do and here was the gotcha for me. Even after I went through all this, I kept getting strange messages out of my test runs, things like:

  "unable to load file ubygems"

(the missing 'r' is not a typo: that was the exact error message) when executing the "require 'rubygems'" statement.

I tried adding the command:

      export RUBYOPT=rubygems

to my profile so that I wouldn't have to add a require 'rubygems' statement but that didn't solve the problem.

BTW, when I went to the RubyGems Guide, they said to use the -rubygems option on the ruby command line, but that also generated the same error.

After a little more digging, I found the problem: MacPorts installs everything in /opt/local, which is not where the normal gem command installs things (if you do a simple "gem install ' they wind up in /usr/bin).  Thus, after MacPorts was installed, ruby got a tad confused and couldn't find everything it needed.  Then I remembered what I had run into when I was working on my Linux box: trying to install a LAMP stack: everything got put into /opt/local, but then it was out of the normal stream of things and there were ongoing problems. I finally bit the bullet in that case, removed the LAMP stack, and manually installed everything into their default locations. Life has been pretty good in Linux-land ever since.

The solution? Install rubygems using MacPorts so that my FXRuby could find what it needed. That was a simple:

     sudo port install rb-rubygems

and now my ruby apps would run again, load the gems, and I had a simple FXRuby app running.

Quite frankly, I'm not sure of all the nitty-gritty details of this problem and I'm not even sure that I won't have further problems downstream when I want to use another gem, but I'll face that problem when I come to it. If anyone has a definitive answer as to WHY this happened and a better approach to Mac OS X gem  installation, I'm all ears.

A Users Guide is a great developer tool...

Today I was reminded just how important documentation is, particularly End-User documentation... you know, the instructions for how folks are actually going to use your system.

In my spare time, I've been developing a very small system for a client (unfortunately not in Ruby/Rails, but that's another story). It's designed to replace one labor-intensive portion of their Commission calculation process, which is spreadsheet-intensive. You've seen them before: someone develops a simple spreadsheet to "save time" and eventually, the spreadsheet is a monster and the whole process takes days to complete and is fraught with potential errors. And with the sales reps watching the numbers like a hawk, nothing slips by.

The piece I'm replacing is for maintaining the Work In Process costs and pricing factors for the products developed  in-house. It's a tiny database application which outputs an Excel workbook that is dropped into the monster spreadsheet and thereby replaces a very time-consuming manual update process. In fact, the process has been so challenging in the past that cost updates have been few and far between. The new system will allow them to make pricing changes at will.

Over the past few weeks, time permitting, I've developed small pieces of the system and run it by the client to make sure that my understanding is correct, and they've approved the prototype pieces that I've provided to them. This is actually part of my proposal process: try out important pieces of the system, and then let the client approve the results.

Today, I started writing the contract for the actual development. As part of the scope of work, I provided screen shots and a high-level overview of the procedural steps for using the system. As I worked through the steps in the workflow, I suddently realized that I was omitting one piece! I had prototyped that portion of the system early on and it worked OK, but the table I had created was only a dummy table, hand-crafted for testing, and I had not provided the necessary support for a user to maintain that table!

Fortunately I found this out before completing development and installing it for my client. At this stage in the game, it was pretty easy to add the required functionality and include it in the Users Guide.

You can call it what you want: Requirements or Design or Testing. Developing a Users Guide can help in all areas of your application life cycle methodology. In this case, I  like to think of the Users Guide as being a form of UI Testing: I used it to test my prototype design (screenshots, data entry forms, reports, and output files) against the user interaction that would be required to actually do something productive with the system.

Does this apply to Rails? You bet! Just watch Jospeh Wilk's Cucumber presentation at Scotland on Rails 2009. He suggests that you "...think about the user interface first " and asks "...how is the business user going to achieve value through the interface?" He also stresses figuring out what's really important to you and to your user (see his example of "Popping the Why Stack"). I particularly like his question: "Have we achieved what the user wants yet?" It's all about a single-minded focus on the user of the system and the value that the system is to provide.

Fortunately my Users Guide test failed and I was able to address the issue today, not later down the road when it would be more time-consuming. And I saved a potentially awkward situation with my client.

So... I'm going to be more careful about writing Users Guides and testing the user workflow in the future...

When do you write your Users Guide?

Snipping the blight... fantastic!

I've been fighting an elusive bug for the past few weeks: I've developed some major enhancements for my passwords site plus upgrading to more recent versions of Rails and associated plugins (improved security and testability being my primary goal).  Things were going along fine until I deployed my app to production: I was able to login without errors, but was then immediately returned to the login screen. Needless to say, the Capistrano deploy:rollback task was just what the doctor ordered (that's why I love Cap: it took a bit to get it setup originally, but it really makes deploying or undeploying applications a snap - if you have an app in production and you don't have Capistrano or something similar, do it now).

Anyway, I setup up a production environment on my development machine and, sure enough, it failed in production mode but ran fine in development mode. I modified the production settings so that I would get error messages [Note to self: put it back like it was!] and got a message saying that a module in the filter chain had returned false... it looked as if it were failing authentication somewhere in the login system.

I started digging into the code and using the debugger to see what was going on: from the symptoms and the error message, it seemed to be a problem with session state. Checking with my hosting tech support and local Rails folks, I learned there were reports of problems with sessions in Rails 2.3.2 and one of the local chaps, Brendon Whateley, reported some serious session issues that he had and chased the code back to Rails initialization and lots of 'magic' aliasing and overriding of ActiveRecord methods (nice stuff when it works). To quote:

"Having spent the best part of a week debugging the sudden failure of sessions recently, I may be able to help.  The first thing to figure out is if the session is saving state and not loading, or loading but not saving.  In my case the ActiveRecordStore was not serializing the session data, but was loading and saving the session object.  This meant that previously set session values still existed, but any changes did not persist across calls.  I was able to validate that the logins were successful, but suffered similar issues to you.

"In the end I tracked the problem down to overriding the ActiveRecord::Base.create_or_update method.  For some reason it completely broke ActiveSessionStore serialization, even if the redefined method contained the exact same code as the original.  I never completely figured out WHY that broke, but if I moved my changes to create and update methods that create_or_update called, I no longer experienced that problem."

Made sense to me, and since I am using some of the ActiveRecord callbacks, it seemed quite reasonable. So off I went, digging thru my code, researching how the callbacks were used, trying to find out if anyone else was using them, more debugging... all of which went nowhere.

Yesterday, I bit the bullet and upgraded to Rails 2.3.3. Very easy; just do

  gem install rails 

and try it again in production mode.

You can imagine my surprise when all of a sudden, the app crashed and emitted the following error messge:

ActiveRecord::StatementInvalid (Mysql::Error: Unknown column 
'assets.type_id' in 'on clause': SELECT count(DISTINCT `assets`.id) AS 
count_all FROM `assets`  LEFT OUTER JOIN `types` ON `types`.id = 
`assets`.type_id WHERE ((user_id = 1 and ( entry like '%' or entry 
like '%' ))) ):
  -e:2:in `load'
  -e:2

I had never received any such error message in Rails 2.3.2 (obviously some kind of bug there which caused this to appear as if it were a filter chain termination), but this made it clear what had happened. In the process of upgrading from Rails 1.x to 2.x, I had experimented with using some database constraint logic. It had been a tad problematic and I wanted to focus on the Rails upgrade so I removed those migrations. It was probably at that point that the development database and the production database got out of sync (I haven't yet gone back to my subversion repository to see exactly what/when it happened, but I may do that yet out of curiosity).  I had added a new "type_id" field that was required to support the new version of ActiveScaffold and it had never been added to the production database, even though the schemo_info table contained the same information in both databases (I know, because I checked before I started the whole deploy process).

So what did I learn from this?

1. Even though there were several other folks looking at this, we all missed the obvious: development and production mode are different not only in the code base but also in the database. I  suffered from a perceptual set when identifying potential sources of the problem (and Paired Programming isn't always a silver bullet). In the future, I'm going to be a little more diligent when exploring potential solution spaces.
   
2. Rails migrations, while very nice, are also fragile. I've had problems with them in the past: simply checking that the schema_info values are the same just isn't always enough. I'm going to add migrations and database state to my checklist of things to review before deployments or when making changes.
3. Rails migrations, while very nice, are also something of a siren song, luring one into complacency about the database. "When they are good they are very, very good, and when they are bad they are horrid." After a long period of no problems, I had assumed (mistake!) that they were under control: in a way, I thought of them as if they were "source code control" for my database. They are not, and I'm going to pay more attention to that in the future.
4. Rails upgrades are in the same category: they can be remarkably easy, such as my switch from 2.3.2 to 2.3.3 or they can be remarkably messy, as has been my switch from 1.3.6 to 2.3.3. On the one hand, I want to stick with a known version: there are just too many changes that I view as frivolous in the Rails code base to want to go through all the deprecations and just plan different stuff unless it's really necesary. At the same time, we do need to stay reasonably up-to-date to take advantage of the definite improvements such as better plugins and greater security. This is an individual decision, based on your particular requirements. Regardless, YMMV. 

So... one more bug snipped out of my app and that much closer to deployment. Some cleanup and more testing and it should be good to go.

Learning RSpec/Cucumber - Day 1

I've wanted to improve my testing for some time now and so I'm pairing on the RSpec book with my friend, Allan Miller. We're going to work through the book/examples in tandem and get together regularly to review what we've learned. We just downloaded the RSpec book by Pragmatic and assigned Chapters 1-3 (this is the book that was highly recommended by our Ruby Meetup folks). We'll work through this and then meet in a couple of days to see how things went and move on to the next group of chapters.

For now, I'm running all this on my MacBook, without any IDE... just straight terminal access and MacVIM. Chapters 1 & 2 are introduction; chapter 3 is where you start actually doing stuff.

First problem (minor) that I ran into was that the rspec installation wasn't clean; I got the following errors:

Successfully installed rspec-1.2.8

1 gem installed

Installing ri documentation for rspec-1.2.8...

Installing RDoc documentation for rspec-1.2.8...

Could not find main page README.rdoc

Could not find main page README.rdoc

Could not find main page README.rdoc

Could not find main page README.rdoc

So far, this doesn't seem to have affected anything -- I just ignored it and moved on (so far, I don't use the ri or RDoc information at all).

The cucumber installation was a little more disconcerting:

Installing ri documentation for builder-2.1.2...

ERROR: While generating documentation for builder-2.1.2

... MESSAGE: Unhandled special: Special: type=17, text=""

... RDOC args: --ri --op /Library/Ruby/Gems/1.8/doc/builder-2.1.2/ri --title Builder -- Easy XML Building --main README --line-numbers --quiet lib CHANGES Rakefile README doc/releases/builder-1.2.4.rdoc doc/releases/builder-2.0.0.rdoc doc/releases/builder-2.1.1.rdoc --title builder-2.1.2 Documentation

(continuing with the rest of the installation)

but again, it was only documentation so I moved on from here as well (have to pick your battles, you know).

Everything went just fine, except for a couple of glitches: one was just some missing default snippets and the other was a result of those missing snippets. Here's what happened:

1. The first attempt at cucumber did NOT display the snippets for the 3 undefined steps (pp 27-28 in my beta copy of the book) as expected. Rather than being able to copy and modify them I had to type them in by hand. [I tried the command twice just to make sure.. sure enough, didn’t display, although a later point where I had only one step undefined, it DID display the snippet template. Go figure.]
2. Because of #1, I made a transcription error in the regexp for step 3: I omitted the ‘^’ exclusion character. This cost me several minutes (regexp’s are like that!). I have a feeling that there may be more of these... hopefully I’ll be able to get the step snippets more reliably in the future and then just copy them into my step definitions as a starting point.

All in all, a successful beginning... But I do have a concern -- given that we're using regexp's, there are MANY opportunities for small errors that can cost time. Seems like a bit of a contradiction: cucumber is a "higher level" testing language than rspec, but it uses one of the lowest level syntaxes to accomplish this.

I'll just have to wait and see what this leads to... and I'm sure going to copy/paste those snippets whenever I can.

Pair programming really works!

I've known this in the past and I had another "Aha!" moment the other day: pair programming (or debugging in this case) is great!

I've had a problem for the past week or so in one of my applications. I'm in the middle of a big upgrade (from Rails 1.3.6 to Rails 2.3.2 and lots of other associated changes), when I hit a hard stop; my screen wasn't getting updated correctly after an Ajax call.

I struggled with this for quite some time and then today called Allan Miller, a good friend of mine over to help me debug it. He sat down, I described he problem and demonstrated it. He said "Well, let's take a look and make sure that you're actually getting an XHR request." so we fired up FireBug and entered a couple of trials: the POST request was properly executed and my application did see it as an XHR request. Everything's working as exepcted so far, but no errors and no screen refresh.

Since he had been focusing on the Ajax call, I decided to show him the code on the page, and used FireBug's feature to "click an element to inspect" and pointed out the code [line breaks inserted for readability; this is all one line of code]:

<a class="ltr-locate" onclick="new Ajax.Updater('asset-content', '/asset/get_assets?ltr=F',
{asynchronous:false, evalScripts:true, parameters:'authenticity_token=
' + encodeURIComponent('xxx')}); return false;" href="/asset/get_assets" align="right">F</a>

"See" I said. "There's the AjaxUpdater call to update the... ... that's it!" The first field in the AjaxUpdater call signifies the named area on the page that will be updated as a result of the call. In this case 'asset-content'. "I'll bet that they changed the name in the intervening releases" I said.

Sure enough, that was the case: the name name is 'as_asset-content' and updating the link_to_remote call solved the problem. What a relief!

I've got 3 take-aways from this:

1. Two heads are at least 10 times better than 1, if you compare 50 total minutes of work (25 minutes times 2 heads) with at least 10 hours of work with my one head. There's just something magical about having to REALLY explain your code to an outside observer that makes you think thru things much more carefully. It's making the thinking process external, rather than internal where your hidden assumptions come into play without your even knowing it.
2. The AjaxUpdater call gives NO ERROR if the named area doesn't exist; it just quietly closes up shop, assuming that you knew what you were doing.
3. I was reminded how great FireBug is for this kind of thing. Quick and easy to see that the Ajax call was firing with the right information so that we could move on to the next question.

Extrapolating that out, I would imagine that -- had there beeen two folks in the seat from the beginning -- this whole thing wouldn't have taken so long... and the code would probably be better for it.

Reminds me of the old "Double your pleasure..." chewing gum ad, only in this case it's really true.

Interesting svn "gotcha"... II

Turns out, there's any easy fix for this situation (see Interesting svn 'gotcha'). All you have to do is:

1. Checkout your entire project into a different directory using your command line client.
2. Run the script/plugin install... command on that working copy.
3. Checkin the updates you just made.
4. Update the IDE working copy to the latest from repository.

Easy; problem solved...

Interesting svn "gotcha"...

I just learned that you have to be careful about which version of subversion you are using if you have an IDE and also want to use the command-line client.

Starting about V1.4, there were changes in the internal structure of the subversion directory hierarchy (i.e., the .svn files that it creates to track things) which means that you have to make sure that your versions are in synch.

For example, I'm using Aptana Studio, which is eclipse-based, and which uses the latest subclipse client (subversion for Eclipse). That latest version is V1.6. However, on my Kubuntu development box (V8.04/Hardy Heron), the latest subversion package right now is V1.4. If I try to run a subversion command from the shell, I get this error message:

jseidel@EDP15:~/AptanaStudio/4MyPasswords$ svn status
svn: This client is too old to work with working copy '.'. You need
to get a newer Subversion client, or to downgrade this working copy.
See http://subversion.tigris.org/faq.html#working-copy-format-change
for details.

The help link above will explain that -- due to new features in the later versions -- they had to introduce new structure which is incompatible with the older clients.

Since I don't want to downgrade my working copy, I went looking for V1.6 command-line clients. Sad to say, they aren't packaged (yet) for Kubuntu/Hardy --so far I only found it for the Karmic version of Ubuntu.

There are other svn clients out there, either as wrappers around the command line client, or full-on clients. One that I looked at was kdesvn. This looked to be a nice client and was at V1.6. Unfortunately, the developer only provides a Fedora package, so I would have to build my own from source.

I went to the subversion site, but the referenced Ubuntu binaries only pointed back to the Ubuntu site and the older version, so I tried building it from source. Bad move! As stated on their site, there are a number of dependencies which I started to work through one at a time. When I reached a point of having to figure this out:

jseidel@EDP15:~/downloads/subversion-1.6.4/subversion-1.6.4$ apr/buildconf
buildcheck: checking installation...
buildcheck: autoconf version 2.61 (ok)
buildcheck: autoheader version 2.61 (ok)
buildcheck: libtool version 1.5.26 (ok)
buildcheck: local copy of find_apr.m4 does not match APR's copy.
An updated copy of find_apr.m4 may need to be checked in.
buildcheck: local copy of PrintPath does not match APR's copy.
An updated copy of PrintPath may need to be checked in.
Copying libtool helper files ...
buildconf: Using libtool.m4 at /usr/share/aclocal/libtool.m4.
Creating include/arch/unix/apr_private.h.in ...
Creating configure ...
configure.ac:185: error: possibly undefined macro: AC_PROG_LIBTOOL
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
Generating 'make' outputs ...
apr/buildconf: 91: build/gen-build.py: not found rebuilding rpm spec file
apr/buildconf: 107: build/get-version.sh: not found
cat: ./build/rpm/apr.spec.in: No such file or directory

I quit trying: I want to work on my application, not have to learn THIS much about the internals of the tool I want to use... Argh! For now, I'm sticking with the Aptana support which is working (with some help from the Aptana support folks), although I do miss the ability to do quick svn commands from the shell.

I think Linux needs to come up with a better installation mechanism...