“What you do for a living is not be creative, what you do is ship” -Seth Godin… And I say, A-MEN!
Great comments by the master Seth Godin…
CustomerServiceFail
This Christmas season I flew back to Vancouver to visit family. I used up the last of my points to fly business class so I could bring extra luggage (for presents). While I must compliment Air Canada on their lounges, I have to say that the web site and baggage handling (at Ottawa in particular) leave a LOT to be desired.
The day I departed from Vancouver, there was some kind of general failure of computer systems at YVR blocking all Web Check-in capability (or at least, that’s the “official” story). Now this was the day AFTER the Nigerian terrorist decided to set his pants on fire on a US airplane, so the “issue” may have been some kind of emergency security upgrade. That being said, I was on my way to the airport when I got word that web-check-in might now be available. I tried it from my Blackberry - an 8310 Curve running OS 4.5. (No, I was NOT driving while doing this).
Now, yes, I don’t have latest model of Blackberry, and a slightly older OS, but there are still TONS of 8300 series devices out there and 4.5 is still a very active OS. You’d think an airline, of all companies, would have their websites optimized to serve business travellers, and mobile devices especially. Well Air Canada isn’t one of them. Sure, the page rendered, (when so many of them don’t), but no matter what I tried, it kept asking me to re-enter my departure city. You see, on desktop, it has one of those javascripted look-ups that offers you the city when you’ve typed a few letters. Lord knows what value it inserts, but typing “Vancouver”, and “YVR” and “Vancouver YVR” and several other variants did not get me past the “please enter your departure city” error on my Blackberry. To add insult to injury, clicking on the “help” link beneath that field throws a Javascript error, so I had NO idea what format it expected the city to be in. (Hint to Air Canada: If nothing else, put the same logic on the server that you have in Javascript and re-render the page with the best guess/first result). I gave up and checked in at the counter when I got to the airport. I didn’t really want to carry around a paper boarding pass, but they gave me no choice. That was frustration number 1. Frustration number 2 is one that I have experienced on any Flights into Ottawa on Air Canada in recent memory.
I swear, that at the Ottawa terminal, Air Canada has ONE baggage handler, and his right arm is broken. That can be the ONLY excuse why my priority checked bags took an HOUR to get to the baggage carousel. I have NOT received my bags at Ottawa International Airport, priority tagged or otherwise, in less than 45 minutes. When possible, I try to avoid checked bags for this very reason. When I spoke to someone at the baggage counter (after 45 minutes of waiting), his excuse was that they had 3 flights come in at the same time.
Oh. My. God. THREE. Three whole flights. Heaven Forfend! At a BRAND NEW international airport, with all new upgraded facilities. And they can’t figure out how to get the bags off in less than an hour? They don’t have scheduling systems for staffing? Let me tell you, that when I arrived, it was our flight and one from Calgary listed on the carousel. I don’t know where this mystical third flight was. And by the sounds of it, several of the Calgary people didn’t get their bags after waiting an hour. Even worse, the bags that DID come off, TRICKLED off… one every 30 seconds or so, and then whole minutes would go by with NO bags, and then a trickle and then 10 minutes or more of no bags and the conveyor stopping entirely. People were livid. The flight got in at 12:15, and I didn’t get my bags till 1:15, and there were STILL bags coming off the carousel when I left. Utterly unacceptable.
I don’t know what is going on with Air Canada at Ottawa International, but they need to *fix* the baggage handling problem. And they need to properly test their website components on Blackberry devices. It isn’t that hard. I managed delivery of a very high traffic site that renders and functions on every type of Blackberry, (and even renders *differently* if you have a newer high-res device), plus on 4 different desktop browsers (with a different, flash/ajax/etc desktop-optimized view). And we delivered it with a team of 3 developers and 1 QA person. It’s not rocket science. KNOW your audience. DESIGN for mobile. TEST your site with simulators if you cannot afford real devices - RIM provides the simulators for free. It’s just good business.
Not all Practice is “Good Practice”…
I’m currently listening to the audio book “Influencer” by Kerry Patterson, Joseph Grenny, and David Maxfield. I’m about half-way through, but thought I’d share a couple of tidbits because the book is truly thought-provoking if you are the kind of person who is interested in effecting positive change in your organization, personal life, community or the world at large.
The dominant meme throughout the book is that to effect change, you need to focus on changing behaviors, and avoid the traditional pitfall of focussing on outcomes. There are actually some very good tangible example stories presented to demonstrate the proposed techniques. These are the same people who produced “Crucial Conversations” and “Crucial Confrontations” – two very good books providing tools for how to have critically important conversations and interactions in high-stakes, emotionally-charged situations.
Another item that came up in the current chapter is that according to Psychologist Anders Erickson “improvement is not just related to practice, but to Deliberate Practice…. prowess is a matter of knowing how to enhance your skills through Deliberate Practice” therefore, “not all practice is good practice.” This is also something that was highlighted in Malcolm Gladwell’s “Outliers”.
Disturbingly, research results have shown that “most professionals progress until the reach an acceptable level and then they plateau” correspondingly, ” Software engineers for instance, usually reach their peak somewhere around 5 years after entering the workforce. Beyond this level of mediocrity, further improvements are not correlated to years of work in the field. ” Stunningly, Erickson has discovered that, “No matter the field of expertise, when it comes to elite status, there is no correlation whatsoever between time in a profession and performance levels.”
In essence, unless we deliberately work on a particular type of focused, deliberate ongoing education and learning, we don’t progress beyond basic proficiency.
Corporate committment to learning and development aside (most small companies lack much budget for that so you have to get creative in building learning opportunities), one question that springs to mind is, is how do you recognize the “elite” from the chaff when hiring senior developers? The dot-com era unfortunately drove a bunch of people into high-tech for the money. This gave us a lot of developers, but we still suffer from a lack of GOOD developers. I personally look for people who have personal initiative, and for whom the paycheque isn’t their primary driver for being in the profession. I want people who love developing, programming and seeing the fruits of their labors go live. Once I’ve established their technical know-how and experience, I personally look for soft qualities – a passion for the technology, a personal committment to ongoing education and self-learning, a willingness to experiment. I will ask interview questions like, “What blogs do you read? What is the coolest thing you have done (in the past/recently)? What kind of projects have you done outside of work? If you had to pick a website out there that has a really great design, what would it be and why? Tell me about your network at home…”
You would be surprised how many people never do anything beyond what is required of them in the workplace. Those people are unlikely to acheive an elite status in their profession.
The power of Social Neworking…
I was away at Web 2.0 Expo in New York this past week – a conference for those interested embracing and extending the opportunities created by Web 2.0 technologies. Ironically, it was the use of one of those technologies, social networking on facebook in this case, that got a huge amount of local press for our company while I was away.
It turns out that one of our games “Ka-glom”, is now the top game in the iTunes Appstore in Japan. We are encouraged to use Facebook, Twitter, digg and any social media tools to spread the word about our company and games, and to actively participate in online social networks and communities (including our own mobile social network). Because of this culture of sharing, a status update in Facebook by one employee about Ka-glom’s recent top status in Japan caught the eye of a network friend of a friend who works for the Canadian Broadcasting Corporation (CBC). By the end of the week we had a radio interview on CBC Radio 1, and were featured on the evening news last night. Apparently an article will also be in the newspaper next week. All this free press and publicity extended out over traditional channels, due to the power of social networks.
This video is the news segment that ran yesterday. At minute 13:34 you can see the segment on Magmic Games. You even get to see them panning around the desks of some of my awesome webdev team in their sanctum sanctorum. Very cool. http://www.cbc.ca/video/#/News/Local_News/Ottawa/ID=1337696899
(And yes, we all have our “Toys” that decorate the environment. I’m a huge fan of ReBoot (the original series), so if you know where I can get more collectbles from that show…)
How Enterprise Lost my business and Budget Won
So, I’m in NYC for the Web 2.0 Expo, and decided that I wanted to rent a car for one of the days I am here. I was on wireless on my BlackBerry 8900 curve, because data roaming is killer on these things. Unfortunately, the G band wireless at the conference leaves a lot to be desired and the organizers are well aware of it. Needless to say, surfing on it was like being on dialup.
First, I tried the Enterprise rent-a-car site as I recalled they had the best prices the last time I looked at rental car options, and I remembered they had a location at Newark airport which is very close to where I am staying. On the positive side, their site, while not tailored for mobile users, at least rendered on my phone (so many non-mobile sites give a “Request Entity too Large” error). On the down side, I could only get two steps into the reservation process before it hung loading a script. I tried going in another way – by city name, but the javascript on the page didn’t play nice with my 8900, so that was a no-go.
So I broke down and PHONED them. Guess what? The cars at the AIRPORT are only for people coming in on a flight. “Oh”, I said, “But I’ve booked online in the past for cars at the airport without needing to specify an inbound flight”. But I was told by the agent that she simply could not book me a car if I was not flying in that day. *argh*. I explained very politely that I was staying at a hotel very near the airport which has a free shuttle there, so it’s my best option for picking up a rental car. She wouldn’t budge from the corporate policy.
I guess they didn’t want my business. I guess they like having excess rental inventory sitting there doing nothing.
So I went to budget.com, and LO! It rendered a mobile site. It was a VERY streamlined experience. It wasn’t fancy, but it was quick and painless. It did not require a credit card – just name and email address and date/time of pickup/drop off. Brilliant. AND it let me rent from the nearby airport. The confirmation email was instantaneous, even if the wireless access on the floor was not.
Did Enterprise not do any kind of user profiling? Wouldn’t it make sense, if you were a car rental company, to cater to one of your primary user demographics – the traveling business person, who is tied to their mobile device and may well try to rent a car using that device?
In any case, I’m sending props out to Budget, and a raspberry to Enterprise.
Network routing is the bane of my existence
A few weeks ago, we put in a hosted VOIP system, whereby we just have some polycom phones here, and the PBX is hosted by a VOIP application service provider. Fortunately, our Astaro firewall had built in VOIP security and SIP support. I simply entered the hosted provider network IP and our internal network IP, plugged the phones into a network drop and they worked. All seemed reasonably good except for a bit of call quality issues, which shouldn’t exist because I set up multipath routing and it should be routing the calls over our Ethernet over Fibre network. The VOIP provider is hosted by the same company that delivers our EOF, so we should be 2-3 hops from their datacenter over that network, assuring pretty good quality. (From there, the VOIP provider connects directly to the telco networks).
Yesterday we upgraded the ASG to 7.501 and all the VOIP phones died. (They’re dead, Jim).
A look at the packet filter logs showed tons of PF violations to the voip network. I enabled everything through to that IP until we could sort out the problem. In doing so, it got some of the phones working but not all. Some phones appear to work – they register with the network; they can make and take calls, but neither person on either end of a call can hear the other. This includes calling internal extensions, EXCEPT, a problem handset can call another problem handset via internal extension and they can hear each other. GO FIGURE. Given that ALL calls, even extension-to-extension are routed over the provider’s network, this is utterly bizarre.
Even worse, I got my VOIP provider to do a log file look-up on our calls, and some of the calls were coming in from our cheap DSL connection, NOT from our fibre one. The multipath routing rules in the ASG are supposed to take care of this. Apparently they aren’t working. I tried to put in a multipath rule for tracert to see if I could force it out over the fibre interface. Guess what? It goes out over our cheap DSL connection.
I tried putting in a STATIC route – anything destined for the VOIP provider network should be forced over the external interface. It killed all the phones. Just couldn’t get them working. (And oh yes, we tried any number of power-cycles/reboots to no avail). I had to take out the static route.
We got some of the non-functional phones (the ones that wouldn’t even register) up and running by killing the DHCP lease on the server and restarting them or by putting in a static IP. This, unfortunately, does not work for the phones that register and make/take calls but won’t let the two parties actually hear each other.
Nothing in the Astaro user forums points to a solution to this problem, but it’s clearly a routing issue.
I’m currently on the phone with Astaro support trying to get this resolved. It obviously isn’t something simple, as I’ve managed to enter this article while the support rep is working on the problem, and I’m still on hold waiting while he asks the engineers questions…
How to keep web developers happy and productive
Web developers are a special breed. I find that they are often far more creative than other types of application developers I have come across – with many having artistic or creative side interests. Because so much of what we do on the web ultimately winds up as a user interface, or affecting the user, a good web developer is by his or her nature, intensely user-focussed and full of ideas on how to make your web application or portal better.
So here are a few pointers on how to keep web developers happy and by extension, productive.
a) toys/gadgets from Think Geek when they’ve finished a big release. Even better if you tailor the toy to each person’s interests/personality.
b) praise. Trivial as it may sound, recognition and positive feedback both privately and in front of the team when they’ve done a good job is essential. Nothing gets a developer beaming more than the announcement in the team meeting that their latest feature has seen a 10% increase in revenue from that portal.
c) challenge – good developers like to work on interesting projects that challenge them to grow their skills. Give them a tough problem and then encourage them to figure out how to solve it.
d) the right tools – you wouldn’t expect a mechanic to fix a modern engine with nothing more than a crescent wrench and a jack, so make sure your developers have the right tools – double monitors, proper seating and lighting, and fast enough processors that they can actually operate in real time. Ensure that they have the software tools they need to get the job done too. Few things make a developer crankier than having to take 2 hours to do something manually that could take 2 minutes with the right tool, not to mention the waste of corporate resources when you look at the actual cost of a developer’s time vs the cost of investing in the right software.
e) open, transparent information – help them to understand what the business objectives are and WHY the requirements are important. People do a far better job delivering features if they know why the feature is important and how it helps the business. Developers do not take kindly to not-so-reassuring/patronizing pats on the head.
f) empowerment – I know it sounds cliche and worn-out, but give them the ability to choose the tools that work best for them, make recommendations on process improvement. Give them the ability to take ownership of a new process to ensure its successful implementation and you’ll see a real boost in morale and productivity. Even if you have a UI designer or design team, let them participate in UI and feature design. Everyone is building sites to allow greater collaboration with customers and partners but many organizations forget to extend that to their process of developement. Let your team collaborate and participate. Don’t just spoon feed them functional specifications.
g) flex hours and the ability to work from home when they are sick or waiting for a delivery.
h) chocolate. Lots of chocolate. Or Swedish Berries. They like those too. Especially at that 3pm low-point in the afternoon…
How did I get here?
You know, looking back over my career, I realized, I have actually migrated servers across datacenters (whether they be self-owned, co-located or fully managed services), six times now. SIX times. 3 of those have been largish migrations, one included 4 nearly full racks of servers.
If you’ve never migrated from one datacenter to another, I can say without hesitation, it is NOT a trivial undertaking, and every move has been for a small-to-medium sized company, so you know there wasn’t a huge budget to do any of them. When migrating, you have to consider a whole host of factors, not the least of which are:
a) Am I going to move my existing servers, or buy new servers for the new location?
(If you are using your own hardware, it’s usually a combination)
b) Are we going with a fully managed situation, or a co-location with some form of hands-on service for hardware installs, etc?
c) Can we move in stages or is this an all-or-nothing situation because of dependencies?
d) Most importantly: how do I stay up and running with all of my services while I migrate my site to the new location and test all the functionality? How can I minimize downtime?
Of course, there are many more factors, like choosing the right provider, and ensuring you don’t use the same vehicles for transporting all your servers, but those ones really jump to the top of the list.
In the end, I tell my team the entire process is like orchestrating a bank heist (not that I’ve ever done THAT, but I’ve seen enough movies), in that there is months of planning involved, rehearsals, and when it comes down to “D” day, a script with every player’s part timed to the minute.
I can’t stress enough how important it is to have a very good map of all the systems in play – hardware, software, applications, layered apps, system configs. You have to bring together all affected parties – from the developers to the business people to ensure you have identified as much “Dark Matter” as possible (as my beloved partner so appropriately coined it), before the move. What is “Dark Matter”? It’s all those hand-offs, couplings and touch-points between systems that everyone thinks are handled by someone else and are generally undocumented. If you don’t identify these touch-points between systems, they will bite you in the ass during a migration.
It’s incredibly stressful, and no matter how good your planning is, you will always have one or two hitches to overcome (Some Dark Matter still slips through). But in the end, it’s incredibly satisfying when the log files are clean, your sites and applications are responding beautifully, you met your schedule, and your QA team gives the “all clear”.
*** Side Note: I’ve never been satisfied with ‘fully managed’ environments. The dynamic nature of many small businesses is such that you very often need to make changes quickly. Of course, we plan things, but it’s a pain to have to ask for log information and WAIT when a crisis is taking place. It’s a pain to be on a 3-hour, 3 way phone call with a technician at the provider and an integration partner, trying to get them to make a firewall configuration work when you KNOW you could get it working yourself much faster if you just had access…)
Poll/Question
If you were a service provider and your customer (who is currently spending something bigger than a breadbox with your company) reported that their accounting clerk had mistakenly paid you nearly $10,000 too much because you kept billing them for a service you were no longer providing to them, would you:
a) take a month to “deliberate”, then stick to the letter of the contract wherein the customer is responsible for notifying you of your billing errors within 60 days, and refuse to provide a full refund.
or
b) provide the customer with a full refund, an apology for the mistake and then look into improving your systems so this kind of thing doesn’t happen in future.
Can you guess which option my current service provider chose initially?
(I now have a verbal proposal for a refund of the overpayment applied as credits to our account over 5 months – still waiting for confirmation in writing - after a lengthy discussion with the VP, wherein I asked if he was appropriately weighing the cost of customer retention against the cost of new customer acquisition…)
And the result is…
I am now in the process of abandoning the Turnkey 32-bit Ubuntu appliance at VPS.NET. It’s been crashing daily, despite the upgrade to Jaunty. VPS.NET has comped me 3 nodes for a month to get a 64-bit virtual going. Unfortunately, the stock LAMP install doesn’t come with things like phpmyadmin, and there is a lot more work to configure the new virtual before beginning the long and arduous journey of reinstalling and reconfiguring all my layered apps and sites. Admittedly I’m totally frustrated by the exercise and not looking forward to the prospect of trying to get this thing ported in my limited spare time. (As I said in a previous post, the original port took a full week of my time, for which I took vacation leave).
Note to all others out there DO NOT USE UBUNTU 32-bit in a Virtualized environment if you need a stable platform.