After several years of programming for the Internet, I have finally decided to start a blog. The theme ‘War Stories’ should be relevant to those who have spent time in the trenches.
This ‘war story’ is about a project which recently passed the Phase I milestone. Getting there was an epic battle.
It all started with the client asking me to re-develop the UI for an existing Flex app. It was an animated greeting card. I can’t go into details about the nature of the card, but it had a clever animation that could be controlled by the user. The client wanted to completely change the UI from their existing prototype, but underneath much of the workings of the app were to be the same. They also wanted to add a Text to Speech feature. The client also told me at the beginning that they wanted me because they were paying too much for the existing designers, who apparently were from a full-service agency. Ok, a FSA does charge more than me, and I’m not sure the client needed that kind of horsepower anyway.
Well, I had done things in Flex that were somewhat similar to this, so I agreed to take a look. But, I needed the existing application source code in order to get a handle on the scope of the problem. When I received the source tree, I noticed two things right off.
1. The project had three components. A Flex project, a Flash project and a php project
2. The Flex project looked like it was built on top of some sort of MVC framework (model-view-controller)
I started by looking into the server component. The php code referenced a database. Well, they probably had a document somewhere that described the database schema and what-not.
And, I have heard that it was easy to load Flash modules from Flex.
And, even though I have never developed a Flex project using an MVC framework, I have read that it is easier to re-implement an app’s UI in a project based on the model-view-controller structure.
And, there must be many sources for Text-to-Speech out there. What with all of the talking apps I have seen. (or have I really seen that many?)
Anyway, I estimated what I thought was a reasonable bid and soon found myself deep in a war the likes of which I had never seen before. They often say: everything you know is wrong. Let’s see how that statement applied in my case…
Let’s start with the server. Looking more closely at the php code, I found that there were no documents describing the project or the database. Just php code. Scanning that, I noticed references to many tables and other server resources. Oh great. That means I’d have to comb through who knows how much code to figure out how to set up a server so that I could replicate and work on the existing project. Rats! Maybe the previous developer will have those documents somewhere.
Oh well… on to the Flex code. I figured that I’d run the app locally and use the previous developer’s server as referenced in the existing code. On first glance, it all looked good. Until I got to the animation, where I got a run-time error. It was complaining that it couldn’t coerce the loaded swf object to the expected interface. Double rats!
I looked into the Flash project and found no source code at all!
Oh well.. I’ll just fire off an email to the previous developer about the problems I’m having and I’m sure they’ll have a ready answer to all my questions.
I suspect you are beginning to see what will happen soon…
But, I needed to make some progress somewhere. So on to the UI. In the MVC architecture, the View component implements the look and feel of the website. I expected to be able to change this code to implement the desired features of the new UI. But, when I really looked into the requirements, I realized that the flow was different and the actual elements of the design had changed a little, plus new features were added like Text-to-Speech and a clear button. So, it became clear that, in order to use the existing code as a base for development, I’d also have to modify the Model and Controller portions of the code.
After some hours of walking through countless files, I realized that I had no idea of how their system worked. It was obviously based on a complex proprietary framework that had been adapted to this particular project. Double rats with nuts!
Fast forward to after many emails to and from the previous developer (one where he suggested that I wasn’t skilled enough to work on this project). I finally got the Flash source code and not much else. So I compiled the swf using the code they sent and used that swf in the existing Flex project. Still the same problem. Aaarrrrggghhh!!!!
OK, I’m in a foxhole and bombs are exploding all around. What should I do next?
I decided to chuck the old project altogether and develop from scratch using the techniques I had used before. At least I’d be making visible progress!
So, I dove in and programmed like a madman for days and nights until I had a pretty good start on the new UI. I put it on my server (no database required, I just used xml) and the client was happy to see something after weeks of nothing but gloomy email.
It looked like I had found a safe bunker as a base of operation. Scanning the battlefield, I knew my enemies were out there… ready for my eventual assault: TTS, a functional client-server interface, and most important, the animation. I knew the animation would prove to be a formidable foe, but I had no idea of the Goliath that was waiting.
Despite the client’s obvious concern about the animation, I decided to tackle the Text-to-Speech problem. I found that there were fewer TTS service providers than you’d think. I found one at: http://www.neospeech.com/ that offered a web service where you posted the text and it would provide a url to the resulting speech file. Sounded cool! And they offered a SOAP interface. (always had wanted to try that). So, I signed up for the trial account and eventually cobbled together a Flex component that talked to the web service using the REST protocol. The Flex app just used the HTTPService component to post the request and to service the resulting reply.
As it turned out, it wasn’t as simple as Request-with-text/Reply-with-speechfile-url. NeoSpeech actually replied with a status message. This message contained a result code and string (success, failure, etc…) and a conversion number (if success). If the result was success, you post again to another method using the conversion number in the request data. The reply to this post contained the actual url to the speech file.
So, after figuring all of this out, I was ready to play the file from Flex. Except, Flash player won’t play anything but mp3 files and NeoSpeech delivered results in .wav format. Hoping against hoppe to find a way for Flash to play a .wav file, I googled this subject until my eyes dropped out; but the answer was still the same.
Double rats with nuts and gerbils!
A quick scan of the NeoSpeech documentation revealed that the trial version limited the format to .wav, but all these other neat formats would be added if a real account was purchased. So, I ordered a real account and would bill the client later. Now I could get the mp3 file I needed… drool drool. But, after carefully reading the list of available formats, I noticed that mp3 was NOT among them. I had to read the list several times. And after that, I even called NeoSpeech and begged them to add mp3 to the list. They said no and they weren’t planning this. (something about patent or copyright… whatever)
I’m running out of rodents…. Now I saw the TTS monster for what it really was. And I felt weak and helpless. Surely I’d met my match.
But, perhaps there was a way to convert a .wav file to a .mp3 file. Perhaps there was even a web service (kinda like NeoSpeech) that would do this for you!
I did not find a web service, but I found that many people were using a utility named FFMPEG to convert audio (and video) files. I searched for web hosting services that offered FFMPEG and found several. I settled on http://www.cirtexhosting.com. So, I opened a hosting account with Cirtex and got to the business of writing a php script that would do the conversion.
Oddly, an exhausting (if not exhaustive) search turned up no concrete examples of how to invoke ffmpeg from a php script. I posted a question to the help desk and I received the answer that they provided a path to ffmpeg and I was to use that to access the utility. They had no examples of how to do this, however.
At some point along the way, I thought I had a glimmer of just what it meant to have the path to the utility. But I don’t really know now. Perhaps I (or someone else) will figure this out. In the meantime, the folks at Cirtex were nice enough to enable the system command in php for me. This is rare on a shared server, and I thank them for doing this!
So, I got it working on a canned file, but now I needed to get it working from a url. It just so happens that php offers a function that works well for this. It is called stream_copy_to_stream. I created a new php file to service the Flex requests, dutifully compartmentalizing all of the details of the TTS server into the php interface file and away from the Flex app. And, it all worked! Most of the time anyway. It turned out that I had to keep requesting the status until I got a valid conversion. In the process of figuring this out, I abandoned the REST methodology and used their SOAP service. In order to do this I used php’s SoapClient (available only in certain versions of php).
Goliath slain… On to the next adversary.
The Flash-based animation was a sticky item to be sure. As loaded from the previous developer’s website, it wouldn’t link properly. The same was true when I loaded the swf I made from either my website or local directory. I was at a loss. I read up on loading Flash swfs from Flex, but nothing was helping. I decided to ask for help. I contacted a colleague I had worked with before (Tom Schreiber) and asked him if he (or anyone he knew) could help with my problem. He referred me to Russ Fergeson who suggested I move the swf file to my bin-debug/release folder. Voila! Thanks Russ! Now all I had to do was isolate the animation code from the old project and port it to my application.
The MVC architecture is generally considered to be a “good thing” because it promotes a programming style that is modular and flexible. Unfortunately, if mis-applied, it turns an otherwise simple task into a programming nightmare. Instead of making a mini MVC module that was a component, the previous developers had intertwined the animation’s M, V and C elements into their overall MVC code. Now, I had to traverse countless other files and extract the relevant portions converting them to my simpler paradigm. This involved a great deal of experimentation. Kind of like hadron colliders, I was smashing code into code and watching the trails left in the aftermath of the destruction.
And, just to make things interesting, my computer got the swine flu and puked all over my office. Luckily I was able to transfer my projects to the laptop and continue on while the main computer went to the hospital.
It came down to just never giving up. Eventually, I had a component that I could use from my application. Things went pretty smoothly as I added controls of the animation almost verbatim from the old code. A sure sign things were going well. I was about ready to relax a little when the next bomb hit.
I had been having a problem with my server-side code where it would mysteriously become unavailable for about 20 minutes. This sounded like a session issue, so I re-wrote my code to eliminate use of session. I thought this had fixed the thing, but all of a sudden I couldn’t access the server at all. Not even via ftp! At this point, I had the application residing on one server (mine) and the TTS proxy on another (Cirtex). It seemed reasonable to port all my code to the Cirtex server and try it there. When I did, it seemed to work once. Then that server became unavailable!
What had I done? I scoured my server-side code for that fatal flaw. Nothing I knew suggested that this code would cause a problem. Had I been writing deadly code all along? Somehow missing the crucial ingredient that all “real” php programmers knew to use?
Of course I had just sent an email to the client announcing an imminent demo. Oh, the irony! This was as classic as the Illiad. Now I was lying wounded in the mud, with tanks rolling in, and I had nothing. But, hope springs eternal!
Ignoring the futility of trying the same thing again a third time, I decided to open yet another hosting account. At least I would be able to do basic experiments in php server technology, if for no other reason than remedial punishment.
But as I was running the app again using the new server, I noticed a funny thing. A new icon in my system tray was blinking red bars at me. It was the recently installed Firewall: Comodo. (I had just gotten my computer back from the shop where they installed it as part of their service).
So I clicked it.
To my surprise, it showed that my computer had spawned hundreds of IP sessions and was in the process of spawning hundreds more. Yikes!! At first I thought that I was still infected, but after looking at the destination IP, I realized that it was pointing to the server I was using for the project. Oooohhh…
I closed my app and they stopped spawning.
OK, so it wasn’t my server-side code after all. I was fighting on the wrong side. The real enemy lay in the Flex code. So what had I done to cause such a problem?
Well, when my server problems started, I had written a component based on HTTPService whose send method was re-invoked by an occurrence of the fault method… if the count was less than some constant. (I used 5). Aha! That must be the problem somehow. So I ripped this code out.
But the problem remained. Now what? When I get flummoxed (which is quite often really) I use a great comparison utility called BeyondCompare. I save my work folder to a backup drive and add an increasing number to the backup’s file name. Yeah, I know this is kindergarten level version control. I have a CVSdude account. But this method works well and is much faster than using cvs for my purposes. I do use CVSdude for when I am on a team however.
But, anyway… walking back in time via the BeyondCompare file comparisons, I noticed a small, seemingly innocuous change in a file that I had overlooked before. I had a problem where a Flex component was loading its data file on the creationComplete event (which only happens once at component creation time), but sometimes the data would change and when the user returned focus to the component, it didn’t show the changed data. So, I had reasoned, add a request for the data when the component gains focus.
This was the culprit. I really don’t know how, but doing this caused the Flex app to keep spawning request after request, with the only symptom being that the server would get slower and more grouchy until it would sign off altogether.
I came up with a more sane way of fixing the data problem which (thankfully?) I won’t go into here. But, now, my original server is down and my TTS server is down and my new server is bombing on file upload (presumably a configuration issue, but who needs this?). And the client is sending emails asking about the demo.
So I get onto live chat with my server company jodohost.com. I ask them to check my account and they say everything’s fine. They can access the website from their browsers. I thought it might be my ISP. So I tried my Blackberry, and sure enough, I connected.
So I call up my ISP gorge.net. I figure that they had some automatic system that detected crazy clients who spawn thousands of connections. But, they said no, they didn’t have any blocks on my account.
Well, it had to be somewhere!
I went back to Jodohost. This time I asked if there was a block on my specific IP. And sure enough, they had sensed the aberrant number of requests from my computer and had blocked that IP.
After apologizing and promising to do my best to avoid making programming errors, they re-enabled my IP and I was back in business. And, just around that time, Cirtex got their server back online. I am still unsure if I had anything to do with their server crash.
The demo went out about a week after Thanksgiving, the target date. But, it looked pretty good and the client seemed happy. If only they knew what I went through to get it to them!
Shannon Vance
12/11/2009