Archive for the ‘Software’ Category

The Next Language Your Engineers Should Learn Is Product

June 25, 2014
 by 

This article was originally published in Forbes.

As a technical co-founder, the second most-asked question I receive (behind “where do I find a technical co-founder”) is from engineers asking me, “What language/framework/ new-hip-thing should I learn?”

This question has always puzzled me, especially when it comes from seasoned veterans who have well-respected jobs. Although my alma mater taught me this way as well, this is not the way I think — and neither should veteran engineers. Why would you want to learn yet another language, when that language is nothing more than a set of instructions that tell a machine how to behave? Any engineer worth his salt can pick up a new language in a matter of weeks. The same goes for a new framework. If you’ve been writing software for 10 years and you’re worried about the new hot framework out in the world, you’ve lost the forest for the trees.

The bigger question is, how do you take your knowledge of language and software systems to hack real-world systems? I look at this as a form of social engineering; instead of the goal being to teach a machine how to behave, you are attempting to create a machine that mimics the way humans behave. Or better yet, teach the machine to behave the way the user expects it to behave.

This is the difference between a good product and a great one, and it has nothing to do with software. Companies like Twitter had to change the way humans behaved. It was a long and arduous road, and people still don’t ‘get Twitter,’ whereas companies like Apple and Facebook have painstakingly studied human beings and purposely built products around their needs.

I can’t tell you the amount of times that I’ve come into an engineering meeting where one engineer is bragging to another that in five lines of beautifully succinct code they have been able to complete their task in record time with flawless execution. But I can count on one hand the number of times an engineer has bragged that they changed five lines of HTML and upped the engagement rate 10% on a feature that everyone perceived as useless. A hack is a hack, and that’s what engineers do. I don’t care if it took zero lines of code or a hundred and a roll of duct tape to complete the task, and guess what? Neither do your customers. The product is all that matters.

Both Facebook and Apple have taken this to extremes by hacking human perception. Facebook recently discovered that by changing their loading animation icon to what iOS and Android natively do, the user couldn’t tell if the slowness was coming from their phone, their carrier, or Facebook itself. This simple, genius hack saved so many man hours and millions of dollars of server time it’s hard to even quantify the impact. Apple has done something similar by creating a stripped progress bar that scrolls at just the right speed so you feel as though it is moving faster than it actually is.

These hackers were so perceptive and audacious that rather than engineer a faster application, they decided it was easier to change the users’ perception of time. All of this by changing very little code and no infrastructure.

But let’s get back to the engineer’s original question. I use my current team of three engineers to prove my point. When I began hiring back in May of 2013, not one of them knew our native language of Python, and the other two had barely written any JavaScript (which we have a lot of). Less than a year later, I now have three full-stack engineers who are capable of writing complex interactions on a JavaScript-heavy front-end stack all the way to organizing complex schemaless datasets. If I had attempted to hire three full-stack engineers out the gate, I would be lucky to end with one. Instead, I found three humans who are willing and able to think about the bigger system and realize that often, the best code is the code not written.

Next time you’re hiring an engineer, don’t just ask them the usual whiteboard questions and what they learned in college 10 years ago. Your job is to find out if they are willing and able to program the way users behave — or just program computers.

Automating and Scaling Business Workflow Pipelines with Humans and Software

June 15, 2012
 by 

Every business has workflows or pipelines it must maintain to keep operations running. Manufacturing companies consume raw materials to create finished products and in turn the sales department pipelines the products out the door to its customers. Internet tech startups aren’t any different. From hiring, lead generation, data onboarding, and sales pipelines all the way down to the software release management just like a brick-and-mortar manufacturing company. Every company produces some sort of good or service and can be broken down into a series of workflows.

People play an incredibly large part in every company workflow; however us humans can be costly and hard to manage which creates a challenge when scaling horizontally. A human can only remember so much and without machines it becomes impossible to grow effectively. With the advent of commodity software in our workflows we can get multiples of scale with the use of less human intervention. Another way to look at it is make the humans you have on staff more efficient.

Software is definitely the answer to hit multiples of scale but on the flip side outsourced human labor has also become a commodity due to services like Amazon’s mTurk to labor on oDesk and eLance. We’ve even successfully hired dozens of folks locally for over a year from Craigslist when we really need to maintain a high degree of data integrity and skill. One could easily achieve another level of scale by just hiring more people.

The big question really is where is the line between human intervention and software automation? The engineer in me immediately screams, “software!” but in reality this is the wrong way to begin to look at the problem. If you’re automating something of a known quantity like a grocery store then go get yourself some point-of-sale system and call it a day. Software is clearly your answer and you can stop reading now. When you’re a fast paced startup and need to pivot quickly you wouldn’t architect an entire software stack around an ever-changing business model. You will spend more time and money rewriting code as your business adapts rather than having a human do the job from the beginning.

This is exactly how UPS scaled their business. They would use humans extensively and slowly automate different pieces of their jobs. One script after another would be written until finally they would string them all together into a pipeline of software. They did this for the same reason we do it; they were in uncharted waters. There’s no reason to double down on engineering that is 10x more expensive than much cheaper labor that can adapt quicker than machines.

Discovering Inefficiencies

Unfortunately every business is different and identifying scaling problems takes proper vision and knowledge of the business. If you’re reading this article then you probably understand all this so I will keep this section brief. To me, there’s one rule you have to follow that will allow all others to build upon it; “that which is measured is improved”. You cannot even begin to optimize until you find the highest level of return or biggest problem area to focus on. Sure you can blindly go into different departments to help them optimize but how do you decide where to start?

The analogy I like to think of is diagnosing car troubles. If your water temperature gauge doesn’t work but your voltage gauge does and you solve your electrical issues your car still may overheat down the road. In reality, you can fix the battery issues later as long as the car starts but the highest value of return is fixing the overheating issues immediately.

Solving Inefficiencies

Taking a scientific approach to solving these problems is the most financially prudent and effective way to find scaling solutions. There are a few ways we have gone about doing this. The easiest (also least likely) way is to have the humans tell the engineers what they spend their day doing. This helps the engineers keep their heads down on other projects without directly getting involved in the nitty-gritty.

Unfortunately, not many non-engineers are able to actually explain what it is they do all day long in enough detail to replicate. We’ve had mixed success but our staff is learning how to distill their use cases down more and more. In order to help them discover inefficiencies we have one golden rule; if you find yourself doing the same task for more than a few hours a day tell your boss or the engineers immediately!

If we step back even further and look at the bigger picture what I am really saying is question your job! Don’t just do something because you are told to do so. Many folks have a hard time with this concept and want to please their boss but in reality what we are after is scale. The only way to get to the next level of scale is to be introspective.

The other way we find inefficiencies is to send in the engineers. We have successfully achieved multiples of scale by having engineers shadow employees that have highly repeatable workflows. In some cases, we have even removed the human from the task for a few weeks while the engineer fills in to really dig deep into the problem. This can be a painful process for the engineer but it achieves extremely good results.

Knowing When to Stop

Always remember that just because you can do something doesn’t mean you should. Much like writing web software, scaling pipelines is an iterative process and it’s much easier to roll forward than to roll back. Removing all the human touch points too fast will mask problems and likely create more. Automation itself needs to be monitored otherwise you will lose sight of what’s really happening.

One of the tricks we employ is to automate things to the point of complete automation minus the final step. We do this for quality control purposes in many cases. For example, let’s say you need to gate an approval process part of your system. Rather than automatically approving an event to happen, we send an email to the administrators with the proper information they need to make the decision and several URL links we dub ‘one-click approval links’. This is our way of gating sensitive or otherwise high-risk events that require human attention. This buys us many levels of scale without sacrificing quality. If we hit the next level of scale and still require humans for quality reasons we can easily outsource this for very little cost.

Some pipelines like the one I just described will always require human interaction but other times we do fully automate. When this does happen we have to write even more software to monitor these events. Whether it be audit trails or reports that are emailed to us we need to keep an eye on the gauges. As we scale up it becomes more and more important to have a complete picture as to the status of your pipelines.

Conclusion

There really is no silver bullet to any of these problems and solutions come from careful study and measurement just like scientists would study behavior in lab. Software alone cannot solve all of your logistics problems to scale just as humans will never be able to achieve the same scale without software. Since humans are smarter than machines and will be for the foreseeable future you will need to weld the two together. Finding out how far to push the automation is really based upon your business needs and how closely you study your logistical systems.

My recommendation for anyone who is attempting to scale up their logistical pipelines from shipping product to onboarding data is to use humans and study their behavior closely. It will cost you less up front and in the long term as well as delivering a better more accurate end result. Measure twice and cut once.

Data collection patent filed

April 3, 2012
 by 

A few weeks ago, right before my 30th birthday, I filed my first patent with the founder of my company Adam Sah and our other engineer and my housemate David Merritt. I dreamt the up the idea while brainstorming lead generation at trade shows that we attend. A large part of my job involves bridging the physical world with the online world and how to share data between the two. We are always looking for ways to make this happen not only faster and easier, which equates to cheaper, but also more accurately.

The patent involves software that is used on commodity hardware such as a cell phone to collect data and use it in a novel way. Unfortunately I cannot share much more information due to the simplicity of the design how easy it is to replicate. We are currently using this technology internally and soon we will be sharing it directly with our customers. Once the provisional is approved I will provide the final document and a demo.

Stale cache serving strategy with reactive flush

November 27, 2011
 by 

This is a cache I have created that allows me to always serve the user cached data, i.e. never serving a miss. There are times when some data is just too complicated to make the user request thread wait and other times you just want to be sure you have high availability with no user facing recompute time at all. This is a great technique for template caches and API wrappers which I have employed more than once in my career.

I achieve the goal of no cache misses by first building what I call a persistent cache; memcache backed by disk or database. This way if memcache ever takes a miss I look in the database and pull the result from there. I then recache that result in memcache for a short amount of time and return the response to the user. At the same time, I then put a task on my task queue (MQ) to go recalculate the data and then update my persistent cache. If all caches are empty then I hit the API and upon success, cache the results and return the data to the user. If the API throws an error I put a task on the queue and hopefully the next time the user requests that data the cache will be warmed upon a successful API query.

Essentially this creates a type of MRU cache, only using memcache for frequently used data. I could use cron or something like it but I like that my users and bots are the ones who flush my caches based on demand rather than time or some other indicator.

Stale cache serving strategy
(click diagram for a larger version)

The diagram above describes how I built the caching for Klout Widget, a hack I created just to demonstrate this strategy. Their API was a little finicky so I decided to create a persistent cache that would be resilient to bad or missing data. Go ahead, give it a try.

Anyway, hope this strategy is useful. If anyone knows the name for this type of cache I’m all ears. As far as I know its something I dreamed up.

Django template tags for SOLR queries

November 2, 2011
 by 

Yes, I know the idea is nuts but when you’re a fast moving bootstrapped startup sometimes you build things like this. That being said, you better have good business reasons for making something this hifalutin as there are many other ways to skin the cat.

The reason we opted to do this is to allow us to quickly and easily make SEO friendly pages without changing backend software. This is also built into our CMS so churning out data driven pages happens at the drop of a hat. Currently most of our SOLR queries are AJAX’ed from frontend so web crawlers wont index that data. The alternative is to push all this logic into the backend but then we have to recompile and deploy code for changes.

Without further adieu, here is the django query language I ended up with. As you can see, they work just like regular filters. Note on line 2 I am using urlparam to take in arguments. Strings also work here.

1
2
3
4
5
6
7
8
9
10
{{ "new"|solr_command }}
{{ "__QUERY__"|solr_set_replacement:urlparam_q }}
{{ "text:(__QUERY__)"|solr_set_query }}
{{ "AND recordtype:(company)"|solr_wrap_query }}
{{ "rows"|solr_set_param:"50" }}
{{ "run"|solr_command }}
 
{% for company in solr_results.response.docs %}
  {{ company.name}}
{% endfor %}

Below is the associated python code. Note there are a few App Engine specific things but everything else is standard. Also, note the use of globals in this example. You will have to handle this in your own way.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
import logging, simplejson as json
from django import template
from google.appengine.ext import webapp
from google.appengine.api import urlfetch
 
# these should be a part of thread local or
# a similar mechanism, i.e. not here!
SOLR_QUERY_WRAP = []
SOLR_REPLACEMENTS = {}
SOLR_PARAMS = {}
SOLR_QUERY = None
TEMPLATE_CONTEXT = None
 
def set_replacement(key, value):
  value = value.replace(':', '\:')
  SOLR_REPLACEMENTS[key] = value
  return ''
 
def set_param(key, value):
  SOLR_PARAMS[key] = value
  return ''
 
def set_query(value): # pylint:disable=W0613
  global SOLR_QUERY
  SOLR_QUERY = value
  return ''
 
def wrap_query(value): # pylint:disable=W0613
  SOLR_QUERY_WRAP.append(value)
  return ''
 
def run_solr_command(command):
  if command == 'run':
    return _run_query()
  elif command == 'new':
    return _new_query()
  else:
    raise Exception('Solr command not found')
 
def _new_query():
  global SOLR_QUERY_WRAP, SOLR_REPLACEMENTS 
  global SOLR_PARAMS, SOLR_QUERY
  SOLR_QUERY_WRAP = []
  SOLR_REPLACEMENTS = {}
  SOLR_PARAMS = {}
  SOLR_QUERY = None
  return ''
 
def _run_query():
  query_string = SOLR_QUERY
  for value in SOLR_QUERY_WRAP:
    query_string = '(%s) %s' % (query_string, value)
  for key in SOLR_REPLACEMENTS:
    query_string = query_string.replace(key,
                        SOLR_REPLACEMENTS[key])
  for key in SOLR_PARAMS:
    query_string = '%s&%s=%s' % (query_string, 
                        key, SOLR_PARAMS[key])
 
  query_string = query_string.replace(' ', '+')
  url = 'http://localhost:8983/solr/select?wt=json&q=%s'
  url = url % query_string
  result = urlfetch.fetch(url)
  if result.status_code != 200:
    raise urlfetch.DownloadError()
  res = json.loads(result.content)
 
  TEMPLATE_CONTEXT['solr_results'] = res
  return ''
 
register = webapp.template.create_template_register()
register.filter('solr_set_replacement', set_replacement)
register.filter('solr_set_query', set_query)
register.filter('solr_wrap_query', wrap_query)
register.filter('solr_set_param', set_param)
register.filter('solr_command', run_solr_command)

I hope someone finds my hack useful. I was going to try the Solango project but it looks like it was abandoned.

Selenium, Python, and Sauce

April 25, 2011
 by 

With a title like that most people are probably thinking I have some sort of strange fetish — but alas, my sickness is far geekier. I’m talking of course about User Acceptance Testing (UAT). Selenium, for those of you who are unaware, is an automated testing system that will run assertions against a web site. In our case, we use Python to invoke these browser commands but there are many other languages you can write them in.

Since there are many posts about this subject (sources listed at the end) I am not going to chronicle all the intricacies of the entire setup, rather some of the tricks I used and my experiences with it.

At Buyer’s Best Friend we are a Python shop. At first when I brought Selenium to the company I wrote them in HTML to demonstrate their effectiveness as tool to replace the lack of QA and speedup our release process. Once their power was demonstrated naturally I chose Python to keep one common language across our codebases.

This added a interesting challenge however. Since we use Python 2.5 we cannot run them concurrently. To overcome this we run our tests using 2.6 and added the nosetest framework. For a more detailed explanation, read Running Your Selenium Tests in parallel: Python on the Sauce Labs blog.

Although Nosetests are a great tool, it adds another level of abstraction on top of my homemade test harness which I had to throwout due to the introspective nature of Nosetests and the way it wanted to run. One of the main issues I came up with was configuration. I ultimately I made my own configuration which was called in the setUp() method of each test. In my case, my base test would call it and set the config object as a member variable on the test itself. Here is my config.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import ConfigParser
 
class Config():
  config = ConfigParser.ConfigParser()
  config.read("../config.cfg")
 
  if "config" not in config.sections(): 
    config = ConfigParser.RawConfigParser()
    config.add_section('config')
    config.set('config', 'host', '10.0.1.20')
    config.set('config', 'port', '4444')
    config.set('config', 'browser', 'iehta')
    config.set('config', 'target', 'my-test-server.com')
    config.add_section('sauce')
    config.set('sauce', 'enabled', 'true')
    config.set('sauce', 'browser-version', '8.')
    with open("../config.cfg", "wb") as configfile:
      config.write(configfile)
 
  # our member variables
  host = config.get('config', 'host')
  port = config.get('config', 'port')
  browser = config.get('config', 'browser')
  target = config.get('config', 'target')
 
  sauce_enabled = config.get('sauce', 'enabled')
  browser_version = config.get('sauce', 'browser-version')

As you can see from this configuration, I have two setups here. One more testing locally, or remotely against and instance or grid, or to run them on Sauce. More on that later.

On the grid

As you create more and more tests it becomes increasingly slow to run them on just one machine. Recently it was taking us 30 minutes to run our entire suite of 62 tests and counting. Enter the Selenium Grid. The grid allows you to run many machines or virtual machines and distribute your tests across a farm. Currently I run them across a laptop or two and I have been able to cut the total runtime nearly in half. Although thats good I know I can do better and deal with less configuration and hardware hassle.

Hitting the Sauce

Sauce Labs, founded by Selenium’s creator Jason Huggins, allows you to run your tests in the cloud. Not only that, they also give you an awesome dashboard, a video recording of each of your tests that run, and logging and diagnostic tools to make your tests run faster.

The configuration is brain-dead easy and only takes a matter of minutes to switch your current setup to a sauce setup. Sauce has a setup wizard to help guide you through the process but here’s the configuration I ended up going with. Below is my BaseTest class which extends unittest.TestCase. This ties in with the configuration I setup in the excerpt above.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
def setUp(self):
  self.config = Config() # from config.py (see above excerpt)
  self.verificationErrors = []
 
  if self.config.sauce_enabled:
    sauce_config = {}
    sauce_config["username"] = "john"
    sauce_config["access-key"] = "xxx"
    sauce_config["os"] = "Windows 2003"
    sauce_config["browser"] = self.config.browser
    sauce_config["browser-version"] = 
      self.config.browser_version
    sauce_config["name"] = self._testMethodName
    self.selenium = 
      selenium('ondemand.saucelabs.com', 80, 
        json.dumps(sauce_config),
        "http://" + self.config.target)
  else:
    self.selenium = 
      selenium(self.config.host, self.config.port, 
        "*" + self.config.browser,
        "http://" + self.config.target)
 
  self.selenium.start()
  self.selenium.set_timeout(90000)

As you can see from the above excerpt, you just need to instantiate your selenium object with a different configuration and you’re running Selenium in the cloud. Now I have my tests running in less than 10 minutes running 4 concurrently.

Conclusion

Selenium is a wonderful tool that I have used many times on the job, this time extensively. It has not only caught more bugs than any other human in our company, it has sped up our release process, and also greatly added to our confidence level at every push. Soon we will most likely add it to our continuous integration to get even earlier warnings. In my mind, Selenium is the key to getting repeatable successful launches week after week — with or without a dedicated QA.

In regards to running your tests, Sauce’s Selenium Grid in the cloud has been a great experience and something I am going to continue to use. I have pushed two major releases using it and I have nothing but great things to say. If you’re considering Selenium, you better consider Sauce otherwise be prepared for more configuration and wiring. Running the grid on your own sure does work and I may try it for continuous integration but Sauce is sure making it hard to go back to running my own machines!

Sources:
Selenium Official Site
Nosetests – Python Unit Test Framework
“Running Selenium Test on Sauce Labs”, Matt Raible
“Running Your Selenium Tests in parallel: Python”, Santiago Suarez Ordoñez

Migrate from SVN to GIT with history

January 1, 2011
 by 

Recently we decided to switch from SVN to GIT at my new gig. There is a plethora of information on the web regrading git-svn hybrids, running them in parallel, etc but thats not what we want. We wanted to just cleanly migrate from SVN keeping all the history and deprecate the old SVN repo. The best info I found was Jon Maddox’s fix but that didnt work entirely so here is the summary of how it all works in a few simple steps.

1
2
cd ~/Desktop
vim users.txt

Insert a list of your users as shown below. If you forget any the fetch will fail but dont worry, the error will contain the missing email address.

johnclarkemills = John Clarke Mills <john@gmail.com>
somedude = Some Dude <dude@gmail.com>

Here comes the fun part:

1
2
3
4
5
6
7
mkdir my_blog_tmp
cd my_blog_tmp
git svn init http://svn.location.com/trunk/ --no-metadata
git config svn.authorsfile ~/Desktop/users.txt
git svn fetch
cd ..
git clone my_blog_tmp my_blog

Now the final step. Your repo is all setup locally, but it isnt linked to any remote GIT repo. Here is the final step that got me all setup.

1
2
cd my_blog
vim .git/config

Now all you have to do is replace the urls in there with your git repo’s url like so: john@git.location.com:my_blog.git. Save the file, and now your all set, history included!

1
2
git pull
git push origin master

Group text messaging weekend startup project

October 26, 2010
 by 

Its 2010 and there still isn’t an ad-free and non-paid way to have many-to-many text messages with a group of people! Almost a year ago I built a script in PHP that leveraged twitter as the middle man to solve this simple problem for my neighborhood buddies; essentially creating a mobile chat room. Just last weekend a few of my industry friends and I decided to sit down and productize the thing as an exersize. Two days, some pizza, and a bit of open source code we had Twitmob.com. Here’s how it works.

Simply create a Twitter account for your group, say @bffs_4_eva, and invite your friends to follow. Then when a follower tweets “@bbfs_4_eva happy hour!”, Twitmob will re-tweet the message. Now all of your other followers will see, “RT @a_follower: happy hour today!”. Once your friends turn on Twitter’s mobile updates for the group you can chat with them anywhere. No smart phone required!

Check out Twitmob.com to see how it works. Its quite simple and only takes a few minutes.

Learning PHP after years of Java

April 17, 2010
 by 

Its been nearly 6 months since my new gig as a lead engineer at Transpond and I havent posted anything about software or much of anything else for that matter. I’ve been busy with many other projects as well as learning and writing PHP. Well, mostly writing as the learning curve isnt bad at all. The more I was able to block Java and what I know of it out of my mind the faster I was able to learn PHP.


Anyway, there are tons of books and blog posts about how to write PHP so Im not going to go into that. What I have compiled though is a list of things that helped me get over the hump. Although Java and PHP have a lot of similarities, their life cycles are different. Here are the things the differences that stood out the most in my mind.

Classes

Now that PHP is fully object oriented we can use classes as well as objects. Lots of new open source projects out there seem to take advantage of them; however, a lot do not, especially older ones. What I find to be very common throughout PHP are the untyped array objects with lots of nested arrays. This appears to be very standard. The sooner you get over this fact the sooner you will be able to code. The point of a dynamic language is to be able to build quickly and not deal with POJO’s.

Instantiation

Just like Java, classes in PHP have constructors and even destructors; however, they are not declared as you may think. You will need to use the special notation for these functions, __construct and __destruct respectively. Also, be careful when building more complicated stacks or frameworks. Objects get destructed at unusual points in a requests lifecycle so getting your debugger running can come in handy.


What is also interesting is that classes cannot be static in the way we would normally think about them. You can make a class that has all static methods which effectively makes a class static but you cannot use the static keyword for a class declaration.

Imports (require or include in PHP terms)

Unlike Java, PHP is file based not package based. This makes for some interesting issues at first. Trying to grok this idea and deal with file paths can be daunting. What we found to be helpful was name spacing which is somewhat recent to PHP. We use that in conjunction with __autoload() which makes class loading very simple. Essentially we can dynamically load classes easily based on their namespaces and not their file location as much (our autoloader handles that).

Methods, functions, and void (or lack thereof)

As you probably already noticed, methods are called functions. These function do not need to declare what type of object they will return, as this is a dynamic language. In fact, the same method could return nothing (void), and object, or a string. This puts the burden of good engineering on the caller.


Functions in PHP (methods in Java) are however similar in the way they are declared. They can have public, private, and protected declarations, as well as static, abstract, and final keywords. This makes the transition from Java pretty simple, however; there is something strange I noticed. Member variables cannot have the final declaration which I find to be frustrating but there are obvious ways around it. The main reason for this being, most objects are passed by value so modifying a member variable is harder than in Java. I will go into this more later.

Conclusions

So far I am really enjoying PHP. The environment is very simple and straightforward, there is a ton of support for it, and there is a plethora of open source projects to choose from. All of this makes for a quick development time and a quick ramp up time. As I mentioned before, the sooner I stopped thinking about how Java worked, the sooner I was about to pickup PHP.

Can you see the concurrency flaw?

November 3, 2008
 by 

A coworker and I came across this block of code which was causing us numerous headaches over the past few weeks. For a while, there was absolutely no time delay which was causing an inordinate amount of CPU cycles but thats besides the point. After that was fixed with a short delay we were left with what you see below. Can you tell what’s wrong?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
   while ( true ) {
        TriggeredSend ts = queue.peek();
        if ( ts != null ) {
            switch( doPost( ts ) ) {
                case SUCCESS:
                    queue.take();
                    ts.complete( true );
                    break;
                case FAILURE:
                    queue.take();
                    ts.complete( false );
                    break;
                case SYSTEM_UNAVAILABLE:
                    systemIsAvailable = false;
                    break;
            }
        }
        Thread.sleep( delay );
   }

As with most concurrency bugs, its hard to determine what’s actually happening without experience. You can turn on your debugger but then you are changing the way the code behaves by slowing it down, essentially taking out any race conditions. Turns out, the same task was getting invoked multiple times well as some other strange conditions. That led us to the block of code you see above.

Figure it out yet? Well, what’s happening is that when the code is peeking into the queue, its assigning the result to a variable which is then being called. According to the java.util.concurrency spec, the peek method is used to see if the queue has tasks in it and not used to take any tasks off the queue. Not only that, but after the task was invoked, another task was being pulled off the queue. So what happens when multiple threads come through that block of code and peek? Well, they can both get the same task and invoke it. Bad bad bad.

Copyright © 2005-2011 John Clarke Mills

Wordpress theme is open source and available on github.