Archive for the ‘Software’ Category

Yes, I know the idea is nuts but when you’re a fast moving bootstrapped startup sometimes you build things like this. That being said, you better have good business reasons for making something this hifalutin as there are many other ways to skin the cat.

The reason we opted to do this is to allow us to quickly and easily make SEO friendly pages without changing backend software. This is also built into our CMS so churning out data driven pages happens at the drop of a hat. Currently most of our SOLR queries are AJAX’ed from frontend so web crawlers wont index that data. The alternative is to push all this logic into the backend but then we have to recompile and deploy code for changes.

Without further adieu, here is the django query language I ended up with. As you can see, they work just like regular filters. Note on line 2 I am using urlparam to take in arguments. Strings also work here.

1
2
3
4
5
6
7
8
9
10

{{ "new"|solr_command }}
{{ "__QUERY__"|solr_set_replacement:urlparam_q }}
{{ "text:(__QUERY__)"|solr_set_query }}
{{ "AND recordtype:(company)"|solr_wrap_query }}
{{ "rows"|solr_set_param:"50" }}
{{ "run"|solr_command }}
 
{% for company in solr_results.response.docs %}
  {{ company.name}}
{% endfor %}

Below is the associated python code. Note there are a few App Engine specific things but everything else is standard. Also, note the use of globals in this example. You will have to handle this in your own way.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

import logging, simplejson as json
from django import template
from google.appengine.ext import webapp
from google.appengine.api import urlfetch
 
# these should be a part of thread local or
# a similar mechanism, i.e. not here!
SOLR_QUERY_WRAP = []
SOLR_REPLACEMENTS = {}
SOLR_PARAMS = {}
SOLR_QUERY = None
TEMPLATE_CONTEXT = None
 
def set_replacement(key, value):
  value = value.replace(':', '\:')
  SOLR_REPLACEMENTS[key] = value
  return ''
 
def set_param(key, value):
  SOLR_PARAMS[key] = value
  return ''
 
def set_query(value): # pylint:disable=W0613
  global SOLR_QUERY
  SOLR_QUERY = value
  return ''
 
def wrap_query(value): # pylint:disable=W0613
  SOLR_QUERY_WRAP.append(value)
  return ''
 
def run_solr_command(command):
  if command == 'run':
    return _run_query()
  elif command == 'new':
    return _new_query()
  else:
    raise Exception('Solr command not found')
 
def _new_query():
  global SOLR_QUERY_WRAP, SOLR_REPLACEMENTS 
  global SOLR_PARAMS, SOLR_QUERY
  SOLR_QUERY_WRAP = []
  SOLR_REPLACEMENTS = {}
  SOLR_PARAMS = {}
  SOLR_QUERY = None
  return ''
 
def _run_query():
  query_string = SOLR_QUERY
  for value in SOLR_QUERY_WRAP:
    query_string = '(%s) %s' % (query_string, value)
  for key in SOLR_REPLACEMENTS:
    query_string = query_string.replace(key,
                        SOLR_REPLACEMENTS[key])
  for key in SOLR_PARAMS:
    query_string = '%s&%s=%s' % (query_string, 
                        key, SOLR_PARAMS[key])
 
  query_string = query_string.replace(' ', '+')
  url = 'http://localhost:8983/solr/select?wt=json&q=%s'
  url = url % query_string
  result = urlfetch.fetch(url)
  if result.status_code != 200:
    raise urlfetch.DownloadError()
  res = json.loads(result.content)
 
  TEMPLATE_CONTEXT['solr_results'] = res
  return ''
 
register = webapp.template.create_template_register()
register.filter('solr_set_replacement', set_replacement)
register.filter('solr_set_query', set_query)
register.filter('solr_wrap_query', wrap_query)
register.filter('solr_set_param', set_param)
register.filter('solr_command', run_solr_command)

I hope someone finds my hack useful. I was going to try the Solango project but it looks like it was abandoned.

Recently we decided to switch from SVN to GIT at my new gig. There is a plethora of information on the web regrading git-svn hybrids, running them in parallel, etc but thats not what we want. We wanted to just cleanly migrate from SVN keeping all the history and deprecate the old SVN repo. The best info I found was Jon Maddox’s fix but that didnt work entirely so here is the summary of how it all works in a few simple steps.

1
2

cd ~/Desktop
vim users.txt

Insert a list of your users as shown below. If you forget any the fetch will fail but dont worry, the error will contain the missing email address.

johnclarkemills = John Clarke Mills <john@gmail.com>
somedude = Some Dude <dude@gmail.com>

Here comes the fun part:

1
2
3
4
5
6
7

mkdir my_blog_tmp
cd my_blog_tmp
git svn init http://svn.location.com/trunk/ --no-metadata
git config svn.authorsfile ~/Desktop/users.txt
git svn fetch
cd ..
git clone my_blog_tmp my_blog

Now the final step. Your repo is all setup locally, but it isnt linked to any remote GIT repo. Here is the final step that got me all setup.

1
2

cd my_blog
vim .git/config

Now all you have to do is replace the urls in there with your git repo’s url like so: john@git.location.com:my_blog.git. Save the file, and now your all set, history included!

1
2

git pull
git push origin master

Its been nearly 6 months since my new gig as a lead engineer at Transpond and I havent posted anything about software or much of anything else for that matter. I’ve been busy with many other projects as well as learning and writing PHP. Well, mostly writing as the learning curve isnt bad at all. The more I was able to block Java and what I know of it out of my mind the faster I was able to learn PHP.

Anyway, there are tons of books and blog posts about how to write PHP so Im not going to go into that. What I have compiled though is a list of things that helped me get over the hump. Although Java and PHP have a lot of similarities, their life cycles are different. Here are the things the differences that stood out the most in my mind.

Classes

Now that PHP is fully object oriented we can use classes as well as objects. Lots of new open source projects out there seem to take advantage of them; however, a lot do not, especially older ones. What I find to be very common throughout PHP are the untyped array objects with lots of nested arrays. This appears to be very standard. The sooner you get over this fact the sooner you will be able to code. The point of a dynamic language is to be able to build quickly and not deal with POJO’s.

Instantiation

Just like Java, classes in PHP have constructors and even destructors; however, they are not declared as you may think. You will need to use the special notation for these functions, __construct and __destruct respectively. Also, be careful when building more complicated stacks or frameworks. Objects get destructed at unusual points in a requests lifecycle so getting your debugger running can come in handy.

What is also interesting is that classes cannot be static in the way we would normally think about them. You can make a class that has all static methods which effectively makes a class static but you cannot use the static keyword for a class declaration.

Imports (require or include in PHP terms)

Unlike Java, PHP is file based not package based. This makes for some interesting issues at first. Trying to grok this idea and deal with file paths can be daunting. What we found to be helpful was name spacing which is somewhat recent to PHP. We use that in conjunction with __autoload() which makes class loading very simple. Essentially we can dynamically load classes easily based on their namespaces and not their file location as much (our autoloader handles that).

Methods, functions, and void (or lack thereof)

As you probably already noticed, methods are called functions. These function do not need to declare what type of object they will return, as this is a dynamic language. In fact, the same method could return nothing (void), and object, or a string. This puts the burden of good engineering on the caller.

Functions in PHP (methods in Java) are however similar in the way they are declared. They can have public, private, and protected declarations, as well as static, abstract, and final keywords. This makes the transition from Java pretty simple, however; there is something strange I noticed. Member variables cannot have the final declaration which I find to be frustrating but there are obvious ways around it. The main reason for this being, most objects are passed by value so modifying a member variable is harder than in Java. I will go into this more later.

Conclusions

So far I am really enjoying PHP. The environment is very simple and straightforward, there is a ton of support for it, and there is a plethora of open source projects to choose from. All of this makes for a quick development time and a quick ramp up time. As I mentioned before, the sooner I stopped thinking about how Java worked, the sooner I was about to pickup PHP.

A coworker and I came across this block of code which was causing us numerous headaches over the past few weeks. For a while, there was absolutely no time delay which was causing an inordinate amount of CPU cycles but thats besides the point. After that was fixed with a short delay we were left with what you see below. Can you tell what’s wrong?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

   while ( true ) {
        TriggeredSend ts = queue.peek();
        if ( ts != null ) {
            switch( doPost( ts ) ) {
                case SUCCESS:
                    queue.take();
                    ts.complete( true );
                    break;
                case FAILURE:
                    queue.take();
                    ts.complete( false );
                    break;
                case SYSTEM_UNAVAILABLE:
                    systemIsAvailable = false;
                    break;
            }
        }
        Thread.sleep( delay );
   }

As with most concurrency bugs, its hard to determine what’s actually happening without experience. You can turn on your debugger but then you are changing the way the code behaves by slowing it down, essentially taking out any race conditions. Turns out, the same task was getting invoked multiple times well as some other strange conditions. That led us to the block of code you see above.

Figure it out yet? Well, what’s happening is that when the code is peeking into the queue, its assigning the result to a variable which is then being called. According to the java.util.concurrency spec, the peek method is used to see if the queue has tasks in it and not used to take any tasks off the queue. Not only that, but after the task was invoked, another task was being pulled off the queue. So what happens when multiple threads come through that block of code and peek? Well, they can both get the same task and invoke it. Bad bad bad.