Archive for the ‘Software’ Category

MooTools Javascript framework

March 23, 2007
 by 

Let me start by saying, I dont like Javascript but I love MooTools. Its not that I dont think Javascript is great and highly powerful, I just hate dealing with it especially when my job is the backend. I would do anything for a stacktrace like there is in Java!

Anywho, MooTools is a Javascript framework developed by a fellow coworker at CNET. Much like Prototype, Dojo, Scriptaculous and others it offers and extensible and extendable framework on which to build upon. I will cover more of my toilings when I have the time. I would highly recommend looking into it even if you have already used other frameworks. Here is a link to the MooTools site.

Solr configuration

March 22, 2007
 by 

Solr is quite easy to setup once you understand it. It is much like any other database setup. So given the following the table, we have to mimic something similar in Solr. It need to know what is “stored” versus what is “indexed” as well as facets and many other options. I will explain more later. Here is my MySQL table representing ‘things’.

things-table.gif

Okay. Simple enough right? So we want to do here is store most of this data; however, for facets we dont really need to store them. What does that mean? Well we want to be able to search through them and index them but when we ask for all columns of a given field it wont return this field. This will come into play later when I discuss faceting. Here is the schema.xml file for the above MySQL table:

<field name=”id” type=”string” indexed=”true” stored=”true”/>
<field name=”name” type=”text” indexed=”true” stored=”true”/>
<field name=”fileName” type=”text” indexed=”false” stored=”true”/>
<field name=”tags” type=”text” indexed=”true” stored=”true” multiValued=”true”/>
<field name=”isBackground” type=”text” indexed=”false” stored=”true”/>
<field name=”dateCreated” type=”text” indexed=”false” stored=”true”/>
<field name=”dateModified” type=”date” indexed=”false” stored=”true”/>
<!– for faceting –>
<field name=”tagsFaceted” type=”string” indexed=”true” stored=”false” multiValued=”true”/>


Now that we are complete with our basic table structure we have to tell Solr a few things about our index. It wants to know the primary key, or in Solr terms, the unique key. Also we have to tell it our default search field.


<uniqueKey>id</uniqueKey>
<defaultSearchField>tags</defaultSearchField>


Now we have one more very important variable that we have to tackle if want proper faceting on tags. We have to make sure that any time we write to the tags text field we also write to the tags string field. The difference is that the ‘tags’ field is stemmed, i.e. searching for ‘kids’ returns ‘kid’ and so forth. The ‘tagsFaceted’ field will return the whole words. One is human readable and the other is for the machines.


<copyField source=”tags” dest=”tagsFaceted”/>

Starting to work with SOLR

March 14, 2007
 by 

solr-head.gifSOLR, a wrapper for Lucene, was developed by a fellow coworker at CNET. It has recently graduated from the Apache incubation cycle and is now a full fledge project. It not only does it wrap Lucene with a simpler interface it more importantly creates a Restful type of API. I only been using this for a few months now but I love it. It will be launching it into production at work and for my own personal project.

So what is SOLR/Lucene

Lucene is an indexing system, much like a database, except faster and narrower. Narrower? Well essentially an index is a single table with a primary key per entry. It allows extremely fast full text searching with stemming that other databases cannot handle, or even support. MySQL for example, especially MySQL 5 will fall over under heavy load no ifs-ands-or-buts. I’ve seen it happen at work in the lab as well as in production.

Lucene, as well as Solr, are built in Java but Solr needs a servlet container to run like Tomcat, Resin, WebLogic, etc. It runs “next to” your database. All the data in the Lucene index is the same as 1 table in your database. When you write to your DB you will write to Lucene (via SOLR). When you delete, same thing. Updating, yes. You get the point.

Faceting

Solr, and Lucene of course, also support faceting which is very powerful. You have seen this on many sites especially in the world of online shopping comparison. It allows you to see how many other entries also share the same common attributes. In the example of shopping comparison you can see how many other products are also, under 20 dollars, made of cotton, and from Amazon Marketplace. This is very powerful feature and it is perfect for allowing users to drill down through the data.


I will post more findings when I have the time, including configurations.

Spring binding many-to-many

March 4, 2007
 by 

Wow. This was a lot more complicated than I thought. After several days of research and questions I finally discovered how to make this work. Its all about custom property editors. Please note that many-to-many relationships are not the best thing to use in a production environment. They dont scale well and they are not easy to manage. I later on replaced with Solr, my new favorite implementation.

In this example, many ‘things’ have many ‘tags’ and inversely many ‘tags’ have many ‘things’.

protected void initBinder(HttpServletRequest request, ServletRequestDataBinder binder) {
   binder.registerCustomEditor(Set.class, “tags”, new CustomCollectionEditor(Set.class) {
     protected Object convertElement(Object element) {
       long id = NumberUtils.parseNumber((String) element, Long.class).longValue();
       Tag tag = tagMgr.getTag(element.toString());

       return new Tag(id, tag.getName);
     }
   });
}

Use Apache Rewrites much like an ‘if’ block

February 23, 2007
 by 

Recently I had to fabricate some nasty rewrite rules involving some unruly query parameters where if one was present do this, whereas if two were present do that, or if none were present redirect somewhere else. I had to step back after a few hours of work to try and look at this problem differently. All of sudden it became clear. Some code is omitted but here are two examples that I am using in production:


RewriteCond %{QUERY_STRING} ^pid=([0-9]+)&pid=([0-9]+)
RewriteRule ^/oldurl.html
   http://%{HTTP_HOST}$1/newurl.html?%{QUERY_STRING} [R=301,L]

RewriteCond %{QUERY_STRING} ^pid=([0-9]+)
RewriteRule ^/oldurl2.html
   http://%{HTTP_HOST}$1/newurl2.html? [R=301,L]


As you can see I am setting up gates or switches. The list went on and on and the order is highly important as you may notice, just as some ‘if’ blocks are (consider refactoring). In the case of a top-down script like mod_rewrite I feel this is not only acceptable but necessary to accomplish this task.

My car was just featured on Jalopnik!

January 2, 2007
 by 

While having a few drinks with a buddy of mine at a local bar in San Francisco I met an editor for Jalopnik, a sister site for gawker.com. After talking to him for a while I mentioned my open source fuel injected 1973 BMW 2002 sitting out on the street. Next thing I know he is taking pictures of the car and taking down my info. A few days later some friends who read the site me this link featuring my car.

jalopnik.jpg

MegaSquirt EFI assembly is almost complete

November 8, 2006
 by 

getting_there.jpg

After 5 days of working on the fuel injection I am finally almost complete. Fortunately my friend let me use his garage for week which has been extremely helpful. I also had the help of my roommate. Right now are are stuck because my computer cannot get a signal from the coil telling it to fire. I have to do some more research but I am sure it will work.

MegaSquirt Electronic Fuel Injection

June 1, 2006
 by 

In the spirit of being open source, I have decided to convert my little BMW to Electronic Fuel Injection (EFI). What prompted this decision was a road trip to Chico when my car didnt perform as well due to the environment, i.e. the altitude. This is due the carburetor, a great invention that works incredibly well to this day, but for me, it wasnt enough. I liked the idea of being able to tune my car with a computer. Even while I’m driving! It sure beats taking the carbs apart over and over to change the jets ;) . As always, I will be managing the project and researching on my wiki.

The MegaSquirt system was built by a bunch of Assembler engineers who realized that there was a place for open source in this industry. If I were to do the same via some of the closed source manufacturers I would be paying thousands with very limited resources. This is exactly how I view software in my day job. I am 100% open source in all of my environments. Now my car is the same. More info on this project can be found on the MegaSquirt website.

Spring pagination support for Hiberate

May 18, 2006
 by 

As I have said before I love Spring and Hibernate; however, there is no Spring support for Hibernate’s pagination API. Why am I am not sure. Maybe they are supporting it now but heres what I did. First I made and abstract object I called ‘Page’. I also have a Solr subclass of this object but I will cover that later.


public abstract class Page {

    public List results;
    private int pageNumber;
    private int resultsPerPage;
    private int totalCount;
    private String queryString;
    private Integer totalPages = null;

    /*
        Getters and setters
    */

}


public class PageHibernate extends Page implements HibernateCallback {

    public PageHibernate() { }

    public PageHibernate(String queryString, int page, int resultsPerPage) {
        this.setPageNumber(page);
        this.setResultsPerPage(resultsPerPage);
        this.setQueryString(queryString);
    }


    public final Object doInHibernate(Session session)
            throws HibernateException {

        Query query = session.createQuery(this.getQueryString());
        query.setFirstResult((getPageNumber() – 1) * getResultsPerPage());
        query.setMaxResults(getResultsPerPage());

        return this.results = query.list();
    }
}


Since Spring abstracts your Hibernate interaction with HibernateDaoSupport you dont have to write straight criteria calls to Hibernate if you dont want to. Spring does create simple implementation to allow to write straight HQL and many other functions but not pagination. Rather than tackle this problem at this angle I decided to make the abstract class ‘Page’ that you see above for all my data stores. Anyway, here’s how easy it is to then hit the database with this object properly constructed and have it returned ready for use.


public Page getPageOfThings(int page, int resultsPerPage, String orderBy) {

    PageHibernate newPage = new PageHibernate(
            ”FROM Thing”, page, resultsPerPage);

    getHibernateTemplate().executeFind(newPage);

    List count = getHibernateTemplate().find(“SELECT count(*) FROM Thing”);

    newPage.setTotalCount(Integer.parseInt(count.get(0).toString()));

    return newPage;

}

Copyright © 2005-2011 John Clarke Mills

Wordpress theme is open source and available on github.