It’s been a dream of mine since I was a child to have a projector screen. Turns it out its not that hard or expensive, it just takes time and careful planning. The project has been complete for almost a year now but I hadn’t gotten around to creating a write-up until recently.
See the whole story, Hidden Projector Screen writeup, on my house blog.
Every business has workflows or pipelines it must maintain to keep operations running. Manufacturing companies consume raw materials to create finished products and in turn the sales department pipelines the products out the door to its customers. Internet tech startups aren’t any different. From hiring, lead generation, data onboarding, and sales pipelines all the way down to the software release management just like a brick-and-mortar manufacturing company. Every company produces some sort of good or service and can be broken down into a series of workflows.
People play an incredibly large part in every company workflow; however us humans can be costly and hard to manage which creates a challenge when scaling horizontally. A human can only remember so much and without machines it becomes impossible to grow effectively. With the advent of commodity software in our workflows we can get multiples of scale with the use of less human intervention. Another way to look at it is make the humans you have on staff more efficient.
Software is definitely the answer to hit multiples of scale but on the flip side outsourced human labor has also become a commodity due to services like Amazon’s mTurk to labor on oDesk and eLance. We’ve even successfully hired dozens of folks locally for over a year from Craigslist when we really need to maintain a high degree of data integrity and skill. One could easily achieve another level of scale by just hiring more people.
The big question really is where is the line between human intervention and software automation? The engineer in me immediately screams, “software!†but in reality this is the wrong way to begin to look at the problem. If you’re automating something of a known quantity like a grocery store then go get yourself some point-of-sale system and call it a day. Software is clearly your answer and you can stop reading now. When you’re a fast paced startup and need to pivot quickly you wouldn’t architect an entire software stack around an ever-changing business model. You will spend more time and money rewriting code as your business adapts rather than having a human do the job from the beginning.
This is exactly how UPS scaled their business. They would use humans extensively and slowly automate different pieces of their jobs. One script after another would be written until finally they would string them all together into a pipeline of software. They did this for the same reason we do it; they were in uncharted waters. There’s no reason to double down on engineering that is 10x more expensive than much cheaper labor that can adapt quicker than machines.
Unfortunately every business is different and identifying scaling problems takes proper vision and knowledge of the business. If you’re reading this article then you probably understand all this so I will keep this section brief. To me, there’s one rule you have to follow that will allow all others to build upon it; “that which is measured is improvedâ€. You cannot even begin to optimize until you find the highest level of return or biggest problem area to focus on. Sure you can blindly go into different departments to help them optimize but how do you decide where to start?
The analogy I like to think of is diagnosing car troubles. If your water temperature gauge doesn’t work but your voltage gauge does and you solve your electrical issues your car still may overheat down the road. In reality, you can fix the battery issues later as long as the car starts but the highest value of return is fixing the overheating issues immediately.
Taking a scientific approach to solving these problems is the most financially prudent and effective way to find scaling solutions. There are a few ways we have gone about doing this. The easiest (also least likely) way is to have the humans tell the engineers what they spend their day doing. This helps the engineers keep their heads down on other projects without directly getting involved in the nitty-gritty.
Unfortunately, not many non-engineers are able to actually explain what it is they do all day long in enough detail to replicate. We’ve had mixed success but our staff is learning how to distill their use cases down more and more. In order to help them discover inefficiencies we have one golden rule; if you find yourself doing the same task for more than a few hours a day tell your boss or the engineers immediately!
If we step back even further and look at the bigger picture what I am really saying is question your job! Don’t just do something because you are told to do so. Many folks have a hard time with this concept and want to please their boss but in reality what we are after is scale. The only way to get to the next level of scale is to be introspective.
The other way we find inefficiencies is to send in the engineers. We have successfully achieved multiples of scale by having engineers shadow employees that have highly repeatable workflows. In some cases, we have even removed the human from the task for a few weeks while the engineer fills in to really dig deep into the problem. This can be a painful process for the engineer but it achieves extremely good results.
Always remember that just because you can do something doesn’t mean you should. Much like writing web software, scaling pipelines is an iterative process and it’s much easier to roll forward than to roll back. Removing all the human touch points too fast will mask problems and likely create more. Automation itself needs to be monitored otherwise you will lose sight of what’s really happening.
One of the tricks we employ is to automate things to the point of complete automation minus the final step. We do this for quality control purposes in many cases. For example, let’s say you need to gate an approval process part of your system. Rather than automatically approving an event to happen, we send an email to the administrators with the proper information they need to make the decision and several URL links we dub ‘one-click approval links’. This is our way of gating sensitive or otherwise high-risk events that require human attention. This buys us many levels of scale without sacrificing quality. If we hit the next level of scale and still require humans for quality reasons we can easily outsource this for very little cost.
Some pipelines like the one I just described will always require human interaction but other times we do fully automate. When this does happen we have to write even more software to monitor these events. Whether it be audit trails or reports that are emailed to us we need to keep an eye on the gauges. As we scale up it becomes more and more important to have a complete picture as to the status of your pipelines.
There really is no silver bullet to any of these problems and solutions come from careful study and measurement just like scientists would study behavior in lab. Software alone cannot solve all of your logistics problems to scale just as humans will never be able to achieve the same scale without software. Since humans are smarter than machines and will be for the foreseeable future you will need to weld the two together. Finding out how far to push the automation is really based upon your business needs and how closely you study your logistical systems.
My recommendation for anyone who is attempting to scale up their logistical pipelines from shipping product to onboarding data is to use humans and study their behavior closely. It will cost you less up front and in the long term as well as delivering a better more accurate end result. Measure twice and cut once.
A few weeks ago, right before my 30th birthday, I filed my first patent with the founder of my company Adam Sah and our other engineer and my housemate David Merritt. I dreamt the up the idea while brainstorming lead generation at trade shows that we attend. A large part of my job involves bridging the physical world with the online world and how to share data between the two. We are always looking for ways to make this happen not only faster and easier, which equates to cheaper, but also more accurately.
The patent involves software that is used on commodity hardware such as a cell phone to collect data and use it in a novel way. Unfortunately I cannot share much more information due to the simplicity of the design how easy it is to replicate. We are currently using this technology internally and soon we will be sharing it directly with our customers. Once the provisional is approved I will provide the final document and a demo.
Recently Jane Dagmi, a writer for Bob Vila, found me on Twitter and began to follow the restoration process of my Victorian house. She approached me along with 3 other DIY renovation bloggers to write about our workshops and our restoration process. It was an honor and the first interview I’ve ever done like this. I hope you enjoy it.
SanFranVic’s John Clarke Mills: In the Workshop
Since then, I have also syndicated my house blog content onto Bob Vila’s crowd-sourced blog for do-it-yourselfers. Check out Bob Vila Nation for a look at what other DIY’ers like myself are building.
This is a cache I have created that allows me to always serve the user cached data, i.e. never serving a miss. There are times when some data is just too complicated to make the user request thread wait and other times you just want to be sure you have high availability with no user facing recompute time at all. This is a great technique for template caches and API wrappers which I have employed more than once in my career.
I achieve the goal of no cache misses by first building what I call a persistent cache; memcache backed by disk or database. This way if memcache ever takes a miss I look in the database and pull the result from there. I then recache that result in memcache for a short amount of time and return the response to the user. At the same time, I then put a task on my task queue (MQ) to go recalculate the data and then update my persistent cache. If all caches are empty then I hit the API and upon success, cache the results and return the data to the user. If the API throws an error I put a task on the queue and hopefully the next time the user requests that data the cache will be warmed upon a successful API query.
Essentially this creates a type of MRU cache, only using memcache for frequently used data. I could use cron or something like it but I like that my users and bots are the ones who flush my caches based on demand rather than time or some other indicator.
(click diagram for a larger version)
The diagram above describes how I built the caching for Klout Widget, a hack I created just to demonstrate this strategy. Their API was a little finicky so I decided to create a persistent cache that would be resilient to bad or missing data. Go ahead, give it a try.
Anyway, hope this strategy is useful. If anyone knows the name for this type of cache I’m all ears. As far as I know its something I dreamed up.
Yes, I know the idea is nuts but when you’re a fast moving bootstrapped startup sometimes you build things like this. That being said, you better have good business reasons for making something this hifalutin as there are many other ways to skin the cat.
The reason we opted to do this is to allow us to quickly and easily make SEO friendly pages without changing backend software. This is also built into our CMS so churning out data driven pages happens at the drop of a hat. Currently most of our SOLR queries are AJAX’ed from frontend so web crawlers wont index that data. The alternative is to push all this logic into the backend but then we have to recompile and deploy code for changes.
Without further adieu, here is the django query language I ended up with. As you can see, they work just like regular filters. Note on line 2 I am using urlparam to take in arguments. Strings also work here.
1 2 3 4 5 6 7 8 9 10 | {{ "new"|solr_command }}
{{ "__QUERY__"|solr_set_replacement:urlparam_q }}
{{ "text:(__QUERY__)"|solr_set_query }}
{{ "AND recordtype:(company)"|solr_wrap_query }}
{{ "rows"|solr_set_param:"50" }}
{{ "run"|solr_command }}
{% for company in solr_results.response.docs %}
{{ company.name}}
{% endfor %} |
Below is the associated python code. Note there are a few App Engine specific things but everything else is standard. Also, note the use of globals in this example. You will have to handle this in your own way.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | import logging, simplejson as json from django import template from google.appengine.ext import webapp from google.appengine.api import urlfetch # these should be a part of thread local or # a similar mechanism, i.e. not here! SOLR_QUERY_WRAP = [] SOLR_REPLACEMENTS = {} SOLR_PARAMS = {} SOLR_QUERY = None TEMPLATE_CONTEXT = None def set_replacement(key, value): value = value.replace(':', '\:') SOLR_REPLACEMENTS[key] = value return '' def set_param(key, value): SOLR_PARAMS[key] = value return '' def set_query(value): # pylint:disable=W0613 global SOLR_QUERY SOLR_QUERY = value return '' def wrap_query(value): # pylint:disable=W0613 SOLR_QUERY_WRAP.append(value) return '' def run_solr_command(command): if command == 'run': return _run_query() elif command == 'new': return _new_query() else: raise Exception('Solr command not found') def _new_query(): global SOLR_QUERY_WRAP, SOLR_REPLACEMENTS global SOLR_PARAMS, SOLR_QUERY SOLR_QUERY_WRAP = [] SOLR_REPLACEMENTS = {} SOLR_PARAMS = {} SOLR_QUERY = None return '' def _run_query(): query_string = SOLR_QUERY for value in SOLR_QUERY_WRAP: query_string = '(%s) %s' % (query_string, value) for key in SOLR_REPLACEMENTS: query_string = query_string.replace(key, SOLR_REPLACEMENTS[key]) for key in SOLR_PARAMS: query_string = '%s&%s=%s' % (query_string, key, SOLR_PARAMS[key]) query_string = query_string.replace(' ', '+') url = 'http://localhost:8983/solr/select?wt=json&q=%s' url = url % query_string result = urlfetch.fetch(url) if result.status_code != 200: raise urlfetch.DownloadError() res = json.loads(result.content) TEMPLATE_CONTEXT['solr_results'] = res return '' register = webapp.template.create_template_register() register.filter('solr_set_replacement', set_replacement) register.filter('solr_set_query', set_query) register.filter('solr_wrap_query', wrap_query) register.filter('solr_set_param', set_param) register.filter('solr_command', run_solr_command) |
I hope someone finds my hack useful. I was going to try the Solango project but it looks like it was abandoned.
For the past nine days my father and I, with the help of my house mate and other friends, built a victorian library in my double parlor. It is built entirely of red oak and stained a dark walnut color to match the banister we built two summers ago. This has been a long time dream of mine even before we had this house and now it’s finally complete. We still have a little bit of decorating left to do but its ready to be lived in.
During the build process I blogged daily and uploaded as many photos as I could. If you’d like to see what it took check out the victorian library posts on sanfranvic.com.
It is with great sadness that I will be missing the burn this year after participating six years in a row. I met the people who I would later cofound the Duck Pond with back in 2005 and I’ve gone every year since. Now its six years later and the camp is still going strong. It’s evolved in every way; scrutinized and perfected down to the smallest detail. Just as it’s grown and changed over the years, so have we. The experience has taught us so much about ourselves, managing others, logistics, and most of all– perseverance. Keeping a cohesive group of about 50 people together throughout the years has proven to be an incredible joy but also a great challenge and responsibility.
Since I am not attending this year I have decided to take this opportunity to share my experiences building a long standing theme camp. This is something that my campmates and I have talked about for years but just never had the time. The endless amount of tribal knowledge at Burning Man takes years to amass; trial and error are a way of life at Burning Man. If someone had taken the time to write down even half of their knowledge I would have been infinitely in their debt. For me, Burning Man is about community building and hopefully this small contribution will help that community that I’ll be missing this year.
Please keep in mind when reading this that there are many ways to build a theme camp. This is what worked for us based on our size and other parameters we want to live by. All camps are different and fun in their own way. They are a reflection of the people who build them. This is a reflection of how we like to do things.
Although anyone can come to Burning Man and find some unoccupied land to pitch a tent, to become a proper registered theme camp with land deemed yours for the week, you must offer something to the community. Whether it be a Slip-n-slide and bar like our camp or a Barbie petting zoo or mini-golf course, you must invoke participation. Once you have your idea and mission you must fill out the official theme camp questionnaire to be considered by the Burning Man Organization. Unfortunately not everyone who applies will be granted theme camp status because there are just too many applicants. I highly advise you to take your theme camp questionnaire seriously, especially if you are a first year camp. More information about theme camps can be found on the Burning Man theme camp page.
To give you an idea about the level of intelligence and insanity of my group (yes, you need both to be a good burner), here is the stop motion video created last year chronicling our camp build. Total time elapsed is 16 hours assembled with 25 people.
Stop motion video by James Park
Trust me when I say, it wasnt always this easy. Our first year out in 2006 we bit off more than we could chew and ended up building in the searing heat from Saturday to Wednesday. Even the next year when we thought we had it all figured out we were still finishing the build on Tuesday. Before I go too deep into what worked and what didnt, I should explain our camp’s mission and scale.
We are a dive bar at heart, with a daytime dance scene and a 150′ slip-n-slide with jazz and ambient music at night. The bar is open 23 hours a day and our bartenders are always happy to help. We can always be found on the 9 o’clock side of the playa, usually at E street. Also, the 40 foot tower topped with a rotating yellow duck makes us pretty easy to find. As for our people, they are always hard working and never more than 50 of them. Since the beginning we’ve always wanted to be an intimate first-name basis camp with family dinners and we’ve been able to stay true to our foundings. I believe this is one of the reasons we’re still together today.
On the flip side, although our size and intimacy are one of our greatest qualities it’s also a hinderance in respect to scaling. With 50 people paying dues each plus fundraisers, we’re left with a $20,000 budget to build our camp. This may sound like a lot of money for a theme camp but putting on a week long party for free in the desert isn’t cheap. Some camps have budgets upwards of $100,000 but for us being resourceful is a big part of making it all happen.
Yes even at Burning Man there are responsibilities. The people are the most important asset to any group, company, or organization and a Burning Man camp is no different. The amount of sweat and tears that go into the months of planning each year can be overwhelming. We’re lucky enough to have over two dozen incredibly dedicated people to spread the load.
From the beginning we’ve instilled a work-hard/party-hard attitude. It’s essential to the core of this group. After the 2005 burn where the other founders and I met, we returned home to San Francisco and merged our friends together. From there those friends invited other friends of similar attitude and worth and the cycle continued. We let anyone in our group as long as they understand what it means to be a Duck.
As for responsibilities, everyone has a job and they’re expected to do it to the best of their abilities. Each project has a team lead and people who help them. Whether you’re a truck packer, kitchen cleaner, or structure builder you always have help.
Once the camp is up and running there are also chores and responsibilities. The camp needs cleaning and maintenance from time to time, especially the bar. We used to have chore lists and shifts for such things as bar tending and cleaning but as the years have gone on they have faded away. People mostly manage themselves and our community hums. There are even times when we have too many bartenders which is not a bad problem to have.
A critical part of our camp is serious organization and structure. We have a head committee who dedicate their time to organizing and leading projects. In the end it is them who take direct responsibility for accomplishing all of their required tasks. This team is also vital in keeping the camp’s vision a reality and not deviate from our foundings. This vision, morale, and sense of camaraderie must come from the top down and in turn it comes back from the bottom up.
Another critical part of our organization is a camp leader to head our operations. Someone directly responsible for the day-to-day activities from camp budgets and collections to fundraising and conflict resolution. We’ve also found that the best two leaders we’ve had, including our current one, have been women; incredible motherly types who can take charge and make all of this possible. Without them we wouldn’t still be here today. Thank you Amanda and our current leader Robin Guido for all your hard work.
Over the years we’ve tried many different methods of communication. Most of it has been based around email but we have had great success with a wiki as well. It really helped us to collaborate and get our thoughts in order when we first started this endeavor. As the years went on and we gained more experience we’ve slowly migrated back to email and Google groups as they fit our simple needs. We also use Facebook to communicate with our fans and campmates for fundraisers and other events. For simple day-to-day operations and communication we use an email list. This is a great place for us to organize, share, and have fun with one another all year round.
As for camp budget and other planning documents, our camp leader uses spreadsheets and Word documents. In case you are wondering, these documents are made public to the camp. Not only are they emailed out from time to time, they are printed and distributed to the camp before we all head out to burn. We believe that it’s in everyone’s best interest to know where their camp dues are going. We feel that transparency is the only way to run a camp.
The most important thing when it comes to infrastructure is reuse. No matter what it is you are building you don’t want to have to build it twice. Some of you are probably saying, “but we won’t go again next year, for reals this time!”. Every veteran burner I know has said this at least once. Even if it is your last year you recoup your money by selling your infrastructure to another camp. I can’t stress this enough. Don’t burn your infrastructure; build it once and build it right. Color code your bolts and poles, label everything, and do it religiously. You’ll thank yourself later.
To give you a better idea of what I’ll be covering in this section I’ve attached a satellite photo from this year’s Burn. We always use satellite photos from Gigapan to review the previous year’s layout and prepare for the next.
A. 5,400 square foot shade structure, 8′ tall.
B. 200 square foot kitchen
C. 1,200 square foot Bar structure
D. Truck (setup behind the bar to block the wind)
E. 150′ slip-n-slide
F. 40′ radio tower with rotating light up duck on top.
G. Two stall showers and evaporation pond
Shade is obviously one of the most important things on the playa. It is especially important that you get your tent under some sort of shade if your desire is to sleep past 8 am. I’m not talking about a rain tarp here, I mean a metal structure wrapped in tarps. Think wedding tents and car ports. Get something strong and durable that will last for years to come. Expect to pay nearly a thousand dollars for larger structures 50′x50′ or more.
We got our many shade structures from army supply type places around the bay area and on the web. For our sleeping shade alone we have 5,400 square feet of space for all of us to sleep under. It also supplies some nice relaxation areas in between areas with no tents. It is made up of 10′ poles for the roof, 8′ poles for the walls, assorted connectors, and the top is wrapped with tarps and stretched with bungees.
Be sure to keep your poles and tarps properly labeled. We spray paint every pole and connector so we can easily identify and inventory them. We also shake out our tarps, re-grommet any of the broken grommets, and fold them back up labeled with duct tape.
Since our first year we have had a meal plan which has been wildly successful. We divide up into teams of 6 or 7 and provide one meal a night and the other nights you eat free. We also believe that the family that eats together sticks together which is why we have a 50 person dining room table. This is a big part of who we are and how we stay cohesive. These dinners are one of the things I am going to miss the most about not attending this year.
This may seem obvious to many but in practice this doesn’t always happen. Once camps start growing past 75 or 100 people the kitchen usually suffers or becomes non-existant due to cleanup problems and other issues of scale. This is yet another reason why 50 campmates is the perfect number for us.
As for the kitchen itself we use a 10′x20′ carport. We fully stock it with everything you could possibly need from a BBQ and deep-fryer all the way down to the utensils, serving trays, and sink. This may sound daunting but if you’re resourceful you can find everything you need at garage sales and build the rest. For example, our sink consists of a 55 gallon drum with a makeshift top and strainer paired with a sun shower bag.
The showers are one of the projects that we haven’t gotten quite right until this past year. We never took the time to properly invest and we paid for it, year after year. Not only did we pay in cash, we paid in time which takes a toll on the team.
This year we stepped up and built proper showers that are nearly 100% reusable consisting of pressure treated color coded wood and stainless steel color coded bolts. It’s a very simple structure consisting of two 4×8′ sheets supported by 2×4″ floor joists that are held up by six 4×4″ posts.
In order to keep the wind out and to provide some privacy we wrapped the showers in vinyl which we source for free. Vinyl billboards are often tossed out or sent to Mexico as house wrap but if you know someone in the business or look hard enough on Craigslist you can find some. We even put grommets in them just like regular tarps.
For the shower sprayers themselves we’ve tried a number of different things from CO2 to hand powered. In the end, the easiest and most sustainable thing are good old gravity sun showers and hand pump sprayers. The simpler the better for something like this that gets used by so many people.
One of the unfortunate side effects of trying to stay mildly hygienic on the playa is having to handle your shower water. Like many of our infrastructure projects we’ve experimented with many different ideas over the years. This is a lengthy topic and there’s many ways to handle it so I’ll just talk about a few of our experiences.
The easiest way to evaporate your gray water is make a pond with 2×4′s for walls and a tarp. Then there’s the more advanced designs that use wind or electricity to speed the evaporation process. We’ve actually had a lot of luck with both. One year we had an incredible setup that would pump water up over vertical bed sheets to spread the water over a large surface area. Recently I’ve seen a couple of new designs which have also proven to be very successful and are much easier to build (video).
In the end we reverted back to a simple evaporation pool with a twist. We bring a simple 120v bilge pump to pump out any extra water into our empty 55 gallon drums that we bring home with us.
It’s quick and effective and costs us very little extra in time or energy. Sometimes it’s the little tricks that save you engineering headaches and let you focus other places.
There are so many ways to solve this problem but for us it’s simply about cost and effectiveness. We use two Honda EU2000i generators that we run in parallel. They can easily be rented from a hardware rental store. They are extremely efficient and very quiet, capable of supplying 4,000 watts of power for a week on only 40 gallons of gas.
One of the best innovations in regards to power was a campmates auto-refueling system for the generator which we’ve perfected over the years. This allows the generators to run for an entire night without having the power cut out. If you’ve ever been to Burning Man, you’ve probably been at a camp or party where the generator goes out. A few dollars and a trip to the hardware store can solve this problem. Here is a video of a high end version of what we built. It utilizes the vacuum inside the generator to siphon gas from an external container.
If you’re anything like us then you love indulging in a few adult beverages. During the day our bar is packed with dancing party goers and at night it’s a great place to kick back and hang out at a neighborhood dive. Depending on what your goal is, you may not want a bar. That being said, if it’s lots of participation you’re after then you’re probably going to need one. You’ll see good camps with good music and no bar that are completely empty but right next door you’ll see a broken down shack with a great little bar teeming with people. This definitely isn’t true for the massive camps like Opulent Temple and the Roots with insane sound stages but for us mere mortals a bar is a necessary addition.
To house our actual bar and sound equipment we’ve always used military surplus tents but last year we upgraded to two large wedding tents. So far it’s been great and the beefy 3″ poles are a huge improvement over the usual 1″ army tent poles that they replaced. Also the fitted top is quite nice and happens to match our camp colors which is always a plus. We bought ours at tents4sale.com.
The bar itself is also reusable. It is 16′ long and built out of 2×4″ boxes and covered with a nice bar top that is bolted down. The front face of the boxes are covered in thin masonite. The key here just like everywhere else is to use bolts instead of screws. Always think reusable. Our first bar lasted us three years and now this bar is going on its third year. They have been built to last and sustain many daily dance sessions on top of it.
When it comes to stocking your bar you’re gonna need a big budget. Depending on your popularity you could run out very quickly. Last year we brought 20 kegs, 200 handles of Vodka, and almost a palette of mixers and we were running low on Thursday. Thankfully our neighbors and patrons donate to us constantly to keep our place open. We keep bringing more and more every year and we are never able to keep up. One last tip, bring more mixers than you can imagine. You’ll end up with random bottles of alcohol with no mixers probably more than once in your camp’s lifetime.
Signage is very important if you’re going to attract patrons, especially if you are off the beaten path. Not only that, once you attract them they need to be able to find their way back and tell their friends the name. For years we had branding problems and people couldn’t figure out our name. Even with a huge light up duck on top of a 40′ radio antenna people would call us the ‘Duck Bar’ or the ‘Slip-n-Slide Camp’. Finally we gave up and made a huge light up sign on the big yellow bar tent that says ‘Duck Pond’. This may seem brutally obvious but it took us years to get our signage just right.
Our tower has definitely been one of the more effective methods of attracting people. One year we even wrote the words ‘Bar’ down two sides. People showed up like moths to a flame. Not only that, it helps us and our neighbors find our way home from a mile away. Below is a photo from one of our earlier towers that was only about 30′ tall.
We have a few talented DJ’s in our camp who cannot only spin, but they also know how to setup a great system and source the components. It’s usually a mix of personal equipment and some rented or borrowed. Unfortunately I can’t speak much to our sound setup as I’ve never worked on this team.
What I do know is that the desert can be very hard on the equipment. My campmates wrap the speakers in sheets and do their best to clean their components. Once they return home they need to be taken apart and cleaned. All of this can definitely add up to a large price tag so budget accordingly.
Trash on the playa is a hard thing to contain and organize properly. Luckily we have an incredible trash tsar, thats right a tsar, who has done a great job keeping us in order. Not only do we bring out huge blue, green, and black bins from San Francisco so they’re recognizable, they are clearly labeled with color photos of what goes in each. All these details matter and we have seen vast improvement over the years with little optimizations like this.
The other major innovation that one of my cofounders pioneered is the air-powered incinerator (documented on our wiki). It burns well over a thousand degrees without any gasoline or other accelerant; so hot that it burns almost all of the pollutants out. Last year we were able to burn 35 bags of trash, leaving us with only 2 inches of ash. That’s almost 1 ton of trash. I’d highly recommend this if you have the skills and/or are crazy enough. In years past we would pack it all home to the dump where you have to pay by the pound.
Early Incinerator protype (Built by Joe Andolina and Mark Rausch)
In regards to the glass, plastic, and metal you have other options. The easiest is to bring them to the recycling camp if your scale is small enough. During certain times of the day you can cart over your empties and they will take some of them (I forget exactly what types they take). If you’re anything like us though, you have too much and it can be a hassle to get it over there. Instead we bring them back to the dump and get money back which pays for some of the nonburnable trash we have to throw away.
Since we are a San Francisco based camp where space is at a premium we rent a storage space for the duration of the year. For the first few years we split up infrastructure between our collective houses but that has its challenges when your campmates are moving across the city all the time. More suburban based camps definitely have an easier time storing and sharing the load. For us, a thousand dollars a year makes the problem go away and gives our infrastructure a safe home. It’s also a great staging area to inventory our camp before the burn.
Each year before the burn we unpack our storage unit, inventory all the parts, and identify what needs to be fixed. We can’t bother to do it when we get back from the burn because we are so exhausted so we just store it for a year. Although this inventory process can be time consuming it’s paramount to ensure that we have everything we need.
For transporting our infrastructure to the playa we rent a 26′ box truck with a lift gate costing us over $3,000. We’ve seen other camps rent shipping containers, piecemeal it out between trucks, RV’s, and trailers, but for us city dwellers this is very effective. We could have bought our own truck 3 times over at this point but without a truck, we are a camp in a box without any other strings attached.
After the first year where we ended up spending 4 days building, we quickly learned we had to build at night. The searing heat led to slow build times and exhausted campmates. Since then we have always brought large halogen work lights to build at night. You can buy them at any hardware store for around $30 dollars and they are well worth it. Building your camp at night comfortably with your buddies and the music blaring is an incredible experience.
Another crucial part of a quick build is proper planning, packing, and ultimately building. This year we used satellite photos and a 200′ tape measure to layout our entire camp before the truck even arrived. With the ground staked with flags for the different structures we would begin to unpack the truck. As you may have seen from the build video at the top of this post, we employ fire lines to quickly unpack the truck. With a line of 15 people we could easily pass poles and other infrastructure anywhere we needed across our land. If you aren’t yet employing fire lines for your camp build I would highly recommend it.
To make the build go even faster we pack the truck in a very specific order. We know exactly what needs to come out and at what time which has greatly increased our efficiency. The same goes for teardown. We have a team of experienced campmates who run the truck load who have their own plan and schedule.
If you’re thinking about creating a theme camp I highly urge you to try it. Get a tight group of hard working friends together, plan, organize, and implement. Creating this camp has been one of the greatest pleasures of my life. The experiences we’ve shared with each other and the community on the playa are priceless. That community even spreads into our daily lives in San Francisco. I meet people all the time who have nothing but great things to say about our camp, our people, and our bar. I think that’s a big part of the reason that the machine keeps turning. It’s all fuel for the fire and hopefully this article will have the same effect.
There’s a part of me that’s glad that I’m going to miss the burn this year because I finally got the chance to write about my experiences, be nostalgic, and get excited for next year. Hopefully at the next burn I’ll get the opportunity to meet someone who has created something better because of the experiences that I’ve shared.
With a title like that most people are probably thinking I have some sort of strange fetish — but alas, my sickness is far geekier. I’m talking of course about User Acceptance Testing (UAT). Selenium, for those of you who are unaware, is an automated testing system that will run assertions against a web site. In our case, we use Python to invoke these browser commands but there are many other languages you can write them in.
Since there are many posts about this subject (sources listed at the end) I am not going to chronicle all the intricacies of the entire setup, rather some of the tricks I used and my experiences with it.
At Buyer’s Best Friend we are a Python shop. At first when I brought Selenium to the company I wrote them in HTML to demonstrate their effectiveness as tool to replace the lack of QA and speedup our release process. Once their power was demonstrated naturally I chose Python to keep one common language across our codebases.
This added a interesting challenge however. Since we use Python 2.5 we cannot run them concurrently. To overcome this we run our tests using 2.6 and added the nosetest framework. For a more detailed explanation, read Running Your Selenium Tests in parallel: Python on the Sauce Labs blog.
Although Nosetests are a great tool, it adds another level of abstraction on top of my homemade test harness which I had to throwout due to the introspective nature of Nosetests and the way it wanted to run. One of the main issues I came up with was configuration. I ultimately I made my own configuration which was called in the setUp() method of each test. In my case, my base test would call it and set the config object as a member variable on the test itself. Here is my config.py:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | import ConfigParser class Config(): config = ConfigParser.ConfigParser() config.read("../config.cfg") if "config" not in config.sections(): config = ConfigParser.RawConfigParser() config.add_section('config') config.set('config', 'host', '10.0.1.20') config.set('config', 'port', '4444') config.set('config', 'browser', 'iehta') config.set('config', 'target', 'my-test-server.com') config.add_section('sauce') config.set('sauce', 'enabled', 'true') config.set('sauce', 'browser-version', '8.') with open("../config.cfg", "wb") as configfile: config.write(configfile) # our member variables host = config.get('config', 'host') port = config.get('config', 'port') browser = config.get('config', 'browser') target = config.get('config', 'target') sauce_enabled = config.get('sauce', 'enabled') browser_version = config.get('sauce', 'browser-version') |
As you can see from this configuration, I have two setups here. One more testing locally, or remotely against and instance or grid, or to run them on Sauce. More on that later.
As you create more and more tests it becomes increasingly slow to run them on just one machine. Recently it was taking us 30 minutes to run our entire suite of 62 tests and counting. Enter the Selenium Grid. The grid allows you to run many machines or virtual machines and distribute your tests across a farm. Currently I run them across a laptop or two and I have been able to cut the total runtime nearly in half. Although thats good I know I can do better and deal with less configuration and hardware hassle.
Sauce Labs, founded by Selenium’s creator Jason Huggins, allows you to run your tests in the cloud. Not only that, they also give you an awesome dashboard, a video recording of each of your tests that run, and logging and diagnostic tools to make your tests run faster.
The configuration is brain-dead easy and only takes a matter of minutes to switch your current setup to a sauce setup. Sauce has a setup wizard to help guide you through the process but here’s the configuration I ended up going with. Below is my BaseTest class which extends unittest.TestCase. This ties in with the configuration I setup in the excerpt above.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | def setUp(self): self.config = Config() # from config.py (see above excerpt) self.verificationErrors = [] if self.config.sauce_enabled: sauce_config = {} sauce_config["username"] = "john" sauce_config["access-key"] = "xxx" sauce_config["os"] = "Windows 2003" sauce_config["browser"] = self.config.browser sauce_config["browser-version"] = self.config.browser_version sauce_config["name"] = self._testMethodName self.selenium = selenium('ondemand.saucelabs.com', 80, json.dumps(sauce_config), "http://" + self.config.target) else: self.selenium = selenium(self.config.host, self.config.port, "*" + self.config.browser, "http://" + self.config.target) self.selenium.start() self.selenium.set_timeout(90000) |
As you can see from the above excerpt, you just need to instantiate your selenium object with a different configuration and you’re running Selenium in the cloud. Now I have my tests running in less than 10 minutes running 4 concurrently.
Selenium is a wonderful tool that I have used many times on the job, this time extensively. It has not only caught more bugs than any other human in our company, it has sped up our release process, and also greatly added to our confidence level at every push. Soon we will most likely add it to our continuous integration to get even earlier warnings. In my mind, Selenium is the key to getting repeatable successful launches week after week — with or without a dedicated QA.
In regards to running your tests, Sauce’s Selenium Grid in the cloud has been a great experience and something I am going to continue to use. I have pushed two major releases using it and I have nothing but great things to say. If you’re considering Selenium, you better consider Sauce otherwise be prepared for more configuration and wiring. Running the grid on your own sure does work and I may try it for continuous integration but Sauce is sure making it hard to go back to running my own machines!
Sources:
Selenium Official Site
Nosetests – Python Unit Test Framework
“Running Selenium Test on Sauce Labs”, Matt Raible
“Running Your Selenium Tests in parallel: Python”, Santiago Suarez Ordoñez
Recently we decided to switch from SVN to GIT at my new gig. There is a plethora of information on the web regrading git-svn hybrids, running them in parallel, etc but thats not what we want. We wanted to just cleanly migrate from SVN keeping all the history and deprecate the old SVN repo. The best info I found was Jon Maddox’s fix but that didnt work entirely so here is the summary of how it all works in a few simple steps.
1 2 | cd ~/Desktop vim users.txt |
Insert a list of your users as shown below. If you forget any the fetch will fail but dont worry, the error will contain the missing email address.
johnclarkemills = John Clarke Mills <john@gmail.com> somedude = Some Dude <dude@gmail.com> |
Here comes the fun part:
1 2 3 4 5 6 7 | mkdir my_blog_tmp cd my_blog_tmp git svn init http://svn.location.com/trunk/ --no-metadata git config svn.authorsfile ~/Desktop/users.txt git svn fetch cd .. git clone my_blog_tmp my_blog |
Now the final step. Your repo is all setup locally, but it isnt linked to any remote GIT repo. Here is the final step that got me all setup.
1 2 | cd my_blog vim .git/config |
Now all you have to do is replace the urls in there with your git repo’s url like so: john@git.location.com:my_blog.git. Save the file, and now your all set, history included!
1 2 | git pull git push origin master |
