I don’t read local Slovenian news a lot, but when I do, all I read is about austerity measures, economic decline and rising unemployment. From our local standpoint it is hard to grasp, that across the pond “the roaring nineties” are back and that we are once again witnessing wealth creation on a gigantic scale. Zemanta is an american company that only happens to have technical development in Slovenia, so at the moment we are very much thrilled by the opportunities present in the USA market, where companies with 13 engineers are worth 1B$ and Apple has enough money in his coffers to buy each of 20.000 square kilometers of Slovenia including all the buildings.
Ever since the Tulip mania in the 17th century, whenever there are prolonged times of rising economy, people start to claim that “this time is different” and that some fundamentals of economy such as technology, political conditions, or whatever has changed and that we have reached something that “looks like a permanently high plateau“. But as Shakespeare, Greek tragedians, and Epic of Gilgamesh have shown to us time and again human nature does not change and after every period of rapid growth, the crash is inevitable.
Thirteen years ago I was a passionate reader of BusinessWeek which was one of the premiere loudmouths of the first dotcom bubble. Everything comes to Slovenia with at least two years of delay so when I started to work on my first startup in the summer of 1999, our hopes were immediately squandered by the technology bubble burst of 2000. So while I don’t believe that the current economical boom in technology sector is any different than previous ones, I’m quite hopeful that “this time it will be different” at least for me. Zemanta has just celebrated its fifth birthday and in this five years we have discovered a business model that works and weathered some rough times. This makes me confident that we are ready to seize the opportunity that the current boom is offering to us.
In the mid-january of this year I’ve met Matthias Alder of Astina. He came to check whether they could relocate some of their operations to Slovenia, since salaries in Zurich have skyrocketed as a consequence of Zurich becoming a premiere financial, technology, and research hub of Europe. When they have researched possible destinations for relocation of a part of their operations, Ljubljana/Slovenia came high on their list. We have similar mentality and work ethics as Swiss, while having substantially lower labor costs. And perhaps even more importantly, the flight from Zurich to Ljubljana takes only one hour and there are three flights per day connecting the two cities!
Three months later I’m glad to report that enthusiasm of many people has brought to fruition and Astina has opened a subsidiary in Slovenia. They plan to do some serious development in Slovenia so they are hiring java, web, and mobile developers. If you’re expert in any of these field, you definitely should check their jobs page. I wish them a lot of success! I think that Astina’s access to Zurich clients and resources coupled with Slovenian technical expertise is a win-win situation for everyone involved.
Let me finish with a free tip to our Ministry of Economic Development and Technology. We should really invest more effort in presenting Slovenia as a high-tech outsourcing destination to Zurich entrepreneurs!
Ljubljana office of Astina
University of Ljubljana, Slovenia
Yesterday me and a few other people from the industry met with some of the members of the faculty of computer science at my alma mater, University of Ljubljana. We have discussed several initiatives how to improve the perception of computer science among youth, women, alumni and also general public. I commend Faculty of Computer Science that it has improved tremendously over the past couple of years in its communication with outside world and it is becoming a true focal point of computer science in Slovenia. I’ve been involved with the faculty for half of my life (that is, 18 years) and I remember well how ivory-towerish faculty felt in the nineties. I hope that at least some of the initiatives that we have discussed yesterday will bear fruit for the common interest of making computer science more respected in our society.
If my writing has made you nostalgic about your student years, you have great opportunity to visit the faculty today, since today at 15:00 there will be another installment of technical talks in the #VBlatu lecture series. This time engineers from CosyLab will present their experize in controlling huge machines. Please RSVP here!
Yesterday I’ve written about our experience with RabbitMQ and our reasons why have we decided to introduce a new piece of technology to our stack. This can serve as a good introduction to a wider question what should (and shouldn’t) be deciding factors in choosing a particular technology.
The first consideration when introducing new technology should be that sufficient number of other people use it. You really do not want to be a crash test dummy when it comes to new technology. Every new piece of technology has not only benefits, but also lot of side effects and hidden defects. It takes lots of time and resources to root them out or, more commonly, to circumvent defects in some ingenious way. I’ve heard that banks have a rule that no new technology should be introduced which is not battle tested for at least three years. Of course, business processes of banks are much more well-defined and understood than business processes at startups, so we cannot enjoy the privilege of using so well-tested technologies in startups. But we should remind ourselves constantly that, just like core business of banks is trading money, the core business of startups is searching for a repeatable and scalable business model and not playing with unreliable technology.
The second consideration when introducing new technology should be if the new technology solves a real and pernicious problem. If you have a problem that no established technology is capable of solving, then you should first contemplate whether you are really so special or you just like to think of yourself as something special. Quite often the most effective solution of a problem is changing your process to fit the existing technology, and not finding the technology that would perfectly fit to idiosyncrasies of your process.
With regards to which factors shouldn’t be deciding when choosing a new technology, you should read the following article:
Today, software developers are faced with a great abundance of options when choosing how to design and implement systems. We are constantly bombarded with choice and are used to dealing with buzzwords like NoSQL, the cloud, REST, Map-Reduce and so on.
One of the technologies that we have introduced to Zemanta‘s technology stack in the past year is RabbitMQ. I have first encountered message queues some ten years ago when I’ve led the project of toll collection software upgrade for Slovenian highway administration. Toll booths are by their nature distributed and they must operate even if the connectivity to the central server is down. By using message queues we were able to build a reliable system that is still in operation today (with some major changes, though). The same rationale of allowing disparate systems to function independently, while still communicating between each other, was also the main reason why we felt the need to introduce the message queues to our system. Zemanta’s backend does a lot of data processing in the background that is mostly asynchronous and consists of a series of steps. For example, to provide thumbnails for related articles, we first extract set of potential image urls from the article, then we download images in order to identify the most suitable thumbnail, and once we find one, we upload it to S3/CloudFront. By using two message queues we have decoupled these three step process into three completely separated modules with a clear separation of responsibilities.
RabbitMQ is written in only 5000 lines of Erlang and it is reliable as hell (our instance of RabbitMQ is up since February 5th). While we had some reservations initially about introducing Erland based software to our technology stack, we have found in the past year that RabbitMQ is such a low-maintenance system that lack of knowledge of Erlang programming is not a problem. Well, on the other hand, AMQP clients have a fair share of problems and we still observe some occasional hiccups in their operations.
At Zemanta we strive to have our code 100% covered by unittests. While unittests are important for achieving the desired quality of service, they are even more important as a safety net for programmers. Incrementally developing a complex service, such as Zemanta’s, requires jumping between parts of the system and doing piecemeal code changes. Quite often you are required to change code not written by you and therefore it would be illusional to require from the developer to fully understand the system that he’s trying to bugfix or augment with new functionality. The main role of unittests is therefore to alert the programmer that the changes he’s implementing have side effects outside of his immediate focus.
While unittests are great for testing small units of functionalities (hence their name), they are not suitable for testing whether the overall system functions as expectedly. A perfect unittests has all its dependencies to outside systems mocked, but no mock is equal to the real system. We are currently dealing with the question how to effectively test our systems as a whole, that is, how to do functional tests. One way in which we have tried to approach this problem is to use unittesting framework also for system tests, that is, we have written a separate set of unittests that do not mock external dependencies, but instead connect to production or staging services. The main problem I see with this approach is that functional tests are notoriously brittle and require constant maintenance. If you include constantly failing unittests in your deployment procedure, you’re causing more harm than there are benefits.
At Zemanta we are measuring most of the aspects of the performance of our system in order to be alerted if something goes wrong. In recent projects we are specifying what we need to measure along with other aspects of functional specifications. I’ve noticed that with the correct measurements in place, we are able to detect and fix defects very quickly. This effect made me think, that instead of spending our effort on devising system tests, we should spend more time on instrumentalizing our code with effective measurements and nice charts for our Operations guys. Real-time measures are not only much easier to maintain, but are also testing the operation of the system continuously and not just at fixed points, as is the case with system tests.
Zemanta was originally an engineering focused company. It was founded by the hackers that surrounded themselves with even more hackers and that naturally implanted engineering way of thinking into the company. While we have talked (and quarreled) about customer needs a lot, this talk came from the mind, not from the heart. When the decisions had to be made, the engineering focus always won. Perhaps the best thing about Zemanta is that we have some of the best advisers in the world. They had seen that after initial success, we have been kind of stalling, and they have recommended to us to hire someone to help us solve our problems.
I have never seen a company so thoroughly changed as was Zemanta after three-day workshop by Marty Cagan in June 2011. Marty is an engineer-turned-to-product-guy himself, so he was able to present us the perils of engineering focused organization in such a way that even hackers could understand. After 10 months of practicing the Marty’s way, I think he’d be proud of how customer and product focused we have become since.
I’ll finish with the quote that “Great products come from great companies.” We are far a way from being a great company (we have lots of fun, though), but I hope we are at least moving in the right direction. The quote is from the following article that is well worth reading:
A treasure trove of unearthed interviews, conducted by the writer who knew him best, reveals how Jobs’s ultimate success at Apple can be traced directly to his so-called wilderness years. All illustrations drawn on iPad by Jorge Colombo If Steve Jobs’s life were staged as an opera, it would be a tragedy in three acts.
Some weeks ago I wrote that we have started experimenting with Trello to support our development process. By now I can clearly state that Zemanta has caught a Trello bug. I’ve seen many process support tools in my lifetime, from MOS that I’ve developed myself at Monolit to Jira that we have used at Najdi.si. What sets Trello apart from other such tools is that developers actually enjoy using it. We’ve pretty much left adoption of Trello functionalities to developers themselves and, quite to my surprise, they’ve started to use advanced functionalities, such as commenting and attachements, all by themselves.
Trello is a very open-ended product. It just gives you boards and cards, and how you organize your workflow, is pretty much up to you. Trello also doesn’t have any analysis and reporting functionalities, but since it has an excellent API, we are in the process of developing them ourselves using Google Apps Scripts. Right now we are experimenting with the Trello workflow that would work best for us. I hope that in a few months our process will become as mature as the one used at UserVoice and that is described in the following article:
Stories about launching and running a startup (while maintaining some semblance of sanity) How we use Trello & Google Docs to make UserVoice better every day Like us on Facebook Join Us on Google+ Last fall I returned from vacation to find that our Product Manager, Dejana, had replaced my precious Google Doc “Roadmap” with a Trello board.
I’ve always held a pretty utilitarian view of code as the best tool to earn (a shitload of) money. Therefore, I find it hard to understand why somebody would spend his energy on such debates and why not just add that f***ing semicolon, since, in accordance with the Zen of Python, ”Explicit is better than implicit”.
Of parser-fetishists and semi-colons Monday, April 16th, 2012 at 4:26 pm TL;DR: if you advocate omitting sensible syntax as parsers will fix that for us, you are not a visionary developer. You waste your and our time. And you come across as a semi-colon.
A few months ago I’ve given a talk about search engines at the #VBlatu lecture series. The lecture was about dirty details of a modern day search engines which are capable of handling billions of search queries and indexing the whole vastness of the web. To make the talk more accessible I’m posting it also here.
Source: Kiberpipa video archive