Interview with the creator of Nginx

This is an interesting interview with a creator of one of the best web servers out there. The original interview is in Russian and the translation (by google) is fairly difficult to read. I took a crack at writing a better translation and I think this might be a bit better.

Igor, can you tell us about your education, how did you become a programmer and got interested in computers in general?

I was born in a small town in Kazakhstan. My father was in the military and when I was around 1, we moved to Alma-Ata and lived there until I was 18. I finished school in 1987 and tried to get into Bauman Higher Technical School, but didn’t succeed on the first try and returned to Alma-Ata to work as a lab assistant in a branch of the Institute of Advanced Training of the ministry of Geology of the USSR. They had old “Iskra (spark)-226” computers and I started programming in BASIC. At the same time, the magazine “Radio” had a series of articles about how to build your own computer “Radio-86PK” and thanks to them I got a pretty decent understanding about how computers are put together and how they work. But my first experience working with computers came a bit earlier: at the end of high school I went to “Palace of Pioneers” and they had Yamaha KYBT (MSX standard) computers. I remember when I was coding my first program, I confused 1 with an I. So I had trouble because of things like that.

Do you remember what was your first program used by other people?

My first independent program was antivirus “AV”, which I wrote sometime in 1989-1990. It was written completely in assembler and had about 100KB lines of code. It could detect some viruses, because it had a built-in database with a few virus signatures that were popular at the time in the USSR. There were maybe 10 of those: Marijuana, Sophia, Vienna and a handful of others. That was my first program that I distributed via binaries – no source code at the time. It spread around the country and was even used in a couple of factories. I got some feedback: people sent me envelopes with disks that had viruses on them. I kept supporting this AV program for a while, but around 1992 I lost interest and the program died. In 1994 I graduated, and started working as a sysadmin a year before in a company that was in an oil trade market. I worked there for almost 7 years until deciding to leave in April of 2000. This was right when NASDAQ crashed and the internet bubble burst and I decided to join the “internet” right at this time. I worked for a half year for a ecommerce site “xxl.ru” and joined Rambler on November 13^th, 2000.

What did you do at Rambler?

I was a sysadmin. But beyond my core job I started programming in my spare time. I want to point out that programming wasn’t part of my responsibilities, but I did it because I had time and desire. I adapted a patch for compressing Apache responses. Unfortunately, mod_gzip was already taken, so I called my version mod_deflate and it was compatible with Apache 1.3 Then I was asked to take a look at the mod_proxy module. After spending some time with it, I decided that it was going to be simpler to write something from scratch, rather than try to adapt the existing code. So, that’s how mod_accel module was created – module and patches for Apache to do reverse proxying. This was in the spring of 2001.

So you wrote all these modules for Rambler, and simultaneously made the code public?

Mostly, yes. Mod_deflate came from a patch written by Dmitry Khrystalev when he worked at RBC. So that patch was the foundation of my module. My own code is maybe only 50% of it. In the fall of 2001, I got an idea about writing a lighter and better performing web server than Apache. There were similar servers at the time, but they couldn’t proxy, they just served static pages. They also had a common weakness – they run in a single process and they didn’t scale to something like a dual proc machine. At the time, I had a lot of experience working with Apache – both as a sysadmin and as a developer. I learned a lot from the two modules that I wrote, since I had to look at Apache source and figure out how it works. So a lot of things in nginx are inspired by Apache. Not code, but the philosophy – all code was written from scratch. But I didn’t like everything in Apache: for example, it’s very simple to create a configuration that is very hard to support. Your site is growing, you add functionality and then working with it becomes impossible. You need to add something and you’re worried about what’s going to break when you add it. I tried to avoid it with nginx. So, sometime in the spring of 2002 I started developing nginx.

How quickly did people outside of Rambler find out about your development? What was the evolution of the project?

In 2003 people outside of Rambler found out about my project. Moreover, some sites started using nginx. The first one was an Estonian site Rate.ee, which still exists today. It’s the most trafficked site in Estonia. Then mamba.ru started using it as well as zvuki.ru, which offered MP3 downloading. In early 2003, Rambler started offering a foto.rambler.ru service and one of my colleagues, Oleg Bunin, asked me to add proxy functionality, so that it could be used for Rambler’s photo service. Before then, the project was mostly academic, I was developing it slowly and perhaps it would have never been used in production. So, I added proxy functionality and sometime in the beginning of 2004 a proxy version was released and foto.rambler.ru was powered by nginx. On October 4^th, 2004, on the anniversary of the first space satellite, I released the first public version: 0.1.0.

Right now the share of nginx is growing very fast. How was it in the beginning?

Right now it is growing pretty fast. It was much more modest in the beginning. During the first year, nginx was mostly popular in Russia, for obvious reasons. Later on people outside the country found out about nginx and some enthusiasts started using it on their own. My mailing list got an English version, various resources popped up that started describing it, and I started receiving wishes and comments from people. I started adding improvements and fixes and the product slowly gained popularity. Right now the project is growing very fast and that was one of the reasons for starting a company. It was just too much for me alone.

So you didn’t promote it at all, the product made itself?

On my part, I didn’t do any special promotions. Although there is an opinion that the best promotion is a good product? So the growth is explained because nginx “just worked” and people shared their positive experience with other sysadmins who shared it with their friends and so on. I think the popularity of nginx is tied to a couple of things. Firstly, it’s effective and free software that allows you to conserve machine resources and money and secondly it works pretty well.

But there are others, like lighttpd?

Well, in reality there are a couple of other reasons. We developed a combination of important features for creation of effective web infrastructure; which I added slowly and which made nginx indispensable  Having said that, nginx isn’t overloaded with unnecessary features and remains a very compact program. Also, the modularity of nginx allowed many companies and 3^rd party developers to create their own modules on top of nginx. You could say that it became a web platform of the sort. As far as lighttpd, it used to be more common than nginx and more popular around the world. Its author is a German guy named Jan Kneschke. The difference in popularity is that Russia is a confusing country with bears and balalaikas and snow and this was from Europe. Also, he had better English as well as documentation in English? By the way, because of lighttpd, FastCGI protocol got a second lease on life. Until 2000-2001, it was something exotic; everyone used interpreters that were built-in to Apache: like PHP, Perl, Python. But because you can’t really execute PHP inside lighttpd process, FastCGI became a solution. It’s precisely because of lighttpd that FastCGI got a second life. Also back in 2000, people said: “Why, what’s that – FastCGI? We have mod_php and it works great”.

What are the most important use cases for Nginx today?

The key use case on a heavily trafficked site is proxying. Nginx is a front end and proxies requests to the backend via HTTP to FastCGI or WSGI. The standard configuration is to use it in combination with Apache – for example at my last job, that exactly how we used it and it was only a couple of years ago that they switched to FastCGI. By the way, in this scenario, the stats show that nginx grows and apache disappears, although in reality both are used, it’s just that nginx is the visible component of the proxy system.

Can you explain why you need to proxy requests?

Why do people use apache with nginx? Seemingly, it’s just an extra piece of software, which is unnecessary. Apache is easy to use in places where you need an interpreter, like mod_php. Imagine if your PHP can generate 100 responses per second and each response is say 100KB. Not every client is on a fast connection: 10 years ago you’ve had modems, now you have mobile clients and some people simply have a bad provider or slow connectivity. So, we have a response that’s 100KB and your real client speed is maybe 80kbps. So that means your client is going to take 10 seconds to receive the response. As a result, while the client is slowly receiving the response, Apache with mod_php is eating 10-20MB or RAM for each client. And instead of doing things that it can do fast, it waits for slow clients to get the response. You use up a lot of RAM and CPU for that. When you place nginx between your clients and Apache, everything works more effectively: nginx gets the response quickly, frees up Apache and then slowly sends it to the clients, without using a lot of RAM. And it doesn’t use a lot of RAM or CPU because it uses a different architecture – non-blocking, and based on asynchronous processing, which allows you to handle thousands of connections in a single process. Also, in addition to that you can move all your static files from the back-end, which is a simple thing that nginx can do easily and more effectively. Nginx can serve tens of thousands static files per second, assuming you have enough RAM and bandwidth.

Can we come back to standard use cases?

So, the first use case is when we do simple acceleration, maybe even for a single site. We had Apache, we put nginx in front and wow – it’s a miracle. People put it in, get surprised and then write about how awesome it is. Second use case is also proxying, but with many back-ends, so we can horizontally scale the entire system, assuming your back-end allows for it. So, nginx acts as a load balancer. One of the weaknesses of the current version is the absence of multiple load balancing algorithms, but people use it, it works and we’ll add functionality later. What else? Well, some people don’t like Apache for one reason or another. They want to only use nginx and don’t want to install Apache. In that case they run their scripts through FastCGI for PHP or WSGI for Python. For example, wordpress.com – they’ve been using nginx for a long time as a load balancer and commercial LiteSpeed as the web server. This year they migrated to nginx and their PHP runs as FastCGI. Another use case would be when nginx serves static files, like MP3, FLB, MPEG4-video, images.

Can we talk about security? Have you had serious vulnerabilities in nginx?

We’ve had vulnerabilities, but none where you could get remote access or execute 3^rd party code. You could drop worker processes, but not code execution. Think about how exploits work? We write something to the server that gets written to the stack. The server is working, comes back with a response and hits that code. In order for that exploit to work, you need to know where the stack is going to be accessed for this process. As a rule, when you have a debian or ubuntu binary, you can reproduce the situation, try to find where the stack is and write an exploit. How do you fight it? They started randomizing the address space – as in modern Windows.

ASLR?

Yes, that’s right. That is randomization. We had a buffer here and now it’s there. So we can’t predict. We grab a packet, but you don’t know where its memory space is going to be. Nginx is simpler there, because we practically don’t put anything in the buffer that’s sent from the clients. You could probably count on one hand the cases where that’s done and in these places our code is fairly secure. What we receive from the client is placed on a heap and memory is allocated via malloc. So even if you overflow that buffer, you won’t get to the stack pointer. That kind of randomization in nginx existed from the very beginning. Basically, even if you could write a working exploit, it would be very difficult. Also, the processes that work with requests, don’t work as root. We had a few security advisories and you can read them on our site. I think in all these cases, you should react calmly and professionally. For example, if you try to ignore the bug after everything is public and say something like “What? Nothing happened, it’s all good” – then it just undermines the trust in the project.

How many people used to work on the project and how many work on it now?

For a long time, it was just me and I practically wrote the entire codebase myself. About 4 years ago, I got more and more help from Maxim Dunin. Besides the two of us, as the product grew, we got patches from other people. And we often get emails that just describe a problem or have some wishes. They say: “There is this mistake and you could solve it like this”. Just words. So we try to do that as far as we can. And now we got a dedicated person – Ruslan Ermilov, who works on documentation. He has a couple of responsibilities: translation of existing Russian documentation to English, notifications and adaptation of the documentation, so that it’s understandable and non-ambiguous for people that read it for the first time. A frequent problem when an author is writing documentation is that he has a context from which he draws on. He thinks that some things are self explanatory and as a result misses a lot of details. So we are trying to fight that: Ruslan looks at nginx with a fresh perspective “from outside” and can write documentation that is better understood. And he has extensive experience with development and documentation of complex projects.

I’d like to switch topics and ask questions about the company Nginx, Inc and how did you decide to create a business?

I’ll tell you everything. I think in about 2008, I got the first letter from an investor, I don’t even remember who it was. Basically, in the last two years I got about a dozen letters like this. People wanted to do something with nginx and create a company. I declined all offers, since I am not much of a businessman. In the end, I understood that I needed to do something, since I couldn’t carry the project forward by myself. I just didn’t have enough time and strength for everything. It took me a lot of time to figure out with whom I wanted to create a company. Typically, I rarely change directions: before Rambler, I worked for a single company for 7 years and I worked for Rambler for a decade. I find change difficult. But regardless, in the spring of this year I finally decided to form a company that could help the development of the project. Partially, I was inspired to take this step by Sergei Belousov, creator of Parallels and Run Capital fund. We talked a few times and I got more comfortable with the idea of forming a company.

Sergei can be convincing, right?

He is a very interesting person, and it’s always fun to discuss business and other things with him. He is generally a very energetic guy. He is also a fairly opinionated manager and I think he directly influences many decisions in his companies. He is an owner that likes to control what happens and be directly involved in the business. In general, negotiations with investors, writing contracts – that’s a difficult thing because there are very many boring details, huge amount of legal paperwork in English, which is difficult to read in Russian, not to mention in another language. Minute details, coordination – maybe we want one thing and they want another thing. It’s difficult psychologically. Though afterwards, if investors understand your business, everything becomes much easier.

I am interested: you were employed for Rambler and worked on nginx. Rambler doesn’t have any rights? It’s a delicate question, but how did you manage to keep the rights for the project?

Yeah, it’s a delicate subject. Of course you’re not the only one that brought it up and we worked on it for a while. In Russia, the law says that the company owns anything that was created within your core responsibilities or with a separate agreement. So you have to have a contract that says: I need this program developed. In Rambler, I was a sysadmin and was doing development in my spare time. From the very beginning, the product was released under the BSD license, as open source. Rambler started using it when core functionality was already done. Moreover, Rambler wasn’t even the first user of nginx, it was Rate.ee and zvuki.ru.

Who else works for Nginx?

We also have Sergei Budnevich, he is a sysadmin that is responsible for our infrastructure. It’s not very big, but it’s still there. We have our mailing lists, mail server, build server, package testing, bug tracking, etc. Sergei helps a lot with all of that. We are also working on packaging nginx for a couple of other distributions: CentOS and Ubuntu. Sergey works on automation of various processes that have to do with development, QA and release. We also have a couple of other people: one does marketing – Andrei Alekseev and Maxim Konovalov – who is in charge of everything. He makes the entire company work.

What’s your official title?

Formally, I am a technical director. I don’t know how to manage people and I focus on the architecture of future products and transferring development to the “team”. It’s a bit difficult to delegate, but the purpose of the company was to improve the product and development process around it, so I am trying to teach myself how to do that. My colleagues work on organizational issues, client communication, marketing, partner relationships, documentation, hiring, etc. We have a fair bit of challenges – sometimes learning how to communicate at a different level isn’t easy. But in reality, we all wear different hats. There are lots of things to do and the company is pretty small.

Was delegation difficult because you thought that it’s easier to do it yourself and others do a bad job?

Yeah, you often think that it’s easier to do it yourself, if you want it done right. Or because it takes too long to explain what to do or it’s just difficult to say: “Go do this”. For me personally, delegation was difficult because of a few reasons. Now, as a technical director, I am mostly responsible for architecture and development quality.

Igor, thank you for the interview. I think you did learn to delegate: you told us to address all business questions to Maxim Konovalov.

By the way, this is my first interview. I only agreed to it because we created the company. In the spring, I was asked to do an interview by another IT publication and I told them: “Sorry, I don’t like it, I don’t want it and I don’t know how to do it”.

Thank you again. Maxim, when you were negotiating with investors, did you have a business plan? How are you planning to make money?

Mostly, our investors saw nginx as a good product. Of course it was important to have a formal business plan, but American investors don’t only make their decision based on some business plan that says we’ll make some amount of money annually, right down to the last cent. What was important was that nginx is already very popular; it’s a complete and real product. As far as our ideas for making money: first of all we want to determine the right balance between free and paid features. We want to do something that many companies couldn’t do in the past. There are a few companies based on open source where they couldn’t strike the right balance, they had to close some features in the product, ask for inordinate amounts of money, which resulted in upsetting their user base and the products stopped growing.

So you want to create a separate commercial product and find a balance between open source and commercial?

We don’t want to make a separate product; we want to have paid features on top of the core open source product. It will keep growing and adding features that the community wants. The money that we got will allow us to raise product development to a new level. Right now, Igor is already not the only person working on the code and we are building a team. We are hiring in Russia and will keep the engineering team in Moscow. Our focus on the open source product is strong and will remain strong. At the same time, we know that there are customers out there of all sizes that have been using nginx for a long time. They built a business on top of nginx and they are thankful to us for that. When we meet, we often hear: “This is a great product – thank you. But we need this and that. Could you do it? We are ready to pay you”. So, from conversations like this, we get an idea of what we could sell, without upsetting the backers of open source product and without undermining trust in the project as a whole. So we collect these feature requests and compare them to feature requests from the community. If they overlap and we see that it’s a feature that’s needed by everyone, we will add it to the open source version. Some companies even say “We’ll pay you for these features so that they are done faster. We want them to be open sourced and don’t want it to be commercial or exclusive to us”. That’s something we call sponsored development. For now, we have some ideas about commercial add-ons that have to do with mass installations of nignx. For example, these add-ons will make it easier to manage thousands of instances, expand monitoring, and add functionality for hosting, cloud and CDN infrastructures.

So you’re focusing on the product. You won’t sell services like implementation or consulting?

Well, we are a small company and want to stay that way. We don’t want to grow to a few hundred people. We are working with partners, systems integrators; hardware and software vendors and we are looking for channels for working with partners. Consulting will be partially done via partners and partially by us. Unfortunately, we won’t be able to directly offer consulting and support services to all users.

What can users expect in the short-term? Are you planning any new features?

From the history of changes in the last three months and the release schedule you can see that since we created the company we accelerated the development and bug fixing processes. We integrated some functions and added new options. For example, we added MP4 streaming, which Igor was asked about for a few years. We are working and adding functionality.

Do I understand correctly that Igor Sysoev is the main and controlling shareholder and a smaller share is owned by investors?

Yes, Igor is the controlling shareholder and we have three founders. Of course there are investors as owners, a group of investors that controls a certain percentage. The process of getting investment money is technically simple – we create some shares according to the corresponding law and investors buy them for a price. You get the money and use it for developing the company. That’s how it works for us.