Why does Quora use Nginx

E-commerce dwarfs on the shoulders of giants

How a single tweet can result in almost 800 words in a blog post: A few days ago I came across one of my colleague Yann Simon - core developer at commercetools - who reported on the consequences of an Elasticsearch update:

upgrade from elasticsearch 5 to 6: the size of most of our indices shrank by 4x or 5x. Impressive! Great work from the Lucene team! Https: //t.co/xIwudu7DHZ

- Yann Simon (@simon_yann) March 2, 2018

As many readers know, Elasticsearch is a search engine based on the Apache Lucene program library, which in turn is being further developed as an open source project by the Apache Software Foundation. At commercetools it is used as the primary search for product data, for example - and you can easily imagine the advantages of being able to reduce the amount of data to be stored as an operator of a PaaS solution in this way. Shopify also uses Elasticsearch, other cloud providers do the same (in the non-e-commerce environment, for example Wikipedia) or use Lucene's sister company Solr in the background.

This search technology is also used in on-premise solutions: According to the documentation, Spryker uses Elasticsearch, there are connectors for Magento, Shopware, OXID eShop and the larger enterprise solutions such as Intershop, SAP Hybris and IBM Websphere will have similar integrations in the drawer .

Software zoo?

Now this is not supposed to be a contribution about Elasticsearch, the technology here is representative of many similar programs and libraries that have long been an integral part of the web and especially the e-commerce landscape. To make this even more clear, here is a post on Quora on the question of the software stack behind the aforementioned Shopify:

Products that make up Shopify's tech stack include: Apache (Commons, Kafka, Zookeeper), Elasticsearch, FAYE, Git, GitHub, Google (Cloud Messaging, Maps), HockeyApp, JavaScript, jQuery, Mapbox, MySQL, Nitrous, OneSky, Redis, Ruby, Ruby on Rails, Swiftype, Twilio, Vagrant, VividCortex, ZapStitch, CircleCI, Datadog, DigiCert, Disqus, Docker, Dyn DNS, Fastly, GoDaddy Domain Registration, HackerOne, HuBoard, New Relic, nginx, Pingdom, StatsD, StatusPage, ThousandEyes, and Unicorn.

It's a pretty wild mix of commercial and open source products and services, and you could spend a whole series of blog posts breaking it down in detail. In this context, however, the list should make one thing clear: the development of such platforms always includes the corresponding knowledge and the tailor-made orchestration of these different technologies and their merging to form the desired end product.

Specialists in the rescue

Of course, Shopify is a special case. The company is now listed and, according to Wikipedia, processes an annual transaction volume of 45 billion USD, which is moved by more than half a million traders on the platform. So it makes sense to rely on tried and tested, specialized software components - enter Buzz-Word "best-of-breed".

Access to these components is also common practice to a lesser extent, which is also quite reasonable. Why spend a lot of time and money developing your own search algorithm when Apache already offers a freely available product with plenty of documentation and a large, worldwide developer community? And what speaks against using a JavaScript framework like React if it promises flexibility and scalability from Facebook?

The scope and implementation depth of such third-party components may vary from project to project and from team to team, but nobody should deny that it makes sense. Even if you “only” use github for your repos, there is no need for the operation and administration of your own versioning infrastructure and the time is freed up, for example, for a really well thought-out discount model.

Keep an eye on the giants

So what exactly does this mean for e-commerce technology? Well, the question for commercial providers of e-commerce software is how to use third-party technology successfully. In the best case scenario, they benefit like the proverbial dwarfs on the shoulders of giants from the external speed of innovation and use it to strengthen the product instead of relying on their own solutions.

This is a special challenge for providers from the very beginning (the disgrace of early birth) such as Intershop, SAP Hybris or IBM Websphere, which were built in the mid-nineties - long before Elasticsearch (2010), React and Docker (2013) and service providers like the Google Cloud Platform (2011). You are faced with the fundamental decision of whether to further develop your own solutions and services yourself or to open your core to these new components.

For brands and retailers who, for certain reasons, cannot or do not want to use standard commerce software and instead rely on in-house development, this strategy of course means that it is not enough to hire Java or PHP rock stars and write their code 100% to build a green meadow. It is more about keeping your eyes open for such components, attending conferences and exchanging ideas with one another. (And rumor has it that not having to maintain a 15-year-old .NET application every day that can coincide with every commit like a house of cards can be a real motivational boost.)

In conclusion, we can state that the actual technical innovations not only take place in our own teams but also on the part of third-party providers. The challenge is to cleverly connect external components with one another and to automate development, test and deployment processes in a smart and efficient way. That is why we will keep referring to developments in these core technologies here in the blog - and also let one or the other giants have their say.

Dr. Roman Zenner has been active in e-commerce since 2001. He has written leading specialist books on well-known shop systems, publishes regularly in specialist magazines on e-commerce technology and has been working as a Technical Partner Manager at Shopify since January 2020.

I like it:

LikeLoading ...