Back to all articles

Building Better Catalogs Together: Our Open-Source Project

Info Setronica October 14th, 2025

As an IT company with over 25 years of experience developing diverse software solutions, we have repeatedly faced the challenge of building catalogs. Over time, we’ve worked with marketplaces, individual online stores, document processing systems, and even large-scale delivery service. Almost every project had one recurring need: a catalog service.

Businesses across industries often need a reliable and flexible way to organize and present information – products, services, or structured data. The question is not whether a catalog is needed – it almost always is. The real challenge lies in defining what kind of catalog serves businesses best.

In over two decades of projects, we had worked on dozens of catalogs – for marketplaces, e-commerce, document management systems, logistics platforms, and more. And in every single product, there was always some form of catalog:

  • a catalog of products,
  • a catalog of services,
  • or a catalog of data/documents.

This recurring pattern led us to question our own approach and examine the deeper issues beneath the surface. What began as technical frustration evolved into a project to solve not just our own challenges, but those faced by developers everywhere. Here’s how that journey unfolded.

Learning the hard way

With every new project, we’ve always tried to do the same thing: listen carefully to our clients, understand their needs in detail, and deliver exactly what they expect.

Of course, there were times when we felt we could see a better solution – but it wasn’t always possible to convince a client to take a longer path than they were ready for.

That experience made us realize: if we truly believe in certain ideas, perhaps we should invest in them ourselves and build a product that can be reused and quickly customized for different business needs.

And so, unexpectedly, we found ourselves on the other side of the table — as clients.

Working with brilliant developers who could deliver fast, high-quality results, we experienced the process from a different perspective. Instead of openness to explore our needs, we often encountered resistance. This was eye-opening and made us reflect on our own approach.

Through these experiences, we’ve learned a hard lesson: if you start building a catalog from scratch for each new project – ignoring past experience – you inevitably hit the same wall. The system lacks flexibility, fails to cover evolving business scenarios, and quickly becomes a bottleneck.

Businesses grow faster than the software meant to support them, and nothing is more frustrating than watching technical limitations slow down real opportunities.

Every time, we built catalogs from scratch. Sometimes we refined older versions, but in the end, we always started over. Which made us ask a simple but powerful question:

👉 Why are we reinventing the wheel every time?

This question became our starting point. As we reflected on our years of building catalog systems, we realized that we weren’t just facing a technical challenge – we were looking at an opportunity to fundamentally rethink how catalogs should be approached. Our setbacks could become the foundation for something better.

From experience to open source: our solution

From our own experiences and from challenges we’ve seen others face, we came up with the idea of creating something like a “handbook for catalogs.” Not as a set of rules, but as a way to collect and share the practices, patterns, and lessons that have worked for us so far.

At the same time, it’s also a chance for us to learn new approaches we might not know yet.

But since we live in the real world, theory alone is not enough. That’s why we decided to transform our accumulated knowledge into a practical solution – an open-source catalog project.

We realized we already had the expertise and the experience:

  • We know which solutions work well at small scale and which scale to millions of records.
  • We know the bottlenecks, like when a “simple” DB query can no longer handle full-text search, and ElasticSearch (or another tool) needs to be added.
  • We’ve seen the pitfalls firsthand and learned how to avoid them.

So why not create a flexible catalog product – one that can serve as a component in almost any system? A product where developers (or businesses) can:

  • Configure the catalog to their exact needs.
  • Decide whether they need advanced search or just a lightweight structure.
  • Handle anything from small datasets to millions of records.
  • Reuse proven patterns instead of starting from scratch.

Of course, writing a catalog isn’t rocket science. Any skilled developer can do it. But doing it again and again is a waste of time. By making our solution open source, we want to:

  • Share our accumulated knowledge.
  • Save others from repeating the same mistakes.
  • Invite the community to give feedback, criticize, and improve it with us.

We’re not just building a tool – we’re building it together.

Technical foundation & capabilities

Our catalog solution follows a modular architecture designed for flexibility and scalability. At its core, the system consists of well-defined components that handle different aspects of catalog management:

Data processing architecture

The system uses a job-based approach with separate import and export workflows:

Import process:

  • Reader: FlatFileItemReader parses CSV files containing catalog items
  • Processor: ValidatingItemProcessor validates items against predefined ProductSchema
  • Writer: ProductItemWriter stores validated items in the ProductEntity database (PostgreSQL by default, but any other could be set there)

Export process:

  • CreateFileStep: Prepares the export operation
  • Reader: JpaPagingItemReader retrieves products from the database with efficient pagination
  • Processor: Transforms database entities into export-ready product objects
  • Writer: FlatFileItemWriter outputs data to CSV format
  • StoreFileStep: Finalizes and stores the exported file

This architecture provides several key advantages:

  1. Separation of concerns. Each component handles a specific task, making the system easier to maintain and extend.
  2. Validation pipeline. Data quality is enforced through schema validation before entering the system.
  3. Flexible storage. The system can work with different database technologies.
  4. Scalable processing: Pagination ensures efficient handling of large catalogs.
  5. Format adaptability. While currently using CSV for demonstration, the readers and writers can be extended to support other formats.

The modular design allows developers to replace or extend specific components without affecting the entire system. For example, you could swap the FlatFileItemReader with an API-based reader, or replace the validation logic with your own custom rules.

Data processing architecture

Join the journey: goals & how to contribute

With this open-source catalog project, we’re trying to create something practical that grows through real-world use and input from other developers.

What we’re trying to do:

  • Build a catalog foundation that works for different business needs
  • Reduce duplicate work by creating reusable components
  • Document what works (and what doesn’t) based on actual experience

How you can help

We’ve put the first version on GitHub, and now we need your input:

  • Tell us what you think: What’s useful? What’s missing?
  • Tell others: Let people know if you find it helpful
  • Add to it: Fix bugs or add features that would help you

Different perspectives make for better software. Your experience might help us see things we’ve missed.

👉 https://github.com/setronica-dev/unify-catalog/tree/prototype

We hope this saves you time and helps your projects grow. Let us know what you think!

Chapters

Related posts