What's the best way to organize the directory structure of a big mixed domain project? (discussion)

10    21 Apr 2016 14:28 by u/starTracer

How would you set up the structure of a big (in our case we have around 2500 executables and libraries) mixed domain project that is not based on a particular third party framework.

If a standard directory structure is employed, then a powerful buildsystem would make many things easy as pie (e.g. automatic globbing of sources, creation of unit tests et.c.).

Some features that we have to support in whatever structure we end up using:

  • "Modules", a logical grouping of artifacts that may share the same namespace.
  • "Packages", a logical grouping of modules (possibly with its own namespace).
  • Multi-language (C, C++, Java, Python, PLC)
  • Multiple runtime targets (e.g. Linux, VxWorks, PPC, Intel). Also sometimes the same module share code between targets.
  • Code generation (e.g. IDL, Qt-mocs, Py-bindings, custom...)
  • Resources (e.g. UI glyphs, sounds)
  • Documentation
  • Unit tests
  • Integration tests (scripts and helper-tools)

Also,

  • Pros/cons of single artifact per source directory (like Maven projects)?
  • With a big project like this, not everyone will want to build the whole source tree just to be able to build their module. How would you go about doing this? Build against a pre-installed header+library tree? What are the alternatives?

8 comments

1

My immediate thought is that you are either making your project way too complex, or you are in way over your head. By the time you start making projects like that, you should have a solid grasp on what works for your team.

0

Well, big projects tend to become complex just by the virtue of its size. And we are in the initial stages of the project (software wise), so I'm sounding the field for what have worked well for others.

1

Hi I am going to answer some of your questions, not all because I don't have all the issues you have.

This is how I structure my multi language projects. I build my projects on Windows and on Linux.

Binaries (all my compiled output ends up here. Note for cpp I also create a msc/gnu folder to store os specific binaries)

~/bin/
~/bin/cpp   (all compiled libraries, shared objects, and executables go here, sometimes in their own sub folders)
~/bin/csharp  (all compiled dlls go here)
~/bin/py3   (all eggs ans pyd's go here)
~/bin/jar    (all jars go here)

Build

~/build/makefile    (my solution make file goes here, it builds all my languages.  You could put an ant/maven project here... but I rock it old school :P)

Source Code (all my source code goes here)

~/code/cpp/<projname>/subdirs..     (I always use the projname here so my include paths match my namespaces)
~/code/csharp/... various solutions files
~/code/java/... various java projects/packages
~/code/python3/<eggname>/  ... here I create a folder for each egg that I want to create

Externals (most projects have external dependencies, be it other repository's, external library dependencies, or resources)

~/externs/makefile      (a makefile that links or junctions to external repos or libraries )
~/externs/_template.config.mk   (a template for the config settings)
~/externs/config.mk    (config file local to this environments required external dependencies - this file is not checked into the repo)
~/externs/postgres9   --> /usr/lib/postgres9/include    (this would be a link created by my makefile to point to a dependency)

Generated Code

~/genes/...     (various code generation files go here, the either generate into my ~/code folders or locally in this folder)

Release/Packages (any final complete packages built from the the ~/bin folder most build end up here)

~/releases/<end package>.zip
~/releases/<end package>.debug.zip

Testing (finally all my unit testing ends up here)

~/testing/  ...   (I use the python unittest module to create a common start point that wraps unittests the other languages)
~/testing/cpp
~/testing/csharp
~/testing/java
~/testing/py

The build steps for my project are:

  1. Check out repo
  2. Configure the config.mk (copy _template.config.mk to config.mk and edit) in the externs folder (only have to do this once)
  3. gmake the makefile in the externs folder (only have to do this once)
  4. gmake the makefile in the build folder

Finally in regards to your question about building specific modules only.. this is what I do.

I generate a makefile for every component for each language. Then then I generate a makefile that build these sub components as groups.

Basically I have makefiles that build "projects/components" and makefiles that collectively run other makefiles that act like "solutions".

For example, if I want to compile all the java code in my project I simply go to my build folder and type:

~/build> gmake java

You could achieve the same thing with ant files or maven however (which is something I do at my job... because a lot people are afraid of make files for some reason).

0

Read up on fhs, then go in your own direction

0

What do you mean? Map the FHS structure for each SW-module or..?

0

Well, you could if you wanted to, but that would be overkill, instead try the reverse; mapping all the modules a single FHS structure inside of a project folder.

0

What in the world are you building?

And please keep us updated.

This is ambitious regardless of your skill level. Good luck.

0

It is scientific instrument control software. Which means a lot of devices like motors, sensors and detectors. So the control software is full stack; from the back-end low level device control to data collection and processing to front-end user interfaces.