NVCC does not define USE_MPI when compiling with omp+mpi+cuda

Created by: cypox

Branch Occurs in branch gpu-hackathlon

Explanation The cuda compier is not seeing the USE_MPI flag when compiling ".cu" files. This raises a serious issue when using cuda+mpi. The ParticlesData struct compiled with mpic++ is different from the one compiled with nvcc. This is not shown as error nor as warning during compilation but is problematic when I added the DeviceParticleData.

How to reproduce?

1. Add: printf("[DEBUG] -- sizeof: %ld\n", sizeof(Dataset)); To file: density.hpp At line: 109 (before cuda::computeDensity<T>(l, taskList, d); )

2. Add: printf("[DEBUG] -- sizeof: %ld\n", sizeof(ParticlesData)); To file: cudaDensity.cu At line: 53 (before const int maz = d.bbox.PBCz ? 2 : 0; )

3. While we should get similar results (the two structures should be the same), we get different sized because the variables declared in ParticlesData.cpp after the USE_MPI definition are not included in the struct passed to the cuda file.

How to reproduce 2?

Just add: #ifdef USE_MPI #warning "BROOKS WAS HERE" #endif To one of the *.cu files and compile with "mpi+omp+cuda".