Synchronization USB Device

Hi guys,
I am looking for some tips.

The following scenario.

Ableton live including a live set, long samples, a Drum Rack, different vsts

Modular triggered by CV/ Gate Outs from the hapax

I wanna use the hapax like the following.

Send midi to the drum rack in live to improvise live
Send midi to a vst synth to play some chords
Send cv gate to different modules and modulation

the hapax can run as a master or a slave I tried both and I can get it in sync with the modular and the ableton metronom, as soon as I want to trigger than the drum rack with the midi sequencer from hapax I get a delay, same as vsts.

At the moment the hapax and Ableton are connected via usb.

What could be a work around? Is there a midi delay setting I can add to the hapax? Or sth?

One option could be to send everything through Ableton and go then back into the modular from Ableton, but this is not what I would like to have.