dbt Core: Prepare the data source files for Shared Storage connection

Prepare the data source files and store them in the folder that you created when you created the Shared Storage connection.

  1. In your dbt Core environment, locate the target/ directory of your dbt project. The target/ directory is created when you run dbt commands such as dbt run or dbt compile. The target/ directory must contain:

    • manifest.json
    • Compiled SQL files

    If you have multiple dbt projects, locate the target/ directory for each project.

  2. If the target/ directory does not exist or does not contain the required files:

    1. Set the dbt profile to the environment from which you want to retrieve lineage information.
    2. Run the dbt compile command for each project to generate the required files, including manifest.json and compiled SQL files. For details, go to About dbt compile command and Manifest JSON file in dbt documentation.

  3. Store the target/ directory in your Shared Storage connection folder. Ensure that you preserve the folder structure.

    Ensure that you preserve the folder structure. Your Shared Storage folder must include all files and subdirectories, for example:

    • target/manifest.json
    • target/compiled/project-name/models/

    If you have multiple projects, store each target/ directory in a separate project folder and maintain the structure:

    Copy 
      shared-storage-folder/                                                                                                                                                                                  
      ├── project-1/
      │   └── target/                                                                                                                                                                                           
      │       ├── manifest.json
      │       └── compiled/                                                                                                                                                                                     
      │           └── project-1/models/                                                                                                                                                                       
      ├── project-2/                                                                                                                                                                                            
      │   └── target/
      │       ├── manifest.json                                                                                                                                                                                 
      │       └── compiled/                                                                                                                                                                                   
      │           └── project-2/models/
      └── ...
    Note If you use Collibra Platform 2024.05 or newer, ensure that you use the Edge CLI tool for creating the Shared Storage connection and storing files in the Shared Storage connection folder. Go to the following Create a Shared Storage connection step for detailed instructions. For more information, go to Edge Command Line Interface (CLI).

What's next

Add the Technical Lineage for dbt capability for Shared Storage connections.