dbt Core: Prepare the data source files for Cloud Storage connection

Prepare the data source files and store them in your cloud-based storage system. The files must be in one of the following:

  • An AWS S3 bucket.
  • An Azure Data Lake Storage container.
  • A Google Cloud Storage bucket.

  1. In your dbt Core environment, locate the target/ directory of your dbt project. The target/ directory is created when you run dbt commands such as dbt run or dbt compile. The target/ directory must contain:

    • manifest.json
    • Compiled SQL files

    If you have multiple dbt projects, locate the target/ directory for each project.

  2. If the target/ directory does not exist or does not contain the required files:

    1. Set the dbt profile to the environment from which you want to retrieve lineage information.
    2. Run the dbt compile command for each project to generate the required files, including manifest.json and compiled SQL files. For details, go to About dbt compile command and Manifest JSON file in dbt documentation.

  3. Store the target/ directory in a bucket or directory in your cloud-based storage system. Ensure that you preserve the folder structure.

    Ensure that you preserve the folder structure. Your Shared Storage folder must include all files and subdirectories, for example:

    • target/manifest.json
    • target/compiled/project-name/models/

    If you have multiple projects, store each target/ directory in a separate project folder and maintain the structure:

    Copy 
      shared-storage-folder/                                                                                                                                                                                  
      ├── project-1/
      │   └── target/                                                                                                                                                                                           
      │       ├── manifest.json
      │       └── compiled/                                                                                                                                                                                     
      │           └── project-1/models/                                                                                                                                                                       
      ├── project-2/                                                                                                                                                                                            
      │   └── target/
      │       ├── manifest.json                                                                                                                                                                                 
      │       └── compiled/                                                                                                                                                                                   
      │           └── project-2/models/
      └── ...

What's next

Add the Technical Lineage for dbt (Cloud) capability for Cloud Storage connections.