The sqlfluff Meltano utility pulls data from SQLFluff that can then be sent to a destination using a loader.
Other Available Variants
- sqlfluff (default)
Getting Started
Prerequisites
If you haven't already, follow the initial steps of the Getting Started guide:
Installation and configuration
-
Add the sqlfluff utility to your project
using
:meltano add
-
Configure the sqlfluff settings using
:meltano config
meltano add utility sqlfluff
meltano config sqlfluff set --interactive
Next steps
Update the pip_url in your meltano.yml by appending your dbt adapter (e.g. dbt-snowflake, etc.).
pip_url: sqlfluff sqlfluff-templater-dbt dbt-core dbt-snowflake
Re-install the plugin with the updated pip_url:
meltano install utility sqlfluff
Create a
.sqlfluff
file in the root directory of your project using the sample content below. You will need to put your dbt adapter name in fordialect
andprofiles_dir
without the dbt prefix (e.g. snowflake). This config file is where you can customize the linting rules to match your team's style guide. SQLFluff comes with logic defaults so you will only need to override configurations you want to change.[sqlfluff] templater = dbt dialect = <your_dialect_name> [sqlfluff:templater:dbt] project_dir = transform profiles_dir = transform/profiles/<your_dialect_name> profile = meltano
Create a
.sqlfluffignore
in the root directory of your project using the sample content below. This makes sure SQLFluff ignores auto generated sql or installed packages..meltano/ utilities/ transform/dbt_packages/ transform/target/ transform/dbt_modules/ transform/macros/
Depending on your dbt adapter you will need to override your dbt environment variables using the
env
key so when SQLFluff calls dbt your profile.yml env vars are properly set. Refer to the Meltano transformer docs for details on what variables are needed for your adapter. An example for Snowflake is shown below.utilities: - name: sqlfluff variant: sqlfluff pip_url: sqlfluff sqlfluff-templater-dbt dbt-core dbt-snowflake settings: - name: user env: DBT_SNOWFLAKE_USER - name: password kind: password env: DBT_SNOWFLAKE_PASSWORD - name: role env: DBT_SNOWFLAKE_ROLE - name: account env: DBT_SNOWFLAKE_ACCOUNT - name: warehouse env: DBT_SNOWFLAKE_WAREHOUSE - name: schema env: DBT_SNOWFLAKE_SCHEMA value: foo # This isnt used so we just put a placeholder by default - name: database env: DBT_SNOWFLAKE_DATABASE value: foo # This isnt used so we just put a placeholder by default
SqlFluff does still need access to your warehouse so you have to supply valid credentials similar to your dbt configs which will live under your meltano.yml config and
.env
file.meltano config sqlfluff set user <your username> meltano config sqlfluff set account <your account> meltano config sqlfluff set role <your role> meltano config sqlfluff set warehouse <your warehouse> meltano config sqlfluff set password <your password>
Run
meltano invoke sqlfluff:lint
to lint your SQL files.Run
meltano invoke sqlfluff:fix
to automatically fix your SQL files based on your linting rules.
If you run into any issues, learn how to get help.
Capabilities
The current capabilities forsqlfluff
may have been automatically set when originally added to the Hub. Please review the
capabilities when using this utility. If you find they are out of date, please
consider updating them by making a pull request to the YAML file that defines the
capabilities for this utility.This plugin has the following capabilities:
You can
override these capabilities or specify additional ones
in your meltano.yml
by adding the capabilities
key.
Settings
This plugin currently has no settings defined. If you know the settings required by this plugin, please contribute!Something missing?
This page is generated from a YAML file that you can contribute changes to.
Edit it on GitHub!Looking for help?
#plugins-general
channel.
Install
meltano add utility sqlfluff
Maintenance Status
Meltano Stats
Keywords