![]() dbt does not have that sprawl, which is good and bad.īut similarly to Rails, dbt has the project skeleton there for pure-SQL transforms, and if that is what you need, it makes it so easy to jump right to focusing on the transforms and leave the project structure to dbt's conventions and config. I definitely had a feeling of "is this it"? I was comparing it to Rails's "batteries included" approach for webdev, and that is such a sprawling ecosystem. I was a little underwhelmed when I first saw dbt. ![]() Ok, the incremental builds also require a little more SQL (three more lines in my case) to handle "fresh build" vs "incremental build", but this is so smooth compared to something custom. Switch a transform from a VIEW to a TABLE to an incrementally built TABLE by changing one configuration value.Sure, this is equivalent to "just" keeping a list of sql files in order, but now I don't even need that list. Reference other transforms, or data sources like the csv above, and dbt figures out the order to run the transforms and/or load the csvs.csv file to the project and dbt loads it as a table. So easy to be able to just add - unique or - not null to a yaml doc and have test run on upstream tables. Declaring existing tables as sources makes it easy to add declarative tests to them.I only use it for about a dozen transforms, but already find some of the more minor seeming features to be useful: I see dbt's strengths all in the testing, and the configuration that enables 100% declarative transforms. dbt doesn't touch loading except for static fixtures like. In my experience so far, dbt doesn't overlap at all with other CI/CD tools because it doesn't address at all when your transformations are run.ĭbt overlaps somewhat with Airflow, since Airflow does have some capability to do transforms as I understand it, but I think Airflow's strength is also sequencing and scheduling? And perhaps better at Loading than Transforming. ![]() I have significant experience in CI/CD for delivering and testing applications, but little experience with Airflow. I've only used the standalone CLI based part of dbt. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |