Mage

Marketing analytics teams waste hours every day hunting down data quality errors. Parker Goldman at Cannonball Advertising engineered a better solution. He used Mage Pro to build dedicated validation pipelines that run every morning, before his team even logs in. These pipelines: ✅ Automate naming convention checks ✅ Validate theme taxonomy (A, B, C) ✅ Detect nulls with automated alerts Exceptions land in their inbox as summaries where they can identify and fix the issue prior to the clients finding the errors. His team went from reactive firefighting to proactive quality control, freeing up his analysts to perform other more essential tasks. According to parker, if you can define the rule, you can automate the check. #marketinganalytics #dataengineering

One wrong backfill script corrupts your entire data warehouse You'll have duplicate records everywhere ↳ And your AI model doesn't know these are mistakes It thinks they are patterns Here's the problem with backfilling: ↳ You hack together manual scripts. ↳ One mistake creates duplicates. ↳ Your data becomes unreliable. ↳ Your AI trains on corrupted data. But Mage treats backfilling as a core feature Creating idempotent pipelines is straightforward with Mage's backfill tools. So what's that mean when you run a backfill in Mage? ✅ No duplicates ✅ No corruption ✅ The same results You have AI-ready data even when your pipelines fail. What's your biggest backfill nightmare? Watch the full demo: link is in the comments 👇 🔔 Follow me for more on building AI-ready data infrastructure. ♻️ Repost if your team struggles with backfills. #dataengineering #ai

When you hit Cancel, does your infrastructure actually stop? 🛑 In a perfect world, stopping a pipeline run is the end of the story. In production, it is often just the beginning of a manual cleanup project. We have all lived through the cancellation hangover 👇 🧹 Temp tables left behind cluttering the warehouse 💸 Cloud resources idling away your budget ⏳ Downstream systems stuck waiting on a status update that never comes because failure logic was skipped A hard stop should not be a messy stop. With the latest Mage Pro release, we are introducing Pipeline Run Cancellation Callbacks. You now get a dedicated hook that fires the moment a run is interrupted, so you can automate your exit strategy. Drop temporary resources 🗑️, send targeted Slack alerts 💬, trigger rollback scripts 🔄, or run any custom cleanup logic without manual intervention. Data engineering is about building systems that handle the unexpected. Now your Stop button is as engineered as your Start button. ✨ 📖 Doc: https://lnkd.in/gM7ApvXa #DataEngineering #Pipelines #Infrastructure #MageAI #DevOps #DataPlatforms

Great news! The storm was... less severe than I thought it would be. I'll be heading down to Huntsville to speak at the NASA - National Aeronautics and Space Administration AI Symposium after all. Here's the good news: Even if you can't be there, I'll be publishing REFLECT on Github, so you can run it yourself, improve it, etc. REFLECT is my love letter to all the public servants who work across hundreds of disciplines of Earth and Climate Science. It takes the combined research of NASA, NOAA, the Forest Service, USGS... and many many more -- and creates a cost, damage, and risk simulation engine. This is useful for... insurance companies. Vacation companies. Logistics companies. People who perhaps just want to understand the risks of purchasing a home in Florida or the gulf coast? I couldn't have made this without the team from Mage, because running 55 data pipelines across climate science ain't easy without the right tool. Mage is the best ETL tool on the market. Period. The geospatial pieces are all running Devii for GraphQL API middleware. Once again, fetching across 55 databases ain't easy, but Devii allowed me to instantly create a GraphQL API... and I mean instantly. 5 minutes and I was done. If you hate building API's, Devii is Oprah-tier helpful. And of course, the climate models are all open source. Special thanks to this little startup called NVIDIA for making DARCY open source so I had an easy starting point. In the coming weeks, Dynamo Technologies, LLC will be publishing an API template with Devii that ANYONE can depoy on their own AWS, Azure or GCP environment. (Or on prem, if you got the hardware. It's all in Docker containers) This will also be open source. So if you're say... trying to better understand weather and climate risks for MDO effects on the battlefield ,(weather changes war -- ask the dudes who fought in the Battle of the Bulge), health risk, etc, we want to make sure you can aggregate it and get the data you need. When I started at Dynamo a few weeks ago, it was because our CEO Alex gave me the green light to do stuff like this. If you happen to download our open source code, and you're able to deliver it at a better value to the American taxpayers... honestly? Good for you. Every federal contractor should be in a race to make things faster, better and cheaper for your tax dollars. With that said, lacing up some J's won't make you Michael Jordan. You still need to know ball, but I'm happy to share my signature shoes with you, so to speak. If we've seen anything the past few days, we all interact with the weather, and risk planning can only happen if we can run simulations to understand what happens, when it happens. Can't wait to see y'all in Alabama!

Your most complex pipeline just disappeared. 😬 Accidental deletes and overwrites happen. Losing hours of work does not have to. We added a new CLI command that lets you instantly recover pipeline and block files from historical versions, even if they were fully overwritten. Build fast, without fear of the Delete key. 🔗 https://lnkd.in/gt_QysxH

The Mojave just opened up in Mage Pro ☢️ Meet the New Vegas release. 🔧 Pipeline Recovery from File Versions 🧹 Cancellation Callbacks & Cleanup 🏗️ Fault-Tolerant Workspace Hooks 📉 Instant Block & Output Collapse 🔐 Self-Serve Password Reset 👾 And more! Dive into the release notes 👉 https://lnkd.in/gARi-V4a #dataengineering #mageai #magepro #prolog #releasenotes #changelog #fallout #newvegas #datarecovery

¿Airflow sigue siendo el rey o es hora de modernizar la orquestación? 🎻 Tener los mejores scripts de datos no sirve de nada si no se ejecutan en el momento correcto. Ya definimos nuestra arquitectura ELT y tenemos modelos sólidos en dbt. Ahora llega la pregunta del millón: ¿Quién coordina todo esto? Durante años, Apache Airflow ha sido el estándar indiscutible. Nos enseñó a pensar en DAGs (Grafos Acíclicos Dirigidos) y nos salvó de la locura de los Cron Jobs manuales. Pero seamos honestos, Airflow puede ser pesado. Configurar el scheduler, manejar dependencias complejas y testear localmente a veces se siente como pilotar un avión cuando solo necesitas ir en bicicleta. ✈️🚲 Por eso, herramientas como Mage.ai, Prefect y Dagster están ganando terreno rápidamente en el Modern Data Stack. ¿Qué traen de nuevo? 1️⃣ Data Awareness: No solo pasan el estado de la tarea ("éxito/fallo"), sino que entienden los datos que fluyen entre ellas. 2️⃣ Experiencia de Desarrollador (DX): Interfaces UI increíbles y testing local mucho más sencillo. 3️⃣ Código como activo: Tratan los pipelines como proyectos de software modernos desde el día 1. Airflow sigue siendo el más robusto y con mayor comunidad, pero la brecha se está cerrando. #DataOrchestration #ApacheAirflow #MageAI #Prefect #DataEngineering #Pipeline #Automation

#ETL dünyasında her geçen gün bir araç ortaya çıkıyor, ancak bunlardan çok azı piyasada tutunabiliyor. #Airflow'un daha modern ve daha kolay versiyonu mottosuyla yola çıkan #MageAi ı duymuş muydunuz? Eğer ilk defa duyuyorsanız, Gizem Dağdeviren'in yazısıyla beraber #Mage ile tanışabilirsiniz https://lnkd.in/dDm2yer9 Veri ile alakalı teknik video ve makale içeriklerini ve veri profosoynelleri ile networking imkanını kaçırmamak için sen de Türkiye Veri Topluluğu'na üye ol!