You can see which pipeline wrote a table.
André Lindenberg님의 업데이트
You can see which pipeline wrote a table. You cannot see which agent read it. OpenMetadata, the open-source data catalog, now stores agent and MCP runs as records: the tables read, the columns, the tool calls, the cost. Ask what an agent touched last month and the answer sits next to the tables themselves. Your runtime posts the record. 14.7k stars, Apache-2.0. #AIAgents #DataGovernance #DataLineage #AIInfrastructure #MCP
Read lineage is the part that actually hurts in production. Write lineage tells you what changed, but when an agent gives you a bad answer you need to know what it read. Having tool calls and column reads sit next to the table turns that into a lookup instead of a dig. The open question is whether you can trust the record. If the runtime reports its own runs, an agent that skips the instrumented path just never shows up. Catching it at the gateway is what makes it auditable.추천[
댓글
So the agent reads the table, writes its own witness statement, reports its own tool calls — and we call the result auditable. The suspect has become the court stenographer. Nitesh points to the first problem: anything outside the instrumented path becomes invisible. A gateway can catch the read, but even then it only proves which column was touched. It still cannot tell us whether the agent preserved the column’s meaning, respected the authority attached to it, or quietly reconstructed both afterwards. Excellent metadata. Rather generous use of the word evidence.추천[
댓글
댓글을 보거나 남기려면 로그인
관련성이 더 높은 업데이트
- #TodayILearned 15 of 365 Table JOINs Day 13, I learned Database normalization, that split one table into many to eliminate anomalies, but it has a cost, which is joins. A JOIN reassembles normalized data. It combines rows from two tables based on a related column (usually a foreign key). Every foreign key that appears in a JOIN needs an index. The constraint is for integrity. The index is for speed. They’re separate concerns. Interested in learning Joins? https://lnkd.in/g6Sw4QQn #TodayILearned #learninpublic #softwareengineering #databases 댓글 1 댓글을 보거나 남기려면 로그인
- One of our customers had a standard way of testing data movement jobs: submit it Friday night, check the results Monday morning. Moving a single VSAM file — 60 million records — off the mainframe took two days. They ran the same file with PropelZ. #Two hours, sustaining 56,000 #records per second into #PostgreSQL in the #cloud. No staging database. No change data capture. #Zero lines of custom code. Vince Re, our CTO, says the most common reaction to numbers like this is disbelief. Customers call to report a failure. We ask whether they’ve looked at the output file yet. They haven’t. The data is always there. Marc Sokol and Vince just recorded a full session walking through how it actually works — the real customer environment, why ETL and CDC break down at enterprise scale, and what changes when getting mainframe data to the cloud stops being a six-month project. ~70% of the world’s transactional data still originates on the mainframe. Every #AI and #analytics initiative in your organization depends on data that is sitting there right now. #mainframe #IBMZ #dataengineering #AI #mainframemodernization VirtualZ Computing
How to Move VSAM Data to PostgreSQL at 56,000 Records per Second — No Code, No CDC
https://www.youtube.com/
댓글을 보거나 남기려면 로그인 - 댓글을 보거나 남기려면 로그인
- We just shipped an 𝗠𝗖𝗣 𝘀𝗲𝗿𝘃𝗲𝗿 𝗳𝗼𝗿 𝗗𝗲𝗰𝘂𝗯𝗲. Most “AI on top of your data” demos show a chatbot answering questions. This is different. Claude can now read from and write back to your data. In the demo we recorded: Claude traces lineage and answers business questions in the same thread, no dashboard hopping. It finds every table missing an owner in one prompt. Before a data engineer pushes a schema change to GitHub, Claude checks downstream impact first. It reconciles business glossary terms against the physical tables that should map to them. It runs a PII sweep across MSSQL, Snowflake, and Databricks in one conversation, and catches classification gaps a manual review would miss. Then it pushes a corrected column description straight back into the catalog. Live. No ticket, no wiki page, no waiting on someone else. That last part is the real unlock. Most catalog tools stop at “here’s what we found.” This one acts on it. If you’re running data governance, lineage, or compliance across a fragmented stack (banking, insurance, telecom, whatever your mess looks like), this is worth five minutes. Full demo here: https://lnkd.in/gUdUhTPG
MCP Server Demo | Claude Talking Directly to Your Data
https://www.youtube.com/
댓글 4 댓글을 보거나 남기려면 로그인 - Fun fact: instant branching is not a function of data size. 10 TB database is also branched / clones in around 1 second. Here is the power of that using “xata scratch”, instant disposable Postgres branch (a scratch pad) that’s automatically deleted the moment your session ends. Most “try this query on prod” workflows still look the same: clone the database, wait, hope the dump finishes, clean up later (or never). xata scratch is a new command in the Xata CLI that short-circuits that. It forks a disposable branch off your real Postgres, including databases at 2TB+, in about a second. You land in psql (or run SQL inline), experiment safely, and when you exit the branch is torn down. Same data shape as production. No leftover environments. No waiting on a full copy. Demo below. 댓글을 보거나 남기려면 로그인
- Can I quickly try this query on prod? Now the answer can be: please don’t, but run xata scratch. A production-like Postgres branch in about a second, automatically deleted when you’re done. Most “try this query on prod” workflows still look the same: clone the database, wait, hope the dump finishes, clean up later (or never). xata scratch is a new command in the Xata CLI that short-circuits that. It forks a disposable branch off your real Postgres, including databases at 2TB+, in about a second. You land in psql (or run SQL inline), experiment safely, and when you exit the branch is torn down. Same data shape as production. No leftover environments. No waiting on a full copy. Demo below. 댓글 1 댓글을 보거나 남기려면 로그인
- Most “try this query on prod” workflows still look the same: clone the database, wait, hope the dump finishes, clean up later (or never). xata scratch is a new command in the Xata CLI that short-circuits that. It forks a disposable branch off your real Postgres, including databases at 2TB+, in about a second. You land in psql (or run SQL inline), experiment safely, and when you exit the branch is torn down. Same data shape as production. No leftover environments. No waiting on a full copy. Demo below. 댓글 2 댓글을 보거나 남기려면 로그인
- We just shipped Monospace, and it’s in public preview 🥳 If you’ve lost a week rebuilding the same CRUD APIs just to reach data locked in three different systems, this fixes that. Register your data sources and you get a full REST API and MCP server immediately. No migration, no duplicating your data. Start building the same day. If you’re building on enterprise data, we’d genuinely like your feedback on this one. https://monospace.io/ 댓글 2 댓글을 보거나 남기려면 로그인
- I’m looking forward to speaking for CTXiUG next Tuesday! I’m so excited about this session that I’m even willing to miss my beloved Tuesday night running group to present it. I’ll be sharing Finding Data in the Forest: Exploring Three-Part Naming in SQL, where we’ll dig into how three-part naming can simplify SQL development across libraries and databases. Whether you’re just getting started or looking to sharpen your SQL skills on IBM i, I hope you’ll join us! #IBMChampion #IBM #IBMi #RPG #SQL #Db2fori #CTXiUG https://lnkd.in/gqZvTcWb 댓글 1 댓글을 보거나 남기려면 로그인
- Types of Views in Snowflake When working with large-scale data, choosing the right type of view can make a huge difference in performance, security, and maintainability. Standard View – Simplifies complex queries without storing data Secure View – Protects sensitive data by hiding underlying logic Materialized View – Boosts performance with precomputed results Dynamic Table – Keeps data fresh with automated refresh (ELT pipelines) External View – Query data directly from external storage 댓글을 보거나 남기려면 로그인
팔로워 68,834명
글쓴이의 다른 업데이트
-
The Most Expensive Rung on the Ladder
André Lindenberg 1일 -
The Half We Don’t Budget For
André Lindenberg 1주 -
From Loops to Graphs
André Lindenberg 2주
- Source: Unknown