etl · GitHub Topics

apache / airflow

#计算机科学#Apache Airflow 是一个workflow工作流调度、编排、监控平台

Python 40.56 k

4 小时前

pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

batch-processing kafka pathway Python streaming 机器学习 real-time data-analytics data-pipelines data-processing dataflow etl etl-framework iot-analytics Rust stream-processing time-series-analysis

Python 26.78 k

2 天前

airbytehq / airbyte

Airbyte 开源 EL(T) 平台，帮助用户将数据从应用程序，API 和数据库中同步到数据仓库

data pipeline 数据分析 data-engineering Java Python etl change-data-capture data-collection data-integration elt BigQuery redshift snowflake data-pipeline sql-server MySQL PostgreSQL s3 自托管

Python 18.43 k

16 小时前

apache / doris

Doris 是百度开源的支持对海量大数据进行快速分析的MPP数据库。

olap 数据库 hadoop hive hudi iceberg real-time SQL BigQuery dbt delta-lake elt etl lakehouse query-engine redshift snowflake Apache Spark

Java 13.81 k

2 天前

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.

data-pipelines dagster workflow 数据科学 workflow-automation Python scheduler data-orchestrator etl analytics data-engineering mlops orchestration data-integration metadata

Python 13.38 k

2 天前

redpanda-data / connect

Fancy stream processing made operationally mundane

message-queue stream-processing streaming-data message-bus logs stream-processor cqrs event-sourcing Go kafka amqp rabbitmq nats etl data-engineering DataOps

Go 8.38 k

2 天前

mage-ai / mage-ai

#计算机科学#🧙 Build, run, and manage data pipelines for integrating and transforming data.

机器学习人工智能 data data-engineering 数据科学 Python elt etl pipelines data-pipelines orchestration data-integration SQL Apache Spark dbt pipeline reverse-etl transformation

Python 8.37 k

2 天前

turbot / steampipe

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

steampipe PostgreSQL postgresql-fdw cloud 安全 Amazon Web Services Azure cis cnapp cspm DevOps devsecops Google 云 Go Kubernetes Terraform etl SQLite Hacktoberfest

Go 7.36 k

3 天前

cloudquery / cloudquery

一个高性能ELT 框架，powered by Apache Arrow

Amazon Web Services Google 云 Azure SQL data-integration elt etl etl-framework BigQuery data-collection data-engineering Kubernetes data airbyte GitHub API 数据分析 Google Go cspm attack-surface-management

Go 6.12 k

2 天前

apache / flink-cdc

Flink CDC Connector 是ApacheFlink的一组数据源连接器

change-data-capture cdc batch data-integration data-pipeline distributed elt etl flink kafka MySQL paimon PostgreSQL real-time schema-evolution

Java 6.1 k

2 天前

rudderlabs / rudder-server

Privacy and Security focused Segment-alternative, in Golang and React

隐私 warehouse-management data-warehouse customer-data-platform data-integration data-synchronization etl BigQuery redshift snowflake data-pipeline elt data-engineering cdp event-streaming

Go 4.21 k

3 天前

orchest / orchest

#编辑器#Build data pipelines, the easy way 🛠️

数据科学机器学习 pipelines ide Jupyter Notebook cloud 自托管 jupyterlab notebooks Docker Python data-pipelines 部署 Kubernetes airflow dag etl etl-pipeline

TypeScript 4.13 k

2 年前

aws / aws-sdk-pandas

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Par...

Python Amazon Web Services pandas apache-arrow apache-parquet data-engineering etl 数据科学 redshift athena lambda aws-lambda emr MySQL modin ray

Python 4.03 k

2 天前

nucleuscloud / neosync

Open Source Data Security Platform for Developers to Monitor and Detect PII, Anonymize Production Data and Sync it across environments.

Docker Go synthetic-data benthos etl orchestration Testing TypeScript Open Source React 自托管 Kubernetes Next test-data-generator fine-tuning synthetic-data-generation faker MySQL PostgreSQL

Go 3.87 k

4 天前

quadratichq / quadratic

Spreadsheet with AI, Code, Connections

Python 数据科学 Spreadsheet SQL quadratic etl data-engineering 数据分析 data WebAssembly webgl 人工智能

Rust 3.7 k

2 天前

Netflix / maestro

#计算机科学#Maestro: Netflix’s Workflow Orchestrator

analytics 自动化 batch-processing dag data-engineering DataOps data-orchestrator data-pipelines 数据科学 elt etl Java 机器学习 mlops orchestration scheduler workflow workflow-engine workflow-orchestration

Java 3.48 k

21 小时前

blockchain-etl / ethereum-etl

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Go...

以太坊 CSV export erc20 erc20-tokens etl transaction SQL Amazon Web Services Google 云 BigQuery erc721

Python 3.03 k

2 个月前

xyflow / awesome-node-based-uis

#Awesome#A curated list with resources about node-based UIs

node-based-ui Awesome Lists etl visual-programming

3.01 k

2 个月前

chonkie-ai / chonkie

#自然语言处理#🦛 CHONK your texts with Chonkie ✨ - The no-nonsense RAG chunking library

人工智能 chunking rag text-processing 自然语言处理 Python semantic-segmentation vector-search etl retrieval

Python 2.87 k

3 个月前

apache / incubator-devlake

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and co...

data 数据分析 data-engineering data-integration data-transfers DevOps domain-layer etl Go integration jira Open Source user-friendly dora Hacktoberfest

Go 2.75 k

5 天前