X Tutup

COLLECTED BY

Organization: Archive Team

Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.

History is littered with hundreds of conflicts over the future of a community, group, location or business that were "resolved" when one of the parties stepped ahead and destroyed what was there. With the original point of contention destroyed, the debates would fall to the wayside. Archive Team believes that by duplicated condemned data, the conversation and debate can continue, as well as the richness and insight gained by keeping the materials. Our projects have ranged in size from a single volunteer downloading the data to a small-but-critical site, to over 100 volunteers stepping forward to acquire terabytes of user-created data to save for future generations.

The main site for Archive Team is at archiveteam.org and contains up to the date information on various projects, manifestos, plans and walkthroughs.

This collection contains the output of many Archive Team projects, both ongoing and completed. Thanks to the generous providing of disk space by the Internet Archive, multi-terabyte datasets can be made available, as well as in use by the Wayback Machine, providing a path back to lost websites and work.

Our collection has grown to the point of having sub-collections for the type of data we acquire. If you are seeking to browse the contents of these collections, the Wayback Machine is the best first stop. Otherwise, you are free to dig into the stacks to see what you may find.

The Archive Team Panic Downloads are full pulldowns of currently extant websites, meant to serve as emergency backups for needed sites that are in danger of closing, or which will be missed dearly if suddenly lost due to hard drive crashes or server failures.

Collection: ArchiveBot: The Archive Team Crowdsourced Crawler

ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records it in a WARC, and then uploads that WARC to ArchiveTeam servers for eventual injection into the Internet Archive (or other archive sites).

To use ArchiveBot, drop by #archivebot on EFNet. To interact with ArchiveBot, you issue commands by typing it into the channel. Note you will need channel operator permissions in order to issue archiving jobs. The dashboard shows the sites being downloaded currently.

There is a dashboard running for the archivebot process at http://www.archivebot.com.

ArchiveBot's source code can be found at https://github.com/ArchiveTeam/ArchiveBot.

TIMESTAMPS

The Wayback Machine - https://web.archive.org/web/20200813031653/https://github.com/topics/stream-processing

#

stream-processing

Here are 596 public repositories matching this topic...

onurakpolat / awesome-bigdata

Star

A curated list of awesome big data frameworks, ressources and other awesomeness.

data-science data awesome database data-stream bigdata series-database data-visualization data-warehouse stream-processing data-analytics awesome-list distributed-database visualize-data streaming-data

Updated Aug 7, 2020

google / mediapipe

Star

MediaPipe is the simplest way for researchers and developers to build world-class ML solutions and applications for mobile, edge, cloud and the web.

android c-plus-plus calculator machine-learning framework computer-vision deep-learning inference pipeline-framework stream-processing video-processing perception mobile-development audio-processing graph-framework graph-based mediapipe

Updated Aug 6, 2020
C++

flink-learning

zhisheng17 / flink-learning

Star

flink learning blog. http://www.54tianzhisheng.cn 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

mysql redis elasticsearch streaming kafka spark rabbitmq hbase stream-processing flink rocketmq

Updated Aug 11, 2020
Java

vector

timberio / vector

Star

A lightweight and ultra-fast router for building observability pipelines

rust parser events router pipeline metrics vector logs stream-processing forwarder observability

Updated Aug 12, 2020
Rust

robinhood / faust

Star

Python Stream Processing

python distributed-systems kafka stream-processing asyncio kafka-streams

Updated Aug 11, 2020
Python

confluentinc / ksql

Star

The event streaming database purpose-built for stream processing applications

real-time sql kafka interactive stream-processing kafka-connect materialized-views

Updated Aug 13, 2020
Java

douban / dpark

Star

Python clone of Spark, a MapReduce alike framework in Python

python spark bigdata stream-processing mapreduce dpark

Updated Jan 23, 2019
Python

benthos

Jeffail / benthos

Star

A stream processor for mundane tasks written in Go

go golang kafka cqrs etl rabbitmq amqp logs message-bus event-sourcing nats stream-processing message-queue streaming-data stream-processor etl-pipeline

Updated Aug 12, 2020
Go

madd86 / awesome-system-design

Star

A curated list of awesome System Design (A.K.A. Distributed Computing) resources.

distributed-systems microservices nosql interview stream-processing microservices-architecture relational-database message-broker hadoop-ecosystem

Updated Aug 11, 2020

pipelinedb / pipelinedb

Star

High-performance time-series aggregation for PostgreSQL

c sql stream time-series analytics zeromq realtime postgresql stream-processing push aggregation pipelinedb

Updated May 1, 2019
C

watermill

ThreeDotsLabs / watermill

Star

Building event-driven applications the easy way in Go.

go golang events reactive kafka cqrs sagas rabbitmq event-sourcing nats stream-processing event-driven watermill

Updated Jul 8, 2020
Go

fluent / fluent-bit

Star

Fast and Lightweight Log processor and forwarder for Linux, BSD and OSX

c log logging data-collector stream-processing fluentd fluent-bit cloudnative sql-queries forwarder

Updated Aug 13, 2020
C

francoispqt / gojay

Star

fastest JSON encoder/decoder with powerful stream API for Golang

go golang json encoder decoder perfomance stream-processing stream-decoder

Updated Aug 6, 2020
Go

manuzhang / awesome-streaming

Star

a curated list of awesome streaming frameworks, applications, etc

list awesome stream-processing awesome-list

Updated Aug 12, 2020

nerevu / riko

Star

A Python stream processing engine modeled after Yahoo! Pipes

cli rss data library etl functional-programming asynchronous parallelism stream-processing featured

Updated Jul 30, 2020
Python

WallarooLabs / wallaroo

Star

Distributed Stream Processing

python go linux api golang framework stream-processing pony-language stream-processing-engine stream-processor

Updated Aug 13, 2020
Pony

apache / hudi

Star

Upserts, Deletes And Incremental Processing on Big Data.

bigdata stream-processing data-integration datalake apachespark hudi apachehudi incremental-processing

Updated Aug 13, 2020
Java

siddhi-io / siddhi

Star

Stream Processing and Complex Event Processing Engine

library cloud cep cncf stream-processing cloud-native complex-event-processing online-learning streaming-data-processing

Updated Aug 11, 2020
Java

edp963 / wormhole

Star

Wormhole is a SPaaS (Stream Processing as a Service) Platform

stream-processing spark-streaming wormhole

Updated Aug 4, 2020
JavaScript

gearpump / gearpump

Star

Lightweight real-time big data streaming engine over Akka

scala akka bigdata stream-processing

Updated Jul 27, 2020
Scala

hazelcast-jet

hazelcast / hazelcast-jet

Star

Open-Source Distributed Stream and Batch Processing

java big-data hazelcast stream-processing low-latency batch-processing event-processing hazelcast-jet

Updated Aug 12, 2020
Java

spring-cloud / spring-cloud-dataflow

Star

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes

orchestration stream-processing cloud-native microservices-architecture predictive-analytics batch-processing portability

Updated Aug 12, 2020
Java

spring-cloud / spring-cloud-stream

Star

Framework for building Event-Driven Microservices

microservices reactive messaging stream-processing event-driven-microservices cloud-native

Updated Aug 6, 2020
Java

vladimirvivien / automi

Star

A stream processing API for Go (alpha)

go golang data-stream stream-processing streaming-api

Updated Mar 29, 2020
Go

nodefluent / kafka-streams

Star

equivalent to kafka-streams 🐙 for nodejs ✨

🐢

🚀

✨

nodejs node kafka big-data streams stream-processing kafka-streams

Updated Aug 12, 2020
JavaScript

lw-lin / streaming-readings

Star

Streaming System 相关的论文读物

streaming apache-spark storm stream-processing spark-streaming dataflow flink heron drizzle millwheel s4 streaming-engine spe stream-processing-engine

Updated Mar 31, 2018

nmaquet / kasper

Star

Kasper is a lightweight library for processing Kafka topics.

kafka stream-processing golang-library

Updated Sep 16, 2017
Go

reugn / go-streams

Star

Go stream processing library

redis kafka pipeline etl data-stream pipelines streams stream-processing pulsar aerospike kafka-streams data-pipeline streaming-data stream-processor apache-pulsar

Updated Jul 9, 2020
Go

infoslack / awesome-kafka

Star

A list about Apache Kafka

infrastructure kafka apache-spark stream-processing apache-kafka kafka-streams data-processing data-pipeline streaming-data

Updated Dec 22, 2019

halaxa / json-machine

Star

Efficient, easy-to-use and fast PHP JSON stream parser

php parsing stream-processing json-parser json-stream json-iterator

Updated May 8, 2020
PHP

Improve this page

Add a description, image, and links to the stream-processing topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the stream-processing topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.

X Tutup