# 2 Minute Serverless > Freshly baked serverless nuggets in 2-minutes. Full text of every published post. --- # Relational Foundation Models: Bridging Databases and AI URL: https://www.serverless.fyi/p/relational-foundation-models-bridging-databases-and-ai/ Published: 2025-11-06 Tags: rfm ## Relational Foundation Models: Engineering Primer Relational foundation models are emerging as a class of foundation models designed to understand, reason over, and predict from **relational databases** – not just single tables or unstructured text. Unlike tabular or LLM-focused models, relational models accept multi-table schemas, foreign keys, temporal rows and cross-table joins as first-class inputs and generalize across unseen schemas and tasks. They aim to enable in-context learning and zero/few-shot predictive performance for common enterprise use cases (fraud, churn, demand forecasting) without per-dataset retraining. See the vision paper [“Towards Foundation Models for Relational Databases”](https://arxiv.org/abs/2305.15321) by Vogel et al. ### Why This Matters to Engineers Enterprises run on relational data. Existing workflows often require bespoke feature engineering, ETL pipelines and per-dataset model training. Relational foundation models promise to (a) collapse engineering effort by offering a single pretrained model for many schemas, (b) support hybrid query/prediction workloads (temporal predictions, per-entity scoring) and (c) enable explainability by surfacing schema-aware reasoning traces. From an engineering perspective that means faster prototyping, fewer bespoke ML pipelines, and an API surface that treats a database as an input modality. ### Leading Projects and Papers A few notable public efforts and research artifacts to follow: - **KumoRFM (**__[Kumo.ai](https://kumo.ai)__**)** — a practical Relational Foundation Model built for enterprise relational databases; demonstrates schema-agnostic in-context learning across multi-table schema and predictive tasks. See [Kumo’s announcement](https://www.prnewswire.com/news-releases/kumo-unveils-worlds-first-relational-foundation-model-rfm-for-instant-predictions-on-enterprise-data-302460899.html) and the technical paper [“A Foundation Model for In-Context Learning on Relational Data”](https://kumo.ai/research/kumo_relational_foundation_model.pdf) - **Griffin** — a graph-centric relational database foundation model that encodes relational structure with cross-attention and novel aggregation layers. See [the arXiv paper](https://arxiv.org/abs/2505.05568) and [GitHub implementation](https://github.com/yanxwb/Griffin) - **Google’s Graph Foundation Models for Relational Data** — Google Research exploring graph-native pretraining to generalize across table sets and tasks. See [Google Research blog](https://research.google/blog/graph-foundation-models-for-relational-data/) - Additional academic work: _“Relational Transformer: Toward Zero-Shot Foundation Models for Relational Data”_ (Ranjan et al.) These repositories and papers matter because they allow engineers to experiment with relational foundation models today: benchmark performance, integrate with existing systems (TLS, VPNs), and begin architecture planning for migration. ### Comparison Matrix (Quick) | **Project / Paper** | **Target Input** | **Schema Generalization** | **Primary Use Case** | **Notes** | | ------------------- | --------------------------- | ------------------------- | ------------------------------------ | ----------------------------------------- | | KumoRFM | Multi-table RDB | Yes (schema-agnostic) | Entity scoring, temporal predictions | Enterprise focus, high performance claims | | Griffin | Graph representation of RDB | Yes | Unified multi-task RDB inference | GitHub code available | | Google GFM | Table sets → graphs | Yes | Scaling to arbitrary schemas | Google blog covers approach | | TabPFN / Prior Labs | Single table / tabular | Limited | Fast tabular predictions, AutoML | Related but narrower scope | ### Use Cases and Engineering Benefits - **Real-time entity scoring**: Risk/fraud scoring at transaction time by querying a model over live relational joins instead of pre-computed features. For example, KumoRFM supports zero-shot predictions out of a relational warehouse. - **Temporal and sequence prediction**: Churn prediction, next-item recommendation, demand forecasting that require cross-table history and time-aware reasoning. Models like Griffin and Google’s approach target these. - **Rapid prototyping and data exploration**: Engineers can test predictive tasks on new datasets without building full feature stores and pipelines. The Vogel et al. paper argues this “collapse” of engineering effort. - **Hybrid query augmentation**: SQL workflows can be augmented with model answers — for example: “Which customers should we call today?” becomes a SQL + model pipeline. ### Engineering Challenges and Practical Notes - **Preprocessing and encoding**: Designing canonical table encodings, handling joins, nulls, categorical cardinality, and schema shifts at scale. Papers like Griffin and Google’s outline this challenge. - **Latency and serving infrastructure**: Real-time scoring requires optimized runtime, caching, or incremental evaluation. Enterprise models like Kumo target production ability. - **Governance and data privacy**: Enterprise relational data often contains sensitive PII. Models must run under governance, audit, and deployment constraints (on-prem, hybrid cloud). - **Schema shifts and transfer learning**: The core challenge of RFMs is supporting unseen schemas. Models like Griffin and Google’s work focus on transfer ability. ### Bottom Line Relational foundation models represent a fast-moving intersection of representation learning, graph neural models and foundation model engineering. For engineering teams, they offer a compelling way to collapse bespoke pipelines into reusable pretrained systems — but they bring practical challenges in encoding, latency and governance. Track KumoRFM, Griffin, Google’s GFM and related academic work, then prototype with relational encoders and inference APIs to understand trade-offs for your stack. --- ### Comprehensive list of papers and technical reports (selected) - **Vogel et al., “Towards Foundation Models for Relational Databases”** — Vision paper outlining why foundation models for relational data matter, recommended pretraining strategies, and research directions. [https://arxiv.org/abs/2305.15321](https://arxiv.org/abs/2305.15321) - **KumoRFM — “A Foundation Model for In-Context Learning on Relational Data” (Kumo technical paper / report)** — Practical technical writeup and evaluation from Kumo describing their relational foundation model, schema-agnostic in-context learning approach, and enterprise integration examples. [https://kumo.ai/research/kumo_relational_foundation_model.pdf](https://kumo.ai/research/kumo_relational_foundation_model.pdf) Announcement / overview: [https://www.prnewswire.com/news-releases/kumo-unveils-worlds-first-relational-foundation-model-rfm-for-instant-predictions-on-enterprise-data-302460899.html](https://www.prnewswire.com/news-releases/kumo-unveils-worlds-first-relational-foundation-model-rfm-for-instant-predictions-on-enterprise-data-302460899.html) - **Griffin — Graph centric Relational Database Foundation Model (preprint and code)** — Research preprint and associated code that treat relational databases as graphs, using graph encoders and cross-attention to support multi-table reasoning and transfer. Paper (arXiv): [https://arxiv.org/abs/2505.05568](https://arxiv.org/abs/2505.05568) Implementation / examples: [https://github.com/yanxwb/Griffin](https://github.com/yanxwb/Griffin) - **Google Research — Graph Foundation Models for Relational Data (research blog and pointers)** — Google Research discussion of graph-native pretraining methods for relational data and scaling strategies across arbitrary schemas. Useful as a conceptual and engineering reference. [https://research.google/blog/graph-foundation-models-for-relational-data/](https://research.google/blog/graph-foundation-models-for-relational-data/) - **Relational Transformer / related preprints (overview pointers)** — A body of emerging preprints exploring transformer based encoders adapted to relational schemas, temporal joins, and schema generalization. For a representative reference on the line of work, see research index listings and citations in the Vogel et al. paper. (See Vogel et al. bibliography for references.) Vogel et al. bibliography / references: [https://arxiv.org/abs/2305.15321](https://arxiv.org/abs/2305.15321) - **Selected project and implementation repos (practical references)** — While not all are pure academic papers, these repositories include engineering reports, implementation notes and benchmarks that are essential reading for engineers experimenting with RFMs: - Kumo project pages and code resources: [https://kumo.ai/research/](https://kumo.ai/research/) and [https://kumo.ai/company/news/kumo-relational-foundation-model/](https://kumo.ai/company/news/kumo-relational-foundation-model/) - Griffin GitHub: [https://github.com/yanxwb/Griffin](https://github.com/yanxwb/Griffin) --- --- # OpenHands: Declarative Automation for Developers URL: https://www.serverless.fyi/p/openhands-declarative-automation-for-developers/ Published: 2025-11-05 Tags: ai-agents ## Unlocking Developer Productivity with OpenHands In the era of AI-driven development, engineering teams face growing expectations to build more, iterate faster, and scale with leaner teams. OpenHands positions itself as _code less, make more_ — an open-source platform of autonomous software agents designed to help developers automate and accelerate engineering workflows. --- ### Moving from Manual Tasks to Autonomous Systems OpenHands treats development work such as code edits, command execution, API calls, or creating pull requests as modular tasks that can be delegated to agents. This allows senior engineers to focus on architecture and problem-solving, while agents handle repeatable or time-consuming tasks. The platform has gained significant traction with tens of thousands of stars and forks on GitHub. --- ### Architecture and Core Components The repository is structured into folders such as frontend, openhands, containers, models, scripts, and tests. **Agent Runtime using CLI and Docker** ```bash docker pull docker.openhands.dev/openhands/runtime:0.60-nikolaik docker run -it --rm \ -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.openhands.dev/openhands/runtime:0.60-nikolaik \ -p 3000:3000 openhands-app ``` This containerized setup ensures reproducible agent behavior and clean environment isolation. **LLM Provider Abstraction** OpenHands supports multiple large language model providers. For example, it references **anthropic/claude-sonnet-4-5-20250929** and allows you to plug in alternative or self-hosted models through the models directory. **Microagents and Workflows** The platform includes prebuilt microagents for code review, bug triage, CI configuration, refactoring, and documentation. These microagents can be composed into custom workflows and adapted to your development lifecycle. **Testing and Evaluation** The tests folder contains performance benchmarks and scripts to evaluate how effectively agents handle tasks. Teams can measure improvements like reduction in cycle times or number of automated pull requests. --- ### Why OpenHands Matters for Engineers **Scale output without adding headcount** Delegate repetitive work and focus engineering time on architecture, technical debt, or complex problem-solving. **Improve code consistency and quality** Agents follow structured templates and project conventions, reducing variability and enforcing standards. **Faster experimentation** Spin up locally, test scenarios, and connect to GitHub or GitLab in a few hours instead of building automation from scratch. **Open-source ecosystem** The MIT license for the core project gives flexibility to audit, customize, and embed OpenHands into enterprise infrastructure without vendor lock-in. --- ### How to Adopt OpenHands Effectively **Start locally** [Clone the repository](https://github.com/OpenHands/OpenHands), run a simple agent via CLI, and experiment with small code-generation or documentation tasks. **Deploy to production environments** Use Docker or Kubernetes for hosting. Connect to your version control systems and enable automated pull requests. **Integrate into workflows** Replace manual tasks with agent-driven flows such as test stub generation, config file updates, or CI script creation. **Measure results** Track metrics related to productivity, task completion time, and agent success using included test scripts. **Customize agents** Develop domain-specific agents such as data pipeline configuration assistants, cloud infrastructure provisioning helpers, or support ticket triage bots. --- ### Conclusion OpenHands is more than an AI utility. It is a framework for scaling engineering operations with intelligent agents. By offloading task-based work, teams gain time to focus on system design, strategy, and innovation. If your engineering organization is exploring AI-enabled development, OpenHands is one of the most practical and extensible platforms to adopt. Start small, measure value, and scale intentionally. --- --- # Post-Quantum Cryptography URL: https://www.serverless.fyi/p/post-quantum-cryptography-fd6d/ Published: 2025-11-04 Tags: pqca ## The Quantum-Safe Imperative: Why Engineers Need Post-Quantum Cryptography Now Quantum computing is no longer experimental fiction. With breakthroughs like Google’s new “Quantum Echoes” algorithm running 13,000× faster than classical supercomputers on its Willow chip, engineers must treat post-quantum cryptography (PQC) as an _engineering requirement_, not a future upgrade. Classical cryptographic systems like RSA and ECC underpin TLS handshakes, VPNs, SSH, banking systems, and blockchain wallets. A sufficiently powerful quantum computer could break these by solving problems like integer factorization and discrete logarithms exponentially faster. The real threat is “store now, decrypt later”: attackers capture encrypted data today and decrypt it when quantum capabilities mature. --- ## From Strategy to Execution: The Role of PQCA The **Post-Quantum Cryptography Alliance (PQCA)**, under the Linux Foundation, is building the roadmap for quantum-safe infrastructure. Their goals include: - Defining migration pathways for organizations and protocols - Building open-source reference implementations of PQC algorithms - Creating tools that support crypto-agility and compliance - Aligning vendors, academia, and security engineers around standards The PQCA Technical Advisory Council (TAC) governs project standards, lifecycles, interoperability, and documentation. --- ## GitHub Repositories Engineers Should Explore Engineers can start experimenting with [PQCA](https://pqca.org/) today using these [open repositories](https://github.com/pqca): - **pq-code-package**: High-assurance C and assembly implementations of ML-KEM (Kyber), ML-DSA (Dilithium), and optimized microcontroller variants. - **cbomkit**: A crypto bill-of-materials generator that scans your source code or container images to list cryptographic assets like RSA, ECC, and PQC usage. - **PQCA GitHub Organization**: A growing collection of reference code, documentation, and tooling. These are production-ready starting points to experiment, benchmark, and integrate hybrid classical plus PQC protocols in CI or staging environments. --- ## Signals from the Field: Quantum Computing Is Accelerating From Reddit discussions to engineering subforums, urgency is growing. One highly upvoted Reddit comment said: Google’s quantum system solved in minutes what classical systems would need longer than the age of the universe to compute. Google’s blog confirmed: Our Willow quantum chip achieved the first-ever verifiable quantum advantage, performing 13,000× faster than classical machines. There is also increasing attention on new world records like a 0.000015 percent quantum error rate and Google’s new algorithm breaking performance barriers. --- ## Why This Matters for Engineers These breakthroughs translate into three core engineering problems: ### 1. Crypto-Agility Design systems to allow seamless replacement of cryptographic algorithms. Decouple protocol logic from crypto primitives. ### 2. Inventory and Visibility You cannot migrate what you cannot measure. Use tools like cbomkit to audit: - TLS configurations - API gateways - Firmware update mechanisms - SSH key distribution - Certificate authorities and code-signing systems ### 3. Hybrid Deployment Until PQC becomes fully standardized and supported by browsers and hardware, use hybrid models. Example: ECDHE + Kyber for key exchange in TLS. --- ## Practical Roadmap for Engineers **Step 1: Audit your stack** Run cryptographic inventories across repositories and infrastructure. Identify where RSA, ECC, and long-lived keys are used. **Step 2: Benchmark PQC libraries** Clone implementations like ML-KEM or Dilithium. Measure handshake time, CPU cycles, binary size, and RAM usage. **Step 3: Prototype hybrid TLS** Implement a hybrid handshake in a test service. Log metrics like latency impact and certificate size. **Step 4: Prioritize migration targets** Focus on systems handling long-lifetime data: medical records, financial contracts, firmware for IoT devices, and digital identities. **Step 5: Monitor quantum and standards progress** Track NIST PQC standardization, PQCA announcements, and hardware error-rate improvements. --- ## Final Thoughts Post-quantum cryptography is no longer theoretical. With Google demonstrating verifiable quantum advantage and industry groups like PQCA building usable standards and codebases, engineers must begin real implementation work. Start small. Test in staging. Build for crypto-agility. Your future systems, users, and data integrity depend on it. --- --- # Understanding the DNS Header URL: https://www.serverless.fyi/p/understanding-the-dns-header/ Published: 2025-10-29 Tags: dns The **Domain Name System (DNS)** is often called the “phonebook of the internet,” translating human-readable domain names like [serverless.fyi](https://serverless.fyi) into IP addresses that computers can understand. While most users interact with DNS through their browsers, the real magic happens in the **DNS header**, a compact 12-byte structure that drives every query and response. --- ## Why the DNS Header Matters Every time you open a website, your device sends a DNS query to a server. The **DNS header** contains the information needed to route the query, track the response, and apply special rules for caching, recursion, or security. Understanding it can be a game-changer for network engineers, penetration testers, and developers working with networking protocols. --- ## Anatomy of a DNS Header A DNS header is exactly **12 bytes**, divided into six key fields: ```text +---------------------+---------------------+ | ID (16) | Flags (16) | +---------------------+---------------------+ | QDCOUNT (16) | ANCOUNT (16) | +---------------------+---------------------+ | NSCOUNT (16) | ARCOUNT (16) | +---------------------+---------------------+ ``` **Breakdown:** - **ID (16 bits):** Unique identifier for the query, used to match requests and responses. - **Flags (16 bits):** Bit-level flags controlling query/response type, recursion, and error codes. - **QDCOUNT (16 bits):** Number of questions in the query section. - **ANCOUNT (16 bits):** Number of answer records. - **NSCOUNT (16 bits):** Number of authority records. - **ARCOUNT (16 bits):** Number of additional records. --- ## A Closer Look at Flags The **Flags** field is packed with bit-level details that affect DNS behavior: ```text QR | Opcode | AA | TC | RD | RA | Z | RCODE ``` - **QR (1 bit):** Query (0) or Response (1) - **Opcode (4 bits):** Type of query (standard, inverse, status) - **AA (1 bit):** Authoritative Answer - **TC (1 bit):** Truncated message - **RD (1 bit):** Recursion Desired - **RA (1 bit):** Recursion Available - **Z (3 bits):** Reserved - **RCODE (4 bits):** Response code (NoError, NXDomain, etc.) **Pro Tip:** You can manipulate RD and RA flags when testing DNS resolvers to see how recursion affects response times and caching. --- ## Hands-On: Inspecting DNS Headers Here’s a sample **packet dump** from a dig command: ```bash dig @8.8.8.8 serverless.fyi +norecurse +tcp ``` ```text ;; HEADER SECTION: ;; opcode: QUERY, status: NOERROR, id: 0x1a2b ;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ``` - **id:** 0x1a2b → matches request and response - **flags:** qr rd → response, recursion desired - **Counts:** QD=1, AN=1 For network engineers, this is a simple way to verify resolver behavior and test DNS configurations. --- ## Tricks and Tips with DNS Headers 1. **Custom Queries:** Modify the ID and flags to test how servers handle out-of-order or spoofed requests. 2. **Load Testing:** Adjust QDCOUNT to batch multiple questions in one query to simulate high DNS load. 3. **Security Analysis:** Inspect the AA, TC, and RCODE bits to detect misconfigured authoritative servers. 4. **Packet Crafting:** Tools like scapy let you build raw DNS packets and control every header bit: ```python from scapy.all import * dns_query = IP(dst="8.8.8.8")/UDP()/DNS(id=0x1234, rd=1, qd=DNSQR(qname="serverless.fyi")) send(dns_query) ``` --- ## Conclusion: Small Header, Big Impact The **DNS header** may only be 12 bytes, but it carries critical instructions that govern the behavior of the most important internet service. From troubleshooting latency issues to testing server security, understanding each field opens a world of possibilities. Whether you are a developer, engineer, or cybersecurity enthusiast, mastering DNS headers gives you deeper insight into the hidden mechanics of the internet. **Next time you open a website, remember:** a tiny header is working behind the scenes to make it all possible. --- # Netskope | Agentic AI: NextGen SRE Meetup URL: https://www.serverless.fyi/p/netskope-agentic-ai-nextgen-sre-meetup/ Published: 2025-03-24 Tags: serverless-meetup ## Post-Event Update On March 6th, 2025, Serverless community came together for an exclusive engineering and technology meetup at Netskope HQ which brought together over 60 industry leaders, innovators, and experts in AI, SRE, security, DevOps, data science, and infrastructure. The event fostered meaningful connections, sparked insightful discussions, and provided a deep dive into emerging trends shaping the industry. With an engaging agenda featuring technical talks, a thought-provoking panel discussion, and plenty of networking opportunities, attendees walked away with valuable insights and new collaborations. --- **Meetup Agenda** The evening kicked off with a warm welcome and introductions, setting the stage for the deep technical conversations to follow. The agenda was structured to provide a mix of expert insights, interactive discussions, and opportunities for networking. - **Opening Keynote:** Krishna Narayanaswamy, CTO, Netskope - **Technical Talks:** - Atul Kulkarni (Amazon Rufus) on AI-driven DevOps pipelines - Jineet Doshi (Intuit) on evaluating generative AI systems - **Panel Discussion:** Industry leaders discussed the intersection of Agentic AI, SRE, security, and DevOps - **Closing Remarks:** John Cho, SVP Engineering, Netskope - **Networking Session:** Unstructured mingling to encourage collaborations --- ### Technical Talk and Panel Discussion Details The first technical talk, **"Smart Ops: Building the AI-Enhanced DevOps Pipeline,"** was delivered by **Atul Kulkarni**, an engineering leader at Amazon Rufus. He provided a deep dive into AI-driven automation in DevOps, explaining how AI can streamline operational workflows, enhance system reliability, and optimize performance. Following this, **Jineet Doshi**, AI Lead at Intuit, presented **"Smart Agents, Smarter Evaluations: Evaluating Generative AI Systems Holistically."** His talk explored best practices for assessing AI agent performance, measuring effectiveness, and ensuring robust real-world applications through comprehensive evaluation frameworks. The panel discussion was a highlight of the evening, featuring industry experts from [AWS](https://aws.amazon.com/), [Sully.AI](https://sully.ai), [OpsVerse](https://opsverse.io/), [PlayerZeroAI](https://playerzero.ai/), [ArizeAI](https://arize.com/), and [KumoAI](https://kumo.ai/). Moderated by a seasoned expert, the discussion explored the challenges and opportunities at the intersection of Agentic AI, SRE, and security, covering topics such as enhancing developer productivity and optimizing large-scale AI systems. ![](https://www.serverless.fyi/_astro/img-39455bf8b81f.D8UqNFBK.jpg) --- **An Evening of Insightful Discussions** The meetup provided a platform for attendees to share experiences, discuss emerging trends, and gain insights from some of the brightest minds in the field. The interactive nature of the talks and panel discussion allowed for engaging Q\&A sessions, enabling participants to explore topics relevant to their work and challenges. ![](https://www.serverless.fyi/_astro/img-a78d6fc11f7c.CfpNp_f_.jpg) ![](https://www.serverless.fyi/_astro/img-0aec17ed3967.BHnqo5Vn.jpg) --- **Expert Panel:** The panel discussion featured: - **Rueben Jimenez**, AI Architect at [AWS](https://aws.amazon.com/) - **Chaitanya Gharpure**, Co-Founder and CTO at [Sully.AI](https://sully.ai) - **Arul Francis**, Co-Founder at [OpsVerse](https://opsverse.io/) - **Animesh Koratana**, Founder at [PlayerZeroAI](https://playerzero.ai/) - **Hakan Tekgul**, AI Solutions at [ArizeAI](https://arize.com/) - **Parth Pareek**, Product at [KumoAI](https://kumo.ai/) These industry leaders shared valuable insights during the panel discussion, enriching the event with their expertise. These experts provided deep insights into the current and future landscape of AI-driven operations, security, and infrastructure management. Discussions revolved around real-world case studies, best practices, and the impact of AI in scaling engineering operations. ![](https://www.serverless.fyi/_astro/img-583c032c221a.BAHiwz-6.jpg) --- **Fostering Connections & Collaboration** Beyond the structured agenda, the event facilitated valuable networking opportunities. Attendees had the chance to connect with industry peers, exchange ideas, and explore potential collaborations. The atmosphere was electric with discussions ranging from AI-powered automation to innovative approaches in security and infrastructure. ![](https://www.serverless.fyi/_astro/img-89c202f0457b.COJ_oRxU.jpg) ![](https://www.serverless.fyi/_astro/img-ea54b9fbafa3.B3zMs_e9.jpg) ![](https://www.serverless.fyi/_astro/img-9dfff4c7afb1.CjCUF3ZA.jpg) --- **A Huge Thank You!** A big thank you to **Netskope** for hosting this fantastic event and to all the speakers and panelists for sharing their expertise. Special appreciation to the attendees for making this gathering a success by contributing to meaningful discussions and fostering a collaborative environment. Stay tuned for future events as we continue to bring together the best minds in AI, SRE, DevOps, and beyond! 🚀 ![](https://www.serverless.fyi/_astro/img-f7997706a6e6.BJLhlsJN.jpg) --- # Serverless ft. AnChainAI | AI in Fintech & Security Meetup URL: https://www.serverless.fyi/p/serverless-ft-anchainai-ai-in-fintech-security-meetup/ Published: 2025-02-09 Tags: serverless-meetup ## Post-Event Update: Serverless ft. AnChainAI | AI in Fintech Meetup This past week on Jan 29th 2025, we hosted an incredible evening at the [**Serverless ft. AnChainAI | AI in Fintech Meetup**](https://lu.ma/08hdn03b), bringing together an exclusive group of AI, data, and fintech leaders for deep discussions, knowledge sharing, and high-impact networking. It was an honor to welcome a carefully curated group of 25 attendees from companies like Wayfair, Netskope, AnChain, Adobe, Meta, SpyCloud, and Capital One, ensuring a meaningful exchange of ideas and insights. ### An Evening of Insightful Discussions We kicked off with warm introductions, setting the stage for an engaging and interactive session. The event featured three thought-provoking technical talks, each offering a unique perspective on AI's evolving role in fintech, security, and compliance. 🔹 [**Dr. Victor Fang**](https://www.linkedin.com/in/drvictorfang/), Founder & CEO of [AnChain.AI](https://anchain.ai), shared how **Agentic AI is disrupting the fintech industry**—reducing operational costs by 3x, improving accuracy by 90%, and addressing $14 billion in annual financial crime fines with unprecedented efficiency. 🔹 [**Aashish Naik**](https://www.linkedin.com/in/naikaa/), Engineering Director, AIOps at [Netskope](https://www.netskope.com/), introduced us to **Macaw, an Agentic AI YAML Coding Bot**, showcasing how AI-driven automation is revolutionizing infrastructure and security management. 🔹 [**Venkatesh Chitnis**](https://www.linkedin.com/in/venkatesh-chitnis/), Engineering Leader, MLOps at [Wayfair](https://www.wayfair.com/), discussed **securing low-latency inferencing for real-time ML use cases**, highlighting the critical balance between speed and security in AI-powered applications. ![](https://www.serverless.fyi/_astro/img-67cbb9bb8328.C1o7o6mU.jpg) ### Expert Panel: AI & Fintech at the Crossroads A highlight of the evening was our panel discussion, where top industry experts explored the evolving landscape of AI in fintech, compliance, and security. The panel featured: - [**Dr. Victor Fang**](https://www.linkedin.com/in/drvictorfang/), Founder, [AnChain.AI](https://anchain.ai) - [**Aashish Naik**](https://www.linkedin.com/in/naikaa/), Director AIOps, [Netskope](https://www.netskope.com/) - [**Venkatesh Chitnis**](https://www.linkedin.com/in/venkatesh-chitnis/), Engineering Leader, MLOps, [Wayfair](https://www.wayfair.com/) - [**Suresh Mandava**](https://www.linkedin.com/in/mandavasuresh/), Founder, [Unovie](https://unovie.ai/) (Edge-based Agentic AI Platform) - [**Kareem Fazal**](https://www.linkedin.com/in/kareem-fazal-30288212/), Principal Program Manager, [Microsoft](https://www.microsoft.com/en-us/) - [**Mukul Bafana**](https://www.linkedin.com/in/mukulbafana/), Co-Founder, [Omuni](https://omuni.com/) The discussion delved into the **pressing challenges and emerging trends** shaping the industry, from the ethical implications of AI-driven compliance to the increasing role of automation in mitigating financial fraud. ![](https://www.serverless.fyi/_astro/img-1f2369bc8180.DvmYxjrT.jpg) ### Fostering Connections & Collaboration One of the most valuable aspects of the event was the **networking session**, where attendees had the chance to connect, exchange ideas, and explore potential collaborations. With a room full of fintech leaders, AI engineers, investors, and startup founders, the conversations were rich with insights and opportunities. --- ### Disrupting FinTech and SaaS: Dr. Victor Fang's Insights on Agentic AI Dr. Victor Fang’s technical talk, “How Agentic AI is Transforming FinTech and Combating Financial Crimes,” was a highlight at the recent Agentic AI event. His insightful presentation explored how Agentic AI is reshaping the financial technology landscape, particularly in the areas of financial crime prevention and risk management. Fang also expressed his gratitude for the engaging panel, which featured thought leaders like Mohinish Shaikh, Aashish Naik, Mukul Bafana, Kareem Fazal, and Don Khamapirad, who sparked rich discussions on pressing AI and tech topics. Looking ahead to 2025, Dr. Fang shared the following predictions for Agentic AI: - **Disruption across sectors**: Agentic AI will impact all sectors, with a focus on FinTech. It will begin with essential services in risk management, setting the stage for broader applications. - **SaaS evolution**: SaaS winners in 2025 will be verticals that successfully leverage open-source AI models on proprietary data, creating customized solutions that offer a competitive advantage. As AI continues to evolve, these trends are expected to shape the future of industries, driving innovation and offering new growth opportunities. Stay tuned for more updates from [AnChain.AI](https://anchain.ai)! ![](https://www.serverless.fyi/_astro/img-d0c2a4ff5061.B5FYb1UH.jpg) --- ### Optimizing Real-Time Personalization for Business Growth In his talk, Venkatesh Chitnis addressed the challenges and opportunities of enabling real-time, session-based matching for use cases like related searches, product recommendations, and personalized experiences. Implementing such systems effectively can unlock significant business value by improving customer engagement and conversion rates. #### Challenges in High-Dimensional Matching Building a real-time matching system requires handling high-dimensional vector spaces with large cardinality. Common industry challenges include: - **Scalability**: Managing one-to-many and many-to-many comparisons efficiently. - **Computational Overhead**: High infrastructure costs for batch pre-compute and real-time model serving. - **Latency Considerations**: Ensuring fast response times while maintaining accuracy in recommendations. - **Cost Constraints**: Optimizing costs, especially when dealing with M \* N cardinality (millions of comparisons). #### Key Outcomes & Impact By leveraging optimized indexing and infrastructure improvements, businesses can achieve: - **Cost Reduction**: Batch pre-compute and efficient online serving can reduce brute-force computing costs from \~$4K to a few hundred dollars. - **Revenue Growth**: More accurate intent capture leads to double-digit million-dollar annual revenue gains. - **Scalable Architecture**: Containerized embedding and indexing solutions support elastic scaling, ensuring efficiency in real-time serving. #### Implementation Considerations To deploy an effective system, teams must evaluate: - Dataset size and distribution patterns. - Quantization techniques to balance performance and precision. - Infrastructure choices (e.g., GPU types) to manage cost and availability. - In-house expertise and time-to-market trade-offs. With these strategies, businesses can unlock powerful real-time personalization while optimizing infrastructure costs. ![](https://www.serverless.fyi/_astro/img-b86b7ea394e0.0ztBdMSR.jpg) --- ### The Future of Edge AI: Open-Source Distilled Models During the meetup, panel guest Suresh Mandava, founder of Unovie—an edge-based agentic AI platform—shared the following insights. DeepSeek-R1 is redefining how we deploy AI at the edge by distilling **large language models (LLMs) into smaller, efficient variants (SLMs)**. This transformation enables organizations to run AI **offline, closer to their data**, within secure private sandboxes. Deep Seek R2 enhances the capabilities of its predecessors by integrating a more advanced retrieval system for long-range information. Following papers provides information on how DeepSeek went about creating these models. 1. Building on _Deep Seek V1_ (_[arxiv:2501.12948](https://arxiv.org/abs/2501.12948)_) 2. _Deep Seek V2_ (_[arxiv:2405.04434](https://arxiv.org/abs/2405.04434)_), 3. R2 leverages _Deep Seek LLM_ (_[arxiv:2401.02954](https://arxiv.org/abs/2401.02954)_) for improved query resolution and accuracy in large-scale data retrieval. As Edge AI gains momentum in 2025, open-source distilled models are set to drive widespread adoption through key advantages: \- **Accessibility & Affordability:** Open-source models lower the barrier to entry, making cutting-edge AI available to startups and enterprises without high costs. \- **Efficiency & Performance:** Optimized for resource-constrained environments, these models reduce latency and power consumption—key for real-time applications. AMD’s partnership with DeepSeek underscores their scalability across edge hardware. \- **Customization & Flexibility:** Developers can fine-tune models to specific use cases, ensuring tailored solutions for industries like healthcare, finance, and security. \- **Collaboration & Innovation:** Open-source fosters rapid advancements, allowing contributors to enhance models and accelerate AI progress. By enabling **scalable, compliant, and efficient AI deployments**, these advancements position Edge AI as a game-changer for **next-gen intelligent applications**. With AI moving **closer to data sources**, 2025 is the year to start **building smarter, decentralized, and secure AI-driven systems**. --- ### Automating YAML Management with Agentic AI In his talk, Aashish Naik explored the challenges of managing YAML configurations in large-scale distributed systems and how Agentic AI can streamline this process. YAML, while widely used for configuration management, introduces several challenges: - **Human Errors**: Indentation issues, typos, and schema mismatches can lead to deployment failures. - **Scalability**: Managing YAML across thousands of services and teams becomes increasingly complex. - **Customization**: Creating flexible yet maintainable templates for diverse use cases is difficult. To address these challenges, Naik introduced **Agentic AI**, an autonomous system capable of reasoning, planning, and executing tasks. By leveraging **YAML Agents**, AI can: - **Understand and enforce schemas, constraints, and best practices** to ensure consistency across configurations. - **Automatically generate, validate, and optimize YAML files**, reducing manual effort and errors. - **Detect, explain, and resolve errors in real-time**, providing transparency and reliability in configuration management. By embedding AI-driven intelligence into configuration workflows, YAML Agents enable seamless automation, reducing the risk of misconfigurations while improving efficiency and scalability. This approach not only minimizes human intervention but also ensures that large-scale AI systems remain adaptable, resilient, and easy to manage. ![](https://www.serverless.fyi/_astro/img-6da1fc8a8c0e.D_vvmyjl.jpg) --- ### AI in Fintech & Security: Insights and 2025 Predictions from the Serverless ft. [**AnChain.AI**](https://anchain.ai) Panel Discussion The _Serverless ft. _*[AnChain.AI](https://anchain.ai)*_ | AI in Fintech & Security_ meetup brought together industry experts to explore how AI is transforming fintech, security, and enterprise SaaS. The panelists shared deep insights into AI-driven automation, fraud prevention, and the future of AI-native businesses. #### AI in Security: Automating Complex Decision-Making Aashish Naik highlighted how AI is alleviating **alert fatigue** among Security Operations Center (SOC) teams. With an overwhelming number of security alerts, teams often miss critical threats. **Agentic AI** and **AI-driven security automation** can detect patterns, block attacks, and take proactive action, reducing human workload and improving response times. #### AI in Fintech: Fraud Prevention & Personalization Venkatesh Chitnis and Kareem Fazal discussed how AI is reshaping **fraud detection**, **regulatory compliance**, and **digital marketing** in fintech. AI models can **instantly detect financial crimes**, enhance personalization, and automate compliance, enabling fintech companies to operate securely at scale. #### Knowledge Graphs & AI for Regulatory Compliance Suresh Mandava emphasized the power of **knowledge graphs and ontologies** in **reducing AI hallucination** and ensuring factual verification. These tools help financial institutions maintain regulatory compliance while improving fraud detection accuracy. #### The Future of AI-Native Enterprises Mukul Bafana explored **Agentic AI beyond fintech**, highlighting its role in reshaping **enterprise architecture** and **high-compliance SaaS businesses**. The panel also debated the future of **hybrid AI-human teams**, where AI acts as a co-pilot, augmenting human decision-making in high-stakes environments. #### Dr. Victor Fang’s 2025 Predictions Dr. Victor Fang, Founder of [AnChain.AI](https://anchain.ai), shared his vision for **AI’s impact in 2025**: - **Agentic AI will disrupt all sectors**, starting with essential services like **risk management** in FinTech. AI-driven automation will become a necessity, not a luxury, for businesses managing financial risks. - **The next generation of SaaS will be built on open-source AI models applied to proprietary data**. Companies that successfully leverage this combination will dominate their verticals. Dr. Fang’s upcoming blog, _"Is SaaS Dead? How Agentic AI Will Redefine SaaS in 2025,"_ will explore how AI is fundamentally reshaping the SaaS landscape. #### The Tough Questions on AI Moderated by Mohinish Shaikh, the discussion also tackled **hot takes on AI**, challenging panelists to address ethical concerns, regulatory hurdles, and the broader implications of AI-driven automation. As AI continues to evolve, its applications in fintech and security will drive innovation, automation, and operational efficiency, shaping the next generation of intelligent, scalable enterprises. ![](https://www.serverless.fyi/_astro/img-646b1071fa5c.B9hh6leA.jpg) --- ### A Huge Thank You! A big thank you to everyone who joined us for this exclusive gathering! We truly appreciate your participation and contributions to the discussions. We hope you found the event insightful, enjoyable, and full of valuable takeaways. We’d love to hear your feedback to continue improving future meetups. **Please share your thoughts and leave a rating on the event page!** Stay tuned for more exciting meetups, and we look forward to seeing you at our next event! 🚀 **Join our community** to stay updated on future events: [_Serverless Community_](https://lu.ma/serverless.fyi) ![](https://www.serverless.fyi/_astro/img-e097026a457f.DLMEIgqX.jpg) --- # Reflecting on 2024: Growth, Learning, and Milestones URL: https://www.serverless.fyi/p/reflecting-on-2024-growth-learning-and-milestones/ Published: 2025-01-02 Tags: 2024 As 2024 came to a close, I find myself reflecting on a year filled with opportunities, challenges, and growth. From deepening my involvement in the tech community to pushing the boundaries of what’s possible in cloud infrastructure and stream processing, this year has been nothing short of transformative. In this post, I’ll share highlights from my professional journey, community contributions, and the personal growth that defined my year. --- ## DeltaStream - The Complete Stream Processing Data Platform. During my time at [DeltaStream](https://www.deltastream.io), I played a key role in building a cutting-edge stream processing platform from the ground up, focused on empowering companies to derive real-time insights from their data.\ \ As a founding member of engineering team, I worked on shaping the product vision, scaling the architecture, and fostering a culture of innovation within the team. Through collaboration and perseverance, we successfully grew the company and achieved Series A funding, establishing DeltaStream as a leader in the stream processing space. As Head of Cloud Infrastructure at DeltaStream, I architected, engineered, and managed the entire cloud ecosystem. This included developing control plane, multi-tenant dataplane, dedicated dataplane, BYOC dataplane, and implementing cloud connectivity solutions to integrate multiple data-planes with the control plane.\ \ Midway through 2024, I decided to take a personal sabbatical, reflecting on and celebrating the completion of this rewarding journey. ### Career Break During my career break, I took the opportunity to travel and reflect on my next steps. Having recently moved to Texas, I explored its diverse cities and towns, including Dallas, Houston, League City, Texas City, Port Lavaca, Galveston, Corpus Christi, Fort Worth, Irving, San Antonio, and Fredericksburg.\ \ Alongside traveling, I spent time reading impactful papers such as _Google File System_, _Dynamo_, _BigTable_, _Cassandra_, _Spanner_, _Kora_, and _ClickHouse_. Check them out:\ \ \- Google File System: A Scalable distributed file system for large workloads\ \- Dynamo: -Amazon's Highly Available Key Value Store\ \- BigTable: Google's Distributed Storage System for structured data\ \- Cassandra: A Scalable NoSQL database with tunable consistency.\ \- Spanner: Google’s globally distributed and strongly consistent database.\ \- Kora: A Cloud-Native Event Streaming Platform For Kafka\ \- ClickHouse - Lightning Fast Analytics for Everyone\ \ Through my network, I've discovered compelling open-source projects like [SlateDB](https://www.slatedb.io) (an embedded Rush database for S3) and [TektiteDB](https://www.tektitedb.io) (a Go-based streaming store with native processing). My focus remains on developing innovative infrastructure, stream processing, and distributed systems, particularly for AI applications. ### GitHub 2024 I maintained an active GitHub presence, contributing to DeltaStream repositories and personal projects. Check out my 2024 Wrapped! [https://git-wrapped.com/profiles/mohnishbasha](https://git-wrapped.com/profiles/mohnishbasha) ![](https://www.serverless.fyi/_astro/img-3cbb0362a933.CW8mEJfN.png) --- ### Serverless Newsletter The [**Serverless Newsletter**](https://www.serverless.fyi) became an integral part of my routine in 2024. Curating and sharing insights about serverless architecture trends allowed me to connect with a broader audience of founders, investors, and early-stage employees.\ \ The Serverless Newsletter is a curated publication I maintain to share insights, trends, and best practices in AI, Data, InfraStructure, Distributed Systems, Startup Hustle and Career Guidance. It serves as a resource for developers, architects, fellow engineers and enthusiasts, highlighting advancements, practical use cases, and innovative ideas in the ecosystem. Alongside the newsletter, I organized serverless social meetups to foster a sense of community among professionals. These gatherings create a space for knowledge-sharing, collaboration, and networking, allowing participants to discuss challenges, solutions, and emerging opportunities in serverless technologies. Together, the newsletter and meetups contribute to community building and advancing the broader adoption of serverless paradigms in the tech industry.\ \ The newsletter’s subscriber base grew by 200%, a testament to the shared curiosity within this community. Writing for the newsletter not only solidified my own understanding but also helped foster meaningful discussions around where technology is headed. --- ### Community Collaboration Professionally, 2024 was a rewarding year. I continued advising pre-seed startups, helping them navigate the complexities of scaling enterprise software and implementing real-time data strategies. One of the standout moments was contributing to a founder’s journey at [Rappo](https://www.buildrappo.com/) to secure a early pre-seed round by refining their pitch deck and product distribution strategy to capture metrics and usage insights. Check out my personalized reflection card and create yours at [https://reflection.buildrappo.com/](https://reflection.buildrappo.com/) [Rappo](https://www.buildrappo.com/) is a platform for **[champions](https://www.buildrappo.com/#who-is-champion)** to explore new tools, products, network with peers and early stage technical founders. ![](https://www.serverless.fyi/_astro/img-5c2cc33a2575.BJFYbsMH.png) --- ### Year of Self-Reflection and Improvements Beyond professional milestones, 2024 was also a year of introspection. Balancing a busy schedule with personal growth, I prioritized fitness, hiking, and paddle boarding—activities that not only kept me physically active but also offered mental clarity. Reading became a source of both relaxation and inspiration, with books on leadership and emerging technologies shaping my perspective. This year also challenged me to be more intentional with my time and energy. Stepping back to reflect on my goals allowed me to align my professional pursuits with my long-term aspirations, ensuring that my work remains fulfilling and impactful. --- **What's Next for 2025** I see 2025 as the year of exponential growth and opportunities for myself, where I can push the boundaries of infrastructure building for AI applications. With rapid advancements in artificial intelligence and distributed systems, I plan to leverage my expertise in cloud infrastructure to tackle emerging challenges. I aim to drive innovations that will support the scalability and reliability of AI-driven solutions, creating more resilient and efficient systems. By working with cutting-edge technologies and collaborating with forward-thinking teams, I expect to make a significant impact and continue growing both professionally and personally. In addition, I want to see my reach grow wider, hopefully unlocking more opportunities to build and collaborate with the community, fostering stronger connections and contributing to projects that will shape the future of technology. --- Looking back, 2024 was a year of building—in every sense of the word. From contributing to open-source projects and growing the **Serverless Newsletter** to achieving professional milestones and deepening personal insights, it’s been a year of meaningful progress. As I look ahead to 2025, I’m excited to continue exploring interesting problem domains, building community, and creating value in the ever-evolving world of technology. Thank you for being part of this journey. Here’s to another year of learning and growth! --- # Using Terraform Null Resource vs External Data vs Custom Provider: A Deep Dive URL: https://www.serverless.fyi/p/using-terraform-null-resource-vs-external-data-vs-custom-provider-a-deep-dive/ Published: 2024-12-25 Tags: iac, terraform During a recent interview, I was asked about the practice of invoking scripts to handle some custom resource management task, lets say creating s3 buckets or ensuring the bucket policies are intact for a set of s3 buckets. When using Terraform, practitioners often encounter scenarios requiring custom logic that isn’t natively supported by Terraform’s resource providers. One such workaround involves leveraging the **null resource** to invoke a Python script. While this method is simple and quick to implement, alternative approaches such as using the **data external resource** or creating a custom **Terraform provider** offer their own unique advantages. This blog explores these three approaches and compares them based on use cases, pros, and cons. Thus integrating Python scripts with Terraform, there are three main approaches: null_resource with local-exec provisioner, data "external" blocks, and custom providers. Let's analyze each approach and their optimal use cases. ## Null Resource with Python Script [Null resources](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) in Terraform act as placeholder resources that don't create any real infrastructure but allow you to trigger provisioners and execute arbitrary commands, making them ideal for running Python scripts during Terraform operations. ```hcl resource "null_resource" "python_executor" { provisioner "local-exec" { command = "python script.py ${var.input_param}" } } ``` The [null_resource](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) approach executes Python scripts as local commands during Terraform's apply phase. This is straightforward but has limitations in state management and data return handling. ### Pros: 1. Quick to implement with minimal setup. 2. No need for additional tools or plugins. 3. Ideal for small, one-off customizations. #### Cons: 1. Tightly coupled to the execution environment, requiring dependencies (e.g., Python) to be installed. 2. Difficult to scale and maintain as complexity grows. 3. Error handling and state management are rudimentary. --- ## External Data Source The [external data source](https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/external) in Terraform provides a standardized interface for executing external programs and incorporating their output into Terraform's state management system, requiring the external program to accept JSON input and return JSON output in a specific format. ```hcl data "external" "python_data" { program = ["python", "script.py"] query = { param = var.input_param } } ``` The [external data source](https://registry.terraform.io/providers/hashicorp/external/latest/docs/data-sources/external) provides a structured way to execute external programs and capture their output in Terraform's state. ### Pros: 1. Provides a cleaner interface for input and output handling. 2. Decouples execution logic from Terraform resources. 3. Output can be used dynamically in other Terraform resources. #### Cons: 1. Slightly more setup compared to `null_resource`. 2. Limited to simple input-output tasks. 3. Performance can be a concern for long-running scripts. --- ## Custom Provider [Custom providers](https://www.hashicorp.com/blog/writing-custom-terraform-providers) in Terraform allow you to encapsulate complex business logic into a dedicated, maintainable Go package that integrates natively with Terraform's plugin system, offering full state management, validation, and resource lifecycle control through a well-defined API. ```hcl provider "custom" { // Configuration options } resource "custom_resource" "example" { input_param = var.input_param } ``` Creating a [custom provider](https://www.hashicorp.com/blog/writing-custom-terraform-providers) requires more initial setup but offers the best integration with Terraform's ecosystem. ### Pros: 1. Highly scalable and maintainable. 2. Offers first-class integration with Terraform workflows. 3. Well-suited for complex, reusable logic. #### Cons: 1. Requires significant time and expertise to develop. 2. Written in Go, not Python. 3. Overhead might be unnecessary for simple use cases. ## Comparision | Aspect | Null Resource | External Data | Custom Provider | | -------------------- | -------------------------------------- | ----------------------------------------- | -------------------------------------- | | Setup Complexity | Low - Simple script execution | Medium - Requires JSON I/O handling | High - Requires Go development | | State Management | Poor - No built-in state tracking | Good - State managed via data source | Excellent - Full state management | | Error Handling | Basic - Exit codes only | Moderate - JSON response validation | Robust - Native error handling | | Reusability | Limited - Script-dependent | Moderate - Reusable across configurations | Excellent - Distributable as package | | Development Time | Quick - Immediate implementation | Medium - JSON interface needed | Long - Provider development required | | Maintenance | Challenging - Manual script updates | Moderate - Version control needed | Simple - Managed via registry | | Input/Output Control | Limited - Environment variables/CLI | Good - Structured JSON | Excellent - Schema-validated | | Testing | Difficult - Manual testing required | Moderate - JSON testing possible | Excellent - Built-in testing framework | | Documentation | Manual - Separate documentation needed | Basic - Data source docs | Excellent - Registry documentation | | Version Control | Manual - Script versioning needed | Good - Script + configuration | Excellent - Provider versioning | Choose based on your requirements: - Use null_resource for simple, one-off script executions - Personally, I would try to avoid the usage of scripts since it introduces security and governance risk with systems where they get invoked. - Use external data for structured data exchange without complex logic - Personally, I would use these for limited use cases. - Develop a custom provider for production-grade, reusable infrastructure components - This is the best approach but requires heavy lifting. The choice ultimately depends on factors like development time, maintenance requirements, and reusability needs. --- # Ensuring Distributed System Reliability with Jepsen URL: https://www.serverless.fyi/p/ensuring-distributed-system-reliability-with-jepsen/ Published: 2024-12-25 Tags: distributed-systems Jepsen is a framework dedicated to enhancing the safety and reliability of distributed systems, such as databases and consensus algorithms. By rigorously testing these systems under various failure scenarios, Jepsen identifies inconsistencies and potential issues, ensuring that distributed systems perform as intended. **Why Use Jepsen** In distributed systems, ensuring consistency and fault tolerance is paramount. Jepsen provides a structured approach to testing these systems by simulating: - Network partitions - Node failures - Other adverse conditions This process helps uncover hidden bugs and inconsistencies that might not surface under normal operating conditions. Jepsen's key benefits include: - An open-source library offering tools for safety testing. - Verification of adherence to specified consistency models. - In-depth analyses to evaluate system claims, uncover new bugs, and provide operator recommendations. Additionally, Jepsen’s publicly available analyses offer valuable insights into the behavior of various distributed systems. This transparency helps developers and organizations make informed decisions about technologies that align with their consistency and availability requirements. **Use Cases** Jepsen has been instrumental in evaluating a wide range of distributed systems, including: - **Bufstream**: In October 2024, Jepsen collaborated with Buf to analyze this Kafka-compatible streaming system. The analysis uncovered safety and liveness issues, such as the loss of acknowledged writes in healthy clusters. These findings led to enhancements in subsequent Bufstream releases. - **MariaDB**: Jepsen’s scrutiny of the `REPEATABLE READ` isolation level revealed it did not provide true repeatable reads. In response, the MariaDB team introduced the `--innodb-snapshot-isolation=true` flag, which offers Snapshot Isolation at the `REPEATABLE READ` level. - **FoundationDB**: Known for its strong ACID guarantees, FoundationDB underwent rigorous Jepsen testing to verify its fault tolerance and transactional consistency under various failure conditions. - **Riak**: This eventually consistent key-value store benefited from Jepsen’s analysis to identify edge cases that could compromise consistency, helping the team refine its operational model. By identifying critical issues and collaborating with system developers, Jepsen contributes to the enhancement of distributed systems, ensuring they can handle real-world challenges effectively. **Projects Using Jepsen’s Approach** Several notable projects have adopted Jepsen’s approach to ensure their systems’ robustness, including: - **CockroachDB**: Extensively uses Jepsen to validate claims of serializable isolation in distributed transactions. - **Apache Cassandra**: Jepsen’s tests highlighted areas for improvement in consistency guarantees, leading to further system refinements. - **FoundationDB**: Validated its strict ACID properties and fault tolerance through Jepsen simulations. - **Riak**: Enhanced its consistency and operational reliability by addressing issues identified through Jepsen tests. These projects exemplify the value of Jepsen’s methodology in building reliable distributed systems. **How to Use Jepsen** Jepsen is implemented as a Clojure library, and tests are written as Clojure programs. To use Jepsen: 1. Set up a test defining the distributed system under examination, specifying: - Operations to execute. - Failure scenarios to simulate. 2. Jepsen orchestrates these operations across the system’s nodes, inducing faults and recording the operation history. 3. Analyze the recorded history to verify consistency guarantees under induced failures, identifying deviations from expected behavior. For detailed guidance, refer to the official documentation and resources on Jepsen’s website. [https://jepsen.io/](https://jepsen.io/) Watch the video below for a quick intro: [Youtube Link to Jespen](https://www.youtube.com/watch?v=ecZp6cWhDjg) **Conclusion** Jepsen plays a crucial role in the development and maintenance of robust distributed systems by providing a framework for rigorous testing under adverse conditions. Its recent analyses of systems like Bufstream, MariaDB, FoundationDB, and Riak highlight its impact on improving system reliability and consistency. By utilizing Jepsen, developers and organizations can ensure their distributed systems are resilient, consistent, and capable of meeting real-world demands. --- # Vector Clocks in Distributed Systems URL: https://www.serverless.fyi/p/vector-clocks-in-distributed-systems/ Published: 2024-12-15 Tags: distributed-systems **Vector clocks** are a mechanism used in distributed systems to track causality between events in a system of distributed processes. They help determine the ordering of events that occur in different nodes or machines and detect whether events are independent, concurrent, or causally related. Vector clocks are particularly useful for resolving conflicts in distributed systems, such as in databases or message queues, where multiple processes might be updating the same data independently. ## Key Concepts: 1. **Event Causality**: - **Happens-before relationship**: In distributed systems, some events may be causally related, meaning one event (e.g., writing data to a database) happens before another (e.g., reading the data). This is known as the "happens-before" relationship. Vector clocks help capture this relationship. - **Concurrent events**: Events that happen independently of each other are considered concurrent. Vector clocks allow us to determine when two events are concurrent or when one event causally precedes another. 2. **Vector Clock Structure**: - A vector clock is essentially a list of integers, one for each process or node in the system. Each process maintains its own vector clock, which tracks the events that it has seen and the events that have occurred in the system. - The vector clock for a process is updated when it performs a local event (e.g., sending or receiving a message). It is also updated when it receives a message from another process. 3. **Operation of Vector Clocks**: - **Initialization**: Each process starts with its vector clock initialized to all zeros (or another default value). - **Event Generation**: When a process generates an event, it increments its own entry in the vector clock. - **Sending Messages**: When a process sends a message, it attaches its current vector clock to the message. - **Receiving Messages**: When a process receives a message, it updates its vector clock by taking the element-wise maximum of its own vector clock and the vector clock of the received message. This ensures that the receiver’s vector clock reflects the latest causal information from both processes. ### Comparing Vector Clocks: 1. **Vector Clock Comparison**: - **A < B**: Vector clock A causally precedes vector clock B if each entry in A is less than or equal to the corresponding entry in B, and at least one entry is strictly less. - **A = B**: Two vector clocks are equal if they are identical, meaning the events are either identical or perfectly synchronized. - **A || B**: Vector clocks A and B are concurrent (independent) if neither A < B nor B < A holds true, meaning the events are independent and do not have any causal relationship. ### Example: Consider a distributed system with three processes: P1, P2, and P3. Each process has its own vector clock to track events. - Initially, all vector clocks are set to `[0, 0, 0]` (assuming three processes). - **P1 generates an event**: The vector clock for P1 becomes `[1, 0, 0]`. - **P2 generates an event**: The vector clock for P2 becomes `[0, 1, 0]`. - **P1 sends a message to P2**: P1’s vector clock `[1, 0, 0]` is sent along with the message. - **P2 receives the message from P1**: P2 updates its vector clock by taking the element-wise maximum of its own vector clock `[0, 1, 0]` and P1’s vector clock `[1, 0, 0]`. The updated vector clock for P2 becomes `[1, 1, 0]`. - **P3 generates an event**: The vector clock for P3 becomes `[0, 0, 1]`. ### Visual Example: | Event | P1 | P2 | P3 | | -------------- | ---------- | ---------- | ---------- | | Initial | \[0, 0, 0] | \[0, 0, 0] | \[0, 0, 0] | | P1 generates | \[1, 0, 0] | \[0, 0, 0] | \[0, 0, 0] | | P2 generates | \[1, 0, 0] | \[0, 1, 0] | \[0, 0, 0] | | P1 sends to P2 | \[1, 0, 0] | \[0, 1, 0] | \[0, 0, 0] | | P2 receives P1 | \[1, 0, 0] | \[1, 1, 0] | \[0, 0, 0] | | P3 generates | \[1, 0, 0] | \[1, 1, 0] | \[0, 0, 1] | ### Applications of Vector Clocks: 1. **Conflict Detection and Resolution**: In distributed databases or file systems, vector clocks are used to detect conflicts between concurrent updates. When an update occurs to the same data from different sources, vector clocks help determine if the updates are independent or if a conflict resolution is needed. 2. **Versioning**: Vector clocks are useful for maintaining versions of data and determining the relationship between different versions in distributed systems (e.g., CRDTs—Conflict-Free Replicated Data Types). 3. **Event Ordering in Distributed Systems**: Vector clocks can track the order of events across multiple systems, helping to maintain consistency and coherence in the system state. ### Advantages of Vector Clocks: - **Causal Ordering**: They help track the causal relationship between events across distributed systems, which is essential for understanding and maintaining consistency. - **Fault Tolerance**: Vector clocks can help detect and manage discrepancies in the system even when parts of the system experience failures or delays. ### Limitations: - **Space Complexity**: Each process needs to maintain a vector for every other process in the system. This can be problematic in large-scale systems with many processes, as the size of the vector grows linearly with the number of processes. - **Communication Overhead**: The vector clock must be sent with each message, which can increase the communication overhead in high-volume systems. In summary, vector clocks are an essential tool in distributed systems for tracking causality between events and resolving conflicts, allowing systems to maintain consistency even in the presence of concurrent operations. --- # Understanding RTMP, HLS, and RTSP: A Guide to Streaming Protocols URL: https://www.serverless.fyi/p/understanding-rtmp-hls-and-rtsp-a-guide-to-streaming-protocols/ Published: 2024-12-11 Tags: video-streaming **Understanding RTMP, HLS, and RTSP: A Guide to Streaming Protocols** When it comes to video streaming, choosing the right protocol can make or break your user experience. Among the most popular protocols are **Real-Time Messaging Protocol (RTMP)**, **HTTP Live Streaming (HLS)**, and **Real-Time Streaming Protocol (RTSP)**. Each has unique strengths and is designed for specific use cases, from live streaming to surveillance and on-demand video. Here's a detailed look at these protocols and when to use them. --- ## 📡 What is RTMP? **Real-Time Messaging Protocol (RTMP)** was developed by Adobe for streaming video, audio, and data in real time. RTMP is highly efficient in transmitting data between servers and clients, making it a preferred protocol for live streaming platforms like YouTube, Twitch, and Facebook Live. **Key Features:** - **Low Latency**: Ideal for real-time interactions in live streams. - **High Performance**: Smooth delivery of video and audio streams to a server or content delivery network (CDN). - **Server-to-Server Streaming**: Commonly used to ingest live streams into platforms before transcoding. However, RTMP relies on Adobe Flash Player, which is now deprecated. While it remains widely used for stream ingestion, it has been largely replaced by other protocols for playback, such as HLS. --- ### 🎥 What is HLS? **HTTP Live Streaming (HLS)**, developed by Apple, is an adaptive bitrate streaming protocol that divides video into small chunks and delivers them over standard HTTP connections. It has become the de facto standard for streaming video on-demand and live events. **Key Features:** - **Device Compatibility**: Supported across virtually all devices, including iOS, Android, and modern web browsers. - **Adaptive Bitrate Streaming**: Automatically adjusts video quality based on a user’s internet bandwidth, ensuring smooth playback. - **Scalability**: Robust enough to handle large-scale audiences, making it a favorite for platforms like Netflix and Hulu. The downside of HLS is its higher latency compared to RTMP or RTSP, making it less suitable for scenarios where real-time interaction is critical. --- ### 📡 What is RTSP? **Real-Time Streaming Protocol (RTSP)** is designed for establishing and controlling media sessions. While not as widely known as RTMP or HLS, RTSP excels in scenarios requiring low-latency streaming, such as IP cameras and real-time applications. **Key Features:** - **Low Latency**: Perfect for real-time video delivery in closed environments. - **Playback Control**: Supports operations like play, pause, and seek, making it versatile for live and on-demand scenarios. - **Closed Network Performance**: Commonly used for LAN environments or with RTSP-compatible players. RTSP is not widely supported by web browsers, requiring specialized players or plugins for playback. --- ### 🌐 Choosing the Right Protocol | **Feature** | **RTMP** | **HLS** | **RTSP** | | ---------------------------- | ------------------------- | ----------------------- | --------------------------- | | **Latency** | Low | High | Low | | **Compatibility** | Limited (Flash-dependent) | High (device-friendly) | Moderate (specialized) | | **Use Case** | Live Streaming | On-demand/Scalable Live | Surveillance/Real-Time Apps | | **Adaptive Bitrate Support** | No | Yes | No | --- ### 🛠️ Conclusion - **Choose RTMP** for ingesting live streams into platforms or CDNs. - **Choose HLS** for delivering high-quality video to large audiences on mobile or web. - **Choose RTSP** for low-latency needs like surveillance or closed-network streaming. Each protocol serves a unique purpose in the streaming ecosystem. Understanding their differences will help you choose the one best suited to your specific needs! 🚀 --- # Distributed File System URL: https://www.serverless.fyi/p/distributed-file-system/ Published: 2024-12-09 Tags: distributed-filesystem A **distributed file system (DFS)** is a system that allows files to be stored and accessed across multiple machines or nodes in a network, enabling the efficient and fault-tolerant storage, retrieval, and management of large datasets. Unlike traditional file systems that operate on a single server, DFS spreads data across several machines (often in a cluster), providing scalability, redundancy, and high availability. DFS allows multiple users or applications to access the same files concurrently from different locations, typically over a network. In a DFS, data is often divided into smaller chunks or blocks and distributed across various nodes, with some systems offering replication for fault tolerance and performance optimization. This architecture improves the scalability of the file system and reduces the risk of data loss due to hardware failures. ## Key Features of a Distributed File System: 1. **Scalability**: - DFS can scale horizontally by adding more nodes to the system, handling increasing amounts of data and requests without significant performance degradation. 2. **Fault Tolerance and Data Redundancy**: - Data in a DFS is typically replicated across multiple nodes to ensure availability in case of failure. This replication can be configured to specify how many copies of each file/block are stored in the system, providing redundancy. 3. **High Availability**: - By distributing data across multiple nodes and often replicating it, DFS ensures that data remains accessible even if part of the system fails. Some DFS solutions can automatically detect failures and reroute requests to healthy nodes. 4. **Distributed Data Access**: - A DFS provides transparent access to files stored on different machines, allowing clients to interact with the system as though they are accessing a local file system. The underlying infrastructure handles the distribution and retrieval of data. 5. **Parallelism**: - DFS enables parallel data processing, allowing multiple clients or applications to read from and write to the file system simultaneously, which is especially useful in big data applications where large datasets need to be processed quickly. 6. **Consistency and Synchronization**: - DFS often implements consistency models to ensure that data is synchronized across the distributed nodes, guaranteeing that updates to files are propagated correctly. ### Examples of Distributed File Systems: 1. **Hadoop Distributed File System (HDFS)**: - **HDFS** is one of the most popular distributed file systems, primarily designed for storing and processing large datasets in the Hadoop ecosystem. It splits files into blocks and stores them across different nodes, providing high throughput for large data access, fault tolerance, and replication of data blocks. **Key Features**: - Block-based storage - Default replication factor of 3 for fault tolerance - Optimized for batch processing with high throughput 2. **Google File System (GFS)**: - **GFS** is a distributed file system developed by Google to support its large-scale data processing requirements. It provides high availability and is designed for managing large files across thousands of machines. **Key Features**: - Data is broken into chunks, which are distributed across the cluster - Handles large-scale data storage and access efficiently - Provides redundancy and fault tolerance through replication 3. **Ceph**: - **Ceph** is an open-source distributed storage system that provides file, block, and object storage in a unified manner. It offers high scalability and fault tolerance by replicating and distributing data across nodes. **Key Features**: - Self-healing and self-managing - Supports file, block, and object storage - Uses a distributed object store to manage data 4. **Amazon S3 (Simple Storage Service)**: - While **S3** is often considered an object store, it can function as a distributed file system. It is designed to scale automatically and provides durability by replicating data across multiple geographic locations. **Key Features**: - Object-based storage accessible via APIs - Scalability and redundancy are built-in - Provides seamless data access across large networks 5. **GlusterFS**: - **GlusterFS** is an open-source, scalable, distributed file system that aggregates storage resources from multiple servers into a single global namespace. **Key Features**: - Supports multiple replication and data redundancy options - Uses a distributed hash table (DHT) for storing files - Highly available and fault-tolerant 6. **SMB (Server Message Block)**: - **SMB** is a network file-sharing protocol commonly used for file and printer sharing in a local area network (LAN). It allows applications and users to read and write to files and request services from server programs on a network. SMB is used by various operating systems like Windows, macOS, and Linux. **Key Features**: - Supports file and printer sharing over a network - Used for accessing remote files and resources - Provides authentication, encryption, and access control mechanisms 7. **NFS (Network File System)**: - **NFS** is a distributed file system protocol that allows a computer to access files over a network as if they were locally stored. NFS is commonly used in Unix/Linux environments and supports file sharing between different machines in a distributed system. **Key Features**: - Provides access to remote files over a network - Widely used in Unix/Linux environments - Allows sharing of files between different operating systems ### How a Distributed File System Works: 1. **Data Distribution**: A DFS divides large files into smaller chunks or blocks, which are then distributed across multiple machines in the network. Each block is stored in one or more locations (replicas) to ensure data redundancy and fault tolerance. 2. **Metadata Management**: A DFS typically uses a **metadata server** or a **name node** to track the locations of data blocks. This server maintains the directory structure and provides clients with the necessary information to locate and access data blocks. 3. **Client Interaction**: When a client requests data, the DFS redirects it to the appropriate node containing the block of data. The client may interact with multiple nodes simultaneously, accessing different parts of a file in parallel. 4. **Replication and Fault Tolerance**: To prevent data loss due to node failures, DFS replicates data across multiple machines. If a failure occurs, the system can recover by reading the data from the replica or other copies of the data. ### Benefits of a Distributed File System: 1. **Scalability**: Can handle petabytes of data by adding more nodes to the system. 2. **Fault Tolerance**: By replicating data across nodes, DFS ensures high availability and prevents data loss. 3. **Parallel Processing**: Facilitates efficient large-scale data processing, especially in big data applications. 4. **Cost-Effective**: Can use commodity hardware to build scalable storage systems. ### Challenges of Distributed File Systems: 1. **Consistency**: Ensuring data consistency across all nodes in a distributed system can be complex, especially for concurrent reads and writes. 2. **Network Latency**: As the system scales, network latency can become a bottleneck. 3. **Complex Management**: Managing large-scale distributed storage systems can be challenging in terms of monitoring, maintenance, and troubleshooting. 4. **Data Integrity**: Ensuring data integrity and handling failures during data writes or updates require sophisticated mechanisms. ### Conclusion: A **distributed file system (DFS)** is essential for managing large volumes of data across multiple machines in a network. By partitioning data, replicating it for fault tolerance, and enabling parallel processing, DFS ensures scalable, high-performance, and fault-tolerant data storage. Systems like HDFS, GFS, Ceph, and cloud-based solutions like Amazon S3 are popular DFS implementations. Additionally, **SMB** and **NFS** are widely used network protocols for file sharing in distributed environments, providing flexible, scalable storage solutions across different operating systems and applications. --- # Aleo and Zero-Knowledge Proofs: The Future of Privacy-Enhanced Blockchain Technology URL: https://www.serverless.fyi/p/aleo-and-zero-knowledge-proofs-the-future-of-privacy-enhanced-blockchain-technology/ Published: 2024-12-09 Tags: crypto In recent years, blockchain technology has revolutionized industries by offering transparent, secure, and decentralized solutions. However, one of the major challenges blockchain faces is privacy. Public blockchains, like Bitcoin and Ethereum, while transparent, also expose transaction details to anyone with access to the network. This is where **Aleo**, a privacy-focused blockchain built on **Zero-Knowledge Proofs (ZKPs)**, comes into play, offering a promising solution to address privacy concerns while maintaining the security and decentralization of traditional blockchains. ## What is Aleo? Aleo is a blockchain designed to provide privacy-preserving smart contracts and decentralized applications (dApps). Unlike traditional blockchains, where transaction details are visible to everyone, Aleo leverages advanced cryptographic techniques, specifically **Zero-Knowledge Proofs**, to keep user data and transaction details private. This approach allows users to interact with the blockchain without exposing their personal information or transaction histories, ensuring that sensitive data remains confidential. Aleo's architecture is built around the concept of **ZK-SNARKs** (Zero-Knowledge Succinct Non-Interactive Arguments of Knowledge), which are a type of Zero-Knowledge Proof. ZKPs are cryptographic proofs that allow one party to prove to another party that they know a value (such as a secret key or the result of a computation) without revealing the value itself. In Aleo’s case, ZKPs allow users to prove the validity of a transaction or computation without revealing the underlying data. ### What are Zero-Knowledge Proofs? Zero-Knowledge Proofs (ZKPs) are a powerful cryptographic tool that enables one party (the prover) to convince another party (the verifier) that they know a certain piece of information, without revealing the actual information. This is particularly useful for maintaining privacy in decentralized systems. ZKPs work in a way that allows one party to prove that they have knowledge of a secret (such as a password or a private key) without revealing the secret itself. This concept can be extended to prove the correctness of computations on a blockchain, such as the validity of a transaction, without revealing any sensitive details about that transaction. There are different types of ZKPs, but the two most common are **ZK-SNARKs** and **ZK-STARKs**. ZK-SNARKs, which Aleo uses, are succinct and non-interactive, meaning they are efficient and require minimal communication between the prover and verifier, making them ideal for blockchain applications. ### How Aleo Uses ZKPs to Enhance Privacy Aleo utilizes Zero-Knowledge Proofs to enable privacy-preserving smart contracts and transactions. On traditional blockchains, the data associated with each transaction, such as the sender, receiver, and amount, is public. However, Aleo allows users to execute smart contracts and transactions where only the validity of the operation is revealed, while the underlying data remains hidden. For example, in a financial transaction on Aleo, a user can prove that they have sufficient funds to make a payment without revealing the amount of the payment or their account balance. This ensures that the transaction is secure and valid but doesn't expose any sensitive information. ### The Benefits of Aleo and ZKPs 1. **Enhanced Privacy**: The core benefit of Aleo’s use of ZKPs is the ability to ensure privacy in blockchain transactions. Users can perform transactions without revealing their data, protecting their financial and personal information. 2. **Scalability**: Zero-Knowledge Proofs can help improve the scalability of blockchain networks. By only revealing the validity of a transaction rather than all the data, ZKPs reduce the amount of information that needs to be stored and transmitted, enhancing blockchain efficiency. 3. **Secure Smart Contracts**: Aleo’s platform enables developers to create private smart contracts that can interact with private data. This opens up new possibilities for decentralized finance (DeFi) and other industries where privacy is paramount, such as healthcare, finance, and supply chain management. 4. **Regulatory Compliance**: Privacy is not just about keeping data safe; it's also about complying with data protection regulations like GDPR. With Aleo, users can maintain their privacy while ensuring that their transactions are verifiable, which can be beneficial for meeting regulatory requirements. ### The Future of Aleo and ZKPs Aleo’s privacy-first approach, powered by Zero-Knowledge Proofs, is paving the way for a new era of blockchain applications that prioritize confidentiality without compromising security. As the demand for privacy-enhanced solutions grows, Aleo’s innovative use of ZKPs will likely play a significant role in the evolution of blockchain technology, driving adoption across industries that require privacy-sensitive solutions. In conclusion, Aleo and Zero-Knowledge Proofs represent the future of blockchain privacy, enabling decentralized systems to provide transparency while maintaining the confidentiality of sensitive data. As the technology matures, we can expect to see more applications leveraging ZKPs to create secure, private, and scalable decentralized systems. --- # Buildpacks vs. Docker OCI: Simplifying "Hello, World!" in Containers URL: https://www.serverless.fyi/p/buildpacks-vs-docker-oci-simplifying-hello-world-in-containers/ Published: 2024-12-04 Tags: containers Containerization is essential in modern software development for packaging and running applications consistently across environments. Two popular approaches are **Buildpacks** and **Docker OCI**, each offering distinct advantages depending on your needs. Let’s compare them and demonstrate how to create a containerized "Hello, World!" application using both methods. --- ## What Are Buildpacks? Buildpacks automate the process of turning application code into runnable container images without requiring a `Dockerfile`. They are particularly well-suited for developers who prefer a streamlined workflow and less focus on container-specific configurations. **Key Features of Buildpacks:** - **Ease of Use**: Automatically detects the runtime and installs dependencies. - **Standardization**: Produces OCI-compliant images with consistent configurations. - **Automation**: Reduces the need for manual updates and maintenance of dependencies. ### Example: "Hello, World!" with Buildpacks Here’s how you can use Buildpacks to containerize a Python application: 1. **Write the Application**:\ Create a `hello.py` file: ```python print("Hello, World!") ``` 2. **Optional: Create a **`Procfile`:\ For better configuration, add: ```text web: python hello.py ``` 3. **Build the Image**:\ Install the `pack` CLI tool and run: ```bash pack build hello-world-app --builder heroku/buildpacks:20 ``` 4. **Run the Container**: ```bash docker run --rm hello-world-app ``` **Output**: ```text Hello, World! ``` --- ### What Is Docker OCI? Docker follows the Open Container Initiative (OCI) standard, giving developers full control over how container images are built using `Dockerfiles`. This approach is ideal for complex or highly customized use cases. **Key Features of Docker OCI:** - **Customizability**: Full control over image specifications, dependencies, and configurations. - **Flexibility**: Supports any runtime or application type. - **Transparency**: Developers explicitly define every step in the `Dockerfile`. #### Example: "Hello, World!" with Docker OCI Here’s how you can achieve the same result using Docker: 1. **Write the Application**:\ Create a `hello.py` file: ```python print("Hello, World!") ``` 2. **Create a Dockerfile**:\ Define how to build the image: ```dockerfile FROM python:3.9-slim COPY hello.py /app/hello.py WORKDIR /app CMD ["python", "hello.py"] ``` 3. **Build the Image**: ```bash docker build -t hello-world-app . ``` 4. **Run the Container**: ```bash docker run --rm hello-world-app ``` **Output**: ```text Hello, World! ``` --- ### Key Differences | **Aspect** | **Buildpacks** | **Docker OCI** | | ----------------- | ------------------------------------------------ | --------------------------------- | | **Ease of Use** | Simplifies the process, no `Dockerfile` needed. | Requires writing a `Dockerfile`. | | **Automation** | Automatically detects runtimes and dependencies. | Manual configuration is required. | | **Flexibility** | Less customizable. | Fully customizable. | | **Best Use Case** | Standard web or microservice apps. | Complex or non-standard setups. | --- ### Choosing the Right Approach - **Buildpacks**: Ideal for developers seeking simplicity and automation. Great for platforms like Heroku or Google Cloud Run. - **Docker OCI**: Best for teams needing precise control over their container builds or working with unique environments. By exploring both Buildpacks and Docker OCI, you can choose the method that aligns with your project needs and team expertise. Whether you’re optimizing for simplicity or control, the containerization journey starts with tools tailored to your requirements. ### Dockerfile and OCI The key difference between a Dockerfile and OCI (Open Container Initiative) lies in what they represent and how they are used in containerization: ### Dockerfile - **Definition**: A Dockerfile is a script or set of instructions used to build Docker images. It tells Docker how to construct an image layer by layer. - **Purpose**: It defines the steps to create a container image, including: - The base image (e.g., `FROM ubuntu:20.04`). - Commands to install software or copy files (`RUN`, `COPY`). - Configuration of environment variables (`ENV`). - Execution commands (`CMD`, `ENTRYPOINT`). - **Scope**: Specifically designed for use with Docker, although the resulting images can often be compatible with OCI-compliant runtimes. ### OCI (Open Container Initiative) - **Definition**: The Open Container Initiative is a standards organization that defines specifications for container runtimes and container images to ensure interoperability. - **Purpose**: OCI creates standards, such as: - **OCI Image Specification**: Defines the format and metadata of container images. - **OCI Runtime Specification**: Specifies how to execute containers using a runtime (e.g., runc). - **Scope**: Aims to make container images and runtimes compatible across different tools and platforms, promoting open standards and avoiding vendor lock-in. Docker images are OCI-compliant, meaning they adhere to these specifications. ### Key Differences | Aspect | Dockerfile | OCI | | ----------- | ---------------------------------------- | ------------------------------------------- | | **Purpose** | Script for building container images | Standards for container images and runtimes | | **Scope** | Specific to Docker’s ecosystem | Ecosystem-agnostic (broader compatibility) | | **Output** | Docker image | OCI-compliant image/runtimes | | **Role** | Developer-facing, used in image creation | Industry standard for interoperability | ### Relationship Docker images conform to OCI standards, meaning they can be used with any OCI-compliant runtime. However, a Dockerfile itself is not an OCI standard; it is a tool Docker provides to build OCI-compliant images. --- # Understanding Tree Search Time Complexity: A Guide for Developers URL: https://www.serverless.fyi/p/understanding-tree-search-time-complexity-a-guide-for-developers/ Published: 2024-12-02 Tags: algorithms Trees are among the most fundamental data structures in computer science, offering versatile solutions for organizing and managing data. However, the efficiency of searching within a tree heavily depends on the tree's type and structure. In this blog, we'll break down the time complexity of searching in different tree types and explain what affects their performance. ## 1. Binary Search Trees (BSTs) Binary Search Trees allow efficient searching when properly structured. - **Best Case (Balanced Tree)**: O(log⁡n)\ When a BST is balanced, its height remains approximately log⁡2n, allowing for efficient traversal to locate a node. - **Worst Case (Unbalanced Tree)**: O(n)\ In unbalanced cases, the tree may resemble a linked list, where every node has only one child, leading to a linear search time. ### 2. Balanced Binary Search Trees (AVL Trees, Red-Black Trees) Balanced trees like AVL or Red-Black trees are designed to maintain a near-perfect balance, ensuring logarithmic height regardless of the input sequence. - **Time Complexity**: O(log⁡n)\ The self-balancing mechanism guarantees efficient searches by keeping the tree height minimal. #### 3. Binary Heaps (Min-Heap/Max-Heap) Binary heaps are optimized for specific operations, such as extracting the minimum or maximum value, but searching is not one of their strengths. - **Time Complexity**: O(n)\ Since heaps lack a specific order between nodes, a search may involve traversing all elements. #### 4. B-Trees and B+ Trees These multi-way trees, often used in database systems, are ideal for handling large datasets. - **Time Complexity**: O(log⁡n)\ By maintaining balance and allowing multiple children per node, B-Trees reduce height and ensure efficient search operations. #### 5. Tries (Prefix Trees) Tries are specialized trees for storing strings, commonly used in autocomplete systems and dictionaries. - **Time Complexity**: O(L)\ The search time depends on the length of the key L, as traversal occurs character by character through the tree. #### 6. General Trees (Non-binary, No Constraints) For general trees without specific ordering, search performance can be less predictable. - **Time Complexity**: O(n)\ In the worst case, you may need to visit every node to find the desired value. --- ### Factors Affecting Search Complexity - **Balance**: Balanced trees like AVL or Red-Black Trees consistently offer O(log⁡n) complexity. In contrast, unbalanced trees may degrade to linear complexity. - **Structure**: Specialized trees like Tries or B-Trees optimize searching for specific use cases, such as string matching or database indexing. - **Branching Factor**: Multi-way trees (e.g., B-Trees) reduce tree height by increasing the number of children per node, enhancing efficiency. --- ### Summary Table | **Tree Type** | **Best Case** | **Worst Case** | | ------------------ | ------------- | -------------- | | Binary Search Tree | O(log⁡n) | O(n) | | Balanced BST | O(log⁡n) | O(log⁡n) | | Binary Heap | O(n) | O(n) | | B-Trees/B+ Trees | O(log⁡n) | O(log⁡n) | | Trie | O(L) | O(L) | | General Tree | O(n) | O(n) | --- ### Final Thoughts Tree search complexity varies widely depending on the tree type and its structure. For applications like database indexing, string matching, or priority queues, understanding these nuances is key to making informed design choices. Whether you’re optimizing for speed or storage, picking the right tree can significantly impact your system’s performance. What’s your go-to tree type for efficient searches? --- --- # Maximizing QSBS Benefits for Early Employees: A Guide to Tax-Smart Equity URL: https://www.serverless.fyi/p/maximizing-qsbs-benefits-for-early-employees-a-guide-to-tax-smart-equity/ Published: 2024-11-11 Tags: tax, qsbs ## QSBS Benefits for Early Employees Early employees can also take advantage of QSBS benefits, though their strategy differs from founders. Here's what early employees need to know: ## Strategic Playbook for Early Employees 1. Stock Acquisition - Must receive stock directly from the company (not secondary market) - Options or restricted stock count if exercised while company qualifies - Stock must be held for 5+ years after exercise 2. Timing Considerations - QSBS status starts from option exercise date, not grant date - Early exercise can start the 5-year clock sooner - Must exercise while company is under $50M in assets 3. Maximum Benefits - Up to $10M in gains can be tax-free - Unlike founders, early employees typically don't benefit from the 10x basis rule - Multiple stock grants/exercises can each qualify separately Early Employee Tax Saving Example ```text - Joins when company is worth $20M - Receives options for 1% of company - Exercises options early at $200K - Company sells for $1B - Employee's share: $10M - Potential tax savings: ~$2.4M (assuming 23.8% federal capital gains rate) ``` ### Strategic Planning for Early Employees 1. Early Exercise Considerations - Pros: - Starts 5-year holding period earlier - Potentially lower exercise price - Better tax treatment (if 83(b) election filed) - Cons: - Requires upfront capital - Risk if company fails - Must file 83(b) election within 30 days 2. Documentation Needs - Keep records of: - Original grant documents - Exercise dates and prices - 83(b) election filing (if applicable) - Proof of company's qualification at exercise time 3. Optimization Strategies - Consider exercising in tranches to spread out risk - Time exercises before major funding rounds - Coordinate with tax advisor on exercise timing ## Early Employee Strategies Early employees have two potential paths to QSBS benefits: ### Path 1: Joining During LLC Phase - Receive equity/options during LLC period - Wait for C-corp conversion - 5-year clock starts at conversion - Benefits: Up to $10M tax-free per block of shares ### Path 2: Joining C-Corp Directly - Join when company is already a C-corp - Must exercise while company is under $50M in assets - 5-year clock starts at exercise - Benefits: Up to $10M tax-free per block of shares ### Early Exercise Strategy for Maximum Tax Benefit Scenario A: Early Exercise Strategy ```text - Join C-corp valued at $30M - Receive options for 0.5% of company - Exercise immediately at $150K - Company sells in 6 years for $2B - Your share: $10M - Entire gain could be tax-free under QSBS ``` Scenario B: Gradual Exercise Strategy ```text - Join C-corp valued at $40M - Receive options for 1% of company - Exercise in tranches while under $50M - Each exercise starts its own 5-year clock - Multiple blocks of QSBS qualification - Can potentially stack multiple $10M exemptions ``` ![](https://www.serverless.fyi/_astro/img-98bf089165fa.BIe1U1f3.png) ### Key Differences from Founder Benefits 1. Basis Calculation - Founders: Can use company value at C-corp conversion - Employees: Use exercise price of options 2. Maximum Benefit - Founders: Greater of $10M or 10x basis - Employees: Generally limited to $10M per block of shares 3. Timing Strategy - Founders: Focus on LLC to C-corp conversion timing - Employees: Focus on exercise timing ![](https://www.serverless.fyi/_astro/img-a41ea4407a29.zKvvF_iR.png) ### Action Items for Early Employees 1. Immediate Steps - Confirm company's QSBS eligibility - Review grant documents - Consider early exercise options - Consult tax advisor 2. Ongoing Monitoring - Track company's asset value - Monitor vesting schedule - Plan exercise timing - Document all transactions 3. Exit Planning - Calculate holding periods for each share block - Understand impact of different exit scenarios - Plan tax strategy for non-qualifying shares ### Best Practices for Maximum Benefit 1. Communication - Stay informed about company's asset value - Understand company's plans for future funding - Keep open dialogue with management about QSBS status 2. Financial Planning - Budget for early exercises if possible - Consider loans or financing options for exercises - Plan for alternative scenarios if QSBS doesn't qualify 3. Professional Support - Work with tax advisor familiar with QSBS - Consider legal counsel for complex situations - Keep detailed records of all transactions ### The Bottom Line for Early Employees While early employees may not access the same magnitude of QSBS benefits as founders, the potential tax savings are still substantial. The key is understanding your options early and planning strategically around exercise timing and holding periods. The most successful early employees treat their equity compensation as a crucial part of their overall financial planning, making informed decisions about when and how to exercise their options to maximize QSBS benefits. Disclaimer: Not tax advice - consult your advisors, but definitely worth exploring! --- # The QSBS Tax Exclusion: How Founders & Early Employees Can Avoid Taxes on Massive Paydays URL: https://www.serverless.fyi/p/the-qsbs-tax-exclusion-how-founders-early-employees-can-avoid-taxes-on-massive-paydays/ Published: 2024-11-10 Tags: tax, qsbs ## Ultimate Guide to Tax-Free Exits As an entrepreneur, the prospect of a lucrative exit is an exciting one. However, the tax implications of such an event can be daunting. That is, unless you know about the Qualified Small Business Stock (QSBS) tax break. This post is primarily tailored for founders, offering insights into QSBS strategies, but early employees and investors will also find valuable information to help them maximize their benefits. Be sure to check out the previous blog for an introduction to QSBS and its core concepts. **[Unlocking Tax Savings with Qualified Small Business Stock (QSBS)](https://www.serverless.fyi/p/unlocking-tax-savings-with-qualified-small-business-stock-qsbs/)****** QSBS is a provision in the tax code that allows founders to pay little to no taxes on a portion of their company's sale proceeds. The specifics are as follows: - QSBS exempts up to $10 million (or 10 times your original investment, whichever is greater) from federal capital gains taxes when you sell your company's stock. - To qualify, your company must have less than $50 million in assets when you acquire the QSBS. - The company must also be a C-corporation, not an LLC or other pass-through entity. Here's where things get interesting. Some founders have found a way to maximize the QSBS benefits by first starting their company as an LLC, and then converting to a C-corp right before their big exit. The strategy goes like this: 1. Launch your company as an LLC while it's in the early, money-losing stages. This allows you to offset your personal taxes with the LLC's losses. 2. Once the company becomes profitable and approaches the $50 million asset cap, convert it to a C-corporation. 3. When you eventually sell the company, you can apply the QSBS exemption to up to 10 times the company's value at the time of the C-corp conversion - which could be well over $400 million. This maneuver is completely legal, though it does come with some caveats: 1. The $10 million (or 10x basis) QSBS exemption only applies to the value of the company at the time of the C-corp conversion, not the full sale price. 2. The five-year holding period for QSBS only begins once the company becomes a C-corp, so you'll need to wait at least that long before selling. 3. You have to be extremely careful not to exceed the $50 million asset cap before the conversion. While this tax strategy is undoubtedly complex, it demonstrates the power of careful tax planning for founders. By understanding the nuances of the tax code, you can potentially save yourself hundreds of millions in taxes when you eventually sell your company. Of course, it's always best to consult with a qualified tax professional before attempting any major tax maneuvers. But for founders looking to maximize their exit proceeds, the QSBS loophole is certainly worth exploring. ## Understanding the QSBS Advantage QSBS is one of the most powerful tax benefits available to entrepreneurs. It allows founders to potentially pay zero federal capital gains taxes on up to $10 million or 10 times their original investment basis, whichever is greater. This means with proper planning, you could walk away from your exit with hundreds of millions in tax-free gains. ## The Strategic Founder's Playbook Here's how forward-thinking founders are maximizing this opportunity: 1. Initial LLC Structure (Years 0-2) - Start your company as an LLC - Use early-stage losses to offset your personal income taxes - Build value while maintaining pass-through tax benefits 2. Strategic Conversion Timing (Year 2-3) - Monitor your company's asset value carefully - Convert to a C-corporation before reaching $50 million in assets - Document your basis value at conversion (this becomes crucial later) 3. Growth and Holding Period (Years 3-8) - Continue growing your company - Maintain qualified business status - Hold the stock for at least 5 years post-conversion 4. Tax-Free Exit (Year 8+) - Sell your company - Pay zero federal capital gains taxes on up to 10x your basis - Example: $40M basis = up to $400M tax-free ![](https://www.serverless.fyi/_astro/img-75db06fcf3f4.Xj5RX4zf.png) ### Key Considerations for Success 1. Scale Matters - The strategy works best for companies aiming for significant exits - Your initial basis needs to be substantial to maximize the 10x benefit 2. Timing is Everything - The 5-year holding period starts only after C-corp conversion - Converting too early or too late can limit your benefits 3. Asset Cap Awareness - Staying under the $50M asset cap before conversion is crucial - Careful monitoring and timing are essential ### Expert Tips for Implementation 1. Document Everything - Keep detailed records of your company's value at conversion - Maintain proof of qualification throughout the holding period 2. Professional Guidance - Work with tax professionals who understand QSBS - Regular consultations ensure you maintain qualification 3. Strategic Planning - Build your exit timeline around the 5-year holding requirement - Consider future funding rounds when planning your conversion ### The Power of Preparation While this strategy wasn't always intentionally planned by founders, those who understand it early can maximize their benefits. The key is combining careful planning with patience – the five-year holding period might seem long, but the potential tax savings make it well worth the wait. Remember: This strategy isn't just about saving money – it's about maximizing the value you've created through years of hard work and innovation. With proper planning and execution, QSBS can help you retain more of what you've earned, providing greater resources for your next venture or personal goals. ### The Bottom Line QSBS represents one of the most significant tax advantages available to entrepreneurs. While the strategy requires careful planning and patience, the potential to save millions in taxes makes it an invaluable tool for founders planning their exit strategy. As with any tax strategy, consult with qualified tax professionals to ensure proper implementation and compliance. In my next blog, I’ll explore how early employees can leverage QSBS to maximize their equity benefits and minimize taxes, unlocking more value from their stock options. Disclaimer: Not tax advice - consult your advisors, but definitely worth exploring! --- # Unlocking Tax Savings with Qualified Small Business Stock (QSBS) URL: https://www.serverless.fyi/p/unlocking-tax-savings-with-qualified-small-business-stock-qsbs/ Published: 2024-11-09 Tags: tax, qsbs Qualified Small Business Stock (QSBS) is a tax incentive under Section 1202 of the U.S. tax code aimed at promoting investment in early-stage companies. By allowing for substantial capital gains tax exclusions, QSBS encourages investments in qualifying startups and small businesses. This benefit can significantly reduce or eliminate federal taxes on profits when the stock is sold, making QSBS a powerful tool for investors, founders, and employees alike. This blog series is tailored for founders, investors, and early employees, offering insights into the unique tax advantages of Qualified Small Business Stock (QSBS) and how they can maximize its benefits. - **Key Qualification Criteria:** - **Company Type**: Must be a C Corporation in a qualified industry (excluding fields like consulting and financial services). - **Gross Assets**: The company’s assets must be under $50 million at issuance. - **Original Issuance**: Stock must be acquired directly from the company, not via a secondary market. - **Holding Period**: Investors need to hold the stock for **at least five years** to claim the tax exclusion. - **Capital Gains Exclusion Levels:** - **100% exclusion** for QSBS purchased after September 27, 2010. - **75% exclusion** for stock bought between February 18, 2009, and September 27, 2010. - **50% exclusion** for purchases before February 18, 2009. - **Exclusion Cap**: The greater of $10 million or **10 times the original investment**. - **Benefits for Different Stakeholders:** - **For Investors**: Substantial tax savings on capital gains with up to a 100% exclusion; also, if sold within five years, gains can be deferred by reinvesting in other QSBS under Section 1045. - **For Founders**: QSBS can make the company more attractive to investors seeking tax benefits, enhancing fundraising efforts and encouraging long-term investment stability. - **For Early Employees**: Employees with QSBS stock options can also enjoy tax savings on any gain after five years, which can lead to substantial post-tax returns. - **State-Specific Considerations**: - Some states, like California and Pennsylvania, do not conform to Section 1202, so investors may still owe state taxes even if they qualify for federal exclusions. Consulting with tax professionals is crucial to understand the complete tax impact. ![](https://www.serverless.fyi/_astro/img-b655fb452a56.CrnktzU8.png) **Conclusion:**\ QSBS provides a significant tax advantage for those investing in qualifying small businesses, making it a valuable tool for investors, founders, and employees. By understanding QSBS eligibility and benefits, stakeholders can better capitalize on this tax-saving opportunity while supporting innovation and growth in the small business ecosystem. Stay tuned for future posts that will dive deeper into QSBS, providing specific insights and strategies tailored for founders, investors, and early employees to make the most of this powerful tax advantage.\ \ Disclaimer: Not tax advice - consult your advisors, but definitely worth exploring! --- # How Apache Pulsar Manages Multi-Tenancy URL: https://www.serverless.fyi/p/how-apache-pulsar-manages-multi-tenancy/ Published: 2024-11-06 Tags: apache-pulsar In today's data-driven world, organizations often need to support multiple teams, departments, or even external clients on a single infrastructure. Apache Pulsar is designed with this flexibility in mind, offering robust, built-in support for multi-tenancy. This makes Pulsar a popular choice for companies that need to segment resources securely and efficiently. Let’s explore how Pulsar manages multi-tenancy to provide a scalable and secure data streaming solution. ## 1. Tenants and Namespaces At the core of Pulsar’s multi-tenancy model are **tenants** and **namespaces**. A **tenant** is the highest level of organizational boundary within Pulsar, typically representing a department, team, or customer. Each tenant has unique configurations and can be administered independently. Within each tenant, Pulsar provides **namespaces**, which allow further segmentation. Namespaces organize topics into logical groups, making it possible to set policies at a granular level. Policies like quotas, time-to-live (TTL) for messages, and backlog retention can all be applied at the namespace level, offering flexibility and fine control over resource allocation. ### 2. Topic Segmentation Topics are the fundamental data streams in Pulsar and are created within namespaces. Each topic inherits the policies set at the tenant and namespace levels, ensuring that each tenant has isolated, well-defined topics. This segmentation allows for organized access control and enables each tenant to operate independently.\ \ To understand this in depth, refer to the post [**Understanding Apache Pulsar Topics: A Deep Dive into Naming Conventions**](https://www.serverless.fyi/p/understanding-apache-pulsar-topics-a-deep-dive-into-naming-conventions/). ### 3. Authentication and Authorization Multi-tenancy demands stringent **authentication** and **authorization** controls, especially when dealing with sensitive data from multiple sources. Pulsar supports several authentication mechanisms, such as TLS, OAuth 2.0, and JWT tokens, to verify user identities. These security measures operate at the tenant level, making it easy to separate permissions between tenants. Additionally, Pulsar offers **role-based access control (RBAC)**, allowing fine-grained permission settings on read, write, and management operations. This ensures that only authorized users have access to specific tenants, namespaces, and topics. ### 4. Resource Isolation and Quotas One of Pulsar's strongest features is its ability to apply **quotas** on resources within namespaces. Quotas limit the amount of storage, message retention, and message rates per tenant, preventing one tenant from overusing resources at the expense of others. Pulsar also enables **dispatch throttling** to manage bandwidth usage, preventing high usage by one tenant from impacting the performance of others. ### 5. Broker and Storage Isolation Pulsar allows organizations to achieve both logical and physical isolation. By running in a **geo-replicated cluster**, specific tenants can have their data spread across multiple regions, while others may remain localized. Pulsar also supports **dedicated clusters** for tenants, providing even greater isolation by assigning specific broker resources. ### 6. Monitoring and Metrics Effective multi-tenancy relies on real-time insights. Pulsar provides **monitoring and metrics** at every level — from tenants to namespaces to topics. This visibility makes it easy for admins to ensure fair resource usage and quickly resolve tenant-specific issues. ### 7. Load Balancing Pulsar’s **intelligent load balancing** helps distribute topics and partitions across brokers, ensuring that resources are shared fairly among tenants. This prevents any single tenant from consuming excessive resources and ensures reliable performance across the board. ### Conclusion Apache Pulsar’s multi-tenancy features provide a scalable, secure, and efficient solution for organizations supporting multiple teams or customers. With its robust tenant, namespace, and topic structure, coupled with advanced authentication, resource quotas, and monitoring, Pulsar empowers organizations to manage data streaming across diverse users with ease and control. Whether it’s for internal departments or external clients, Pulsar's multi-tenancy framework offers the flexibility and isolation needed to maintain efficient operations.\ \ For an in-depth comparison and insights on why Apache Pulsar may be a better fit for your organization than Kafka, download our comprehensive white paper **[Decoding Kafka Challenges: Addressing Common Pain Points in Kafka Deployments Whitepaper](https://streamnative.io/whitepapers/decoding-kafka-challenges-addressing-common-pain-points-in-kafka-deployments)**. --- # Understanding Apache Pulsar Topics: A Deep Dive into Naming Conventions URL: https://www.serverless.fyi/p/understanding-apache-pulsar-topics-a-deep-dive-into-naming-conventions/ Published: 2024-11-04 Tags: apache-pulsar In the world of distributed messaging systems, Apache Pulsar stands out for its sophisticated approach to topic organization. Unlike other messaging systems, Pulsar implements a hierarchical naming structure that brings clarity and organization to your messaging architecture. Let's dive into how Pulsar topics work and how to effectively use them. ## The Anatomy of a Pulsar Topic A Pulsar topic follows this format: ```text persistent://tenant/namespace/topic-name ``` This three-level hierarchy isn't just for show – it's a powerful feature that enables better organization and control of your messaging infrastructure. Let's break down each component: ### 1. Persistence Prefix The `persistent://` prefix indicates that messages will be durably stored. For scenarios where persistence isn't needed, you can use `non-persistent://` instead. This choice affects how Pulsar handles your messages and their storage. ### 2. Tenant The tenant represents the highest level of isolation in Pulsar. Think of it as your organization's private space. For example, if your company is called Acme, you might use `acme-corp` as your tenant name. ### 3. Namespace Namespaces provide logical grouping within a tenant. They're perfect for separating different departments, environments, or regions. For instance, you might have namespaces like `marketing`, `production`, or `us-east`. ### 4. Topic Name The final component is your actual topic name, which describes the specific message stream, such as `order-events` or `user-clicks`. ## Real-World Examples Here are some practical examples of Pulsar topics: ```text persistent://trading-company/market-data/stock-quotes persistent://ecommerce/order-processing/purchase-events persistent://gaming-platform/us-west/player-statistics ``` ## Best Practices for Topic Organization Consider these patterns when structuring your topics: 1. **Regional Separation** - `persistent://company/us-west/events` - `persistent://company/eu-central/events` 2. **Environment Isolation** - `persistent://company/production/user-events` - `persistent://company/staging/user-events` 3. **Team-Based Organization** - `persistent://company/marketing/email-campaigns` - `persistent://company/sales/lead-tracking` This hierarchical structure not only provides clear organization but also enables granular control over authentication, authorization, and resource quotas at each level. Whether you're building a small application or a large-scale distributed system, understanding Pulsar's topic structure is key to effectively organizing your messaging architecture. --- # Insights from Current 2024: Reflections from Austin, Texas URL: https://www.serverless.fyi/p/current2024-austin-texas/ Published: 2024-09-20 Tags: apache-kafka, apache-flink [Current 2024](https://current.confluent.io/), held at the Austin Convention Center from September 17-18, 2024, brought together industry experts, tech leaders, and startups to explore the latest innovations in real-time applications, event-driven architectures, and AI. The event focused on data streaming technologies, particularly Apache Kafka® and Apache Flink®, and how they are being used to power real-time streaming and generative AI use cases. Attendees gained valuable insights into how these technologies are transforming businesses across various industries. This was my second time attending the two-day event at the Austin Convention Center. At the conference, I reconnected with familiar faces and made plenty of new friends from the Kafka, Flink, and real-time data community. ## Visionaries Unite: Shaping Tomorrow’s Data Landscape with Real-Time Insights Day 1 of the event kicked off with an impactful [keynote](https://www.youtube.com/watch?v=Sn6fVsOzrSU) featuring Jay Kreps, CEO and Co-Founder of Confluent, alongside industry leaders like Seema Acharya (Mercedes Benz R\&D), Richie Artoul (WarpStream by Confluent), Shaun Clowes (Confluent), Kushal Khandelwal (Viacom18), and Adam Tybor (Accenture). Each speaker shared insights on the future of data, AI, and real-time streaming, highlighting their experiences and innovations across various industries. ## Engineering the Future: The Data Streaming Revolution and Its Pioneers Day 2's [keynote](https://www.youtube.com/watch?v=ccupkhcLioM) explored the transformative impact of data streaming on the industry, highlighting the emergence of the Data Streaming Engineer role that unifies the work of application developers, data engineers, data scientists, and architects. Speakers included David Anderson, Tim Berglund, Lucia Cerchie, Danica Fine, Sandon Jacobs, Adi Polak, Anna Povzner, and Matthew Seal, all from Confluent. Attendees gained valuable insights into the current technologies shaping the Data Streaming Platform and learned how to prepare for their implementation in the coming year. ## Breaking New Ground with BYOC Confluent formalized the announcement of [acquiring WarpStream,](https://www.confluent.io/blog/confluent-acquires-warpstream/) a Kafka-compatible streaming solution with innovative architecture, to expand its cloud-native streaming offerings. This addition enhances Confluent's ability to meet diverse customer needs with advanced features like offset-preserving replication, cluster quotas and next-gen BYOC architecture. In just 13 months, [WarpStream](https://www.warpstream.com/blog/warpstream-is-dead-long-live-warpstream) grew its tea,, launched new products, expanded cloud regions, achieved SOC2 Type II certification, and onboarded major customers like Grafana Labs, Zomato, and CharacterAI. During the keynote, WarpStream founder Richie Artoul discussed the adoption of BYOC (Bring Your Own Cloud) as a cloud service, emphasizing its ability to ensure data sovereignty and security by keeping all data securely within the customer's account. ![](https://www.serverless.fyi/_astro/img-f8a29a48be91.Cyuq0ciJ.jpg) Confluent Acquires WarpStream ## Spotlighting the Innovators At the conference, I reconnected with several friends from various companies and made some great new connections. [DeltaStream](http://www.deltastream.io) As one of the [founding engineers,](https://www.linkedin.com/posts/mohinishbasha_current24-streamingdata-funding-activity-7242260038663241728-RFY0) I've been part of DeltaStream’s journey since the very beginning, joining the team 3.5 years ago. After raising an initial $10 million in [seed funding](https://www.deltastream.io/seed-funding-post/) from NEA, I played a key role in building out our cloud-based infrastructure, which includes multi-tenant, dedicated, and BYOC offerings. DeltaStream is founded by [Hojjat Jaffarpour](https://www.linkedin.com/in/hojjatjafarpour/), the original creator of KSQL. Our platform is a serverless, unified stream processing solution designed to simplify the development of stream processing applications by eliminating unnecessary complexity. At the conference, DeltaStream unveiled their [Series A funding round ](https://www.deltastream.io/the-road-to-raising-deltastreams-series-a/)raising $15 million, led by NEA, Galaxy Interactive and Sanabil Investments. ![](https://www.serverless.fyi/_astro/img-e5a938e1dbdb.Dn5qeocm.jpg) [StreamNative](https://streamnative.io/) I met [Sijie Guo](https://www.linkedin.com/in/sijieg/), the founder of StreamNative, at their booth and had an engaging conversation about the importance of BYOC and serverless solutions in their go-to-market strategy. Sijie is a member of the Apache Software Foundation (ASF), serving as a PMC member, committer, and contributor for projects like Apache Pulsar, Apache Bookeeping and Apache Doris. StreamNative has re-architected data streaming from the ground up for cloud-native environments using Apache Pulsar. Their ONE StreamNative Platform, powered by the Ursa engine, provides a scalable, high-performance, and cost-efficient data streaming solution that is compatible with the Kafka API and can run on any cloud. ![](https://www.serverless.fyi/_astro/img-d19268acfd08.Bqz2lOvN.jpg) Sijie Guo, creator of Apache Pulsar, Founder @ StreamNative [RisingWave](https://risingwave.com/) RisingWave is on my favorite [stream processing runtime](https://github.com/risingwavelabs/risingwave), that unified both batch and streaming data and is postgres compatible. RisingWave aims to democratize stream processing by making it simple, affordable, and accessible. Yingjun, the talented founder of Rising Wave, previously was a software engineer on the Redshift team at AWS and a researcher at IBM Almaden Research Center, holds a PhD from the National University of Singapore and was a visiting PhD at Carnegie Mellon University. RisingWave has 2 product lines, [the Rising Wave Cloud](https://risingwave.com/cloud/) and [the Rising Wave streaming database](https://risingwave.com/database/), all built in RUST to provide continuous analytics, build event-driven applications, real-time ETL pipelines, and feature stores in minutes. I met up with [Rayees Pasha](https://www.linkedin.com/in/rayees-pasha/) at the conference, who leads product at RisingWave. ![](https://www.serverless.fyi/_astro/img-4b0c46a0c74d.BnR0RD0T.jpg) Rayees Pasha @ RisingWave [Restate](https://restate.dev/) [Stephan Ewen](https://www.linkedin.com/in/stephanewen/), [Till Rohrmann](https://www.linkedin.com/in/tillrohrmann/) and [Igal Shilman](https://www.linkedin.com/in/igal-shilman-2027a76/) are the co-founders of Restate. Till also serves as an advisor for various companies using Apache Flink, including DeltaStream. Restate at the core has build a durable state execution engine to provide the simplest way to build resilent applications. Restate is on a mission to simplify modern distributed applications by enhancing their resilience, reliability, and fault tolerance. Their team features co-creators of Apache Flink and distributed processing experts from renowned companies like Apple, Stripe, Amazon, and Meta. ![](https://www.serverless.fyi/_astro/img-3504d90f47a9.Tfz9PmkY.jpg) Restate provides durable state execution engine using kafka [Responsive](https://www.responsive.dev/) Responsive is yet another company emerging from the Confluent mafia cohort, and if you ever found KStreams to be challenging to work with, they’ve created a product that makes it incredibly simple.\ \ I met the three remarkable founders—[Apurva Mehta](https://www.linkedin.com/in/apurvarmehta/), [Rohan Desai](https://www.linkedin.com/in/rohanpd/), [Almog Garva](https://www.linkedin.com/in/agavra/)—at their booth. In their previous roles at Confluent, they were pivotal in the development of KStreams and KSQL. [AntiThesis](https://antithesis.com/) This was my top discovery product on the floor. In the past, we relied heavily on system tests at Confluent to evaluate Kafka, but these tests often fell short in catching edge cases across the wide range of Kafka configurations. AntiThesis addresses this challenge by offering autonomous testing for distributed systems. Their product is utilized for black box testing by notable companies such as Ethereum, Palantir, and MongoDB. Discover how [WarpStream leverages AntiThesis](https://www.warpstream.com/blog/deterministic-simulation-testing-for-our-entire-saas) in this blog post. [Timeplus](https://www.timeplus.com/) Timeplus is a SQL stream processing and analytics engine that combines columnar and row stores within a single binary. It enables the creation of real-time applications for both edge and cloud environments. I had a fantastic time chatting with [Jove Zhong](https://www.linkedin.com/in/jovezhong/), Co-founder and Head of Product at TimePlus, about the inner workings of TimePlus and especially their [Proton engine](https://docs.timeplus.com/proton). ![](https://www.serverless.fyi/_astro/img-a1cea29734c7.V99Lu24H.jpg) TimePlus Team - Sarwar Bhuiyan, Wei Q [FactorHouse](https://factorhouse.io/) FactorHouse aims to empower engineers with tools that simplify working with Apache Kafka and Apache Flink, making real-time streaming compute a seamless and enjoyable experience. They provide an integration and observability platform for stream processing with products like [Flow](https://factorhouse.io/flow/) Integration platform, [KPow](https://factorhouse.io/kpow/) for Apache Kafka and [Flex](https://factorhouse.io/flex/) for Apache Flink. Founded by Derek and Kylie Troy-West in 2019, FactorHouse leverages extensive experience with Apache Kafka, Cassandra, and Storm to deliver data platforms for global banking, insurance, and fintech. Their tools are immensely popular, with hundreds of thousands of installations worldwide. Fun fact: they’ve bootstrapped the company since 2019, and a little birdie told me they’re about to announce their seed funding round soon. [Datorios](http://datorios.com) Coming from Israel, Datorios aims to democratize real-time data processing with a development console for Apache Flink, empowering engineering teams to make stream processing standard and unlock their companies' real-time potential. While visiting the booth, I learned Datorios builds and develops their own distribution of Apache Flink. [LittleHorse](https://littlehorse.dev/) I had a great lunch with [Colt McNealy](https://www.linkedin.com/in/colt-mcnealy-900b7a148/) at the conference. If you've worked with Kafka Streams, you're likely familiar with Colt or [LittleHorse](https://littlehorse.dev/)!\ \ [LitteHorse](http://littlehorse.io) enables the creation of workflow-driven applications by offering durable state execution with Kafka Streams, allowing for the rapid composition and deployment of microservices to enhance developer agility in meeting business demands. [ThatDot](https://www.thatdot.com/) Have you ever considered writing stream processing applications using Graph instead of Apache Kafka? Enter [thatDot](https://www.thatdot.com/), the creators of Quine OSS, who present the world’s first streaming graph analytics engine, developed over eight years for advanced persistent threat detection at Darpa and funded by Crowdstrike. [Ryan Wright](https://www.linkedin.com/in/wrightryan/), the CTO and Founder of thatDot, always offers engaging conversations about graph-based stream processing, threat detection, and fraud analysis. [Platformatory](https://platformatory.io/) Based out of Bengaluru, Platformatory provides consulting, system integration, and platform engineering for streaming data solutions. [Pavan Keshavamurthy](https://www.linkedin.com/in/pavankmurthy/), the founder of Platformatory is a quite popular speaker in the streaming circles. His talk [Events at the (API) Horizon: How to synthesize domain events & changes from your HTTP/REST APIs](https://events.bizzabo.com/599116/agenda/session/1336705)** **at the conference introduced an alternative approach that utilizes standard API proxies or gateway solutions to process and push events to Kafka directly from the API data plane, reducing the need for extensive code changes and complex CDC implementations. [Feldera](https://www.feldera.com/) [Lalith Suresh](https://www.linkedin.com/in/lalith-suresh-34bb8911/), CEO and Founder of Feldera, has developed an incremental computational data engine that enables rapid analysis of live and historical data with full SQL processing, handling millions of events per second and scaling to datasets larger than memory for AI use cases. An incremental computational data engine is essential for RAG and AI use cases as it allows for real-time analysis and processing of vast amounts of data, enabling timely insights and decisions. [ShadowTraffic](https://shadowtraffic.io/) You might have seen a [walking QR code](https://www.linkedin.com/posts/michael-drogalis_it-didnt-make-sense-to-get-an-expo-booth-activity-7241420971436343296-FAsv) invite for a free Kafka test stream, encouraging you to check out ShadowTraffic, created by the talented [Michael Drogralis](https://www.linkedin.com/in/michael-drogalis/), who previously managed the KStreams and KSQL products at Confluent. Follow his journey as a solo bootstrapped entrepreneur running ShadowTraffic (yet another company by Confluent Mafia #yacbcm), achieving around $100k in ARR, on LinkedIn. [InfinyOn](https://infinyon.com/) InfinyOn, a real-time event streaming company, has secured $5 million in seed funding led by Gradient Ventures and Fly Ventures, with participation from Bessemer Venture Partners and others. The company aims to enhance customer experiences through its modern cloud-native event streaming platform, which features in-line computation powered by WebAssembly, enabling large-scale multi-channel dataflows. With strong traction across various industries and recognition on G2’s list for Event Stream Processing Software, InfinyOn leverages its Fluvio Open Source platform, built in Rust, to simplify data architectures and improve efficiency while maintaining high security and low memory usage. [Debadyuti Roy Chowdhury](https://www.linkedin.com/in/debadyutirc/), who is a great friend, runs the product organization at InfinyOn. ## Final Thoughts The data streaming world gathered in Austin, Texas for Current 2024, uniting the community in an exhilarating celebration of all things Apache Kafka® and Apache Flink®. Attendees were treated to insights from tech leaders, industry giants, and startups, who filled the air with a wealth of knowledge about real-time event streaming and beyond. What an exciting time to connect and learn together on topics like private saas, serverless, byoc, durarable state execution, streaming lineage, streaming observability, making kafka protocol more efficient! Innovation in real time data opens up exciting opportunities that inspire creativity to drive transformative changes across industries. --- # Stream Processing: The Future of the Modern Data Stack URL: https://www.serverless.fyi/p/stream-processing-future-modern-data-stack/ Published: 2024-07-01 Tags: apache-kafka, apache-flink ## Stream Processing: The Future of the Modern Data Stack Today marks approximately 10 years of my experience building infrastructure with Kafka and its ecosystem. In 2014, I began using the Apache distribution of Kafka at SAP Ariba while building cloud infrastructure as the organization transitioned from a SOA architecture to a microservices-based architecture. Soon, we were deploying Confluent's Kafka distribution to production, onboarding teams to leverage Kafka as the central nervous system for the microservices. The reason for this transition was that Confluent's distribution included Schema Registry and several other proprietary features that were more effective for running Kafka in production. Witnessing the rise in Kafka adoption both internally and externally, I joined Confluent in 2017. There, I had the opportunity to build developer productivity and platform infrastructure for Kafka, Connect, KSQL, Cloud, and the rest of the engineering organization. At Confluent, I witnessed firsthand the rise in adoption of the overall Kafka ecosystem. ## The Evolution of Database Models from 1960 to 2010 In the 1960s, network-based and hierarchical models for computing and storing databases rapidly gained popularity among organizations. Examples include CODASYL, IMS, and SABRE. In 1970, E.F. Codd's seminal paper introduced the relational database, revolutionizing database thinking by separating the database schema—the logical organization—from physical data storage. This innovation set the standard for modern database systems. Between 1974 and 1977, two significant prototypes of relational database systems emerged: Ingres, developed at UBC, and System R, created at IBM San Jose. Ingres employed the QUEL query language, paving the way for systems such as Ingres Corp., MS SQL Server, Sybase, Wang’s PACE, and Britton-Lee. System R introduced the SEQUEL query language, shaping the development of SQL/DS, DB2, Allbase, Oracle, and Non-Stop SQL. This era also marked the recognition and adoption of the term Relational Database Management System (RDBMS). In 1976, Peter Chen proposed a new database model called Entity-Relationship, or ER. This model enabled designers to prioritize data application over the traditional focus on logical table structure. Structured Query Language, known as SQL, was adopted as the standard query language by the American National Standards Institute in 1986 and by the International Organization for Standardization in 1987. Since then, we have witnessed a significant evolution as the popularity of RDBMS rose in contrast to existing network-based and hierarchical-based data management tools. In subsequent years, a multitude of products emerged, including DB2, PARADOX, RBASE 5000, RIM, dBASE III and IV, OS/2 Database Manager, and Watcom SQL. The 1990s played a crucial role in advancing databases and database software, particularly towards application development. New client tools like Oracle Developer, MySQL, PowerBuilder, and Visual Basic (VB) were released. Additionally, tools enhancing developer productivity such as ODBC, Excel, and Access were developed. Prototypes for Object Database Management Systems (ODBMS) emerged in the early 1990s. By 2010, there was a notable shift in how data was generated and utilized, both within organizations and externally, with row-based and column-based databases finding market acceptance with NoSQL solutions. ## Modern Data Stack From 2010 to the present day, the modern data stack embodies a comprehensive ecosystem of products crafted for efficient data management and analytics. Products in this category typically encompass robust components such as cloud storage solutions like Amazon S3 or Google Cloud Storage, providing scalable data storage. Data pipelines are orchestrated using tools such as Apache Kafka or Apache Airflow, ensuring smooth data ingestion and processing. Platforms like Snowflake or Google BigQuery provide scalable and high-performance solutions for data warehousing and analytics. The stack is agile and scalable, accommodating diverse data sources and formats, including structured, semi-structured, and unstructured data. Real-time analytics capabilities are enhanced through technologies like Apache Flink or Apache Spark Streaming. Overall, the modern data stack empowers organizations to extract actionable insights, drive innovation, and maintain competitive advantage in today's data-driven world. There is an evolutionary surge of products in this category, with solutions like ClickHouse, Atlan, MonteCarlo, Telmai, AirByte, RockSet, Materialize and numerous others gaining significant popularity. ## Revolutionizing Data Efficiency: Stream Processing and Streaming Stores Stream processing holds the promise of efficiently handling data with low latency and compute costs, even at massive volumes and high cardinality. With the aid of streaming stores, they eliminate the necessity for additional computations during data ingestion and can operate directly on the data source across multiple streams. In addition to Kafka, emerging streaming stores like Redpanda and Warpstream are increasingly gaining traction. Today, companies like DeltaStream, RisingWave, Decodable, Arroyo etc are developing products aimed at simplifying the complexity involved in building stream processing applications. With each new evolution of a software stack, it commoditizes the stack that came before it. As Stream processing improves efficiency by processing data closer to its source with low latency and handling high cardinality, it will surpass existing use cases currently served by the modern data stack by the end of this decade. The adoption of AI/ML across organizations will drive this transformation. AI/ML applications demand real-time data processing for timely decisions and predictions. Stream processing facilitates handling continuous data streams with low latency, crucial for dynamic environments such as financial trading, IoT networks, and real-time recommendations. In summary, stream processing is indispensable for AI/ML applications to achieve real-time analytics, adaptive decision-making, and operational efficiency across various industries. _“Data streaming is the central nervous system for data, while AI/ML algorithms are the brain. \~ Confluent”_ --- # 2 Minute Serverless: Newsletter for a Bite-Sized Tech Feast! URL: https://www.serverless.fyi/p/2-minute-serverless-newsletter-bitesized-tech-feast/ Published: 2023-08-04 Tags: newsletter ## Subscribe to "2 Minute Serverless" Newsletter for a Bite-Sized Tech Feast! Are you craving a serving of freshly baked serverless nuggets with a side of Distributed Systems, Stream Processing, and Cloud Infrastructure? Look no further! Introducing ["2 Minute Serverless"](https://2minuteserverless.beehiiv.com/subscribe) a delightful newsletter curated to cater to your technical appetite, providing you with the juiciest insights in just two minutes! ### A Flavourful Blend of Serverless Cloud Platforms: Explore the world of serverless computing with us! From AWS Lambda to Microsoft Azure Functions and Google Cloud Functions, our newsletter delves into the nuances of different serverless cloud platforms. Learn how to build scalable, cost-efficient applications without the burden of managing infrastructure. Become a serverless virtuoso and master the art of Function as a Service (FaaS) in just a few sips! ### Savour the Taste of Distributed Systems: Dive into the realm of Distributed Systems and discover the magic behind high-performing, fault-tolerant applications. Our expertly crafted content serves up key concepts like Consistency, Availability, and Partition Tolerance (CAP theorem) and explores the inner workings of distributed databases, message brokers, and microservices architectures. Whether you're a seasoned developer or a curious enthusiast, our newsletter promises to keep you engaged with digestible insights. ### Whisked Away by Stream Processing: Sip on the knowledge of real-time data processing with Stream Processing. We stir up discussions on popular stream processing frameworks like Apache Kafka, Apache Flink, and Amazon Kinesis. Understand the intricacies of event-driven architectures and how to process data at lightning speed. Unravel the potential of event streams and how they revolutionize modern applications. ### A Taste of Cloud Infrastructure: Indulge in a buffet of cloud infrastructure offerings. Delve into the world of containers, Kubernetes, and serverless containers. Understand how cloud providers manage their data centers and ensure data security. Our newsletter serves up practical tips for optimizing cloud resources, minimizing costs, and maximizing performance. ### Seasoned with Open Source Goodness: Explore the open-source landscape with us! Discover trending projects, learn about emerging technologies, and join the vibrant open-source community. We sprinkle our newsletter with the latest tools and frameworks to help you stay ahead in the ever-evolving tech world. ### A Side Dish of Career Tips, Engineering Culture, and Startup Life: It's not just about technology; we also add some spice to your engineering career! Our newsletter occasionally features advice on career growth, cultivating a healthy engineering culture, and insights into startup life. Elevate your professional journey with valuable guidance and inspiration. ### Why Subscribe to "2 Minute Serverless"? Our newsletter is more than just a reading experience—it's a knowledge feast delivered right to your inbox! In just two minutes, you'll receive bite-sized chunks of wisdom, crafted with care and precision. Say goodbye to lengthy articles and time-consuming research. With ["2 Minute Serverless"](https://2minuteserverless.beehiiv.com/subscribe) you can easily stay informed, level up your technical skills, and quench your thirst for knowledge without breaking a sweat. ### Don't Miss Out! Subscribe now to ["2 Minute Serverless"](https://2minuteserverless.beehiiv.com/subscribe) and embark on an adventure into the heart of modern tech! Our weekly newsletter will be your guide through the vast landscape of serverless cloud platforms, distributed systems, stream processing, open source, and beyond. We promise to make your tech journey exciting, informative, and most importantly, time-efficient. Let's build a community of tech enthusiasts who value every second and make the most of their two minutes!