Meetups/Infra/2025-08-25

Discussion of NB wiki infra (traceroute, doggo, whois) to learn about scrapers overloading the wiki (and block/rate limiting plans), k8s and load balancer discussion, and ETL / batch ML frameworks covered.

Introductions

• [name] - [background]. [goals for meetup, or interests to explore]

• Loren - platform engineering, scraping, goal: sharing NB infrastructure
• Elan - interested in web scale technologies, and migrating NB infrastructure (first yaml banging head against the wall issu)E
• Omelia - does graphics, is here, is a vampire
• Zachae - also a vampire, working on compilers, for Neural processing units, thinks C++ is a mistake "free as in speech"
• Doug - evangelist for streamlet/caddy/cloudflare
• Frank - software background, homelabbing, NB infrastructure
• Mark -- here by chance, developed several apps, plan to just listen
• Jacob - just here to listen, developed apps
• Michael - self-hosted, devops,networking
• Alex - background in ML, fighting with data processing frameworks this week. Looking for reccs "BEAM"
• ChineseMan - background Chinese, does random web/mobile/infra work, known for his quantum high/low spending habits "free as in beer"
• Erik - interested in HW infrastructure but also some software stuff

Lesson or Demo

• Read aloud: clarify for meetup. We are taking notes in a riseup pad (or I am--help appreciated, and links). We have meeting notes posted to the wiki. noisebridge.net, search Infra, or Meetups/Infra. (the Infrastructure page has a disambiguation link.)
• Shell, web services, self-hosting, networking!

• caddy m3 ## caddy - nginx alternative.
  • doggo: https://doggo.mrkaran.dev/

• • https://github.com/noisebridge/infrastructure

• load balancing - k8s in nb

berry0.noise/srv/git  -- back up soon
berrybush / bramble cluster 
k8s -- use static ips
k8s node, err pods on the nodes, can take over ips on the k8s network, "via" ARP broadcasts
https://kubernetes.io/docs/concepts/overview/components/

• ETL & ml batch platforms

Alex - working using Spark for ETL stuff, parquet in+out. https://spark.apache.org/
https://ray.io data  -- not at great for heavy etl, great for batch inference, great for e.g. quick protobuf extraction to other structured data formats
issues around the edges of ray data, esp. talking back & forth with Spark. Issues with writing empty files, but breaking on reading empty files.
Using pandas & numpy under the hood, a little concern, desire to defensively program (fragility / NaN concerns) 
other libraries -- Polars - Dask

Alternative/related techs:
    * https://pola.rs/ -- pandas dataframe alternative, in rust. Has conversion methods back and forth to Pandas DataFrames
    * Dask (https://www.dask.org/)
    * Apache Beam (https://beam.apache.org/) - good at doing batch/realtime ETL

• Doug -- demo at the end

Takeaways

Alex -- beam
Frank -- the way the k8s lb (load balancer) uses ARP to let nodes, err pods, or services, take over and migrate IP addresses
Erik -- gpu server -- for gpu services, for k8s
Doug - polars
Elan? - doggo

Questions, Discussion, or Coworking

• [Issue]

For next time

Readings & Exercises

• Readings
• Exercises

Join online

• Try it yourself!
  • Join libera.chat #nb-meetup-infra

https://www.noisebridge.net/wiki/Meetups/Infra